Data processing method and related apparatus, system, electronic device and storage medium

Abstract

The application discloses a data processing method and related device, system, electronic equipment and storage medium, comprising: in response to the target data being knowledge source material, calling a cleaning agent based on a coordination agent to process the knowledge source material according to the corresponding data type to obtain standardized text, and then calling an analysis agent based on the coordination agent to update the target knowledge base of the target scene according to the semantic analysis result of the standardized text; in response to the target data being a to-be-queried statement, calling a retrieval agent based on the coordination agent to retrieve the target knowledge base according to the to-be-queried statement to obtain target knowledge data, and then calling a fusion agent based on the coordination agent to process the target knowledge data to obtain a target query result for replying to the to-be-queried statement. The above scheme can improve the comprehensiveness of the knowledge about the target scene in the target knowledge base, improve the response accuracy of the to-be-queried task in the target scene, and adapt to the processing requirements in complex business scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method and related device, system, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the automation response of query demand by combining a neural network model has become the development direction in complex business scenarios.

[0003] In the prior art, a large model is usually used to uniformly train knowledge data of a business scenario, and after receiving a query request, the large model directly outputs a corresponding response result. However, in this way, the capability of the large model is limited by the parameter quantity of the model itself and the quality of the training data, and when the business scenario is complex, there are many new knowledge data and the data sources are complex, it is difficult to respond to the knowledge update demand of the business scenario in a timely manner, and it is also easy to cause knowledge omission and inaccurate response results, which is difficult to adapt to the processing demand in complex business scenarios. Therefore, how to improve the response accuracy of the query task in the target scenario to adapt to the processing demand in complex business scenarios has become a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a data processing method and related device, system, electronic equipment and storage medium, which can improve the comprehensiveness of the target knowledge in the target knowledge base, improve the response accuracy of the query task in the target scenario, and adapt to the processing demand in complex business scenarios.

[0005] To solve the above technical problem, the first aspect of the present application provides a data processing method, comprising: obtaining target data about a target scenario, and obtaining a plurality of execution agents; wherein the plurality of execution agents include a cleaning agent, an analysis agent, a retrieval agent and a fusion agent, and each execution agent is connected to a coordination agent as an edge node in a star topology; in response to the target data being knowledge source materials, the cleaning agent is called by the coordination agent to process the knowledge source materials according to the corresponding data type to obtain standardized text, and the analysis agent is called by the coordination agent to update the target knowledge base of the target scenario based on the semantic analysis result of the standardized text; in response to the target data being a query statement, the retrieval agent is called by the coordination agent to retrieve the target knowledge base according to the query statement to obtain target knowledge data, and the fusion agent is called by the coordination agent to process the target knowledge data to obtain a target query result for replying to the query statement.

[0006] To solve the above technical problems, the second aspect of the present application provides a data processing device, comprising: an acquisition module, an update module and a retrieval module, the acquisition module is used for acquiring target data about a target scene, and acquiring a plurality of execution agents; wherein the plurality of execution agents comprise: a cleaning agent, an analysis agent, a retrieval agent and a fusion agent, each execution agent is respectively connected with a coordination agent as an edge node in a star topology; the update module is used for responding to the target data being knowledge source materials, calling the cleaning agent based on the coordination agent to process the knowledge source materials according to the corresponding data type, obtaining standardized text, and then calling the analysis agent based on the coordination agent to update the semantic analysis result of the standardized text, and updating the target knowledge base of the target scene; the retrieval module is used for responding to the target data being a query statement to be queried, calling the retrieval agent based on the coordination agent to retrieve the target knowledge base according to the query statement to be queried, obtaining target knowledge data, and then calling the fusion agent based on the coordination agent to process the target knowledge data, and obtaining a target query result used to reply to the query statement to be queried.

[0007] To solve the above technical problems, the third aspect of the present application provides an electronic device, at least comprising a memory and a processor coupled with each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to realize the data processing method in the first aspect.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a data processing system, comprising a coordination agent and a plurality of execution agents connected with the coordination agent as edge nodes in a star topology, the data processing system is used to realize the data processing method in the first aspect.

[0009] To solve the above technical problems, the fifth aspect of the present application provides a computer readable storage medium, which stores program instructions capable of being executed by a processor, and the program instructions are used to realize the data processing method of the first aspect.

[0010] The above scheme obtains target data about a target scene, and obtains a plurality of execution agents, which specifically include a cleaning agent, an analysis agent, a retrieval agent and a fusion agent. Each execution agent is connected to a coordination agent in a star topology. When the target data is knowledge source material, the coordination agent calls the cleaning agent to process the knowledge source material according to a corresponding data type to obtain standardized text. The coordination agent calls the analysis agent to update a target knowledge base of the target scene based on a semantic analysis result of the standardized text. When the target data is a query statement, the coordination agent calls the retrieval agent to retrieve the target knowledge base according to the query statement to obtain target knowledge data. The coordination agent calls the fusion agent to process the target knowledge data to obtain a target query result for responding to the query statement. On the one hand, each execution agent is connected to the coordination agent as an edge node, and the coordination agent can schedule each execution agent to independently process corresponding functions, which is suitable for complex business scenarios with frequent knowledge updates and complex sources. On the other hand, the cleaning agent and the analysis agent update the target knowledge base through step-by-step processing, and process the knowledge source material according to the corresponding data type, which can flexibly adapt to the processing requirements of different sources and different types of knowledge in the target scene, and facilitate timely response to the knowledge update requirements of the target scene, thereby improving the comprehensiveness and accuracy of the target knowledge base of the corresponding target scene. In addition, for the query statement, the retrieval agent retrieves the target knowledge data from the target knowledge base of the corresponding target scene, and the fusion agent processes the target query result, which can retrieve the knowledge data related to the query statement based on the continuously updated target knowledge base in the target scene, thereby improving the response accuracy of the query task in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base is improved, and the response accuracy of the query task in the target scene is improved to adapt to the processing requirements in complex business scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of an embodiment of the data processing method of the present application; Figure 2 is a framework diagram of an embodiment of the knowledge source material acquisition of the present application; Figure 3 is a framework diagram of an embodiment of the data processing of the cleaning agent of the present application; Figure 4 is a framework diagram of an embodiment of the target knowledge data generation of the present application; Figure 5 is a framework diagram of an embodiment of the data processing device of the present application; Figure 6 is a framework diagram of an embodiment of the data processing system of the present application; Figure 7 is a frame schematic diagram of an embodiment of the electronic device of the present application. Figure 8 is a frame schematic diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0013] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the sake of limitation.

[0014] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is merely an associative relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there can be three cases: A alone, A and B together, and B alone. In addition, the segment " / " herein generally represents an "or" relationship between the associated objects. In addition, "multiple" herein means two or more than two.

[0015] If the technical scheme of the present application involves personal information, the product applying the technical scheme of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical scheme of the present application involves sensitive personal information, the product applying the technical scheme of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent sign is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the individual voluntarily enters the collection range, it is considered to agree to collect the personal information; or in the case of using obvious signs / information to inform the personal information processing rules on the device for processing personal information, the personal authorization is obtained by means of pop-up information or asking the individual to upload his / her personal information, etc. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

[0016] Please refer to Figure 1 , Figure 1 is a flow schematic diagram of an embodiment of the data processing method of the present application. It should be noted that the flow operation in the present disclosure embodiment can be executed by an electronic device with computing capability or a related device containing an electronic device, and the specific structure or specific type is not limited here. Specifically, the present disclosure embodiment can include the following steps: Step S11: obtaining target data about a target scene, and obtaining a plurality of execution agents.

[0017] In the embodiments of the present disclosure, the several execution agents include a cleaning agent, an analysis agent, a retrieval agent, and a fusion agent, and each execution agent is connected in a star topology with the coordination agent as an edge node. The agent is an agent that can perceive the environment and take action to achieve a specific goal. Specifically, the agent can be software, hardware, or a system, and has autonomy, adaptability, and interaction ability. The agent perceives changes in the environment through data input, makes judgments and decisions based on learned knowledge and algorithms, and then performs actions to affect the environment or achieve predetermined goals.

[0018] In one implementation scenario, the target scenario is an enterprise-level data governance scenario. Specifically, in the enterprise-level data governance scenario, different business departments generate a large amount of heterogeneous knowledge materials from business systems, offline documents, external suppliers, and other multi-channel sources. These knowledge materials not only have various data types, but also continue to be newly added with business iteration. The coordination agent can uniformly schedule execution agents with different functions, call the cleaning agent to complete the corresponding format standardization processing for different types of knowledge source materials, and then update the target knowledge base of the enterprise-level governance scenario after the semantic analysis of the analysis agent. This can flexibly adapt to the dynamic knowledge update needs of different business lines of the enterprise, and ensure that the enterprise governance knowledge base always matches the latest business situation. When receiving a consultation query such as a governance matter, accurate knowledge can be retrieved based on the updated target knowledge base, and then the fusion agent integrates to form an accurate reply result, thereby improving the accuracy of the enterprise data governance related consultation response.

[0019] In the embodiments of the present disclosure, the data types of the target data include knowledge source materials and query statements. Different data processing operations are performed for different data types to meet different business needs of knowledge update and information query.

[0020] In one implementation scenario, the data source types of the knowledge source materials include at least one of a database, a file, a web page, and an audio system related to the target scenario.

[0021] In one specific implementation scenario, the data source types of the knowledge source materials are determined based on the target scenario. Specifically, in an enterprise service scenario, the knowledge source materials can come from the business database of the enterprise, the offline documents sorted by the employees, the industry information web pages publicly disclosed, and the audio transcription content generated by enterprise training and customer consultation. In a medical auxiliary diagnosis scenario, the knowledge source materials can come from a medical case database, medical literature files, medical popularization web pages, and expert diagnosis audio. Each source of knowledge source materials can contain several data types, and the cleaning agent can be called to perform corresponding format standardization operations according to different data types.

[0022] In an embodiment, the data types of the knowledge source material include structured text, PDF, picture, HTML, audio transcription text, and the like.

[0023] In an embodiment, after accessing the raw data source material of different data source types about the target scenario, the target data of the data source material is obtained based on the parser corresponding to the raw data source material, specifically, the parser is biased towards the bottom layer, certainty and format conversion, for example, a SQL parser, a document parser, an HTML extractor, an OCR parser, an ASR parser, and the like can be used to complete the conversion from the raw data (i.e., the raw data source material) to the machine-processable text or structured segment (i.e., the target data).

[0024] In an embodiment, the query statement can be a business consultation, knowledge query statement initiated by the user for the target scenario, for example, the user's consultation on the business handling process in the enterprise service scenario, the user's query on the disease performance and medication precautions in the medical auxiliary diagnosis scenario, and the like. Specifically, the query statement can be an input statement in natural language form, or a structured query statement, which is not limited in the present application.

[0025] In an embodiment, the knowledge source material and the query statement can be accessed based on the corresponding interface, for example, the knowledge source material can be accessed in batches and at regular intervals through the data synchronization interface, and the query statement can be accessed in real time through the user interaction interface, to meet the access requirements of different types of target data.

[0026] Please refer to Figure 2 , Figure 2 is a framework diagram of an embodiment of the knowledge source material acquisition of the present application. In an embodiment, the raw knowledge source material contained in the database, file access (text, word file, PDF, picture, and the like), web HTML, and the like can be accessed through the data access component Data Connector, the structured data is sent to the SQL parser, the document data is sent to the document parser, the web data is sent to the HTML extractor, the audio data is sent to the speech recognition parser, and the machine-processable data output by each parser is taken as the target data, and the cleaning agent is called by the coordination agent to process the target data.

[0027] Step S12: in response to the target data being the knowledge source material, the knowledge source material is processed according to the corresponding data type based on the cleaning agent called by the coordination agent, to obtain the standardized text, and the semantic analysis result of the standardized text is updated based on the analysis agent called by the coordination agent, to update the target knowledge base of the target scenario.

[0028] In an implementation scenario, the cleaning agent can further denoise and regularize the standardized text of different data types, such as removing HTML format tags, removing irrelevant watermark text, correcting transcription errors, filtering redundant content, unifying content format, and the like, and can filter invalid information in the knowledge source material, thereby improving the efficiency and accuracy of subsequent analysis and updating. After the cleaning agent completes processing to obtain the standardized text, the coordination agent schedules the standardized text to the analysis agent, which performs semantic block splitting, entity recognition, relationship extraction, and the like on the standardized text, extracts knowledge units conforming to the storage format of the target knowledge base, and stores the extracted knowledge units in the target knowledge base, thereby completing updating of the target knowledge base.

[0029] In an implementation scenario, the cleaning agent includes a data type scheduling module, an image processing module, a text processing module, and an audio processing module. Specifically, the data type scheduling module schedules the corresponding processing module to perform cleaning operations according to the data type corresponding to the input standardized text.

[0030] In a specific implementation scenario, the coordination agent calls the data type scheduling module to identify the data type of the knowledge source material, and obtains an identification result. Specifically, the identification result includes one of image, text, and audio as the data type of the knowledge source material.

[0031] Please refer to Figure 3 , Figure 3 is a framework diagram of an embodiment of the cleaning agent data processing of the present application. The data type scheduling module schedules the corresponding processing module to perform cleaning operations according to the data type corresponding the input standardized text. In a specific implementation scenario, in response to the identification result indicating that the data type of the knowledge source material is an image, the data type scheduling module calls the image processing module to perform optical character recognition and layout correction on the knowledge source material, and obtains standardized text. Specifically, the optical character recognition is used to extract the text content contained in the image, and the layout correction is used to adjust the text arrangement order, adapt to the content layout of multiple columns and irregular layout in the picture, and avoid disorder of the extracted text content. After the image data is subjected to optical character recognition and layout correction, the processed data can be converted into markdown text as the standardized text, facilitating subsequent extraction of knowledge units and improving the accuracy of analysis processing.

[0032] In another specific implementation scenario, in response to the identification result representing that the data type of the knowledge source material is text, the data type scheduling module calls a text processing module to perform semantic analysis and arrangement on the knowledge source material, to obtain standardized text. Specifically, the text processing module can perform the following processing on the text type knowledge source material: removing format tags, removing redundant repetitions, removing irrelevant watermark content, etc. For example, for HTML text extracted from a webpage, irrelevant content such as style control codes, navigation bar advertisements, footer copyright information, etc. are removed, or for repeated knowledge fragments, only one logically complete content is retained to avoid subsequent knowledge base storage redundancy and improve knowledge retrieval and storage efficiency. After the text data is subjected to semantic analysis and arrangement, the processed data can be converted into markdown text as the standardized text, facilitating subsequent extraction of knowledge units and improving the accuracy of analysis and processing.

[0033] In another specific implementation scenario, in response to the identification result representing that the data type of the knowledge source material is audio, the data type scheduling module calls an audio processing module to perform text conversion and semantic analysis and arrangement on the knowledge source material, to obtain standardized text. Specifically, the audio processing module first converts the audio content into text through automatic speech recognition, and then corrects speech transcription errors, removes emotional words, pauses, repetitions, redundant errors, etc. to arrange the smoothness of the sentence, and finally outputs the markdown format standardized text after arrangement, for subsequent analysis of intelligent agents.

[0034] In one implementation scenario, the execution intelligent agent further includes a message queue writing intelligent agent. After the coordination intelligent agent calls the cleaning intelligent agent to process the knowledge source material according to the corresponding data type to obtain the standardized text, the coordination intelligent agent calls the message queue writing intelligent agent to obtain the intermediate process data of the cleaning intelligent agent processing the knowledge source material to obtain a target message queue. For example, the intermediate process data of the cleaning intelligent agent processing the knowledge source material is the standardized text, and the target message queue is used to be called by the coordinated intelligent agent to realize asynchronous processing, failure rollback, and parallel expansion of the cleaning intelligent agent. Specifically, when multiple knowledge source materials need to be processed at the same time, the target message queue can be used for caching and queuing to avoid congestion of multiple tasks at the same time, which reduces the processing efficiency. When processing of a knowledge source material fails, the standardized text corresponding to the material can be directly taken out from the target message queue for retry, without the need to call the cleaning intelligent agent to process from the beginning again, which can improve the processing efficiency and reduce the scheduling pressure of the coordination intelligent agent.

[0035] In one implementation scenario, the analysis agent includes a segmentation module, a keyword extraction submodule, a vector extraction module, a relationship extraction module, and a knowledge base updating module. The segmentation module is called by the coordination agent to analyze and segment the standardized text to obtain target subtexts. The keyword extraction submodule, the vector extraction module, and the relationship extraction module are respectively called by the coordination agent to perform keyword extraction, vectorization processing, and entity relationship extraction on the target subtexts to obtain keyword indexes, target vectors, and knowledge graphs of the target subtexts. The knowledge base updating module is called by the coordination agent to update the target knowledge base of the target scene in combination with the keyword indexes, the target vectors, and the knowledge graphs of the target subtexts. The above scheme can update the target knowledge base in different data dimensions by analyzing the standardized text to extract multi-dimensional knowledge information, thereby improving the accuracy of subsequent retrieval results.

[0036] In one specific implementation scenario, the keyword indexes of the target subtexts are index files generated by the keyword extraction submodule from the corresponding target subtexts. Each core keyword records the storage location of the target subtext containing the keyword in the target knowledge base, which facilitates subsequent quick positioning of related target subtext content according to the keywords of the query statement, thereby improving the response speed and accuracy of knowledge retrieval.

[0037] In one specific implementation scenario, the target vector of the target subtext is a fixed-dimensional vector obtained by the vector extraction module by performing semantic vectorization on the corresponding target subtext as a whole. The vector can represent the overall semantic information of the target subtext. After storing the target vector in the target knowledge base, the semantic similarity between the vectorization result of the query statement and the target vector of each target subtext can be calculated directly when performing semantic-based retrieval matching, thereby quickly matching the semantically related target subtext and improving the accuracy of semantic retrieval.

[0038] In one specific implementation scenario, the knowledge graph of the target subtext is a structured knowledge network constructed by the relationship extraction module from the corresponding target subtext. Each node corresponds to a knowledge entity, and each edge corresponds to the association relationship between two entities. The knowledge graph can intuitively present the semantic logical relationship contained in the target subtext. After being stored in the target knowledge base, the knowledge graph can support association reasoning based on the knowledge graph to obtain deeper associated knowledge, thereby further improving the completeness and accuracy of query responses.

[0039] In a specific implementation scenario, the target knowledge base of the target scene is divided into different storage areas according to the data types of the target knowledge data, and three types of target knowledge data, keyword index, target vector and knowledge graph, are stored respectively. Different storage areas are independent of each other and can be associated with indexes, which facilitates quick retrieval of knowledge data in corresponding dimensions according to different retrieval requirements, improves the orderliness of storage management, and accelerates the response speed of knowledge retrieval in different scenarios.

[0040] Step S13: In response to the target data being a query statement, the retrieval agent is called based on the coordination agent to retrieve the target knowledge base according to the query statement, to obtain target knowledge data, and the fusion agent is called based on the coordination agent to process the target knowledge data, to obtain a target query result for replying to the query statement.

[0041] In one implementation scenario, the retrieval agent can call corresponding retrieval modules in combination with the retrieval requirements of the query statement, to obtain matching keyword index results, semantic similar vectors and knowledge graph associated entities from different storage areas of the target knowledge base, to generate the target query result.

[0042] In one implementation scenario, the retrieval agent includes an intent recognition module, a full-text retrieval module, a vector retrieval module and a graph relationship retrieval module. Specifically, the intent recognition module is used to identify the query intent of the query statement, the full-text retrieval module is used to match relevant knowledge from the keyword index storage area of the target knowledge base based on keyword matching, the vector retrieval module is used to match semantically related knowledge from the vector storage area of the target knowledge base based on semantic similarity, and the graph relationship retrieval module is used to retrieve associated entities and relationships from the knowledge graph storage area of the target knowledge base. In the actual retrieval process, one or more retrieval modules can be selected and called for retrieval according to the identified query intent, which can obtain more comprehensive and accurate target knowledge data by combining the advantages of different retrieval methods, and improve the accuracy of the retrieval result.

[0043] In a specific implementation scenario, the intent recognition module is called based on the coordination agent to identify the query intent of the query statement, and at least one of the full-text retrieval module, the vector retrieval module and the graph relationship retrieval module is selected and called based on the intent recognition result to retrieve the target knowledge base according to the query statement, to obtain target knowledge data.

[0044] In one specific implementation scenario, the intent recognition module sequentially performs: obtaining, based on the field attribute of the to-be-queried statement, a first judgment result indicating whether the retrieval based on the deterministic retrieval manner produces a positive effect, obtaining, based on the relationship requirement of the to-be-queried statement, a second judgment result indicating whether the retrieval based on the similarity retrieval manner produces a positive effect, obtaining, based on the query type of the to-be-queried statement, a third judgment result indicating whether the retrieval based on the association retrieval manner produces a positive effect, and obtaining, based on the conflict degree of the to-be-queried statement, a fourth judgment result indicating whether the retrieval based on at least two of the deterministic retrieval manner, the similarity retrieval manner, and the association retrieval manner produces a positive effect. Specifically, judging whether the retrieval based on the corresponding retrieval manner produces a positive effect represents whether it is suitable to use the retrieval manner to meet the current query requirement. The deterministic retrieval manner represents the retrieval based on keyword matching, is suitable for query requirements with explicit limited fields, such as querying the handling location corresponding to a specific business or the applicable disease of a specific drug, and can directly obtain accurate hit results through keyword matching. The similarity retrieval manner represents the retrieval based on semantic vector similarity, is suitable for natural language consultation with ambiguous semantics and no explicit keywords, such as querying related causes after a user describes his / her own symptoms, and can obtain related knowledge with similar semantics through semantic matching. The association retrieval manner represents the relationship retrieval based on a knowledge graph, is suitable for query requirements requiring mining of the association relationship between entities, such as querying the investment relationship between a certain enterprise and related enterprises, and can obtain complete association results through path reasoning of the knowledge graph. The above scheme sequentially performs the judgment of retrieval levels in progression, can match and adapt the retrieval manner according to the specific query characteristics of the to-be-queried statement, avoids introducing invalid retrieval results due to inappropriate retrieval manners, improves the retrieval efficiency, and improves the accuracy of matching of target knowledge data.

[0045] In one specific implementation scenario, in response to the latest obtained judgment result indicating that the corresponding retrieval manner does not produce a positive effect during the sequential execution process, the generation of the next judgment result is continued, and in the case that the fourth judgment result is the latest obtained judgment result and the fourth judgment result indicates that the corresponding retrieval manner does not produce a positive effect, the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module are called correspondingly as the intent recognition result. Specifically, if none of the three retrieval manners meets the retrieval requirement of the to-be-queried statement, the three types of retrieval modules are called in parallel to perform retrieval at the same time, which can cover all possible matching knowledge dimensions, avoid missing related knowledge, guarantee the comprehensiveness of the retrieval result, and improve the accuracy of the target query result as much as possible.

[0046] In one specific implementation scenario, in response to the latest obtained judgment result representing that the corresponding retrieval manner produces a positive effect in the sequential execution process, the sequential execution is ended, and the corresponding intent recognition result is generated based on the latest obtained judgment result. Specifically, if the deterministic retrieval manner is obtained to be adaptive, the full-text retrieval module is called for retrieval, if the similarity retrieval manner is obtained to be adaptive, the vector retrieval module is called for retrieval, if the association retrieval manner is obtained to be adaptive, the graph relationship retrieval module is called for retrieval, and if at least two kinds of retrieval manners need to be combined to meet the query requirement of the to-be-queried sentence, the corresponding adaptive multiple retrieval modules are called in parallel for retrieval, which can reduce unnecessary retrieval operations on the premise of meeting the query requirement, can narrow the retrieval range, improve the retrieval efficiency, can focus on the knowledge content of the adaptive dimension, and improve the accuracy of the retrieval result matching.

[0047] In one specific implementation scenario, in response to the fact that the field attribute of the to-be-queried sentence contains a special term related to the target scenario, the first judgment result is obtained to be a positive effect. Specifically, if the special term belongs to the keyword system defined in the target scenario knowledge base, it can be directly located to the corresponding knowledge content through keyword matching, so that the deterministic retrieval manner can be used to quickly obtain accurate results, and other retrieval modules do not need to be called to occupy processing resources. Therefore, in the case where the first judgment result is a positive effect, the intent recognition result is obtained to be the corresponding call of the full-text retrieval module.

[0048] In one specific implementation scenario, in response to the fact that the relationship requirement of the to-be-queried sentence contains context semantic dependence, the second judgment result is obtained to be a positive effect. Specifically, if the user's query needs to combine semantic context to understand the content requirement, there is no explicit and directly matchable limited keyword, and it is easier to match the knowledge content that meets the user's requirement through similarity retrieval based on semantic vector similarity. Therefore, in the case where the second judgment result is a positive effect, the sequential judgment is ended, and the intent recognition result is obtained to be the corresponding call of the vector retrieval module.

[0049] In one specific implementation scenario, in response to the fact that the query type of the to-be-queried sentence is a global query, the third judgment result is obtained to be a positive effect. Specifically, the global query needs to sort out the association paths between different knowledge entities, and mine the hierarchical and logical relationships hidden in the content. Through the association retrieval based on the knowledge graph, the association paths between entities can be traversed, and the knowledge association in the global range can be completely presented. Therefore, in the case where the third judgment result is a positive effect, the sequential judgment is ended, and the intent recognition result is obtained to be the corresponding call of the graph relationship retrieval module.

[0050] In a specific implementation scenario, in response to the conflict degree of the to-be-queried statement representing that the to-be-queried statement is a multi-entity jump statement, a fourth determination result is that a positive effect is generated. Specifically, when multiple different types of knowledge entities are involved in the to-be-queried statement, and there is a cross-dimension association requirement between different entities, it is determined that the to-be-queried statement is a multi-entity jump statement. A single-dimension retrieval manner cannot completely cover all matching requirements, and at least two different retrieval manners need to be combined to obtain comprehensive and accurate knowledge content. Therefore, in the case where the fourth determination result is that a positive effect is generated, the sequential determination is ended, and an intention recognition result is obtained, that is, a plurality of matched retrieval modules are called in parallel for retrieval, different retrieval manners are supplemented to each other, and knowledge matching requirements in multiple dimensions are covered.

[0051] In a specific implementation scenario, in the case where the fourth determination result is that a positive effect is generated, the intention recognition result is that at least two of the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module are called.

[0052] Please refer to Figure 4 , Figure 4 is a framework schematic diagram of an embodiment of the target knowledge data generation of the present application. In a specific implementation scenario, the intention recognition module performs query intention recognition on the to-be-queried statement, determines whether the to-be-queried statement field attribute contains a special term related to the target scenario, in response to containing the special term related to the target scenario, extracts the keywords of the to-be-queried statement and calls the full-text retrieval module to perform deterministic retrieval in the target knowledge base in combination with the extracted keywords, in response to not containing the special term related to the target scenario, determines whether the context semantic dependency is contained in the relationship requirement of the to-be-queried statement, in response to containing the context semantic dependency, the intention recognition module calls the vector retrieval module to perform similarity retrieval in the target knowledge base in combination with the to-be-queried statement, in response to not containing the context semantic dependency, determines whether the query type of the to-be-queried statement is a global query, in response to being a global query, the intention recognition module calls the graph relationship retrieval module to perform association retrieval in the target knowledge base in combination with the to-be-queried statement, in response not being a global query, determines whether the to-be-queried statement is a multi-entity jump statement, in response to being a multi-entity jump statement, it is determined that at least two retrieval manners need to be used at the same time, and a plurality of adapted retrieval modules are called in combination for retrieval, in response to not being a multi-entity jump statement, the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module are called in parallel for retrieval, so as to guarantee that the retrieval result covers all knowledge dimensions.

[0053] In one implementation scenario, the fusion agent includes a knowledge specification module, a conflict detection module, and a verification and correction module. Specifically, the knowledge specification module is configured to normalize the target knowledge data obtained by different retrieval modules in a unified format, eliminate format differences between knowledge data obtained by different storage areas and different retrieval methods, and facilitate subsequent unified processing. The conflict detection module is configured to detect content conflicts in the normalized multi-source target knowledge data, and identify conflicts such as content contradictions and expression differences in different source knowledge data. The verification and correction module is configured to verify and correct the detected conflict content in combination with the authoritative knowledge entries in the target knowledge base, eliminate incorrect and contradictory knowledge content, and retain accurate and consistent knowledge content. Finally, the knowledge content is integrated into a complete and unified target query result, effectively improving the accuracy and consistency of the final reply content, and avoiding misleading of the reply result caused by multi-source knowledge conflicts.

[0054] In one specific implementation scenario, the coordination agent calls the knowledge specification module to normalize the target knowledge based on the preset prompt word, and obtains the fusion knowledge data. The coordination agent calls the conflict detection module to detect content conflicts in the fusion knowledge data, and obtains the conflict detection result.

[0055] In one specific implementation scenario, the fusion knowledge data can be obtained based on a large language model and a preset prompt word. Specifically, the large language model arranges fragmented knowledge content from different sources into complete knowledge segments that meet unified expression specifications according to the requirements of the preset prompt word, automatically eliminates format differences and expression style differences in different retrieval results, and retains source evidence and conflict explanations, thereby improving the efficiency of knowledge fusion processing.

[0056] In one specific implementation scenario, in response to the conflict detection result indicating that there is no content conflict in the fusion knowledge data, the coordination agent outputs the fusion result as the target result data. Specifically, if there is no expression or logical conflict in the knowledge content obtained by multi-source retrieval, the normalized fusion knowledge data can be directly integrated and output without additional correction operations. This not only ensures the accuracy of the result, but also shortens the processing procedure and improves the query response efficiency.

[0057] In another specific implementation scenario, in response to the conflict detection result being that there is a content conflict in the fused knowledge data, the coordination agent calls a verification correction module to process the fused knowledge data to obtain new fused knowledge data after correction, and the coordination agent outputs the new fused knowledge data as the target result data. Specifically, if the knowledge content obtained through multi-source retrieval has contradictory expressions or logical differences, the verification correction module is used to compare the knowledge entries marked as authoritative sources in the target knowledge base, correct the conflicting content according to the authoritative knowledge content, eliminate the incorrect content that is inconsistent with the authoritative knowledge, and finally generate the target query result with consistent and accurate content, taking into account the comprehensiveness and content accuracy of the retrieval result, and avoiding adverse effects of conflicting knowledge on query response.

[0058] In one implementation scenario, the coordination agent can also call a message queue writing agent to obtain the asynchronous database storage capability of the target query result. Specifically, when the target query result is generated, the coordination agent can trigger the message queue writing agent to write the generated target query result into a preset message queue, and the message queue asynchronously completes the database storage operation of the result data. The coordination agent does not need to wait for the completion of the storage operation to release the processing resources, which can effectively improve the concurrent processing capability of the system, reduce the interface response time, and adapt to the multi-user query demand in a high-concurrency scenario. At the same time, with the peak shaving and valley filling capability of the message queue, the storage pressure caused by a large number of instantaneous query requests can be smoothed, and the system processing exception caused by the storage resources being occupied by the instantaneous peak request can be avoided, thereby ensuring the stability of the overall operation of the system.

[0059] The above scheme obtains target data about a target scene, and obtains a plurality of execution agents, which specifically include a cleaning agent, an analysis agent, a retrieval agent, and a fusion agent. Each execution agent is respectively connected to a coordination agent in a star topology. When the target data is knowledge source material, the coordination agent calls the cleaning agent to process the knowledge source material according to a corresponding data type to obtain standardized text. The coordination agent calls the analysis agent to update a target knowledge base of the target scene based on a semantic analysis result of the standardized text. When the target data is a query statement, the coordination agent calls the retrieval agent to retrieve the target knowledge base according to the query statement to obtain target knowledge data. The coordination agent calls the fusion agent to process the target knowledge data to obtain a target query result for responding to the query statement. On the one hand, each execution agent is connected to the coordination agent as an edge node, and the coordination agent schedules each execution agent to independently process corresponding functions, which is suitable for complex business scenarios with frequent knowledge updates and complex sources. On the other hand, the cleaning agent and the analysis agent update the target knowledge base through step-by-step processing, and process the knowledge source material according to the corresponding data type, which is suitable for processing requirements of different sources and different types of knowledge in the target scene, and facilitates timely response to knowledge update requirements of the target scene, and improves the comprehensiveness and accuracy of the target knowledge base of the corresponding target scene. On the other hand, for the query statement, the retrieval agent retrieves the target knowledge data from the target knowledge base of the corresponding target scene, and the fusion agent processes the target query result, which is based on the continuously updated target knowledge base of the target scene to retrieve knowledge data related to the query statement, thereby improving the response accuracy of the query task in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base is improved, and the response accuracy of the query task in the target scene is improved to adapt to the processing requirements in the complex business scenario.

[0060] Please refer to Figure 5 , Figure 5is a frame schematic diagram of an embodiment of the data processing apparatus. The data processing apparatus 50 comprises an acquisition module 51, an update module 52 and a retrieval module 53. The acquisition module 51 is configured to acquire target data about a target scene and acquire a plurality of execution agents. The plurality of execution agents comprises a cleaning agent, an analysis agent, a retrieval agent and a fusion agent. Each execution agent is connected to a coordination agent in a star topology as an edge node. The update module 52 is configured to, in response to the target data being knowledge source materials, call the cleaning agent based on the coordination agent to process the knowledge source materials according to corresponding data types to obtain standardized texts, and call the analysis agent based on the coordination agent to update a target knowledge base of the target scene based on semantic analysis results of the standardized texts. The retrieval module 53 is configured to, in response to the target data being a query statement to be queried, call the retrieval agent based on the coordination agent to retrieve the target knowledge base according to the query statement to be queried to obtain target knowledge data, and call the fusion agent based on the coordination agent to process the target knowledge data to obtain a target query result for replying to the query statement to be queried.

[0061] The above scheme, the data processing apparatus 50 obtains target data about the target scene, and obtains a plurality of execution agents, the plurality of execution agents specifically include a cleaning agent, an analysis agent, a retrieval agent and a fusion agent, each execution agent is respectively connected with the coordination agent in a star topology communication as an edge node, when the target data is a knowledge source material, the cleaning agent is called based on the coordination agent to process the knowledge source material according to the corresponding data type, and the standardized text is obtained, then the analysis agent is called based on the coordination agent to update the semantic analysis result of the standardized text, and the target knowledge base of the target scene, when the target data is a query statement to be queried, the retrieval agent is called based on the coordination agent to retrieve the target knowledge base according to the query statement to be queried, and the target knowledge data is obtained, and then the fusion agent is called based on the coordination agent to process the target knowledge data, and the target query result used to reply to the query statement to be queried is obtained. On the one hand, by connecting each execution agent with the coordination agent in communication as an edge node, the independent processing of the corresponding function can be realized by scheduling each execution agent through the coordination agent, which is suitable for complex business scenarios with frequent knowledge addition and complex sources. On the other hand, the updating of the target knowledge base is completed through the step-by-step processing of the cleaning agent and the analysis agent, and the knowledge source material is processed according to the corresponding data type, which can flexibly adapt to the processing needs of different sources and different types of knowledge in the target scene, facilitate timely response to the knowledge update needs of the target scene, and improve the comprehensiveness and accuracy of the target knowledge base of the corresponding target scene. In another aspect, for the query statement to be queried, the target knowledge data is retrieved from the target knowledge base of the corresponding target scene through the retrieval agent, and then the target query result is obtained through the processing of the fusion agent, which can retrieve the knowledge data related to the query statement to be queried based on the target knowledge base continuously updated in the target scene, thereby improving the response accuracy of the query task to be queried in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base can be improved, and the response accuracy of the query task to be queried in the target scene can be improved to adapt to the processing needs in complex business scenarios.

[0062] In some disclosed embodiments, the retrieval agent includes an intent recognition module, a full-text retrieval module, a vector retrieval module, and a graph relationship retrieval module. The retrieval module 53 further includes an intent recognition module (not shown) for calling the intent recognition module based on the coordination agent to identify the query intent of the query statement to be queried, and obtain the intent recognition result of the query statement to be queried. The retrieval module 53 further includes a retrieval calling module (not shown) for selecting at least one of the full-text retrieval module, the vector retrieval module and the graph relationship retrieval module based on the intent recognition result, retrieving the target knowledge base according to the query statement to be queried, and obtaining the target knowledge data.

[0063] In some disclosed embodiments, the intent recognition module (not shown) further comprises a sequential judgment module (not shown) for sequentially performing, by the intent recognition module, the following: obtaining, based on the field attribute of the to-be-queried statement, a first judgment result of whether a positive effect is produced by retrieval based on a deterministic retrieval manner; obtaining, based on the relationship requirement of the to-be-queried statement, a second judgment result of whether a positive effect is produced by retrieval based on a similarity retrieval manner; obtaining, based on the query type of the to-be-queried statement, a third judgment result of whether a positive effect is produced by retrieval based on an association retrieval manner; and obtaining, based on the conflict degree of the to-be-queried statement, a fourth judgment result of whether a positive effect is produced by retrieval based on at least two of the deterministic retrieval manner, the similarity retrieval manner, and the association retrieval manner. The intent recognition module (not shown) further comprises a first response module (not shown) for, in response to the latest obtained judgment result representing that the corresponding retrieval manner does not produce a positive effect during the sequential execution process, continuing to execute generation of the next judgment result, and in the case that the fourth judgment result is the latest obtained judgment result and the fourth judgment result represents that the corresponding retrieval manner does not produce a positive effect, calling the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module as the intent recognition result. The intent recognition module (not shown) further comprises a second response module (not shown) for, in response to the latest obtained judgment result representing that the corresponding retrieval manner produces a positive effect during the sequential execution process, ending the sequential execution, and generating a corresponding intent recognition result based on the latest obtained judgment result.

[0064] In some disclosed embodiments, the sequential judgment module (not shown) further comprises a first judgment submodule (not shown) for, in response to the field attribute of the to-be-queried statement containing a special term related to the target scenario, obtaining the first judgment result as producing a positive effect. The sequential judgment module (not shown) further comprises a second judgment submodule (not shown) for, in response to the relationship requirement of the to-be-queried statement containing context semantic dependency, obtaining the second judgment result as producing a positive effect. The sequential judgment module (not shown) further comprises a third judgment submodule (not shown) for, in response to the query type of the to-be-queried statement being a global query, obtaining the third judgment result as producing a positive effect. The sequential judgment module (not shown) further comprises a fourth judgment submodule (not shown) for, in response to the conflict degree of the to-be-queried statement representing that the to-be-queried statement is a multi-entity jump statement, obtaining the fourth judgment result as producing a positive effect.

[0065] In some disclosed embodiments, the data processing apparatus 50 further comprises a first calling module (not shown) for obtaining the intent recognition result as corresponding calling the full-text retrieval module in the case that the first judgment result is to produce a positive effect; the data processing apparatus 50 further comprises a second calling module (not shown) for obtaining the intent recognition result as corresponding calling the vector retrieval module in the case that the second judgment result is to produce a positive effect; the data processing apparatus 50 further comprises a third calling module (not shown) for obtaining the intent recognition result as corresponding calling the graph relationship retrieval module in the case that the third judgment result is to produce a positive effect; the data processing apparatus 50 further comprises a fourth calling module (not shown) for obtaining the intent recognition result as corresponding calling at least two of the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module in the case that the fourth judgment result is to produce a positive effect.

[0066] In some disclosed embodiments, the fusion agent comprises a knowledge specification module, a conflict detection module, a verification and correction module, the retrieval module 53 further comprises a data fusion module (not shown) for normalizing the target knowledge based on the coordination agent calling the knowledge specification module combined with the preset prompt word to obtain the fusion knowledge data; the retrieval module 53 further comprises a conflict detection module (not shown) for content conflict detection on the fusion knowledge data based on the coordination agent calling the conflict detection module to obtain the conflict detection result; the retrieval module 53 further comprises a third response module (not shown) for, in response to the conflict detection result being that the fusion knowledge data does not have content conflict, the coordination agent outputting the fusion result as the target result data; the retrieval module 53 further comprises a fourth response module (not shown) for, in response to the conflict detection result being that the fusion knowledge data has content conflict, the coordination agent calling the verification and correction module to process the fusion knowledge data to obtain new fusion knowledge data after correction, and the coordination agent outputting the new fusion knowledge data as the target result data.

[0067] In some disclosed embodiments, the analysis agent comprises a segmentation module, a keyword extraction sub-module, a vector extraction module, a relationship extraction module, a knowledge base updating module, the updating module 52 further comprises a segmented analysis module (not shown) for content analysis segmentation of the standardized text based on the coordination agent calling the segmentation module to obtain the target subtext; the updating module 52 further comprises a text processing module (not shown) for calling the keyword extraction sub-module, the vector extraction module, and the relationship extraction module based on the coordination agent to perform keyword extraction, vectorization processing, and entity relationship extraction on the target subtext to obtain the keyword index of the target subtext, the target vector, and the knowledge graph; the updating module 52 further comprises a knowledge base updating module (not shown) for updating the target knowledge base of the target scene based on the coordination agent calling the knowledge base updating module combined with the keyword index, the target vector, and the knowledge graph of each target subtext.

[0068] In some disclosed embodiments, the executing agent further comprises a message queue writing agent, after the cleaning agent processes the knowledge source material according to the corresponding data type to obtain the standardized text, the data processing apparatus 50 further comprises a message queue obtaining module (not shown) for calling the message queue writing agent based on the coordinating agent to obtain the intermediate process data of the knowledge source material processed by the cleaning agent to obtain a target message queue; wherein the target message queue is used to be called by the coordinated agent to realize asynchronous processing, failure rollback and parallel expansion of the cleaning agent.

[0069] In some disclosed embodiments, the data source type of the knowledge source material in the data processing apparatus 50 comprises at least one of a database, a file, a webpage and an audio system related to the target scene; and / or the target scene is an enterprise-level data governance scene.

[0070] Please refer to Figure 6 , Figure 6 is a schematic diagram of the framework of an embodiment of the electronic device. The electronic device 60 at least comprises a memory 61 and a processor 62 coupled with each other, the memory 61 at least stores program instructions, and the processor 62 is configured to execute the program instructions to implement the steps in any of the above data processing method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. Exemplarily, the electronic device 60 can include but is not limited to a smart phone, a tablet computer and the like, and the specific type of the electronic device 60 is not limited here.

[0071] Specifically, the processor 62 is configured to control itself and the memory 61 to implement the steps in any of the above data processing method embodiments. The processor 62 can also be referred to as a CPU (Central Processing Unit). The processor 62 can be an integrated circuit chip with processing capability. The processor 62 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0072] The above scheme, the electronic device 60 obtains target data about the target scene, and obtains a plurality of execution agents, the plurality of execution agents specifically include: a cleaning agent, an analysis agent, a retrieval agent and a fusion agent, each execution agent is respectively connected with the coordination agent as an edge node in a star topology communication, when the target data is a knowledge source material, the cleaning agent is called based on the coordination agent to process the knowledge source material according to the corresponding data type, and the standardized text is obtained, then the analysis agent is called based on the coordination agent to update the semantic analysis result of the standardized text, and the target knowledge base of the target scene, when the target data is a query statement to be queried, the retrieval agent is called based on the coordination agent to retrieve the target knowledge base according to the query statement to be queried, and the target knowledge data is obtained, and then the fusion agent is called based on the coordination agent to process the target knowledge data, and the target query result used to reply to the query statement to be queried is obtained. On the one hand, by connecting each execution agent with the coordination agent as an edge node in communication, the independent processing of the corresponding function can be realized by scheduling each execution agent through the coordination agent, which is suitable for complex business scenarios with frequent knowledge addition and complex sources. On the other hand, the updating of the target knowledge base is completed through the step-by-step processing of the cleaning agent and the analysis agent, and the knowledge source material is processed according to the corresponding data type, which can flexibly adapt to the processing needs of different sources and different types of knowledge in the target scene, facilitate timely response to the knowledge update needs of the target scene, and improve the comprehensiveness and accuracy of the target knowledge base in the corresponding target scene. In another aspect, for the query statement to be queried, the target knowledge data is retrieved from the target knowledge base of the corresponding target scene through the retrieval agent, and then the target query result is obtained through the processing of the fusion agent, which can retrieve the knowledge data related to the query statement to be queried based on the target knowledge base continuously updated in the target scene, thereby improving the response accuracy of the query task to be queried in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base can be improved, and the response accuracy of the query task to be queried in the target scene can be improved to adapt to the processing needs in complex business scenarios.

[0073] Please refer to Figure 7 , Figure 7 is a framework diagram of an embodiment of the data processing system of the present application. The data processing system 70 includes a coordination agent 71 and a plurality of execution agents 72 connected with the coordination agent 71 as an edge node in a star topology communication, and the data processing system 70 is used to implement the steps in any of the above data processing method embodiments.

[0074] In one implementation scenario, the plurality of execution agents 72 include a cleaning agent (not shown), an analysis agent (not shown), a retrieval agent (not shown) and a fusion agent (not shown), which can be specifically combined with the foregoing detailed embodiments. For the sake of brevity, they will not be described here.

[0075] Therefore, the data processing system 70 acquires target data about a target scene, and acquires a plurality of execution agents 72, which specifically include a cleaning agent, a parsing agent, a retrieval agent, and a fusion agent, each of which is communicatively connected to the coordination agent 71 as an edge node in a star topology. When the target data is knowledge source material, the cleaning agent is invoked based on the coordination agent 71 to process the knowledge source material according to a corresponding data type to obtain standardized text. The parsing agent is invoked based on the coordination agent 71 to update a target knowledge base of the target scene based on a semantic analysis result of the standardized text. When the target data is a to-be-queried statement, the retrieval agent is invoked based on the coordination agent 71 to retrieve the target knowledge base according to the to-be-queried statement to obtain target knowledge data. The fusion agent is invoked based on the coordination agent 71 to process the target knowledge data to obtain a target query result for responding to the to-be-queried statement. On the one hand, by communicatively connecting each of the execution agents 72 to the coordination agent 71 as an edge node, the coordination agent 71 can schedule each of the execution agents 72 to independently process corresponding functions, which is suitable for complex business scenarios with frequent knowledge updates and complex sources. On the other hand, by step-by-step processing of the cleaning agent and the parsing agent to update the target knowledge base, and by processing the knowledge source material according to the corresponding data type, the processing requirements of different sources and different types of knowledge in the target scene can be flexibly adapted, which facilitates timely response to the knowledge update requirements of the target scene and improves the comprehensiveness and accuracy of the target knowledge base of the corresponding target scene. On the other hand, for the to-be-queried statement, the retrieval agent retrieves the target knowledge data from the target knowledge base of the corresponding target scene, and the fusion agent processes the target query result, which can retrieve the knowledge data related to the to-be-queried statement based on the target knowledge base that is continuously updated in the target scene, thereby improving the response accuracy of the to-be-queried task in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base can be improved, and the response accuracy of the to-be-queried task in the target scene can be improved to adapt to the processing requirements in complex business scenarios.

[0076] Please refer to Figure 8 , Figure 8 is a frame diagram of an embodiment of the computer-readable storage medium. The computer-readable storage medium 80 stores program instructions 81 capable of being executed by a processor, which are used to implement the steps in any of the above data processing method embodiments.

[0077] The above scheme, the computer readable storage medium 80 obtains target data about a target scene, and obtains a plurality of execution agents, the plurality of execution agents specifically include a cleaning agent, an analysis agent, a retrieval agent and a fusion agent, each execution agent is respectively connected with a coordination agent in a star topology communication as an edge node, when the target data is a knowledge source material, the cleaning agent is called based on the coordination agent to process the knowledge source material according to the corresponding data type, and standardized text is obtained, then the analysis agent is called based on the coordination agent to update the semantic analysis result of the standardized text, and the target knowledge base of the target scene, when the target data is a to-be-queried statement, the retrieval agent is called based on the coordination agent to retrieve the target knowledge base according to the to-be-queried statement, and target knowledge data is obtained, and then the fusion agent is called based on the coordination agent to process the target knowledge data, and target query results for replying to the to-be-queried statement are obtained. On the one hand, by connecting each execution agent with the coordination agent in communication as an edge node, the independent processing of the corresponding functions can be realized by scheduling each execution agent by the coordination agent, which is suitable for complex business scenarios with frequent knowledge addition and complex sources. On the other hand, the updating of the target knowledge base is completed through the step-by-step processing of the cleaning agent and the analysis agent, and the knowledge source material is processed according to the corresponding data type, which can flexibly adapt to the processing requirements of different sources and different types of knowledge in the target scene, facilitate timely response to the knowledge update requirements of the target scene, and improve the comprehensiveness and accuracy of the target knowledge base of the corresponding target scene. On the other hand, for the to-be-queried statement, the target knowledge data is retrieved from the target knowledge base of the corresponding target scene by the retrieval agent, and then the target query results are obtained by the fusion agent, which can retrieve the knowledge data related to the to-be-queried statement based on the target knowledge base updated in the target scene, thereby improving the response accuracy of the to-be-queried task in the target scene. Therefore, the comprehensiveness of the knowledge about the target scene in the target knowledge base can be improved, and the response accuracy of the to-be-queried task in the target scene can be improved to adapt to the processing requirements in complex business scenarios.

[0078] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and specific implementations can refer to the descriptions of the above method embodiments. For brevity, they will not be described here.

[0079] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, they will not be described here.

[0080] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely an example, and the division of the modules or units can be different, for example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0082] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0083] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

Claims

1. A data processing method, characterized by, The method comprises: acquiring target data about a target scene, and acquiring a plurality of execution agents; wherein the plurality of execution agents comprise a cleaning agent, an analysis agent, a retrieval agent, and a fusion agent, and each of the execution agents is connected to a coordination agent in a star topology as an edge node; in response to the target data being knowledge source materials, calling the cleaning agent based on the coordination agent to process the knowledge source materials according to corresponding data types to obtain standardized texts, and then calling the analysis agent based on the coordination agent to update a semantic analysis result of the standardized texts to obtain a target knowledge base of the target scene; in response to the target data being a query statement, calling the retrieval agent based on the coordination agent to retrieve the target knowledge base according to the query statement to obtain target knowledge data, and then calling the fusion agent based on the coordination agent to process the target knowledge data to obtain a target query result for responding to the query statement.

2. The method of claim 1, wherein, The retrieval agent comprises an intent recognition module, a full-text retrieval module, a vector retrieval module, and a graph relationship retrieval module, and the calling of the retrieval agent based on the coordination agent to retrieve the target knowledge base according to the query statement comprises: calling the intent recognition module based on the coordination agent to identify a query intent of the query statement to obtain an intent recognition result of the query statement; based on the intent recognition result, calling at least one of the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module to retrieve the target knowledge base according to the query statement to obtain target knowledge data.

3. The method of claim 2, wherein, The calling of the intent recognition module based on the coordination agent to identify a query intent of the query statement to obtaining an intent recognition result of the query statement comprises: the intent recognition module sequentially performs: based on a field attribute of the query statement, obtaining a first judgment result of whether a positive effect is produced by a deterministic retrieval mode; based on a relationship requirement of the query statement, obtaining a second judgment result of whether a positive effect is produced by a similarity retrieval mode; based on a query type of the query statement, obtaining a third judgment result of whether a positive effect is produced by a relevance retrieval mode; and based on a conflict degree of the query statement, obtaining a fourth judgment result of whether a positive effect is produced by at least two of the deterministic retrieval mode, the similarity retrieval mode, and the relevance retrieval mode; in response to the latest obtained judgment result representing that a corresponding retrieval mode does not produce a positive effect in the sequential execution process, continuing to execute generation of a next judgment result, and in a case where the fourth judgment result is the latest obtained judgment result and the fourth judgment result represents that the corresponding retrieval mode does not produce a positive effect, calling the full-text retrieval module, the vector retrieval module, and the graph relationship retrieval module as the intent recognition result. In response to a latest obtained judgment result representing that the corresponding search method produces a positive effect in the sequential execution process, the sequential execution is ended, and a corresponding intent recognition result is generated based on the latest obtained judgment result.

4. The method of claim 3, wherein, The first judgment result of whether the search based on the deterministic search method produces a positive effect is obtained based on a field attribute of the to-be-queried sentence, including: In response to the field attribute of the to-be-queried sentence containing a special term related to the target scene, the first judgment result is obtained as producing a positive effect; The second judgment result of whether the search based on the similarity search method produces a positive effect is obtained based on a relationship requirement of the to-be-queried sentence, including: In response to the relationship requirement of the to-be-queried sentence containing context semantic dependency, the second judgment result is obtained as producing a positive effect; The third judgment result of whether the search based on the association search method produces a positive effect is obtained based on a query type of the to-be-queried sentence, including: In response to the query type of the to-be-queried sentence being a global query, the third judgment result is obtained as producing a positive effect; The fourth judgment result of whether the search based on at least two search methods of the deterministic search method, the similarity search method, and the association search method produces a positive effect is obtained based on a conflict degree of the to-be-queried sentence, including: In response to the conflict degree of the to-be-queried sentence representing that the to-be-queried sentence is a multi-entity jump statement, the fourth judgment result is obtained as producing a positive effect.

5. The method of claim 3, wherein, The method further includes: In a case where the first judgment result is a positive effect, an intent recognition result is obtained as corresponding to calling the full-text search module; In a case where the second judgment result is a positive effect, an intent recognition result is obtained as corresponding to calling the vector search module; In a case where the third judgment result is a positive effect, an intent recognition result is obtained as corresponding to calling the graph relationship search module; In a case where the fourth judgment result is a positive effect, an intent recognition result is obtained as corresponding to calling at least two of the full-text search module, the vector search module, and the graph relationship search module.

6. The method according to any one of claims 1 to 5, characterized in that, The fusion agent includes a knowledge specification module, a conflict detection module, and a verification and correction module, and the coordinated agent calls the fusion agent to process the target knowledge data, to obtain a target query result for replying to the to-be-queried sentence, including: The coordinated agent calls the knowledge specification module to perform normalized processing on the target knowledge in combination with a preset prompt word, to obtain fusion knowledge data; The coordinated agent calls the conflict detection module to perform content conflict detection on the fusion knowledge data, to obtain a conflict detection result; In response to the conflict detection result being that the fusion knowledge data does not have a content conflict, the coordinated agent outputs the fusion result as target result data. In response to the conflict detection result being that the fusion knowledge data has content conflict, the coordination agent calls the verification and correction module to process the fusion knowledge data to obtain new fusion knowledge data after correction, and the coordination agent outputs the new fusion knowledge data as target result data.

7. The method according to any one of claims 1 to 6, characterized in that, The analysis agent includes a segmentation module, a keyword extraction submodule, a vector extraction module, a relationship extraction module, and a knowledge base updating module. The semantic analysis result of the standardized text is updated based on the coordination agent calling the analysis agent to update the target knowledge base of the target scene, including: The coordination agent calls the segmentation module to perform content analysis segmentation on the standardized text to obtain target subtext; The coordination agent respectively calls the keyword extraction submodule, the vector extraction module, and the relationship extraction module to perform keyword extraction, vectorization processing, and entity relationship extraction on the target subtext to obtain the keyword index, target vector, and knowledge graph of the target subtext; The coordination agent calls the knowledge base updating module to update the target knowledge base of the target scene in combination with the keyword index, target vector, and knowledge graph of each target subtext.

8. The method according to any one of claims 1 to 7, characterized in that, The execution agent further includes a message queue writing agent. After the cleaning agent processes the knowledge source material according to the corresponding data type to obtain the standardized text based on the coordination agent, the method further includes: The coordination agent calls the message queue writing agent to obtain the intermediate process data of the cleaning agent processing the knowledge source material to obtain a target message queue. The target message queue is used to be called by the coordination agent to realize asynchronous processing, failure rollback, and parallel expansion of the cleaning agent.

9. The method according to any one of claims 1 to 8, characterized in that, The data source type of the knowledge source material includes at least one of a database, a file, a webpage, and an audio system related to the target scene. And / or, the target scene is an enterprise-level data governance scene.

10. A data processing apparatus, characterized by The method includes: An acquisition module is configured to acquire target data related to a target scene and acquire a plurality of execution agents. The plurality of execution agents include a cleaning agent, an analysis agent, a retrieval agent, and a fusion agent. Each execution agent is connected to a coordination agent in a star topology. An updating module is configured to, in response to the target data being a knowledge source material, call the cleaning agent to process the knowledge source material according to a corresponding data type based on the coordination agent to obtain a standardized text, and call the analysis agent to update a target knowledge base of the target scene based on the semantic analysis result of the standardized text. The retrieval module is configured to, in response to the target data being a query statement, call the retrieval agent to retrieve the target knowledge base according to the query statement, obtain target knowledge data, call the fusion agent to process the target knowledge data according to the coordination agent, and obtain a target query result for responding to the query statement.

11. An electronic device, comprising: At least comprising a memory and a processor coupled with each other, the memory at least stores program instructions, and the processor is configured to execute the program instructions to implement the data processing method of any one of claims 1 to 9.

12. A data processing system, characterized by The data processing system comprises a coordination agent and a plurality of execution agents connected with the coordination agent in a star topology, and is configured to implement the data processing method of any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The memory stores program instructions capable of being executed by the processor, and the program instructions are configured to implement the data processing method of any one of claims 1 to 9.

Images