Data processing method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as deep learning, large models, natural language processing and AI security. According to the specific scheme, firstly, a received first input statement is analyzed, description information corresponding to the first input statement is determined, a target data processing mode is determined on the basis of the description information, then a first reference information set corresponding to the first input statement is obtained, then the first reference information set is processed on the basis of the target data processing mode, and the first input statement is obtained. And finally, inputting the second reference information set into the large model to obtain a response result generated by the large model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as deep learning, big models, natural language processing, and AI security, and specifically to a big model-based data processing method, device, electronic device, and storage medium. Background Art

[0002] Currently, model services can generate more accurate content that better meets user needs through Retrieval Augmented Generation (RAG) technology, which integrates the three key stages of retrieval, enhancement, and generation. However, the accuracy of related model services is low when processing tasks involving timeliness and multi-source information. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] The first embodiment of the present disclosure proposes a data processing method based on a large model, including:

[0005] Parsing the received first input sentence to determine description information corresponding to the first input sentence;

[0006] Based on the description information, determining a target data processing mode;

[0007] Acquire a first reference information set corresponding to the first input sentence;

[0008] Based on the target data processing mode, the first reference information set is processed to obtain a second reference information set;

[0009] The second reference information set is input into the large model to obtain a response result generated by the large model.

[0010] The second aspect of the present disclosure provides a data processing device based on a large model, including:

[0011] A parsing module, configured to parse the received first input sentence and determine description information corresponding to the first input sentence;

[0012] A determination module, used to determine a target data processing mode based on the description information;

[0013] An acquisition module, configured to acquire a first reference information set corresponding to the first input sentence;

[0014] A processing module, configured to process the first reference information set based on the target data processing mode to obtain a second reference information set;

[0015] The input module is used to input the second reference information set into the large model to obtain a response result generated by the large model.

[0016] The third aspect of the present disclosure provides an electronic device, including:

[0017] at least one processor;

[0018] and, a memory communicatively coupled to the at least one processor;

[0019] Among them, the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the large model-based data processing method proposed in the embodiment of the first aspect of the present disclosure.

[0020] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the large model-based data processing method proposed in the first aspect embodiment of the present disclosure.

[0021] The fifth aspect of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the large model-based data processing method as provided in the first aspect of the present disclosure.

[0022] The data processing method, device, electronic device and storage medium based on the big model provided by the present disclosure have the following beneficial effects:

[0023] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and then the first reference information set is processed based on the target data processing mode to obtain the second reference information set, and finally a response result is generated based on the second reference information set. Thus, by determining the processing mode based on the description information corresponding to the input sentence, and then processing the obtained first reference information set based on the processing mode to obtain the second reference information set, and then generating the corresponding response result based on the second reference information set, the reference information is processed and then handed over to the large model for generating the response result, which improves the accuracy of the model in processing timeliness and multi-source data, and improves the accuracy and reliability of data processing.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 A flowchart of a data processing method based on a large model provided by an embodiment of the present disclosure;

[0027] Figure 2 A flowchart of a data processing method based on a large model provided by an embodiment of the present disclosure;

[0028] Figure 3 A flowchart of a data processing method based on a large model provided by an embodiment of the present disclosure;

[0029] Figure 4 A flowchart of a data processing method based on a large model provided by an embodiment of the present disclosure;

[0030] Figure 5 A flowchart of a data processing method based on a large model provided by an embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of the structure of a data processing device based on a large model provided in an embodiment of the present disclosure;

[0032] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] The present disclosure relates to artificial intelligence technology fields such as deep learning, large models, natural language processing, and AI security.

[0035] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0036] Deep Learning (DL) is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0037] The large model can also be called the Foundation Model. The model extracts knowledge through billions of corpora or images, learns and then produces a large model with billions of parameters.

[0038] Natural Language Processing (NLP) is an interdisciplinary subject in the fields of computer science, artificial intelligence and linguistics. It mainly studies how to enable computers to understand, process, generate and simulate human language, so as to achieve the ability to have natural conversations with humans.

[0039] AI security refers to ensuring that artificial intelligence systems can prevent potential security risks and abuse during development, deployment and operation, while ensuring that their decision-making process is transparent, fair and unbiased, and can effectively respond to external attacks or internal vulnerabilities. AI security not only focuses on the technical aspects of the system (such as preventing hackers or data leaks), but also includes ensuring that AI can make decisions that comply with ethical and legal norms and avoid adverse effects on individuals or society.

[0040] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0041] The following describes the large model-based data processing method, device, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0042] Figure 1 A flowchart of a data processing method based on a large model provided in an embodiment of the present disclosure.

[0043] like Figure 1 As shown, the data processing method based on the large model may include the following steps:

[0044] Step 101: parse a received first input sentence to determine description information corresponding to the first input sentence.

[0045] In some possible implementation forms, the description information may include one or more of the following: first time information corresponding to the first input sentence; type of the first time information; entities included in the first input sentence; type of entities; relationship between entities. By determining the time information and entity information of the first input sentence, a basis is provided for improving the accuracy and reliability of data processing.

[0046] The first time information may be specific time information in the first input sentence, or may be time information calculated based on the current time. For example, when the first input sentence is "What movies were released last year", the first time information is last year. When the first input sentence is "What movies were released in the last three months", assuming that the current time is June 12 of a certain year, the first time information is from March 12 of a certain year to June 12 of a certain year. This disclosure does not limit this.

[0047] The type of the first time information may be an absolute time type or a relative time type. For example, when the first time information is specific time information, its type is an absolute time type, and when the first time information is time information calculated based on the current time, its type is a relative time type, which is not limited in the present disclosure.

[0048] The entities included in the first input sentence can be determined according to actual needs. For example, if the first input sentence is "In which year did event A occur?", the entity included in it is event A; if the first input sentence is "In which year was movie B released?", the entity included in it is movie B, etc., and the present disclosure does not limit this.

[0049] It should be noted that the type of entity contained in the first input sentence can be any type of entity determined according to actual needs, for example, it can be an event type, a product type, etc., which is not limited in the present disclosure.

[0050] In the present disclosure, after receiving the first input sentence, when parsing the first input sentence, if the first input sentence contains first time information, the first time information can be parsed through natural language processing and deep learning models (such as BERT) to determine the type of the first time information. In addition, the parsing accuracy of the first time information can be improved through regular expressions and context analysis, which is not limited in the present disclosure.

[0051] Among them, BERT is the abbreviation of Bidirectional Encoder Representations from Transformers (BERT).

[0052] In the present disclosure, when parsing the first input sentence, the entities in the first input sentence and the relationships between the entities can be identified through the knowledge graph and the relationship extraction method, and the accuracy and completeness of the identification can be improved. For example, when the first input sentence is "Is the teacher of character C character D?", the entities contained therein are "character C" and "character D", and the relationship between the entities is a teacher-student relationship, etc., and the present disclosure does not limit this.

[0053] It should be noted that when parsing the first input statement, the events in the first input statement can also be matched with known events in the database through entity recognition and linking technology, thereby improving the efficiency and accuracy of parsing the first input statement, wherein the database can be pre-set, and the present disclosure does not limit this.

[0054] Step 102: determine the target data processing mode based on the description information.

[0055] The target data processing mode may be a mode for processing at least one of the first input statement, the search result corresponding to the first input statement, and the query result.

[0056] Among them, the target data processing mode may include one or more of the following: information update, information identification, information fusion, information calculation and information supplement, information exemption, etc., which is not limited in this disclosure.

[0057] Among them, information update can be used to update information with high timeliness requirements and real-time changing information. For example, when the information queried by the first input statement is real-time changing information such as population data, job changes, character movements, news, etc., in order to ensure the timeliness and accuracy of the information, the information can be updated, and this disclosure does not limit this.

[0058] Among them, information authentication can be used to identify the source and semantics of information, thereby ensuring the reliability of the information.

[0059] Among them, information fusion can be used to fuse information with associations, or to derive and fuse information, etc. For example, based on the associations between multiple documents that describe the same entity and do not conflict, multiple documents can be fused, or when the information in document #1 is: b can be derived from a, and the information in document #2 is: c can be derived from b, document #1 and document #2 can be fused to obtain information that c can be derived from a, etc., and the present disclosure does not limit this.

[0060] Among them, information calculation can be used to calculate the causal relationship between information, which may include direct calculation, intervention calculation, counterfactual calculation and other calculation methods, which are not limited in this disclosure.

[0061] Among them, information supplementation can be used to combine context prediction, supplement the time-event chain in the information, or supplement the relative time corresponding to the first input sentence, etc., which is not limited in the present disclosure.

[0062] Among them, information disclaimer can be used to make disclaimers for outdated or incomplete information, which may include disclaimers for time and events, so as to effectively avoid the spread of misleading information and thus improve user experience. This disclosure does not limit this.

[0063] In the present disclosure, after determining the descriptive information corresponding to the first input sentence, in order to improve the accuracy of the large model in processing timeliness and multi-source information, the target data processing mode can be determined based on the descriptive information, thereby providing conditions for improving the accuracy of data processing.

[0064] It should be noted that the specific type and structure of the large model can be determined according to actual needs. For example, the data processing method based on the large model disclosed in the present invention can be applied to a large language model combined with RAG technology, etc., and the present disclosure does not limit this.

[0065] RAG is the abbreviation of Retrieval Augmented Generation (RAG).

[0066] Step 103: Acquire a first reference information set corresponding to the first input sentence.

[0067] The first reference information set may be a set of search information corresponding to the first input sentence.

[0068] It should be noted that the specific source of the reference information in the first reference information set can be determined according to actual conditions. For example, the information source can be a preset database, or a real-time updated data source (such as a website, etc.), or a knowledge graph, etc., and this disclosure does not limit this.

[0069] In the present disclosure, after determining the target data processing mode, a search may be performed based on the first input sentence to obtain the corresponding first reference information set.

[0070] Step 104: Process the first reference information set based on the target data processing mode to obtain a second reference information set.

[0071] In the present disclosure, after obtaining the first reference information set corresponding to the first input sentence, in order to improve the accuracy of the reference information set, the first reference information set may be processed based on the target data processing mode to obtain a second reference information set.

[0072] It should be noted that when the target data processing mode includes information update, the timeliness information in the first reference information set can be updated through the stream processing framework and API interface, and the change detection algorithm can be used to identify and update the change trends in the first reference information set, such as job changes or demographic updates and other real-time change information, which is not limited to this in the present disclosure.

[0073] Among them, API is the abbreviation of Application Programming Interface (API).

[0074] It should be noted that when the target data processing mode includes information identification, the blockchain verification mechanism can be used to ensure the credibility of the information source in the first reference information set. The scoring system can also be used to score and sort the information. Semantic analysis and logical reasoning can also be performed on the information in the first reference information set to identify and avoid false correlation matches, such as identifying situations where the information in the first reference information set does not match the entity link of the first input sentence, to ensure that the information is semantically consistent with the first input sentence, thereby performing fine-grained information identification.

[0075] It should be noted that when the target data processing mode includes information fusion, a fusion algorithm based on a graph structure can be used to achieve fusion of multi-source information in the first reference information set. For example, multiple documents can be fused based on the implicit relationship between them to ensure the accuracy of information fusion. Information fusion can also be performed through causal reasoning technology, etc., and the present disclosure does not limit this.

[0076] It should be noted that when the target data processing mode includes information calculation, causal reasoning tools (such as DoWhy) can be used to implement causal analysis of the information in the first reference information set. Causal diagrams and structural equation models can also be used to perform direct calculations, intervention calculations or counterfactual calculations, which can be used to calculate deceased people or historical events, or historical event simulation tools can be used to test the causal relationship of past events through a simulated environment, thereby improving the reasoning ability of the large model. The present disclosure does not limit this.

[0077] Among them, DoWhy is a framework for causal inference, which can analyze and understand causal relationships.

[0078] It should be noted that when the target data processing mode includes information supplementation, the context information can be combined to automatically inject timestamps to supplement the information without time subjects in the first reference information set and improve the time-event chain. It is also possible to establish a cross-document time span chain based on graph database technology to support the supplementation and reasoning of complex time queries in the first input sentence. The first time information of the first input sentence can also be supplemented through external retrieval and time information. The present disclosure does not limit this.

[0079] It should be noted that when the target data processing mode includes information disclaimer, it can automatically generate a disclaimer for outdated or incomplete information through preset rules and semantic analysis. It can also evaluate the obsolescence of information through statistical analysis and historical data comparison, and assist in generating accurate disclaimer scripts for variable events (such as "Who is the current agent of Country A?") and large-scale events (such as "Which movies are released recently?"). The present disclosure does not limit this.

[0080] Step 105: input the second reference information set into the large model to obtain a response result generated by the large model.

[0081] The response result may be a query result corresponding to the first input statement.

[0082] In the present disclosure, after the first reference information set is processed based on the target data processing mode to obtain the second reference information set, the second reference information set can be input into the large model to obtain the response result generated by the large model, thereby obtaining the response result generated by the large model by inputting the processed reference information set into the large model, thereby improving the accuracy and timeliness of the generated response result.

[0083] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and then the first reference information set is processed based on the target data processing mode to obtain the second reference information set, and finally the second reference information set is input into the large model to obtain the response result generated by the large model. Thus, by determining the processing mode based on the description information corresponding to the input sentence, and then processing the obtained first reference information set based on the processing mode to obtain the second reference information set, and then generating the corresponding response result based on the second reference information set, the accuracy of the large model in processing timeliness and multi-source data is improved, and the accuracy and reliability of data processing are improved.

[0084] Figure 2 A flowchart of a data processing method based on a large model provided in one embodiment of the present disclosure.

[0085] like Figure 2 As shown, the data processing method based on the large model may include the following steps:

[0086] Step 201: parse the received first input sentence to determine description information corresponding to the first input sentence.

[0087] Step 202: determine the target data processing mode based on the description information.

[0088] The specific implementation forms of step 201 to step 202 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0089] Step 203: When the target data processing mode includes a first processing mode associated with the input sentence, the first input sentence is updated based on the first processing mode to obtain a second input sentence.

[0090] In some possible implementation forms, the first processing mode may be one or more of the following: obtaining the relationship between entities in the first input sentence; obtaining the first time information corresponding to the first input sentence.

[0091] In the present disclosure, after determining the target data processing mode, in the case where the target data processing mode includes the first processing mode associated with the input sentence, it can be determined that there is outdated information or erroneous information in the first input sentence, which may be that the relationship between entities is outdated or erroneous, or the first time information is erroneous, etc. At this time, based on the first processing mode, the relationship between entities in the first input sentence can be obtained, and / or the first time information corresponding to the first input sentence can be obtained, and based on the obtained relationship between entities and / or the first time information, the first input sentence is updated to obtain the second input sentence.

[0092] For example, suppose user D published article D in a certain year, and the first input sentence is "In which year did user B publish article D?" The entity relationship in the first input sentence is: user B published article D, which is an incorrect entity relationship. At this time, the incorrect entity relationship can be clarified. Thus, the entity relationship of the first input sentence can be obtained as: user D published article D, and based on the obtained entity relationship, the first input sentence is updated to obtain the second input sentence "In which year did user D publish article D?". Therefore, when there is incorrect information in the input sentence, such as incorrect entity relationship and / or incorrect time information, the input sentence can be updated by obtaining the correct information corresponding to the input sentence, thereby improving the reliability and accuracy of the input sentence.

[0093] Step 204: Acquire a first reference information set corresponding to the second input sentence.

[0094] In the present disclosure, after the first input statement is updated based on the first processing mode to obtain the second input statement, a search can be performed based on the second input statement to obtain the corresponding first reference information set, thereby performing a search based on the updated second input statement, thereby improving the accuracy of the obtained first reference information set.

[0095] Step 205: Process the first reference information set based on the target data processing mode to obtain a second reference information set.

[0096] Step 206: input the second reference information set into the large model to obtain a response result generated by the large model.

[0097] The specific implementation forms of step 205 to step 206 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0098] In the disclosed embodiment, a received first input statement is first parsed to determine descriptive information corresponding to the first input statement, and based on the descriptive information, a target data processing mode is determined. Then, when the target data processing mode includes a first processing mode associated with the input statement, the first input statement is updated based on the first processing mode to obtain a second input statement. Thereafter, a first reference information set corresponding to the second input statement is obtained, and based on the target data processing mode, the first reference information set is processed to obtain a second reference information set. Finally, the second reference information set is input into a large model to obtain a response result generated by the large model. Therefore, after determining the processing mode based on the descriptive information of the input sentence, when the processing mode includes a processing mode associated with the input sentence, the input sentence is updated based on the processing mode associated with the input sentence, and a first reference information set is obtained based on the updated input sentence, and the first reference information set is processed based on the processing mode to obtain a second reference information set, and the second reference information set is input into the big model to obtain a response result generated by the big model, thereby retrieving through the updated input sentence to obtain the reference information set, thereby improving the reliability of the reference information set, and after processing the reference information set based on the processing mode, the big model generates a response result based on the processed reference information set, thereby improving the accuracy of the generated response result, thereby improving the accuracy and reliability of data processing based on the big model.

[0099] Figure 3 A flowchart of a data processing method based on a large model provided in one embodiment of the present disclosure.

[0100] like Figure 3 As shown, the data processing method based on the large model may include the following steps:

[0101] Step 301: parse the received first input sentence to determine description information corresponding to the first input sentence.

[0102] Step 302: determine the target data processing mode based on the description information.

[0103] Step 303: Acquire a first reference information set corresponding to the first input sentence.

[0104] The specific implementation forms of steps 301 to 303 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0105] Step 304 : Based on the target data processing mode and according to the source of each first reference information in the first reference information set, determine a first weight of the first reference information.

[0106] It should be noted that the source of the first reference information may be a preset database, or a knowledge graph, or a knowledge base, or a real-time updated data source (such as a website, etc.), etc., and the present disclosure does not limit this.

[0107] The first weight may be used to indicate the credibility of the source of the first reference information. The higher the credibility of the source of the first reference information, the greater the corresponding first weight. This disclosure does not limit this.

[0108] It should be noted that when determining the credibility of the source of the first reference information, the credibility of the source of the first reference information can be determined through a blockchain-based verification mechanism, and the present disclosure does not limit this.

[0109] In the present disclosure, after obtaining the first reference information set corresponding to the first input sentence, the first weight of each first reference information can be determined according to the source of each first reference information in the first reference information set. Thus, by determining the weight of each first reference information, the large model can be helped to select information with reliable sources and improve the reliability of data processing.

[0110] Step 305: Associating the first weight of each first reference information with the first reference information set to obtain a second reference information set.

[0111] In the present disclosure, after determining the first weight of each first reference information in the first reference information set, in order to ensure the reliability of the information source during data processing, the first weight of each first reference information can be associated and added to the first reference information set to obtain a second reference information set.

[0112] Optionally, when the first weight of each first reference information is associated and added to the first reference information set to obtain the second reference information set, the trust scoring system and the first weight can also be combined to score and sort the information in the second reference information set, thereby helping the model select the latest and most reliable information. The present disclosure does not limit this.

[0113] Optionally, when the first weight of each first reference information is associated and added to the first reference information set to obtain the second reference information set, the information in the second reference information set can also be filtered based on the first weight to remove information with a lower first weight, thereby effectively reducing noise information in the second reference information set and improving data processing efficiency. The present disclosure does not limit this.

[0114] Step 306: input the second reference information set into the large model to obtain a response result generated by the large model.

[0115] The specific implementation of step 306 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0116] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and based on the target data processing mode, the first weight of the first reference information is determined according to the source of each first reference information in the first reference information set, and then the first weight of each first reference information is associated and added to the first reference information set to obtain the second reference information set, and finally the second reference information set is input into the large model to obtain the response result generated by the large model. Thus, after obtaining the first reference information set corresponding to the input sentence, the weight of the reference information is determined based on the source of each first reference information, and is associated with the reference information to obtain the second reference information set, and the second reference information set with the associated weight is input into the large model to obtain the response result generated by the large model, thereby improving the reliability and accuracy of the large model when processing multi-source data.

[0117] Figure 4 A flowchart of a data processing method based on a large model provided in one embodiment of the present disclosure.

[0118] like Figure 4 As shown, the data processing method based on the large model may include the following steps:

[0119] Step 401: parse the received first input sentence to determine description information corresponding to the first input sentence.

[0120] Step 402: determine the target data processing mode based on the description information.

[0121] Step 403: Acquire a first reference information set corresponding to the first input sentence.

[0122] The specific implementation forms of steps 401 to 403 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0123] Step 404 , based on the target data processing mode, when there is an association relationship between multiple first reference information in the first reference information set, the multiple first reference information are merged according to the association relationship to obtain second reference information.

[0124] It should be noted that the correlation relationship between the first reference information may be a causal relationship, or a temporal relationship, etc., which is not limited in the present disclosure.

[0125] In the present disclosure, after obtaining the first reference information set corresponding to the first input sentence, based on the target data processing mode, when there is a correlation relationship between multiple first reference information in the first reference information set, these multiple first reference information can be fused according to the correlation relationship to obtain second reference information, thereby obtaining the second reference information by fusing the multiple first reference information based on the correlation relationship between the multiple first reference information, thereby improving the accuracy and completeness of the obtained second reference information.

[0126] Step 405: input the second reference information into the large model to obtain a response result generated by the large model.

[0127] In the present disclosure, after a plurality of first reference information are fused according to the association relationship between the plurality of first reference information in the first reference information set to obtain the second reference information, the second reference information can be input into the large model to obtain a response result generated by the large model, thereby generating a response result by fusing a plurality of first reference information with an association relationship and inputting the second reference information obtained after the fusion into the large model, thereby improving the accuracy and completeness of the generated response result.

[0128] The specific implementation form of step 405 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0129] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and then based on the target data processing mode, when there is an association relationship between multiple first reference information in the first reference information set, the multiple first reference information are fused according to the association relationship to obtain the second reference information, and finally the second reference information is input into the large model to obtain the response result generated by the large model. Thus, after obtaining the first reference information set of the input sentence, the first reference information with an association relationship is fused to obtain the second reference information, and the second reference information is input into the large model to obtain the response result generated by the large model, thereby improving the accuracy and completeness of data processing based on the large model by fusing the reference information with an association relationship and inputting the fused reference information into the large model, and generating the response result by the large model.

[0130] Figure 5 A flowchart of a data processing method based on a large model provided in one embodiment of the present disclosure.

[0131] like Figure 5 As shown, the data processing method based on the large model may include the following steps:

[0132] Step 501: parse the received first input sentence to determine description information corresponding to the first input sentence.

[0133] Step 502: Determine the target data processing mode based on the description information.

[0134] Step 503: Acquire a first reference information set corresponding to the first input sentence.

[0135] Step 504: Process the first reference information set based on the target data processing mode to obtain a second reference information set.

[0136] Step 505: input the second reference information set into the large model to obtain a response result generated by the large model.

[0137] The specific implementation forms of steps 501 to 505 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0138] Step 506: Determine the confidence level of the response result according to the second weight corresponding to each second reference information in the second reference information set.

[0139] The second weight may be used to indicate the credibility of the second reference information, which may be the credibility of the source of the second reference information, and the present disclosure does not limit this.

[0140] In the present disclosure, after the response result is generated, in order to improve the reliability of the response result, the confidence of the response result may be determined according to the second weight corresponding to each second reference information in the second reference information set.

[0141] Step 507: When the confidence level is less than or equal to the threshold, generate clarification information.

[0142] The threshold value may be a confidence critical value used to determine whether to generate clarification information, which may be preset or determined according to actual needs, and the present disclosure does not limit this.

[0143] The clarification information may be information used to make a disclaimer for the response result, and its specific content and format may be determined as needed. For example, the clarification information may be information used to make a disclaimer for the time of the response result, such as "The information I can obtain is up to a certain year and month. If you want to obtain the latest information, please visit the official website to check", or it may be information used to make a disclaimer for the event of the response result, such as "The above content may change over time. If you want to obtain the latest information, please visit the official website to check", etc. This disclosure does not limit this.

[0144] In the present disclosure, when the confidence level is less than or equal to a threshold, it can be determined that the accuracy of the response result is not high. At this time, in order to improve the user experience and increase user satisfaction, clarification information can be generated for the response result.

[0145] Optionally, when the confidence level is less than or equal to a threshold, when generating clarification information, the clarification information can be generated based on one or more of the following: the maximum value of the second weight, the business type to which the first input statement belongs, and the source of the second reference information corresponding to the maximum value of the second weight, thereby improving the accuracy of the generated clarification information and enhancing the user experience.

[0146] It should be noted that the business type to which the first input statement belongs can be determined according to the specific needs of the user. For example, the business type to which the first input statement belongs can be news type, legal analysis type, scientific research type, financial analysis type, public security management type, etc., and this disclosure does not limit this.

[0147] It should be noted that when generating clarification information according to the maximum value in the second weight, the smaller the maximum value in the second weight, the lower the confidence of the response result can be determined. At this time, the more content of the generated clarification information is, and the present disclosure does not limit this.

[0148] It should be noted that when clarification information is generated based on the business type to which the first input statement belongs, when the business type to which the first input statement belongs is a business type with higher timeliness (such as news type, financial analysis type, legal analysis type, etc.), clarification information including a time disclaimer can be generated for the response result, and the present disclosure does not limit this.

[0149] It should be noted that when generating clarification information based on the business type to which the first input statement belongs, when the business type to which the first input statement belongs is a business type with higher timeliness, such as "Who is the current agent in country A?", it is also possible to evaluate the obsolescence of the information through statistical analysis and historical data comparison to assist in generating accurate clarification information, and the present disclosure does not limit this.

[0150] It should be noted that when generating clarification information, when the type of the first time information corresponding to the first input sentence is a relative time type, such as "What movies are released recently?", the timeliness of the information can be evaluated through statistical analysis and historical data comparison, and clarification information can be assisted in generation. The present disclosure does not limit this.

[0151] Step 508: Output clarification information and response results.

[0152] In the present disclosure, after generating clarification information, the clarification information and the response result can be output to the user, so that the spread of misleading information can be effectively reduced through the clarification information, the reliability of data processing can be improved, and the user experience can be enhanced.

[0153] It should be noted that the big model-based data processing method proposed in the present disclosure can be flexibly applied to a variety of different application scenarios, for example, it can be applied to news analysis platforms, financial market analysis, public security management, legal consulting services, etc. Among them, when applied to news analysis platform scenarios, the big model-based data processing method proposed in the present disclosure can be used to achieve instant tracking and multi-dimensional analysis of events through real-time information fusion and semantic fusion, thereby improving the timeliness and accuracy of information. It can also provide in-depth analysis of the background and potential impact of news events through causal reasoning for decision-making and reporting. When applied to financial market analysis scenarios, the big model-based data processing method proposed in the present disclosure can be used to quickly obtain the latest market trends through information updates, and can generate clarification information Provide risk warnings to users, and use causal reasoning to chalk out causal relationships between economic indicators for users to make investment decisions. When applied to public safety management scenarios, accurate time-event information is crucial for decision-making in crisis management and emergency response. Through information fusion, a comprehensive event situation analysis can be provided. Causal reasoning can also be used to evaluate the management effects of past events to provide a reference for future emergency measures. When applied to legal consulting service scenarios, the big model-based data processing method of the present disclosure can be used to automatically organize and supplement case-related timelines through entity recognition and information supplementation, thereby improving the efficiency of legal analysis and consultation. Changes in laws and regulations can also be tracked in real time to ensure the accuracy and timeliness of consulting content, etc. The present disclosure does not limit this.

[0154] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and then the first reference information set is processed based on the target data processing mode to obtain the second reference information set, and the second reference information set is input into the large model to obtain the response result generated by the large model, and finally the confidence of the response result is determined according to the second weight corresponding to each second reference information in the second reference information set, and when the confidence is less than or equal to the threshold, clarification information is generated, and the clarification information and the response result are output. Thus, after the second reference information set of the input sentence is input into the large model and the response result generated by the large model is obtained, the confidence of the response result is determined based on the weight of each second reference information set, and when the confidence is less than or equal to the threshold, clarification information is generated, and the clarification information and the query result are output, thereby effectively reducing the spread of misleading information by generating clarification information for the input sentence, improving the reliability of data processing based on the large model, and enhancing the user experience.

[0155] In order to implement the above embodiments, the present disclosure also proposes a data processing device based on a large model.

[0156] Figure 6 A schematic diagram of the structure of a large model-based data processing device provided in an embodiment of the present disclosure.

[0157] like Figure 6 As shown, the large model-based data processing device 600 includes: a parsing module 601, a determining module 602, an acquiring module 603, a processing module 604, and an input module 605.

[0158] The parsing module 601 is used to parse the received first input sentence and determine the description information corresponding to the first input sentence;

[0159] A determination module 602 is used to determine a target data processing mode based on the description information;

[0160] An acquisition module 603 is used to acquire a first reference information set corresponding to the first input sentence;

[0161] Processing module 604, used in a base target data processing mode, processes the first reference information set to obtain a second reference information set;

[0162] The input module 605 is used to input the second reference information set into the large model to obtain a response result generated by the large model.

[0163] In a possible implementation of the present disclosure, the above description information includes one or more of the following:

[0164] first time information corresponding to the first input sentence;

[0165] Type of first-time information;

[0166] entities contained in the first input sentence;

[0167] The type of entity;

[0168] Relationships between entities.

[0169] In a possible implementation of the present disclosure, the acquisition module 603 is specifically configured to:

[0170] In a case where the target data processing mode includes a first processing mode associated with the input sentence, based on the first processing mode, the first input sentence is updated to obtain a second input sentence;

[0171] A first reference information set corresponding to the second input sentence is obtained.

[0172] In a possible implementation of the present disclosure, the first processing mode is one or more of the following:

[0173] Obtaining relations between entities in a first input sentence;

[0174] The first time information corresponding to the first input sentence is obtained.

[0175] In a possible implementation of the present disclosure, the processing module 604 is specifically configured to:

[0176] Determining a first weight of each first reference information according to a source of the first reference information in the first reference information set;

[0177] The first weight of each first reference information is associated and added to the first reference information set to obtain a second reference information set.

[0178] In a possible implementation of the present disclosure, the processing module 604 is specifically configured to:

[0179] In the case where there is an association relationship between multiple first reference information in the first reference information set, the multiple first reference information are merged according to the association relationship to obtain the second reference information.

[0180] In a possible implementation of the present disclosure, the input module 605 is further used to:

[0181] Determining the confidence level of the response result according to the second weight corresponding to each second reference information in the second reference information set;

[0182] When the confidence level is less than or equal to the threshold, generating clarification information;

[0183] Output clarification information and response results.

[0184] In a possible implementation of the present disclosure, the input module 605 is further used to:

[0185] The clarification information is generated according to one or more of the following: a maximum value among the second weights, a business type to which the first input sentence belongs, and a source of the second reference information corresponding to the maximum value of the second weight.

[0186] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, and will not be repeated here.

[0187] In the disclosed embodiment, the received first input sentence is first parsed to determine the description information corresponding to the first input sentence, and based on the description information, the target data processing mode is determined, and then the first reference information set corresponding to the first input sentence is obtained, and then the first reference information set is processed based on the target data processing mode to obtain the second reference information set, and finally the second reference information set is input into the large model to obtain the response result generated by the large model. Thus, by determining the processing mode based on the description information corresponding to the input sentence, and then processing the obtained first reference information set based on the processing mode to obtain the second reference information set, and then generating the corresponding response result based on the second reference information set, the accuracy of the model in processing timeliness and multi-source data is improved, and the accuracy and reliability of data processing are improved.

[0188] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0189] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0190] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0191] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0192] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a data processing method based on a large model. For example, in some embodiments, the data processing method based on a large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or a communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data processing method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a data processing method based on a large model in any other appropriate manner (e.g., by means of firmware).

[0193] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0195] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0197] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0198] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0200] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0201] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method based on a large model, comprising: Parsing the received first input sentence to determine description information corresponding to the first input sentence; Based on the description information, determining a target data processing mode; Acquire a first reference information set corresponding to the first input sentence; Based on the target data processing mode, the first reference information set is processed to obtain a second reference information set; The second reference information set is input into the large model to obtain a response result generated by the large model.

2. The method of claim 1, wherein: The description information includes one or more of the following: first time information corresponding to the first input sentence; type of the first time information; entities contained in the first input sentence; the type of the entity; Relationships between entities.

3. The method of claim 1, wherein: The acquiring of a first reference information set corresponding to the first input sentence includes: In a case where the target data processing mode includes a first processing mode associated with an input sentence, updating the first input sentence based on the first processing mode to obtain a second input sentence; A first reference information set corresponding to the second input sentence is obtained.

4. The method of claim 3, wherein: The first processing mode is one or more of the following: Obtaining relationships between entities in the first input sentence; Acquire first time information corresponding to the first input sentence.

5. The method of claim 1, wherein: The step of processing the first reference information set based on the target data processing mode to obtain a second reference information set includes: Determine a first weight of each first reference information according to a source of the first reference information in the first reference information set; The first weight of each of the first reference information is associated and added to the first reference information set to obtain the second reference information set.

6. The method of claim 1, wherein: The step of processing the first reference information set based on the target data processing mode to obtain a second reference information set includes: In the case where there is an association relationship between multiple first reference information in the first reference information set, the multiple first reference information are merged according to the association relationship to obtain second reference information.

7. The method according to any one of claims 1 to 6, wherein: After obtaining the response result generated by the large model, the method further includes: determining the confidence level of the response result according to a second weight corresponding to each second reference information in the second reference information set; generating clarification information when the confidence level is less than or equal to a threshold; The clarification information and the response result are outputted.

8. The method of claim 7, wherein: The generating of clarification information includes: The clarification information is generated according to one or more of the following: a maximum value among the second weights, a business type to which the first input sentence belongs, and a source of the second reference information corresponding to the maximum value of the second weights.

9. A data processing device based on a large model, wherein: The device comprises: A parsing module, configured to parse the received first input sentence and determine description information corresponding to the first input sentence; A determination module, used to determine a target data processing mode based on the description information; An acquisition module, configured to acquire a first reference information set corresponding to the first input sentence; A processing module, configured to process the first reference information set based on the target data processing mode to obtain a second reference information set; The input module is used to input the second reference information set into the large model to obtain a response result generated by the large model.

10. The device of claim 9, wherein: The description information includes one or more of the following: first time information corresponding to the first input sentence; type of the first time information; entities contained in the first input sentence; the type of the entity; Relationships between entities.

11. The device of claim 9, wherein: The acquisition module is specifically used for: In a case where the target data processing mode includes a first processing mode associated with an input sentence, updating the first input sentence based on the first processing mode to obtain a second input sentence; A first reference information set corresponding to the second input sentence is obtained.

12. The device of claim 11, wherein: The first processing mode is one or more of the following: Obtaining relationships between entities in the first input sentence; Acquire first time information corresponding to the first input sentence.

13. The device of claim 9, wherein: The processing module is specifically used for: Determine a first weight of each first reference information according to a source of the first reference information in the first reference information set; The first weight of each of the first reference information is associated and added to the first reference information set to obtain the second reference information set.

14. The device of claim 9, wherein: The processing module is specifically used for: In the case where there is an association relationship between multiple first reference information in the first reference information set, the multiple first reference information are merged according to the association relationship to obtain second reference information.

15. The device according to any one of claims 9 to 14, wherein: The input module is also used for: determining the confidence level of the response result according to a second weight corresponding to each second reference information in the second reference information set; generating clarification information when the confidence level is less than or equal to a threshold; The clarification information and the response result are outputted.

16. The device of claim 15, wherein: The input module is further used for: The clarification information is generated according to one or more of the following: a maximum value among the second weights, a business type to which the first input sentence belongs, and a source of the second reference information corresponding to the maximum value of the second weights.

17. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images