Data optimization method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses a data optimization method and device, equipment, a storage medium and a product. The method comprises the steps that associated data and reference information of a to-be-optimized statement are obtained, the associated data comprise an analysis process of the to-be-optimized statement, the reference information comprises a rewriting example of a reference statement, supplemental content retrieval is conducted on the to-be-optimized statement, and supplemental content of the to-be-optimized statement is obtained; and rewriting the analysis process based on the to-be-optimized statement, the rewriting example of the reference statement and the supplementary content to obtain a rewritten analysis process, and generating an optimization result of the to-be-optimized statement through the rewritten analysis process. It can be seen that the analysis process of the to-be-optimized statement is rewritten by adding the rewriting example and the supplementary content of the reference statement, automatic optimization of the to-be-optimized statement can be achieved, and the optimization efficiency of the to-be-optimized statement is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data optimization method, a data optimization device, a computer device, a computer-readable storage medium, and a data optimization product. Background Technology

[0002] With the advancement of scientific research, computer technology has developed rapidly. People can quickly obtain the information they need through computers; for example, information they need to know, documents they need to use, etc. For non-professionals, natural language is usually used to interact with computer devices when obtaining the required information. Research has found that there are statements that need optimization in the human-computer interaction process, and the response information corresponding to these statements does not meet actual expectations (such as irrelevant answers, incorrect answers, etc.). These statements need to be manually optimized by sales personnel. How to improve the optimization efficiency of these statements has become a hot research topic. Summary of the Invention

[0003] This application provides a data optimization method, apparatus, device, computer-readable storage medium, and product that can improve the optimization efficiency of the statement to be optimized.

[0004] On the one hand, embodiments of this application provide a data optimization method, including:

[0005] Obtain the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes the statement rewriting examples of the reference statements.

[0006] The supplementary content of the statement to be optimized is retrieved, and the vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold.

[0007] The analysis process is rewritten based on the statement to be optimized, the statement rewriting examples of the reference statement, and supplementary content, resulting in the rewritten analysis process.

[0008] The rewritten analysis process generates optimized results for the statements to be optimized.

[0009] On one hand, embodiments of this application provide a data optimization apparatus, which includes:

[0010] The acquisition unit is used to acquire the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes the statement rewriting examples of the reference statements.

[0011] The processing unit is used to retrieve supplementary content for the statement to be optimized, and obtain the supplementary content of the statement to be optimized. The vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold.

[0012] And it is used to rewrite the analysis process based on the statement to be optimized, the reference statement, and supplementary content, so as to obtain the rewritten analysis process;

[0013] And it is used to generate optimization results for the statements to be optimized through the rewritten analysis process.

[0014] In one implementation, the processing unit is configured to rewrite the analysis process based on the statement to be optimized, a statement rewriting example, and supplementary content to obtain a rewritten analysis process, specifically configured to:

[0015] Constraint information is generated based on the supplementary content. The constraint information is used to constrain at least one of the business requirements, business scenarios and business logic corresponding to the statement to be optimized.

[0016] The statement to be optimized, a rewritten example of the statement, and the analysis process are input into a large-scale language model so that the large-scale language model can generate a rewritten analysis process according to the constraint information.

[0017] In one implementation, the processing unit is configured to input the statement to be optimized, a statement rewriting example, and an analysis process into a large-scale language model, so that the large-scale language model generates a rewritten analysis process according to the constraint information, specifically for:

[0018] Input the statement to be optimized, the statement rewriting example, and the analysis process into the large-scale language model so that the large-scale language model generates M rewriting results of the analysis process according to the constraint information, where M is an integer greater than 1;

[0019] Calculate the average similarity of the rewritten results of each process. The average similarity of the rewritten results of any process is the mean of the similarity between the rewritten results of that process and the rewritten results of the remaining M-1 processes.

[0020] Based on the process rewriting results where the average similarity is higher than the similarity threshold, the rewritten analysis process is determined.

[0021] In one implementation, the processing unit is configured to generate constraint information based on the supplementary content, specifically for:

[0022] Obtain a learning sample, which contains P sub-problems decomposed from the sample problem, and the reasoning steps corresponding to the P sub-problems, where P is an integer greater than 1;

[0023] Constraint information is generated by learning samples and supplementary content. The learning samples are used to constrain the business logic corresponding to the statement to be optimized, and the supplementary content is used to constrain the business requirements or business scenarios corresponding to the statement to be optimized.

[0024] In one implementation, the processing unit is used to generate an optimization result for the statement to be optimized through a rewritten analysis process, specifically for:

[0025] The large-scale language model is invoked to generate N rewritten results of the sentence to be optimized, based on the sentence to be optimized, the supplementary content, and the rewritten analysis process, where N is a positive integer;

[0026] Based on the rewrite results of N statements, the optimized results of the statement to be optimized are generated.

[0027] In one implementation, N is greater than 1, and the description methods corresponding to the N rewritten statement results are different; the processing unit is used to generate the optimized result of the statement to be optimized based on the N rewritten statement results, specifically for:

[0028] The quality score of each statement rewrite result is calculated. The quality score of any statement rewrite result is obtained by comparing the quality of that statement rewrite result with the quality of the other N-1 statement rewrite results.

[0029] The rewritten N statements are sorted in descending order of quality score, and the top k statements are selected as the optimized results for the statements to be optimized, where k is a positive integer less than or equal to N.

[0030] In one implementation, the process by which the processing unit calculates a quality score for each statement rewrite result includes:

[0031] Compare the quality of the rewritten result of the i-th statement with the rewritten result of the j-th statement, where i and j are both positive integers less than or equal to N, and i is not equal to j;

[0032] If the quality of the rewritten result of the i-th statement is better than that of the rewritten result of the j-th statement, then increase the quality score of the rewritten result of the i-th statement and decrease the quality score of the rewritten result of the j-th statement.

[0033] If the quality of the rewritten result of the i-th statement is the same as that of the rewritten result of the j-th statement, then the quality scores of the rewritten result of the i-th statement and the rewritten result of the j-th statement remain unchanged.

[0034] In one implementation, the processing unit is configured to perform supplementary content retrieval on the statement to be optimized to obtain supplementary content for the statement to be optimized, specifically for:

[0035] Extract key information from the statement to be optimized to obtain the key information of the statement to be optimized;

[0036] The vector matching degree between the statement to be optimized and each candidate content is calculated from both the key information dimension and the statement dimension.

[0037] Based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, the comprehensive matching degree between the statement to be optimized and each candidate content is determined.

[0038] Candidate content with a comprehensive matching degree higher than the second matching degree threshold is identified as supplementary content for the statement to be optimized.

[0039] In one implementation, the process by which the processing unit determines the overall matching degree between the statement to be optimized and each candidate content based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, includes:

[0040] The vector matching degree between the statement to be optimized and the h-th candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and the h-th candidate content in the statement dimension are weighted and summed to obtain the comprehensive matching degree between the statement to be optimized and the h-th candidate content, where h is a positive integer.

[0041] In one implementation, the optimization result is in natural language or structured language; the processing unit is further configured to:

[0042] Based on the optimization results, a database query language is generated.

[0043] By using a database query language to retrieve content from the database, target data related to the problem indicated by the optimization results can be obtained.

[0044] Based on the target data, generate response information for the statement to be optimized.

[0045] In one embodiment, the processing unit is further configured to:

[0046] Obtain the optimization result corresponding to the statement to be optimized. The optimization result is obtained by calling the optimization model to rewrite the statement to be optimized.

[0047] Based on the difference between the optimization result and the result to be optimized, the model to be optimized is optimized to obtain the optimized model.

[0048] Accordingly, this application provides a computer device comprising:

[0049] Memory, which stores computer programs;

[0050] The processor is used to load computer programs to implement the aforementioned data optimization methods.

[0051] Accordingly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data optimization method.

[0052] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data optimization method.

[0053] In this embodiment, associated data and reference information of the statement to be optimized are obtained. The associated data includes the analysis process of the statement to be optimized, and the reference information includes a rewritten example of a reference statement. Supplementary content is retrieved for the statement to be optimized to obtain supplementary content. Based on the statement to be optimized, the rewritten example of the reference statement, and the supplementary content, the analysis process is rewritten to obtain the rewritten analysis process. The optimization result of the statement to be optimized is generated through the rewritten analysis process. It is evident that by adding a rewritten example of a reference statement and supplementary content to rewrite the analysis process of the statement to be optimized, automatic optimization of the statement to be optimized can be achieved, improving the optimization efficiency. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A data optimization scenario diagram provided in an embodiment of this application;

[0056] Figure 2 A flowchart illustrating a data optimization method provided in this application embodiment;

[0057] Figure 3 An architecture diagram of a data optimization system provided in this application embodiment;

[0058] Figure 4 A flowchart illustrating another data optimization method provided in this application embodiment;

[0059] Figure 5a An architecture diagram of another data optimization system provided for implementation of this application;

[0060] Figure 5bThis is a schematic diagram of the interactive page of the business analysis system provided in the embodiments of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a data optimization device provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] This application relates to technologies related to large-scale language models. The following is a brief introduction to the relevant terms and concepts of large-scale language models:

[0065] Large-Scale Language Model (LLM): This is an artificial intelligence model that learns to understand and generate human language by learning from large amounts of text data. Such models typically use deep learning techniques, such as neural networks, to train and generate text.

[0066] Contextual learning, in the fields of machine learning and artificial intelligence, refers to enabling models to understand and process the contextual information of input data. Contextual information includes semantic relationships, syntactic structure, and other relevant information within the input data. Through contextual learning, models can more accurately understand the meaning of input data and make more appropriate predictions and responses. Contextual learning is particularly important in Natural Language Processing (NLP) tasks because it helps models better understand and generate human language. In this application's embodiments, the focus is primarily on enhancing the background information and context of the statement to be optimized through retrieved supplementary content, ensuring that the optimization result conforms to the corresponding business needs or scenarios.

[0067] Based on the aforementioned technologies related to large-scale language models, this application provides a data optimization scheme that can improve the optimization efficiency of the statement to be optimized. Figure 1 A data optimization scenario diagram provided for an embodiment of this application, such as Figure 1As shown, the data optimization scenario provided in this application includes a terminal device 101 and a server 102. The data optimization scheme provided in this application can be executed by the server 102. The terminal device may include, but is not limited to: smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (Mobile Internet Devices, abbreviated as MID), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc. This application embodiment does not limit this. The server may be an independent physical server, or a server cluster or distributed system composed of two or more physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application embodiment does not limit this.

[0068] It should be noted that, Figure 1 The number of terminal devices and servers is for illustrative purposes only and does not constitute an actual limitation of this application. Terminal device 101 and server 102 can be connected via wired or wireless means, and this application does not impose any restrictions on this.

[0069] The general process of the data optimization scheme provided in this application is as follows:

[0070] Server 102 obtains the associated data and reference information of the statement to be optimized; wherein, the associated data includes the analysis process of the statement to be optimized, and the reference information includes a statement rewriting example of the reference statement. In this embodiment, the statement to be optimized can be understood as a statement whose rewriting result does not match the expectation. Next, server 102 performs supplementary content retrieval on the statement to be optimized to obtain supplementary content of the statement to be optimized; wherein, the vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold, and the supplementary content is used to enhance the background information and context of the statement to be optimized, so that server 102 can more fully and accurately understand the statement to be optimized. Further, server 102 rewrites the analysis process based on the statement to be optimized, the statement rewriting example of the reference statement, and the supplementary content to obtain the rewritten analysis process. On the one hand, through the statement rewriting example of the reference statement, server 102 can obtain the analysis process of the reference statement, and rewrite the analysis process of the statement to be optimized by referring to the analysis process of the reference statement; on the other hand, when rewriting the analysis process of the statement to be optimized, server 102 will also refer to the supplementary content to ensure that the rewritten analysis process conforms to the business scenario or business requirements corresponding to the statement to be optimized. After obtaining the rewritten analysis process, server 102 generates optimization results for the statement to be optimized through the rewritten analysis process; wherein, the optimization results include at least one statement rewriting result corresponding to the statement to be optimized, and each statement rewriting result is in natural language or structured language; when the optimization results include two or more statement rewriting results corresponding to the statement to be optimized, the description methods (wording) of each statement rewriting result are different.

[0071] In this embodiment, associated data and reference information of the statement to be optimized are obtained. The associated data includes the analysis process of the statement to be optimized, and the reference information includes a rewritten example of a reference statement. Supplementary content is retrieved for the statement to be optimized to obtain supplementary content. Based on the statement to be optimized, the rewritten example of the reference statement, and the supplementary content, the analysis process is rewritten to obtain the rewritten analysis process. The optimization result of the statement to be optimized is generated through the rewritten analysis process. It is evident that by adding a rewritten example of a reference statement and supplementary content to rewrite the analysis process of the statement to be optimized, automatic optimization of the statement to be optimized can be achieved, improving the optimization efficiency.

[0072] Based on the above data optimization scheme, this application proposes a more detailed data optimization method. The data optimization method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0073] Please see Figure 2 , Figure 2 A flowchart illustrating a data optimization method provided in this application embodiment, which can be executed by a computer device; for example, by... Figure 1The server 102 shown is executing.

[0074] like Figure 2 As shown, the data optimization method may include the following steps S201-S204:

[0075] S201. Obtain the associated data and reference information of the statement to be optimized.

[0076] The statement to be optimized, also known as a bad case, refers to a statement whose rewritten result does not meet the expected outcome (e.g., the rewritten result incorrectly expresses the intended meaning of the original statement). Statement rewriting can be understood as supplementing and improving a statement. The statement to be optimized is in natural language or structured language, and the rewritten result of the statement to be optimized is also in natural language or structured language.

[0077] For example, statement A is "Query the turnover of XX in the first half of the year". The actual expectation of statement A is "Query the turnover of XX company in the first half of this year". If the statement A is rewritten as "Query the turnover of XX company in the first half of last year" or "Query the turnover of XX store in the first half of the year" (neither of which does not meet the actual expectation), then statement A is a statement that needs to be optimized.

[0078] The associated data for the statement to be optimized includes the analysis process of the statement, which is used to analyze the actual question the target audience wants to ask. Furthermore, it can be used to decompose the actual question into two or more sub-questions and deduce the answer to the actual question step by step according to the logical relationships between the sub-questions. Understandably, when the statement to be optimized is a non-interrogative statement, the computer device needs to determine the associated questions based on that non-interrogative statement. In addition to the analysis process, the associated data for the statement to be optimized may also include the rewritten result of the statement (to be optimized).

[0079] In one implementation, the computer device can preprocess the statement to be optimized to obtain associated data of the statement; wherein, the preprocessing may include, but is not limited to, data cleaning, format standardization, and outlier handling. In one embodiment, on the one hand, the computer device performs data cleaning on the statement to be optimized to remove duplicates, null values, invalid data, etc.; on the other hand, the computer device performs format standardization on the statement to be optimized; for example, based on the statement to be optimized, it determines four parts of text content: "supplementary content", "user questions", "analysis process", and "statement rewriting results"; furthermore, the computer device performs outlier identification and processing on the statement to be optimized, specifically by using statistical methods such as z-score and interquartile range (IQR) to identify and process outliers in the statement to be optimized.

[0080] The reference information includes a rewritten example of the reference statement, which belongs to the same type as the statement to be optimized. The type of the statement to be optimized can be included in the above preprocessing results; that is, the computer device can determine the type of the statement to be optimized based on the preprocessing results. The specific statement classification method can be dynamically adjusted based on actual needs; for example, it can be classified based on business scenarios, business requirements, etc., or it can be classified based on statement characteristics. This application does not impose any restrictions on this.

[0081] S202. Perform a supplementary content retrieval on the statement to be optimized to obtain the supplementary content of the statement to be optimized.

[0082] The supplementary content enhances the background information and context of the statement to be optimized, enabling the computer device to more fully and accurately understand the statement, thereby improving the quality of the rewritten analysis process (e.g., making the rewritten analysis process more aligned with the intent of the interactive object). In one implementation, the computer device may represent the statement to be optimized as a vector and filter supplementary content from candidate content based on the vector representation of the statement to be optimized, wherein the vector matching degree between the supplementary content and the statement to be optimized is higher than a first matching degree threshold.

[0083] In one embodiment, the computer device extracts key information from the statement to be optimized, obtaining key information (such as keywords and summaries). This can be achieved using key information extraction algorithms such as Term Frequency-Inverse Document Frequency (TF-IDF) and TextRank. After obtaining the key information of the statement to be optimized, the computer device calculates the vector matching degree between the statement to be optimized and each candidate content from both the key information dimension and the statement dimension. Based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, the overall matching degree between the statement to be optimized and each candidate content is determined. In one implementation, the process by which the computer device determines the comprehensive matching degree between the statement to be optimized and each candidate content based on the vector matching degree of the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree of the statement to be optimized and each candidate content in the statement dimension, includes: performing a weighted summation of the vector matching degree of the statement to be optimized and the h-th candidate content in the key information dimension, and the vector matching degree of the statement to be optimized and the h-th candidate content in the statement dimension, to obtain the comprehensive matching degree of the statement to be optimized and the h-th candidate content, where h is a positive integer. The calculation method for the comprehensive matching degree of the h-th candidate content can be expressed as:

[0084] score h_final=αscore h1 +βscore h2

[0085] Among them, score h_final The score represents the overall matching degree of the h-th candidate content. h1 The score represents the vector matching degree between the statement to be optimized and the h-th candidate content in the statement dimension. h2 α represents the vector matching degree between the statement to be optimized and the h-th candidate content in the key information dimension, and β represents the weighting parameters.

[0086] After obtaining the overall matching degree between the statement to be optimized and each candidate content, the computer device identifies candidate content with an overall matching degree higher than the second matching degree threshold as supplementary content for the statement to be optimized. In practical applications, the calculation of vector matching degree at the key information dimension and vector matching degree at the statement dimension can be achieved through a dual-tower model (including an offline tower and an online tower); the offline tower is used to cache the vector representation results of some or all candidate content offline, while the online tower is used to calculate the vector representation result of the statement to be optimized in real time. By pre-calculating the vector representation results of the candidate content, the dual-tower model can achieve efficient retrieval of supplementary content.

[0087] In another embodiment, the computer device calculates the vector matching degree between the statement to be optimized and each candidate content from the statement dimension, and determines the candidate content whose vector matching degree at the statement dimension is higher than a first matching degree threshold as supplementary content of the statement to be optimized.

[0088] In another embodiment, the computer device extracts key information from the statement to be optimized, obtains key information of the statement to be optimized, calculates the vector matching degree between the statement to be optimized and each candidate content from the dimension of key information, and determines the candidate content whose vector matching degree of key information dimension is higher than the third matching degree threshold as supplementary content of the statement to be optimized.

[0089] S203. Based on the statement to be optimized, the statement rewriting examples of the reference statement, and the supplementary content, the analysis process is rewritten to obtain the rewritten analysis process.

[0090] On the one hand, by using the example of rewriting the reference statement, the computer device can obtain the analysis process of the reference statement and rewrite the analysis process of the statement to be optimized by referring to the analysis process of the reference statement; on the other hand, when rewriting the analysis process of the statement to be optimized, the computer device will also refer to supplementary content to ensure that the rewritten analysis process conforms to the business scenario or business requirements corresponding to the statement to be optimized.

[0091] In one implementation, the computer device generates constraint information based on supplementary content. This constraint information constrains at least one of the business requirements, business scenarios, and business logic corresponding to the statement to be optimized. Then, the statement to be optimized, a rewritten example of the reference statement, and the analysis process are input into a large-scale language model. The large-scale language model then generates a rewritten analysis process according to the constraint information. Therefore, in the above implementation, the powerful semantic understanding of the large-scale language model can be used to analyze the rewritten examples of the statement to be optimized and the reference statement. Furthermore, the language generation capabilities of the large-scale language model can be utilized to automatically rewrite the analysis process of the statement to be optimized under the constraints of the corresponding constraint information (generated based on supplementary content), thus improving the efficiency of the analysis process rewriting.

[0092] Optionally, during the rewriting of the analysis process, the computer device can also interact with the object to further determine the intent of the interacting object. For example, the computer device can output multiple different analysis processes (such as those involving different business scenarios, different business objects, or different business requirements) for the interacting object to choose from. Alternatively, the computer device can output questions related to intent (such as business scenario questions, business requirement questions, business object questions, etc.) to determine the intent of the interacting object and further improve the accuracy of the rewritten analysis process.

[0093] S204. Through the rewritten analysis process, the optimization results of the statement to be optimized are generated.

[0094] The optimization results include at least one rewritten statement corresponding to the statement to be optimized, and each rewritten statement is in natural language or structured language. When the optimization results include two or more rewritten statements corresponding to the statement to be optimized, the description (wording) of each rewritten statement is different.

[0095] In one implementation, a computer device invokes a large-scale language model to generate N rewritten statements of the statement to be optimized, supplementary content, and a rewritten analysis process. Based on these N rewritten statements, an optimized statement is generated, where N is a positive integer. For example, when N = 1, the computer device can directly determine the generated rewritten statement as the optimized statement; when N > 1, the computer device can compare the quality of each rewritten statement and, based on the quality, select k rewritten statements as the optimized statement, where k is a positive integer less than or equal to N.

[0096] Figure 3 This is an architecture diagram of a data optimization system provided in an embodiment of this application. Figure 3As shown, after the computer device obtains the statement to be optimized (badcase), it first preprocesses the statement, then performs supplementary content retrieval on the preprocessed statement, then rewrites the analysis process of the statement to be optimized based on the supplementary content, and generates the optimized result (goodcase) of the statement to be optimized through the rewritten analysis process.

[0097] In this embodiment, associated data and reference information of the statement to be optimized are obtained. The associated data includes the analysis process of the statement to be optimized, and the reference information includes rewritten examples of reference statements. Supplementary content is retrieved for the statement to be optimized to obtain supplementary content. Based on the statement to be optimized, the rewritten examples of reference statements, and the supplementary content, the analysis process is rewritten to obtain the rewritten analysis process. The optimized result of the statement to be optimized is then generated through the rewritten analysis process. It is evident that by adding rewritten examples of reference statements and supplementary content to rewrite the analysis process of the statement to be optimized, automatic optimization of the statement to be optimized can be achieved, improving the optimization efficiency. Furthermore, during the analysis process rewriting stage, human-computer interaction can further improve the accuracy of the rewritten analysis process, thereby improving the quality of the optimization result of the statement to be optimized.

[0098] Please see Figure 4 , Figure 4 A flowchart illustrating another data optimization method provided in this application embodiment, which can be executed by a computer device; for example, by... Figure 1 The server 102 shown is executing.

[0099] like Figure 4 As shown, the data optimization method may include the following steps S401-S406:

[0100] S401. Obtain the associated data and reference information of the statement to be optimized.

[0101] S402. Perform a supplementary content retrieval on the statement to be optimized to obtain the supplementary content of the statement to be optimized.

[0102] For specific implementation details of steps S401 and S402, please refer to [reference needed]. Figure 2 The implementation methods of steps S201 and S202 will not be described in detail here.

[0103] S403. Generate constraint information based on supplementary content.

[0104] The constraint information is used to constrain at least one of the business requirements, business scenarios, and business logic corresponding to the statement to be optimized. In one implementation, the computer device acquires a learning sample, which includes P sub-problems decomposed from the sample problem and the reasoning steps corresponding to the P sub-problems, where P is an integer greater than 1. The computer device generates constraint information using the learning sample and supplementary content; wherein, the learning sample is used to constrain the business logic corresponding to the statement to be optimized, and the supplementary content is used to constrain the business requirements or business scenarios corresponding to the statement to be optimized.

[0105] In another implementation, the computer device generates constraint information directly based on the supplementary content; for example, if the statement to be optimized contains “XX”, in addition to its literal meaning, based on the supplementary content, “XX” may also refer to `XX Entertainment` (`group name`), or `XX Media Co., Ltd.` (`customer name`), or `XX Technology Co., Ltd.` (`customer name`).

[0106] S404. Input the statement to be optimized, the statement rewriting example of the reference statement, and the analysis process of the statement to be optimized into the large-scale language model so that the large-scale language model can generate the rewritten analysis process according to the constraint information.

[0107] In one implementation, a computer device inputs the statement to be optimized, a statement rewriting example, and an analysis process into a large-scale language model. The large-scale language model then generates M rewriting results for the analysis process based on constraint information. These M rewriting results are obtained by calling the large-scale language model M times, using the same statement to be optimized, a statement rewriting example of a reference statement, the analysis process for the statement to be optimized, and constraint information, where M is an integer greater than 1. After obtaining the M rewriting results, the computer device calculates the average similarity of each rewriting result. The average similarity of any rewriting result is the average of the similarities between that rewriting result and the remaining M-1 rewriting results. After calculating the average similarity of each rewriting result, the computer device determines the rewritten analysis process based on the rewriting results whose average similarity is higher than a similarity threshold.

[0108] For example, assuming M=3, the similarity between process rewriting result 1 and process rewriting result 2 is 91%, the similarity between process rewriting result 1 and process rewriting result 3 is 25%, and the similarity between process rewriting result 2 and process rewriting result 3 is 17%. Then, the average similarity of process rewriting result 1 is (91% + 25%) / 2 = 58%, the average similarity of process rewriting result 2 is (91% + 17%) / 2 = 54%, and the average similarity of process rewriting result 3 is (25% + 17%) / 2 = 21%. If the similarity threshold is 55%, the computer device will determine process rewriting result 1 as the rewritten analysis process; if the similarity threshold is 40%, the computer device will determine process rewriting result 1 and process rewriting result 2 as the rewritten analysis process.

[0109] The above implementation method utilizes the characteristic that "for the same question, large-scale language models will repeat more responses with higher confidence" to screen the process rewriting results, so as to ensure the quality of the rewritten analysis process.

[0110] In one implementation, a computer device can generate first prompt information for a large-scale language model based on the statement to be optimized, a statement rewriting example of a reference statement, the analysis process of the statement to be optimized, and constraint information. Then, it can use the first prompt information to invoke the large-scale language model to generate the analysis process after rewriting the statement to be optimized. Table 1 shows examples of the first prompt information provided in embodiments of this application.

[0111] Table 1

[0112]

[0113]

[0114] As shown in Table 1, the first prompt information of a large-scale language model can be written according to the technical approach of thought chains, which is a method for representing thought processes and knowledge structures. It connects a series of related concepts, viewpoints, questions, or ideas into an ordered chain. In the fields of artificial intelligence and machine learning, thought chains can be used to represent the model's reasoning process and knowledge representation. By constructing and analyzing thought chains, we can better understand how the model works and determine the direction of model optimization. Constraint information is used to generate the "#requirement" in the first prompt information. Specifically, the learning samples in the constraint information can generate the "**key components**" and "**problem understanding**" parts in the "#requirement," and supplementary content can generate the "**supplementary knowledge comparison**" part in the "#requirement." In one implementation, the computer device can generate constraint information based on supplementary content and a single learning sample. Using a single learning sample can improve the large-scale language model's ability to follow the constraint information (first prompt information) and avoid negative interference from multiple learning samples. The sentence to be optimized, the sentence rewriting example of the reference sentence, and the analysis process of the sentence to be optimized are the input data, and the rewritten analysis process is the output result of the large-scale language model. It should be noted that the specific content in Table 1 is only used to help understand the data optimization method provided in this application and does not constitute an actual limitation of this application.

[0115] S405. Call a large-scale language model to generate N rewritten results of the statement to be optimized based on the statement to be optimized, the supplementary content, and the rewritten analysis process.

[0116] N is a positive integer. When N>1, the description (wording) corresponding to the rewritten results of each statement is different.

[0117] In one implementation, the computer device can invoke a large-scale language model N times to generate N rewritten statements of the statement to be optimized, based on the same statement to be optimized, supplementary content, and rewritten analysis process.

[0118] In another implementation, the computer device invokes a large-scale language model to generate a rewritten statement result (one in number) of the statement to be optimized, based on the statement to be optimized, supplementary content, and the rewritten analysis process. This rewritten statement result can be obtained from two or more rewritten statement results generated based on the large-scale language model. For specific implementation details, please refer to the implementation of "determining the rewritten analysis process based on M rewritten process results" in step S404, which will not be repeated here. After obtaining a unique rewritten statement result for the statement to be optimized, the computer device can further expand the unique rewritten statement result to obtain multiple rewritten statement results corresponding to the statement to be optimized, thereby improving the diversity of the rewritten statement results.

[0119] Similar to the method of generating the first prompt information in step S404, the computer device can generate the second prompt information of the large-scale language model based on the unique statement rewriting result of the statement to be optimized, and call the large-scale language model through the second prompt information to generate the expanded result of the statement to be optimized (i.e., multiple statement rewriting results corresponding to the statement to be optimized). Table 2 shows an example of the second prompt information provided in the embodiments of this application.

[0120] Table 2

[0121]

[0122] As shown in Table 2, the second cue information of a large-scale language model can be written according to the technical approach of the thought chain; in addition, the understanding of the second cue information by the large-scale language model can be further improved by using Markdown text format. It should be noted that the specific content involved in Table 2 is only used to help understand the data optimization method provided in this application and does not constitute an actual limitation of this application.

[0123] Optionally, the computer device can also expand the rewritten results of the statement to be optimized based on transfer learning and reinforcement learning techniques to further improve the quality of the expanded results.

[0124] S406. Based on the rewrite results of N statements, generate the optimized results of the statement to be optimized.

[0125] In one implementation, N=1, the computer device can directly determine the unique statement rewriting result as the optimization result of the statement to be optimized.

[0126] In another implementation, N>1, the computer device calculates a quality score for each statement rewrite result. The quality score of any statement rewrite result is obtained by comparing its quality with the remaining N-1 statement rewrite results. In one embodiment, the process of calculating the quality score of the statement rewrite results by the computer device includes: comparing the quality of the i-th statement rewrite result with the j-th statement rewrite result, where i and j are both positive integers less than or equal to N, and i is not equal to j. The quality measurement standard can be dynamically adjusted based on the actual scenario, and this application does not impose any restrictions on this. If the quality of the i-th statement rewrite result is better than that of the j-th statement rewrite result, the computer device increases the quality score of the i-th statement rewrite result (e.g., increases the quality score by one) and decreases the quality score of the j-th statement rewrite result (e.g., decreases the quality score by one); if the quality of the i-th statement rewrite result and the j-th statement rewrite result are the same, the computer device can keep the quality scores of the i-th and j-th statement rewrite results unchanged. After calculating the quality scores of all rewritten statements, the computer sorts the N rewritten statements in descending order of quality score, and determines the top k rewritten statements as the optimized results for the statements to be optimized, where k is a positive integer less than or equal to N. Therefore, by filtering the rewritten statements, the quality of the optimized results for the statements to be optimized can be guaranteed.

[0127] Figure 5a An architecture diagram of another data optimization system provided for the implementation of this application. (See diagram below.) Figure 5a As shown, in Figure 3 Based on the data optimization system shown, the quality of the rewritten analysis process (such as accuracy and stability) is further improved through the self-consistency module, the diversity of the rewritten results is increased through the expansion module, and the quality of the optimization results of the statements to be optimized is ensured through the filtering module.

[0128] Furthermore, after obtaining the optimization result of the statement to be optimized, the computer device can process the optimization result based on the scenario requirements to obtain the corresponding processing result.

[0129] In one embodiment, the above data optimization method can be applied to a business analysis system (such as ChatBI), which supports interactive objects to query business data through natural language and generate corresponding analysis results for the business data; for example, the business analysis system can generate analysis results and visualization charts of the business indicators queried by the interactive objects.

[0130] Figure 5b This is a schematic diagram of the interactive page of the business analysis system provided in an embodiment of this application. Figure 5bAs shown, the interactive page 501 includes an input field 5011, which is used to obtain data (such as natural language statements) input by the interactive object. In response to the submission of input data in the input field 5011, the submitted input data 5013 is displayed in the interactive content display area 5012; furthermore, the statement rewriting result 5014 of the input data can also be displayed in the interactive content display area 5012. Further, response information 5015 of the input data is displayed in the interactive content display area 5012. The response information 5015 may carry a style switching control 5016, which is used to switch the display style of some or all of the response information.

[0131] During the process of an interactive object querying business data using natural language, the business analysis system needs to rewrite (and structure) the natural language input by the interactive object, and generate a database query statement from the rewritten statement. It is understandable that rewriting the natural language can improve the accuracy of the generated database query statement. The data optimization method provided in this application can automatically optimize the rewritten result of the natural language statement when the result does not meet expectations. In one implementation, the data optimization method provided in this application can be encapsulated in an automatic optimization tool. The computer device provides the interactive object with the application programming interface (API) of this automatic optimization tool, and the interactive object can call this API to automatically optimize the statement to be optimized. Table 3 shows an example of fields from one type of API provided in this application.

[0132] Table 3

[0133] Fields Required type query yes str knowledge yes Dict[str,str] true_label yes str num_cases no int

[0134] As shown in Table 3, query represents the statement to be optimized, knowledge represents supplementary content, true_label represents a rewritten example of the reference statement, and num_cases controls the number of rewritten results for the statement to be optimized. It should be noted that the fields in Table 3 are for illustrative purposes only and do not constitute an actual limitation of this application.

[0135] In one implementation, after obtaining the optimization result of the statement to be optimized, the computer device generates a database query language based on the optimization result, and performs content retrieval on the database using the database query language to obtain target data related to the problem indicated by the optimization result. Then, based on the target data, the computer device generates response information for the statement to be optimized (such as analysis results, charts of the target data, etc.).

[0136] In another embodiment, the above data optimization method can be applied during model optimization. Specifically, the computer device obtains the result to be optimized (i.e., the rewritten result of the statement that does not meet expectations) corresponding to the statement to be optimized. The result to be optimized is obtained by calling the model to be optimized to rewrite the statement to be optimized. After obtaining the result to be optimized corresponding to the statement to be optimized, the computer device optimizes the model to be optimized (e.g., adjusts the model parameters) based on the difference between the optimization result and the result to be optimized (e.g., constructing a loss function) to obtain the optimized model.

[0137] The embodiments of this application are as follows: Figure 2 Based on the implementation examples, constraint information is generated through learning samples to constrain the business logic corresponding to the statement to be optimized; the process rewriting results are screened by leveraging the characteristic that "for the same question, large-scale language models will repeat more responses with higher confidence" to ensure the quality of the rewritten analysis process; the statement rewriting results of the statement to be optimized are expanded to improve the diversity of the statement rewriting results; and the quality of the optimization results of the statement to be optimized is ensured by screening the statement rewriting results.

[0138] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0139] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a data optimization device provided in an embodiment of this application. Figure 6 The data optimization device shown can be mounted in a computer device, specifically a server. Figure 6 The data optimization device shown can be used to perform the above. Figure 2 and Figure 4 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 6 The data optimization device includes:

[0140] The acquisition unit 601 is used to acquire the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes the statement rewriting example of the reference statement.

[0141] The processing unit 602 is used to perform supplementary content retrieval on the statement to be optimized, and obtain the supplementary content of the statement to be optimized. The vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold.

[0142] And it is used to rewrite the analysis process based on the statement to be optimized, the statement rewriting example, and supplementary content to obtain the rewritten analysis process;

[0143] And it is used to generate optimization results for the statements to be optimized through the rewritten analysis process.

[0144] In one implementation, the processing unit 602 is configured to rewrite the analysis process based on the statement to be optimized, a statement rewriting example, and supplementary content to obtain a rewritten analysis process, specifically for:

[0145] Constraint information is generated based on the supplementary content. The constraint information is used to constrain at least one of the business requirements, business scenarios and business logic corresponding to the statement to be optimized.

[0146] The statement to be optimized, a rewritten example of the statement, and the analysis process are input into a large-scale language model so that the large-scale language model can generate a rewritten analysis process according to the constraint information.

[0147] In one implementation, the processing unit 602 is configured to input the statement to be optimized, a statement rewriting example, and an analysis process into a large-scale language model, so that the large-scale language model generates a rewritten analysis process according to the constraint information, specifically for:

[0148] Input the statement to be optimized, the statement rewriting example, and the analysis process into the large-scale language model so that the large-scale language model generates M rewriting results of the analysis process according to the constraint information, where M is an integer greater than 1;

[0149] Calculate the average similarity of the rewritten results of each process. The average similarity of the rewritten results of any process is the mean of the similarity between the rewritten results of that process and the rewritten results of the remaining M-1 processes.

[0150] Based on the process rewriting results where the average similarity is higher than the similarity threshold, the rewritten analysis process is determined.

[0151] In one embodiment, the processing unit 602 is configured to generate constraint information based on the supplementary content, specifically for:

[0152] Obtain a learning sample, which contains P sub-problems decomposed from the sample problem, and the reasoning steps corresponding to the P sub-problems, where P is an integer greater than 1;

[0153] Constraint information is generated by learning samples and supplementary content. The learning samples are used to constrain the business logic corresponding to the statement to be optimized, and the supplementary content is used to constrain the business requirements or business scenarios corresponding to the statement to be optimized.

[0154] In one implementation, the processing unit 602 is used to generate an optimization result for the statement to be optimized through a rewritten analysis process, specifically for:

[0155] The large-scale language model is invoked to generate N rewritten results of the sentence to be optimized, based on the sentence to be optimized, the supplementary content, and the rewritten analysis process, where N is a positive integer;

[0156] Based on the rewrite results of N statements, the optimized results of the statement to be optimized are generated.

[0157] In one implementation, N is greater than 1, and the description methods corresponding to the N statement rewriting results are different; the processing unit 602 is used to generate the optimization result of the statement to be optimized based on the N statement rewriting results, specifically for:

[0158] The quality score of each statement rewrite result is calculated. The quality score of any statement rewrite result is obtained by comparing the quality of that statement rewrite result with the quality of the other N-1 statement rewrite results.

[0159] The rewritten N statements are sorted in descending order of quality score, and the top k statements are selected as the optimized results for the statements to be optimized, where k is a positive integer less than or equal to N.

[0160] In one implementation, the process by which the processing unit 602 calculates the quality score for each statement rewrite result includes:

[0161] Compare the quality of the rewritten result of the i-th statement with the rewritten result of the j-th statement, where i and j are both positive integers less than or equal to N, and i is not equal to j;

[0162] If the quality of the rewritten result of the i-th statement is better than that of the rewritten result of the j-th statement, then increase the quality score of the rewritten result of the i-th statement and decrease the quality score of the rewritten result of the j-th statement.

[0163] If the quality of the rewritten result of the i-th statement is the same as that of the rewritten result of the j-th statement, then the quality scores of the rewritten result of the i-th statement and the rewritten result of the j-th statement remain unchanged.

[0164] In one implementation, the processing unit 602 is used to perform supplementary content retrieval on the statement to be optimized to obtain supplementary content for the statement to be optimized, specifically for:

[0165] Extract key information from the statement to be optimized to obtain the key information of the statement to be optimized;

[0166] The vector matching degree between the statement to be optimized and each candidate content is calculated from both the key information dimension and the statement dimension.

[0167] Based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, the comprehensive matching degree between the statement to be optimized and each candidate content is determined.

[0168] Candidate content with a comprehensive matching degree higher than the second matching degree threshold is identified as supplementary content for the statement to be optimized.

[0169] In one implementation, the process by which the processing unit 602 determines the overall matching degree between the statement to be optimized and each candidate content based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, includes:

[0170] The vector matching degree between the statement to be optimized and the h-th candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and the h-th candidate content in the statement dimension are weighted and summed to obtain the comprehensive matching degree between the statement to be optimized and the h-th candidate content, where h is a positive integer.

[0171] In one implementation, the optimization result is natural language or structured language; the processing unit 602 is further configured to:

[0172] Based on the optimization results, a database query language is generated.

[0173] By using a database query language to retrieve content from the database, target data related to the problem indicated by the optimization results can be obtained.

[0174] Based on the target data, generate response information for the statement to be optimized.

[0175] In one embodiment, the processing unit 602 is further configured to:

[0176] Obtain the optimization result corresponding to the statement to be optimized. The optimization result is obtained by calling the optimization model to rewrite the statement to be optimized.

[0177] Based on the difference between the optimization result and the result to be optimized, the model to be optimized is optimized to obtain the optimized model.

[0178] According to one embodiment of this application, Figure 2 and Figure 4 The data optimization method shown can be partially implemented by... Figure 6 The data optimization is performed by each unit in the illustrated data optimization device. For example, Figure 2 Step S201 shown can be performed by Figure 6 The acquisition unit 601 shown is executed, and steps S202-S204 can be performed by... Figure 6 The processing unit 602 shown executes the operation; Figure 4 Step S401 shown can be performed by Figure 6 The acquisition unit 601 shown is executed, and steps S402-S406 can be performed by... Figure 6 The processing unit 602 shown executes. Figure 6 The data optimization device shown can be composed of individual or combined units into one or more other units, or some of the units can be further divided into two or more functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by two or more units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data optimization device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by two or more units.

[0179] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 2 and Figure 4 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The data optimization apparatus shown herein, and the data optimization method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0180] Based on the same inventive concept, the principle and beneficial effects of the data optimization device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data optimization method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0181] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a server. Figure 7As shown, the computer device includes at least a processor 701, a communication interface 702, and a memory 703. The processor 701, communication interface 702, and memory 703 can be connected via a bus or other means. The processor 701 (or Central Processing Unit, CPU) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data. For example, the CPU can parse power-on / off commands issued by objects to the computer device and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device, and so on. The communication interface 702 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and can be used to send and receive data under the control of the processor 701; the communication interface 702 can also be used for data transmission and interaction within the computer device. The memory 703 is the storage device in the computer device, used to store programs and data. It can be understood that the memory 703 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 703 provides storage space for storing the operating system of the computer device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0182] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by the processor 701. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0183] In one embodiment, processor 701 performs the following operations by running a computer program stored in memory 703:

[0184] Obtain the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes the statement rewriting examples of the reference statements.

[0185] The supplementary content of the statement to be optimized is retrieved, and the vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold.

[0186] The analysis process is rewritten based on the statement to be optimized, the statement rewriting example, and the supplementary content, resulting in the rewritten analysis process.

[0187] The rewritten analysis process generates optimized results for the statements to be optimized.

[0188] As an optional embodiment, processor 701 rewrites the analysis process based on the statement to be optimized, the statement rewriting example, and supplementary content, resulting in the following specific embodiment of the rewritten analysis process:

[0189] Constraint information is generated based on the supplementary content. The constraint information is used to constrain at least one of the business requirements, business scenarios and business logic corresponding to the statement to be optimized.

[0190] The statement to be optimized, a rewritten example of the statement, and the analysis process are input into a large-scale language model so that the large-scale language model can generate a rewritten analysis process according to the constraint information.

[0191] As an optional embodiment, the processor 701 inputs the statement to be optimized, a statement rewriting example, and the analysis process into a large-scale language model, so that the large-scale language model generates the rewritten analysis process according to the constraint information. A specific embodiment of this is as follows:

[0192] Input the statement to be optimized, the statement rewriting example, and the analysis process into the large-scale language model so that the large-scale language model generates M rewriting results of the analysis process according to the constraint information, where M is an integer greater than 1;

[0193] Calculate the average similarity of the rewritten results of each process. The average similarity of the rewritten results of any process is the mean of the similarity between the rewritten results of that process and the rewritten results of the remaining M-1 processes.

[0194] Based on the process rewriting results where the average similarity is higher than the similarity threshold, the rewritten analysis process is determined.

[0195] As an optional embodiment, a specific embodiment in which the processor 701 generates constraint information based on supplementary content is as follows:

[0196] Obtain a learning sample, which contains P sub-problems decomposed from the sample problem, and the reasoning steps corresponding to the P sub-problems, where P is an integer greater than 1;

[0197] Constraint information is generated by learning samples and supplementary content. The learning samples are used to constrain the business logic corresponding to the statement to be optimized, and the supplementary content is used to constrain the business requirements or business scenarios corresponding to the statement to be optimized.

[0198] As an optional embodiment, the processor 701 generates the optimization result of the statement to be optimized through the rewritten analysis process in the following specific embodiment:

[0199] The large-scale language model is invoked to generate N rewritten results of the sentence to be optimized, based on the sentence to be optimized, the supplementary content, and the rewritten analysis process, where N is a positive integer;

[0200] Based on the rewrite results of N statements, the optimized results of the statement to be optimized are generated.

[0201] As an optional embodiment, N is greater than 1, and the description methods corresponding to the N statement rewriting results are different; the specific embodiment in which the processor 701 generates the optimization result of the statement to be optimized based on the N statement rewriting results is as follows:

[0202] The quality score of each statement rewrite result is calculated. The quality score of any statement rewrite result is obtained by comparing the quality of that statement rewrite result with the quality of the other N-1 statement rewrite results.

[0203] The rewritten N statements are sorted in descending order of quality score, and the top k statements are selected as the optimized results for the statements to be optimized, where k is a positive integer less than or equal to N.

[0204] As an optional embodiment, the process by which processor 701 calculates the quality score of each statement rewrite result includes:

[0205] Compare the quality of the rewritten result of the i-th statement with the rewritten result of the j-th statement, where i and j are both positive integers less than or equal to N, and i is not equal to j;

[0206] If the quality of the rewritten result of the i-th statement is better than that of the rewritten result of the j-th statement, then increase the quality score of the rewritten result of the i-th statement and decrease the quality score of the rewritten result of the j-th statement.

[0207] If the quality of the rewritten result of the i-th statement is the same as that of the rewritten result of the j-th statement, then the quality scores of the rewritten result of the i-th statement and the rewritten result of the j-th statement remain unchanged.

[0208] As an optional embodiment, the processor 701 performs supplementary content retrieval on the statement to be optimized to obtain the supplementary content of the statement to be optimized. A specific embodiment of this is as follows:

[0209] Extract key information from the statement to be optimized to obtain the key information of the statement to be optimized;

[0210] The vector matching degree between the statement to be optimized and each candidate content is calculated from both the key information dimension and the statement dimension.

[0211] Based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, the comprehensive matching degree between the statement to be optimized and each candidate content is determined.

[0212] Candidate content with a comprehensive matching degree higher than the second matching degree threshold is identified as supplementary content for the statement to be optimized.

[0213] As an optional embodiment, the processor 701 determines the comprehensive matching degree between the statement to be optimized and each candidate content based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension. The process includes:

[0214] The vector matching degree between the statement to be optimized and the h-th candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and the h-th candidate content in the statement dimension are weighted and summed to obtain the comprehensive matching degree between the statement to be optimized and the h-th candidate content, where h is a positive integer.

[0215] As an optional embodiment, the optimization result is in natural language or structured language; the processor 701, by running the computer program in the memory 703, also performs the following operations:

[0216] Based on the optimization results, a database query language is generated.

[0217] By using a database query language to retrieve content from the database, target data related to the problem indicated by the optimization results can be obtained.

[0218] Based on the target data, generate response information for the statement to be optimized.

[0219] As an optional embodiment, the processor 701, by running a computer program in the memory 703, also performs the following operations:

[0220] Obtain the optimization result corresponding to the statement to be optimized. The optimization result is obtained by calling the optimization model to rewrite the statement to be optimized.

[0221] Based on the difference between the optimization result and the result to be optimized, the model to be optimized is optimized to obtain the optimized model.

[0222] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data optimization method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.

[0223] This application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and execute the data optimization method described in the above method embodiments.

[0224] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data optimization method described above.

[0225] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0226] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0227] In the embodiments of this application, the term "module" or "unit" refers to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, at least one processor (or memory) can be used to implement at least one module or unit. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0228] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0229] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A data optimization method, characterized in that, The method includes: Obtain the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes a statement rewriting example of a reference statement. The supplementary content retrieval is performed on the statement to be optimized to obtain the supplementary content of the statement to be optimized. The vector matching degree between the supplementary content and the statement to be optimized is higher than the first matching degree threshold. The analysis process is rewritten based on the statement to be optimized, the statement rewriting example, and the supplementary content to obtain the rewritten analysis process. The rewritten analysis process generates the optimized result of the statement to be optimized.

2. The method as described in claim 1, characterized in that, The process of rewriting the analysis process based on the statement to be optimized, the statement rewriting example, and the supplementary content to obtain the rewritten analysis process includes: Constraint information is generated based on the supplementary content. The constraint information is used to constrain at least one of the business requirements, business scenarios, and business logic corresponding to the statement to be optimized. The statement to be optimized, the statement rewriting example, and the analysis process are input into a large-scale language model so that the large-scale language model generates the rewritten analysis process according to the constraint information.

3. The method as described in claim 2, characterized in that, The step of inputting the statement to be optimized, the statement rewriting example, and the analysis process into a large-scale language model, so that the large-scale language model generates a rewritten analysis process according to the constraint information, includes: The statement to be optimized, the statement rewriting example, and the analysis process are input into a large-scale language model so that the large-scale language model generates M process rewriting results of the analysis process according to the constraint information, where M is an integer greater than 1; Calculate the average similarity of the rewritten results of each process. The average similarity of the rewritten results of any process is the mean of the similarity between the rewritten results of that process and the rewritten results of the remaining M-1 processes. Based on the process rewriting results where the average similarity is higher than the similarity threshold, the rewritten analysis process is determined.

4. The method as described in claim 2, characterized in that, The generation of constraint information based on the supplementary content includes: Obtain a learning sample, which includes P sub-problems decomposed from the sample problem, and the reasoning steps corresponding to the P sub-problems, where P is an integer greater than 1; Constraint information is generated using the learning samples and the supplementary content. The learning samples are used to constrain the business logic corresponding to the statement to be optimized, and the supplementary content is used to constrain the business requirements or business scenarios corresponding to the statement to be optimized.

5. The method as described in claim 1, characterized in that, The process of generating the optimization result of the statement to be optimized through the rewritten analysis includes: The large-scale language model is invoked to generate N rewritten results of the statement to be optimized, based on the statement to be optimized, the supplementary content, and the rewritten analysis process, where N is a positive integer; Based on the rewrite results of the N statements, the optimized result of the statement to be optimized is generated.

6. The method as described in claim 5, characterized in that, N is greater than 1, and the description methods corresponding to the N rewritten statement results are different; the step of generating the optimization result of the statement to be optimized based on the N rewritten statement results includes: The quality score of each statement rewrite result is calculated. The quality score of any statement rewrite result is obtained by comparing the quality of that statement rewrite result with the quality of the other N-1 statement rewrite results. The N rewritten statements are sorted in descending order of quality score, and the top k rewritten statements are determined as the optimized results of the statements to be optimized, where k is a positive integer less than or equal to N.

7. The method as described in claim 6, characterized in that, The process of calculating the quality score for each statement rewrite result includes: Compare the quality of the rewritten result of the i-th statement with the rewritten result of the j-th statement, where i and j are both positive integers less than or equal to N, and i is not equal to j; If the quality of the rewritten result of the i-th statement is better than the quality of the rewritten result of the j-th statement, then the quality score of the rewritten result of the i-th statement is increased, and the quality score of the rewritten result of the j-th statement is decreased. If the quality of the rewritten result of the i-th statement is the same as that of the rewritten result of the j-th statement, then the quality scores of the rewritten result of the i-th statement and the rewritten result of the j-th statement remain unchanged.

8. The method as described in claim 1, characterized in that, The step of performing supplementary content retrieval on the statement to be optimized to obtain supplementary content for the statement to be optimized includes: Key information is extracted from the statement to be optimized to obtain the key information of the statement to be optimized. The vector matching degree between the statement to be optimized and each candidate content is calculated from both the key information dimension and the statement dimension. Based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, the comprehensive matching degree between the statement to be optimized and each candidate content is determined. Candidate content with a comprehensive matching degree higher than the second matching degree threshold is identified as supplementary content for the statement to be optimized.

9. The method as described in claim 8, characterized in that, The process of determining the comprehensive matching degree between the statement to be optimized and each candidate content based on the vector matching degree between the statement to be optimized and each candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and each candidate content in the statement dimension, includes: The vector matching degree between the statement to be optimized and the h-th candidate content in the key information dimension, and the vector matching degree between the statement to be optimized and the h-th candidate content in the statement dimension are weighted and summed to obtain the comprehensive matching degree between the statement to be optimized and the h-th candidate content, where h is a positive integer.

10. The method as described in claim 1, characterized in that, The optimization result is in natural language or structured language; the method further includes: Based on the optimization results, a database query language is generated; The database is searched using the database query language to obtain target data related to the problem indicated by the optimization results. Based on the target data, generate response information for the statement to be optimized.

11. The method as described in claim 1, characterized in that, The method further includes: Obtain the optimization result corresponding to the statement to be optimized, wherein the optimization result is obtained by calling the optimization model to rewrite the statement to be optimized. Based on the difference between the optimization result and the result to be optimized, the model to be optimized is optimized to obtain the optimized model.

12. A data optimization device, characterized in that, The data optimization device includes: The acquisition unit is used to acquire the associated data and reference information of the statement to be optimized. The associated data includes the analysis process of the statement to be optimized, and the reference information includes a statement rewriting example of a reference statement. The processing unit is used to perform supplementary content retrieval on the statement to be optimized, and obtain supplementary content of the statement to be optimized, wherein the vector matching degree between the supplementary content and the statement to be optimized is higher than a first matching degree threshold. And to rewrite the analysis process based on the statement to be optimized, the statement rewriting example, and the supplementary content to obtain the rewritten analysis process; And the optimized result is generated by the rewritten analysis process to produce the statement to be optimized.

13. A computer device, characterized in that, include: A memory, wherein a computer program is stored; A processor for loading the computer program to implement the data optimization method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the data optimization method as described in any one of claims 1-11.

Images