Data processing method and device based on large model, electronic equipment and medium

Abstract

The invention provides a data processing method and device based on a large model, electronic equipment and a medium, and relates to the field of data processing, in particular to the field of large models and databases. The method comprises the steps of obtaining an original processing instruction; obtaining a prompt statement for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction; based on the prompt statement, a data processing statement corresponding to the original processing instruction is obtained through the large language model, and the data processing statement can be used for executing corresponding processing on data in a target data source.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to large models and databases, and specifically to a method, device, electronic device, computer-readable storage medium, and computer program product for data processing based on large models. Background Art

[0002] As enterprise data assets accumulate, the demand for structured data management and in-depth analysis is increasing. In these projects, multi-source data management and complex statement generation are usually involved. A method for accurately generating data processing statements is desired.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for data processing based on a large model.

[0005] According to one aspect of the present disclosure, a method for data processing based on a large model is provided, comprising: obtaining an original processing instruction; obtaining a prompt statement for a large language model based on the original processing instruction and a grammatical rule associated with the original processing instruction; and obtaining a data processing statement corresponding to the original processing instruction through the large language model based on the prompt statement, wherein the data processing statement can be used to perform corresponding processing on data in a target data source.

[0006] According to another aspect of the present disclosure, there is provided a device for data processing based on a large model, comprising: an instruction obtaining unit, for obtaining an original processing instruction; a prompt statement obtaining unit, for obtaining a prompt statement for a large language model based on the original processing instruction and a grammatical rule associated with the original processing instruction; and a processing statement obtaining unit, for obtaining a data processing statement corresponding to the original processing instruction through the large language model based on the prompt statement, wherein the data processing statement can be used to perform corresponding processing on data in a target data source.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of large model-based data processing according to one or more embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of large model-based data processing according to one or more embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for large model-based data processing according to one or more embodiments of the present disclosure.

[0010] According to one or more embodiments of the present disclosure, data processing statements may be obtained more accurately and efficiently.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0014] Figure 2 A flowchart of a method for data processing based on a large model according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram showing a scenario to which the method for data processing based on a large model according to an embodiment of the present disclosure may be applicable;

[0016] Figure 4 A schematic diagram showing a data flow of a method for data processing based on a large model according to an embodiment of the present disclosure;

[0017] Figure 5 A structural block diagram of a device for data processing based on a large model according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0021] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0022] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0024] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the method of large model-based data processing according to the present disclosure.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform data processing, etc. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0029] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0033] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0034] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system. Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0036] Reference below Figure 2 A method 200 of large model-based data processing according to an exemplary embodiment of the present disclosure is described.

[0037] In step S201, an original processing instruction is obtained.

[0038] At step S202, a prompt sentence for a large language model is obtained based on the original processing instruction and a grammar rule associated with the original processing instruction.

[0039] At step S203, based on the prompt statement, a data processing statement corresponding to the original processing instruction is obtained through the large language model, and the data processing statement can be used to perform corresponding processing on the data in the target data source.

[0040] According to the method described in the embodiment of the present disclosure, data processing statements can be obtained more accurately and effectively.

[0041] Exemplarily, the original processing instruction may be an instruction, an operation description, a query word, or a query statement (query) indicating that corresponding processing is to be performed on the data in the target data source, and a prompt statement (prompt) for a large language model may be obtained based on the original processing instruction and the grammatical rules associated with the target data source, so as to better generate a data processing statement of the grammatical rules that matches the target data source. It is to be understood that throughout this article, although the original instruction is sometimes referred to as a query query according to the custom of those skilled in the art, this does not require that the corresponding processing operation must be a data query operation in a narrow sense.

[0042] As an exemplary scenario, the exemplary embodiments according to the present disclosure can be applied to product system acceptance scenarios, and the generated data processing statements can be used as test data.With the accumulation of enterprise data assets, structured data management and in-depth analysis have become the core requirements of major customer privatization projects.

[0043] These projects usually involve multi-source data management and complex statement generation. Figure 3As shown, an exemplary application scenario diagram is shown, which may involve various types of data sources, such as MYSQL, Oracle, HIVE, HDFS, etc. To ensure the complete acceptance of the product system, the complexity of the data type and the business logic and adaptability requirements all bring technical challenges. On the one hand, customer business data includes a variety of data source storage system types, and it is necessary to ensure that the test covers all types to ensure authenticity and comprehensiveness. On the other hand, the generated data should conform to the business logic and fully cover the data fields and grammatical rules involved in the project to verify the adaptability of the product system in different business scenarios. According to one or more embodiments of the present disclosure, prompt statements can be obtained based on the corresponding grammatical rules, so that the generated test data is more targeted and accurate.

[0044] It is understandable that data processing statements may include statements for querying, changing or performing various other operations on data. As a specific non-limiting example, DDL and DML syntax may be used. The data definition language DDL (Data Definition Language) is a subset of SQL used to define the structure of a database. It contains a series of operations for creating, modifying and deleting objects such as tables, views, indexes, etc. in a database, and common statements include CREATE, ALTER, DROP, etc. The data manipulation language DML (Data Manipulation Language) is used to query, insert, update and delete data in a database, and is the main syntax for processing data in SQL. Generally speaking, DML operations do not change the structure of the database, but target the data content, and common statements include SELECT, INSERT, UPDATE, DELETE, etc. It is understandable that the above two languages ​​are only examples, and the present disclosure is not limited thereto.

[0045] According to some embodiments, obtaining a prompt statement for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction may include: determining the type of the target data source based on the original processing instruction; obtaining the grammar rule based on the type of the target data source; and obtaining the prompt statement based on the grammar rule and the original processing instruction.

[0046] In such an embodiment, the type of the target data source can be determined, and the corresponding grammatical rules and specifications can be obtained (e.g., retrieved, identified, or determined in other ways), thereby obtaining a prompt statement that complies with the grammatical rules. For example, based on the original processing instruction, the type of the target data source that matches the original processing instruction can be determined.

[0047] According to some embodiments, determining the grammar rule based on the type of the target data source may include: obtaining the grammar rule by searching a first knowledge base.

[0048] Thus, corresponding grammatical rules, including rules and templates, can be obtained based on the open source knowledge base of the large language model.

[0049] According to some embodiments, obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction may include: concatenating the grammar rule with the original processing instruction to obtain the prompt sentence.

[0050] According to such an embodiment, a prompt statement conforming to the grammar of the target data source can be generated by concatenating or integrating.

[0051] According to some embodiments, obtaining a prompt statement for a large language model based on the original processing instruction and the grammatical rules associated with the original processing instruction may include: obtaining a prompt statement example, the prompt statement example follows the grammatical rules associated with the original processing instruction; and obtaining the prompt statement based on the prompt statement example through a second large language model.

[0052] For complex business statement examples, the system can provide the examples to the large language model and automatically generate prompts with similar grammatical structure and logic. This method enables the large language model to imitate prompts based on complex examples, achieving adaptive support for complex queries in various business scenarios.

[0053] It is understandable that the second largest language model may be the same as the first largest language model, or may be a different model.

[0054] According to some embodiments, obtaining a hint statement for a large language model based on the original processing instruction and the grammatical rules associated with the original processing instruction may include: obtaining a first intermediate statement based on the original processing instruction and the grammatical rules associated with the original processing instruction; and obtaining the hint statement based on the first intermediate statement.

[0055] Multiple intermediate sentences can be obtained, and a final prompt sentence can be obtained based on the intermediate sentences. The grammar rule can be used as a part of the intermediate sentences in the multiple intermediate sentences, thereby presenting the prompt part of the grammar rule.

[0056] For example, at least one sentence part of the prompt sentence for the large language model can be obtained based on the original processing instruction and the grammar of the database corresponding to the original processing instruction, and the prompt sentence for the large language model for the original processing instruction is obtained based on the at least one sentence part. For example, obtaining at least one sentence part of the prompt sentence for the large language model based on the original processing instruction includes obtaining a first prompt sentence part based on the grammar corresponding to the original processing instruction.

[0057] In some embodiments, the scheme for obtaining relevant grammar rules can be referred to as a grammar expansion technology based on large model open source knowledge. For example, the open source knowledge base of the large model can be used to realize the automatic expansion and generation of data source grammar. As described above, and as will be further expanded below, the present disclosure provides two exemplary ways of performing grammar processing.

[0058] As an example, in the process of data source syntax query, the original query will first match the corresponding data source and identify the syntax specifications and rules of the target data source. After the matching is completed, the system automatically retrieves the syntax rules and templates of the data source from the open source knowledge base of the big model, integrates them with the original query, generates prompt words that conform to the specific syntax of the data source, and then generates DDL and DML statements that are highly compatible with the data source syntax.

[0059] As another example, for complex business statement examples, the system can provide the examples to the big model and automatically generate prompts with similar grammatical structure and logic. This method enables the big model to imitate prompts based on complex examples, and realize adaptive support for complex queries in various business scenarios.

[0060] Some specific non-limiting generation examples are given below.

[0061]

[0062]

[0063] It can be seen that when the method of the embodiment of the present disclosure is not applied, the generated DDL statement only supports mainstream data source types such as MySQL, and the coverage rate of the generated data field types is not high.

[0064]

[0065] As another example of a method not applying the embodiments of the present disclosure, it can be seen that the generated DML statement only covers common GROUP BY, ORDER, and HAVING keywords.

[0066] On the contrary, according to the first example described above, a DDL statement may be generated by matching the data source type.

[0067] Taking Oracle as an example, the created table has a coupling relationship and covers all field types, especially the uncommon XMLType, UROWID, CLOB, etc.

[0068]

[0069]

[0070] As another non-limiting example, when the method of Example 2 is adopted, a complex business example can be added to generate DML as a few-shot.

[0071] In this example, the data source type is MySQL, and the effect is shown in the following table.

[0072]

[0073]

[0074]

[0075] An example in which the data source type is Oracle according to an embodiment of the present disclosure is given below.

[0076]

[0077] As can be seen from the above examples, according to the embodiments of the present disclosure, the complexity of multi-table joins is improved, such as subqueries and data type conversions, and reasonable calculation logic is added to the query. In addition, the keyword coverage is significantly improved, such as aggregate functions, window functions, partition functions, and SQL statements such as NCLOB and XMLTABLE that require a deep understanding of Oracle to write.

[0078] According to some embodiments, obtaining a prompt statement for a large language model based on the original processing instruction and a grammatical rule associated with the original processing instruction may include: obtaining a second intermediate statement based on business knowledge related to the original processing instruction; and obtaining a prompt statement for the large language model for the original processing instruction based on the first intermediate statement and the second intermediate statement.

[0079] The intermediate statement can also be obtained based on business-related knowledge, thereby limiting at the business level. For example, obtaining at least one statement portion of the prompt statement for the large language model based on the original processing instruction includes obtaining a second prompt statement portion based on business knowledge related to the original processing instruction. For example, the second prompt statement portion can be obtained based on a closed source knowledge base related to the business corresponding to the original processing instruction.

[0080] According to some embodiments, obtaining the second intermediate statement based on the business knowledge related to the original processing instruction may include: obtaining the second intermediate statement by searching a second knowledge base corresponding to the business related to the original processing instruction.

[0081] By searching the business-related knowledge base to obtain statements, constraints at the business level can be added. The second knowledge base can be a closed-source knowledge base related to the business corresponding to the original processing instruction.

[0082] According to some exemplary embodiments, such a method may be referred to as using RAG enhanced generation technology to interface with business closed-source knowledge.

[0083] According to some embodiments, the second knowledge base is obtained by the following operations: cleaning the business grammar documents related to the original processing instructions to obtain business grammar content; segmenting the business grammar content; and converting the segmented business grammar content into vectors, and persistently storing them in a vector database as the second knowledge base.

[0084] By analyzing the structure of business grammar documents, business-related knowledge and grammar rules can be integrated into the generation process of the large language model. For example, segmentation can be based on topics, grammar points, and data format requirements.

[0085] As a specific non-limiting example, the business grammar document can be cleaned first to remove redundant information and irrelevant content, and the content can be segmented according to specific topics, grammar points and data format requirements. Then it is converted into vectors and persistently stored in a vector database to form a closed source knowledge base to supplement and enhance the large model's ability to understand and respond to specific business scenarios.

[0086] According to such an embodiment, the original query can be vectorized so that the vector-based query method can accurately match the business keywords and hit the document block where the keywords are located. The model will optimize the structure of the prompt word based on these matching results, and expand and improve the original query to generate data that is more in line with specific business scenarios.

[0087] Exemplarily, according to an embodiment of the present disclosure, by inputting "string search function" or "type conversion function", non-high-frequency keywords such as extractALL can be matched to generate statements that fit the customer's actual application scenario.

[0088] According to some embodiments, obtaining a hint statement for the large language model for the original processing instruction based on the first intermediate statement and the second intermediate statement may include: performing a deduplication operation on at least one of the first intermediate statement and the second intermediate statement to obtain a deduplicated intermediate statement; and concatenating the deduplicated intermediate statements to obtain the hint statement.

[0089] Multiple intermediate statements are deduplicated and concatenated to obtain integrated prompt statements, thereby constraining at different levels to obtain more targeted prompt statements.

[0090] According to some embodiments, the data processing statement may include at least one of a data definition language statement and a data manipulation language statement.

[0091] Data definition language and data manipulation language can be generated to meet different query requirements.

[0092] According to some embodiments, the method may further include, after obtaining the data processing statement corresponding to the original processing instruction, performing error correction on the data to obtain an error-corrected data processing statement.

[0093] According to such an embodiment, error correction can also be performed, thereby reducing errors in statements generated by large models and enabling test data to be dynamically adjusted according to business needs.

[0094] According to some embodiments, correcting the data processing statement to obtain a corrected data processing statement may include: obtaining the corrected data processing statement based on the correction prompt statement and the data processing statement through a third language model.

[0095] According to such an embodiment, the large model automatically corrects errors through a feedback mechanism, performs a first round of verification before execution, and identifies potential grammatical errors.

[0096] According to some embodiments, correcting the data processing statement to obtain a corrected data processing statement may include: obtaining error information corresponding to the data processing statement; and obtaining the corrected data processing statement based on the error information and the data processing statement through a fourth language model.

[0097] According to some embodiments of the present disclosure, a second round of verification can be performed in conjunction with the product system output to correct errors in a targeted manner. For example, during the acceptance execution process, a second round of verification can be performed in conjunction with the product system output to identify syntax compatibility issues and dynamically reload and modify related scenarios.

[0098] According to one or more embodiments of the present disclosure, a dynamic automatic error correction mechanism and a data self-healing mechanism are provided. Test data can be dynamically adjusted according to business needs. On the one hand, automatic error correction can be performed through a feedback mechanism, and a first round of verification can be performed before execution to identify potential syntax errors. On the other hand, during the acceptance execution process, a second round of verification can be performed in combination with the product system output to identify syntax compatibility issues, and dynamically reload and modify related scenarios.

[0099] The effects of error correction are now described in conjunction with some specific non-limiting embodiments. As comparative examples, some direct generation effects with errors are given.

[0100]

[0101]

[0102] It can be seen that there is incorrect SQL because the business product system implementation feature does not support external references in subqueries.

[0103] According to one or more embodiments of the present disclosure, error correction can be performed. For example, the writing method can be changed to output the correct SQL without changing the query result.

[0104] The exemplary effect after model error correction can be shown as follows.

[0105]

[0106]

[0107]

[0108] According to one or more embodiments of the present disclosure, a large model-enabled structured data generation method framework can be provided, which can improve the efficiency and integrity of product acceptance.

[0109] It is understandable that self-correction can allow the model to find out whether there is an error by giving the model some judgment conditions, compatible versions, etc. On the other hand, the input data, error reports, etc. can be integrated as prompts and input into the model, such as "You are an xxx engineer, and you have the following error reports for xx data...". Writing rules can be input into the model, such as "The table structure of the data set I created is like this, and my query syntax under this input is xxxx; under this query syntax, this error is included, please correct it."

[0110] Figure 4 The data flow according to an exemplary embodiment of the present disclosure is shown, which integrates RAG enhanced generation technology to connect with business closed-source knowledge, syntax expansion technology based on large model open source knowledge, and dynamic automatic error correction mechanism and data self-healing. However, it can be understood that these elements do not need to exist together, and one or more of them can be omitted and the desired effect can still be achieved.

[0111] According to one or more embodiments of the present disclosure, it can be applied to structured data generation scenarios. Exemplarily, the method according to the embodiments of the present disclosure can support the generation of CRUD DDL and DML statements, especially in external data sources and self-developed business scenarios.

[0112] According to one or more embodiments of the present disclosure, the diversity of generated data can be ensured. The generated statements can directly adapt to a variety of data sources, can fit the business logic, and generate high-quality complex statements.

[0113] According to one or more embodiments of the present disclosure, the automated error correction mechanism reduces the time investment in manual review and can reduce labor costs.

[0114] According to one or more embodiments of the present disclosure, prompt words with data source characteristics can be obtained, while reducing the learning cost of having a deep understanding of the data source. According to one or more embodiments of the present disclosure, various statements including data creation and insertion operations can be supported, and can be applied to complex business scenarios with poor execution effects. According to one or more embodiments of the present disclosure, since the large model can access business closed-source knowledge, a higher generation quality can be obtained, and efficient automated processing can be achieved.

[0115] According to one or more embodiments of the present disclosure, the flexibility of the system can be enhanced. Through multiple prompt strategies, the system can cope with various business requirements.

[0116] Reference now Figure 5The apparatus 500 for data processing based on a large model according to an embodiment of the present disclosure is described. The apparatus 500 for data processing based on a large model may include an instruction obtaining unit 501 , a prompt statement obtaining unit 502 , and a processing statement obtaining unit 503 .

[0117] The instruction obtaining unit 501 can be used to obtain an original processing instruction. The prompt statement obtaining unit 502 can be used to obtain a prompt statement for a large language model based on the original processing instruction and a grammatical rule associated with the original processing instruction. The processing statement obtaining unit 503 can be used to obtain a data processing statement corresponding to the original processing instruction based on the prompt statement through the large language model, and the data processing statement can be used to perform corresponding processing on the data in the target data source.

[0118] According to the device described in the embodiment of the present disclosure, data processing statements can be obtained more accurately and effectively.

[0119] According to some embodiments, the prompt statement obtaining unit may include: a unit for determining the type of the target data source based on the original processing instruction; a unit for obtaining the grammar rule based on the type of the target data source; and obtaining the prompt statement based on the grammar rule and the original processing instruction.

[0120] According to some embodiments, the unit for determining the grammar rule based on the type of the target data source may include: a unit for obtaining the grammar rule by searching a first knowledge base.

[0121] According to some embodiments, the prompt sentence obtaining unit may include: a unit for concatenating the grammatical rule with the original processing instruction to obtain the prompt sentence.

[0122] According to some embodiments, the prompt statement obtaining unit may include: a unit for obtaining a prompt statement example, wherein the prompt statement example complies with a grammatical rule associated with the original processing instruction; and a unit for obtaining the prompt statement through a second largest language model based on the prompt statement example.

[0123] According to some embodiments, the hint statement obtaining unit may include: a unit for obtaining a first intermediate statement based on the original processing instruction and a grammar rule associated with the original processing instruction; and a unit for obtaining the hint statement based on the first intermediate statement.

[0124] According to some embodiments, the prompt statement obtaining unit may include: a unit for obtaining a second intermediate statement based on business knowledge related to the original processing instruction; and a unit for obtaining a prompt statement for the large language model for the original processing instruction based on the first intermediate statement and the second intermediate statement.

[0125] According to some embodiments, the unit for obtaining the second intermediate statement based on the business knowledge related to the original processing instruction may include: a unit for obtaining the second intermediate statement by searching a second knowledge base corresponding to the business related to the original processing instruction.

[0126] According to some embodiments, the second knowledge base may be obtained by the following operations: cleaning the business grammar document related to the original processing instruction to obtain business grammar content; segmenting the business grammar content; and converting the segmented business grammar content into vectors, and persistently storing them in a vector database as the second knowledge base.

[0127] According to some embodiments, a unit for obtaining a hint statement for the large language model for the original processing instruction based on the first intermediate statement and the second intermediate statement may include: a unit for performing a deduplication operation on at least one of the first intermediate statement and the second intermediate statement to obtain a deduplicated intermediate statement; and a unit for concatenating the deduplicated intermediate statements to obtain the hint statement.

[0128] According to some embodiments, the data processing statement may include at least one of a data definition language statement and a data manipulation language statement.

[0129] According to some embodiments, the apparatus may further include a unit for, after obtaining the data processing statement corresponding to the original processing instruction, performing error correction on the data to obtain an error-corrected data processing statement.

[0130] According to some embodiments, the unit for correcting the data processing statement to obtain a corrected data processing statement may include: a unit for obtaining a corrected data processing statement based on the correction prompt statement and the data processing statement through a third language model.

[0131] According to some embodiments, a unit for correcting the data processing statement to obtain a corrected data processing statement may include: a unit for obtaining error information corresponding to the data processing statement; and a unit for obtaining the corrected data processing statement based on the error information and the data processing statement through a fourth language model.

[0132] In the technical solution disclosed herein, the collection, acquisition, storage, use, processing, transmission, provision and public application of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0133] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0134] refer to Figure 6 , a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0135] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0136] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0137] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as method 200 and its variants, etc. For example, in some embodiments, method 200 and its variants, etc. may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method 200 and its variants, etc. described above may be executed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method 200 and its variations, etc., in any other appropriate manner (eg, by means of firmware).

[0138] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0140] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0145] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for data processing based on a large model, comprising: Get the original processing instructions; Obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction; as well as Based on the prompt statement, a data processing statement corresponding to the original processing instruction is obtained through the large language model, and the data processing statement can be used to perform corresponding processing on the data in the target data source.

2. The method according to claim 1, wherein: Obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction includes: determining the type of the target data source based on the original processing instruction; Obtaining the grammar rule based on the type of the target data source; and The prompt sentence is obtained based on the grammatical rule and the original processing instruction.

3. The method according to claim 2, wherein: The determining of the grammar rule based on the type of the target data source comprises: The grammatical rules are obtained by searching the first knowledge base.

4. The method according to claim 2 or 3, wherein: The step of obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction comprises: The grammar rule is concatenated with the original processing instruction to obtain the prompt statement.

5. The method according to any one of claims 1 to 4, wherein: Obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction includes: obtaining a hint statement example, the hint statement example following a grammar rule associated with the original processing instruction; and Based on the prompt sentence example, the prompt sentence is obtained through a second language model.

6. The method according to any one of claims 1 to 5, wherein: Obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction includes: obtaining a first intermediate statement based on the original processing instruction and a grammar rule associated with the original processing instruction; and The prompt sentence is obtained based on the first intermediate sentence.

7. The method according to claim 6, wherein: Obtaining a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction includes: Obtaining a second intermediate statement based on business knowledge related to the original processing instruction; and A hint sentence for the large language model with respect to the original processing instruction is obtained based on the first intermediate sentence and the second intermediate sentence.

8. The method according to claim 7, wherein: Obtaining the second intermediate statement based on the business knowledge related to the original processing instruction includes: obtaining the second intermediate statement by searching a second knowledge base corresponding to the business related to the original processing instruction.

9. The method according to claim 8, wherein: The second knowledge base is obtained by the following operations: Cleaning the business grammar document related to the original processing instruction to obtain business grammar content; Segmenting the service grammar content; as well as The segmented business grammar content is converted into vectors and persistently stored in a vector database as the second knowledge base.

10. The method according to any one of claims 7 to 9, wherein: Obtaining a prompt statement for the large language model for the original processing instruction based on the first intermediate statement and the second intermediate statement includes: performing a deduplication operation on at least one of the first intermediate statement and the second intermediate statement to obtain a deduplicated intermediate statement; and The deduplicated intermediate sentences are concatenated to obtain the prompt sentence.

11. The method according to any one of claims 1 to 10, wherein: The data processing statement includes at least one of a data definition language statement and a data manipulation language statement.

12. The method according to any one of claims 1-11, further comprising, after obtaining the data processing statement corresponding to the original processing instruction, performing error correction on the data to obtain an error-corrected data processing statement.

13. The method according to any one of claims 1 to 12, wherein: Correcting the data processing statement to obtain a corrected data processing statement includes: obtaining a corrected data processing statement based on the error correction prompt statement and the data processing statement through a third language model.

14. The method according to any one of claims 1 to 13, wherein: Correcting the data processing statement to obtain a corrected data processing statement includes: Obtaining error information corresponding to the data processing statement; and The corrected data processing statement is obtained based on the error information and the data processing statement through a fourth language model.

15. A device for data processing based on a large model, comprising: An instruction obtaining unit, used for obtaining original processing instructions; a prompt sentence obtaining unit, configured to obtain a prompt sentence for a large language model based on the original processing instruction and a grammar rule associated with the original processing instruction; as well as A processing statement obtaining unit is used to obtain a data processing statement corresponding to the original processing instruction based on the prompt statement through the large language model, and the data processing statement can be used to perform corresponding processing on the data in the target data source.

16. The device according to claim 15, wherein: The prompt sentence obtaining unit comprises: means for determining a type of said target data source based on said original processing instruction; means for obtaining the grammar rules based on the type of the target data source; and The prompt sentence is obtained based on the grammatical rule and the original processing instruction.

17. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

19. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 14 when executed by a processor.

Images