Data processing method and device, equipment and medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a medium. The method can comprise the following steps: after obtaining data table information and task demand information for processing data corresponding to the data table information, determining a plurality of candidate links for processing the data; wherein the type of the code adopted by any candidate link in the plurality of candidate links is different from the types of the codes adopted by other candidate links; determining a first processing code set based on the task demand information and the plurality of candidate links; wherein the first processing code set comprises processing codes obtained by performing code conversion on the task demand information according to code types corresponding to the candidate links; afterwards, semantic consistency verification is conducted on simulation execution results of all the processing codes in the first processing code set, a second processing code set is obtained, and then a target processing code is determined from the second processing code set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular relates to a data processing method and device, equipment and medium. BACKGROUND

[0002] Enterprise-level data architecture is naturally heterogeneous, and coexists with a relational database that needs to be accessed by a structured query language (SQL), an analysis engine that needs to be processed by a domain-specific language (DSL), and a data platform that needs to be scientifically calculated by Python. Therefore, the traditional single link solution (such as using only a structured query language to execute a data processing task) cannot be compatible with all systems, resulting in limited coverage scenarios, and low success rate and accuracy of data processing.

[0003] In view of this, it is necessary to provide a data processing scheme with high accuracy and high success rate. SUMMARY

[0004] In view of this, the embodiments of the present application provide a data processing method, device, equipment and medium to provide a data processing scheme with high accuracy and high success rate.

[0005] To solve the above technical problems, the embodiments of the present application provide a data processing method, comprising:

[0006] Obtaining data table information and task demand information for processing data corresponding to the data table information;

[0007] Determining a plurality of candidate links for processing the data; any candidate link in the plurality of candidate links adopts a code type different from that of other candidate links;

[0008] Based on the task demand information and the plurality of candidate links, determining a first processing code set; the first processing code set includes processing codes obtained by code conversion of the task demand information according to the code type of each candidate link;

[0009] Performing semantic consistency verification on the simulation execution result of each processing code in the first processing code set to obtain a second processing code set; the second processing code set includes at least one processing code that meets a preset condition in the semantic consistency verification;

[0010] Determining a target processing code from the second processing code set.

[0011] The embodiments of the present application also provide a data processing device, comprising:

[0012] an acquisition module, configured to acquire data table information and task requirement information of processing data corresponding to the data table information;

[0013] a first determination module, configured to determine a plurality of candidate links for processing the data; any candidate link in the plurality of candidate links adopts a code type different from a code type adopted by other candidate links;

[0014] a second determination module, configured to determine a first processing code set based on the task requirement information and the plurality of candidate links; the first processing code set includes processing codes obtained by code conversion of the task requirement information according to the code types corresponding to the candidate links;

[0015] a verification module, configured to perform semantic consistency verification on simulation execution results of the processing codes in the first processing code set to obtain a second processing code set; the second processing code set includes at least one processing code satisfying a preset condition in the semantic consistency verification;

[0016] a third determination module, configured to determine a target processing code from the second processing code set.

[0017] Embodiments of the present specification also provide a data processing device, comprising:

[0018] at least one processor; and

[0019] a memory in communication connection with the at least one processor; wherein

[0020] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0021] acquire data table information and task requirement information of processing data corresponding to the data table information;

[0022] determine a plurality of candidate links for processing the data; any candidate link in the plurality of candidate links adopts a code type different from a code type adopted by other candidate links;

[0023] determine a first processing code set based on the task requirement information and the plurality of candidate links; the first processing code set includes processing codes obtained by code conversion of the task requirement information according to the code types corresponding to the candidate links;

[0024] The semantic consistency check is performed on simulation execution results of each processing code in the first processing code set, to obtain a second processing code set; the second processing code set includes at least one processing code that meets a preset condition in the semantic consistency check;

[0025] A target processing code is determined from the second processing code set.

[0026] The embodiments of the present specification further provide a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the above data processing method.

[0027] At least one of the embodiments provided in the present specification can achieve the following beneficial effects:

[0028] In the embodiments of the present specification, after obtaining the data table information and the task demand information for processing the data corresponding to the data table information, a plurality of candidate links for processing the data corresponding to the data table information can be determined; the type of code used by any one of the plurality of candidate links is different from the type of code used by other candidate links; then, based on the task demand information and the plurality of candidate links, a first processing code set is determined; the first processing code set includes processing codes obtained by code conversion of the task demand information according to the type of code corresponding to each candidate link; then, semantic consistency check is performed on simulation execution results of each processing code in the first processing code set, to obtain a second processing code set; finally, a target processing code is determined from the second processing code set. Thus, the data processing task can be executed based on the determined target processing code, to obtain an accurate data processing result. Since the plurality of candidate links are determined at the same time and then selected from them, rather than using a fixed single processing link, the problem of limited coverage scenario and low success rate of data processing caused by a single link can be avoided, and a link with a high success rate and suitable for a scenario can be selected from the plurality of candidate links for execution, which is beneficial to improve the scenario applicability and success rate of data processing. In addition, by performing semantic consistency check on the simulation execution results of each processing code, the inaccurate processing code that does not meet the semantic consistency check standard can be filtered out, which is beneficial to improve the accuracy of data processing. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in the embodiments of this specification;

[0031] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this specification;

[0032] Figure 3 This is an overall flowchart of a data processing method provided in the embodiments of this specification;

[0033] Figure 4 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a data processing device;

[0034] Figure 5 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a data processing device. Detailed Implementation

[0035] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0036] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0038] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in the relevant region, and provide corresponding operation portal for user to choose authorization or refusal.

[0039] Enterprise data architecture is usually heterogeneous by nature, and there are relational databases that need structured query language (SQL) access, analysis engines that need domain-specific language (DSL) processing, and data platforms that need Python for scientific computing. When users need to query, analyze, calculate, and other processes on the data in the data system, they can issue data processing tasks to the data processing platform in the form of natural language.

[0040] In the prior art, after the data processing platform receives the data processing task, it usually uses a fixed single link to execute the data processing task (such as using only structured query language to execute the data processing task), so it cannot be compatible with all data systems, resulting in limited application scenarios that can be covered, and low success rate and accuracy of data processing. Especially for complex data processing tasks, the single link solution has low success rate and accuracy for executing data processing tasks.

[0041] In order to solve the defects in the related art, the present application provides the following embodiments.

[0042] Figure 1 An application scenario diagram of a data processing method provided by an embodiment of the present application is shown.

[0043] As shown in Figure 1 The data processing platform can include a data processing platform client 101 and a data processing platform server 102. The data processing platform client 101 can be mounted in the terminal device of the user, and the user can issue a data processing task at the data processing platform client 101, which can include data table information and task requirement information, wherein the task requirement information is the requirement information for processing the data corresponding to the data table information.

[0044] After receiving the data processing task issued by the user, the data processing platform client 101 can send the data table information and the task requirement information to the data processing platform server 102. After receiving the data table information sent by the data processing platform client 101 and the task requirement information for processing the data corresponding to the data table information, the data processing platform server 102 can determine a plurality of candidate links for processing the data, and determine a first processing code set based on the task requirement information and the plurality of candidate links; wherein the first processing code set includes processing codes obtained by code converting the task requirement information according to the code categories corresponding to each candidate link. Further, the data processing platform server 102 can further perform semantic consistency verification on the simulation execution results of each processing code in the first processing code set to obtain a second processing code set; wherein the second processing code set includes at least one processing code that meets the preset condition in the semantic consistency verification.

[0045] Then, the data processing platform server 102 can further determine a target processing code from the second processing code set, and further execute the target processing code to obtain a data processing result of the data processing task. The data processing platform server 102 can feed back the data processing result to the data processing platform client 101. After receiving the data processing result, the data processing platform client 101 can show the data processing result to the user.

[0046] The terminal device of the user can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, an object content computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC).

[0047] The data processing platform server 102 can include but is not limited to any device, equipment, platform, server cluster, etc. with computing and processing capabilities. Specifically, the server at the data processing platform server 102 can be a standalone physical server, or a server cluster composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0048] Data processing platform client 101 and data processing platform server 102 can transmit data through local area network connection, wide area network connection, Internet connection or other types of data network connection, or through other means, without specific limitations.

[0049] In addition, although Figure 1 In at least some of the embodiments shown in this specification, the data processing method provided in the embodiments of this specification can be jointly executed by the data processing platform client 101 and the data processing platform server 102. However, in other optional embodiments, if the running resources of the data processing platform client 101 can meet the processing conditions required by the data processing task, the data processing method provided in the embodiments of this specification can also be completed by the data processing platform client 101. No specific limitation is made in this regard.

[0050] Figure 1 The method described involves a data processing platform acquiring data table information and task requirements for processing the corresponding data. This allows for the identification of multiple candidate processing paths. Each candidate path uses a different code type than the others. Based on the task requirements and the candidate paths, a first set of processing codes is determined. This first set includes processing codes derived from the code transformation of the task requirements based on the code types corresponding to each candidate path. The simulated execution results of each processing code in the first set are then semantically consistent to obtain a second set of processing codes. Finally, the target processing code is determined from this second set. This allows for the execution of data processing tasks based on the determined target processing code, resulting in accurate data processing results. Because the data processing platform identifies multiple candidate processing paths simultaneously and then selects the best one, rather than using a fixed single processing path, it avoids the problems of limited coverage and low success rate caused by a single path. By selecting a path with suitable scenarios and a high success rate from multiple candidate paths, it improves the applicability and success rate of data processing. Furthermore, by performing semantic consistency checks on the simulated execution results of each processing code, inaccurate processing code that does not meet the semantic consistency check criteria can be filtered out, thus improving the accuracy of data processing.

[0051] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this specification. From a hardware perspective, the entity executing this process can be a server or a terminal. From a program perspective, the entity executing this process can be a program hosted on the server or terminal.Figure 2 As shown, the flow can include the following steps:

[0052] Step 202: Obtain data table information and task requirement information for processing data corresponding to the data table information.

[0053] In actual application, a user can issue a data processing task, which can include data table information and task requirement information for processing data corresponding to the data table information. The data table information can include but is not limited to a reference data table and query path information of the reference data table.

[0054] The reference data table can be a data table uploaded by the user at a data processing platform client, and the reference data table can be a data table required by the data processing task in the data processing process. For example, the user uploads data table A at the data processing platform client and issues the task requirement information: "statistical male and female student ratio of each class in data table A". The data processing platform can take data table A as the reference data table, and perform statistical processing based on data in data table A to obtain the male and female student ratio data of each class in data table A.

[0055] The query path information of the reference data table can be related information about the query path of the reference data table uploaded by the user at the data processing platform client, such as storage location information of the reference data table, database information where the reference data table is located, name information of the reference data table, etc. The data processing platform can retrieve the reference data table from the storage location of the reference data table or query data in the reference data table based on the query path information of the reference data table provided by the user. For example, the user specifies the query path information of data table A at the data processing platform client: "data table A is in database B", and issues the task requirement information: "please statistical male and female student ratio of each class in data table A". The data processing platform can obtain data table A from database B or query data in data table A from database B, and perform statistical processing based on data in data table A to obtain the male and female student ratio data of each class in data table A.

[0056] In the embodiments of the present application, the task requirement information can be personal requirement information specified by the user in natural language when the user publishes the data processing task at the data processing platform client. The task requirement information can be information specified by the user for processing data corresponding to the data table information, wherein if the data table information includes a reference data table, the data corresponding to the data table information can be data in the reference data table; if the data table information includes query path information of the reference data table, the data corresponding to the data table information can be data in the reference data table obtained after the query. Specifically, the task requirement information can include but is not limited to query requirement for part of the data in the reference data table, statistical requirement for part of the data in the reference data table, calculation requirement for part of the data in the reference data table, etc.

[0057] In actual application, when the user specifies the task requirement information, the user can specify in the form of natural language, thereby improving the convenience of the user publishing the data processing task, and being conducive to improving the user experience. In addition, the language of the natural language is not limited in the embodiments of the present application, which can be English, Chinese or other languages, and no specific limitation is made.

[0058] Step 204: determining a plurality of candidate links for processing the data; any candidate link in the plurality of candidate links adopts a code of a different type from the codes adopted by other candidate links.

[0059] In actual application, after the data processing platform obtains the data processing task published by the user, the data processing platform can use the reinforcement learning strategy model to determine a plurality of candidate links that can be used to process the data corresponding to the data table information based on the data table information and the task requirement information in the data processing task from a plurality of preset alternative processing links. Since the subsequent embodiments of the present application will explain in detail the process of determining the candidate link by using the reinforcement learning strategy model, no further description is made here.

[0060] The reinforcement learning strategy model can be a large language model (LLM). The large language model is a deep learning model trained using a large amount of text data. The large language model can generate natural language text or understand the meaning of language text. The large language model can provide relevant knowledge about various topics by being trained on a large dataset. The core idea of the large language model is to learn the patterns and structures of natural language through large-scale unsupervised training, to a certain extent, to simulate human language cognition and generation process. The large language model performs well in various application scenarios. It can not only perform simple language tasks such as spelling check and grammar correction, but also handle complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation. Through pre-training on a large-scale dataset, the large language model has strong general modeling ability and generalization ability.

[0061] In actual applications, a plurality of alternative processing links can be preset, and the types of codes used by each alternative processing link are different. Specifically, the alternative processing links can include, but are not limited to, a structured processing language link, a domain-specific language link, and a Python link. The reinforcement learning strategy model can determine a plurality of candidate links that can process the data corresponding to the data table information based on the data table information and the task requirement information from the plurality of preset alternative processing links, so that the type of code used by any one of the plurality of candidate links is also different from the type of code used by the other candidate links. For example, candidate link 1 is a structured processing language link, and the type of code used by candidate link 1 is a structured processing language format code; candidate link 2 is a Python link, and the type of code used by candidate link 2 is a Python format code.

[0062] In the embodiments of the present disclosure, when determining the candidate links for processing the data, a plurality of candidate links are determined at the same time, and then the optimal one is selected from the plurality of candidate links, rather than using a fixed single processing link. Therefore, the problem of limited coverage scenarios and low success rate of data processing caused by a single link can be avoided, and a link with a high success rate and suitable for a scenario can be selected from the plurality of candidate links for execution, which is beneficial to improve the scenario applicability and success rate of data processing.

[0063] Step 206: determining a first processing code set based on the task requirement information and the plurality of candidate links; the first processing code set includes processing codes obtained by code conversion of the task requirement information according to the code types corresponding to the candidate links.

[0064] In actual application, after the data processing platform determines the plurality of candidate links for processing the data corresponding to the data table information, the data processing platform can utilize the link code generation model corresponding to each candidate link to code convert the task demand information in natural language form to obtain the processing code in the code type corresponding to the candidate link. Since the subsequent embodiments in the present specification will explain the content of code converting the task demand information by utilizing the link code generation model in detail, the explanation will not be repeated here.

[0065] In actual application, each candidate link can correspond to a link code generation model, and the link code generation model can code convert the task demand information according to the code type corresponding to the candidate link to obtain the processing code in the corresponding code type. For example, the candidate link 1 is a structured processing language link, and the link code generation model corresponding to the candidate link 1 can convert the task demand information in natural language form into the processing code in structured processing language format; the candidate link 2 is a Python link, and the link code generation model corresponding to the candidate link 2 can convert the task demand information in natural language form into the processing code in Python format.

[0066] In the embodiments of the present specification, after each candidate link utilizes the link code generation model to obtain the corresponding processing code, the processing codes can be summarized to obtain the first processing code set. That is, the number of processing codes included in the first processing code set is consistent with the number of candidate links determined in step 204.

[0067] Step 208: performing semantic consistency verification on the simulation execution result of each processing code in the first processing code set to obtain a second processing code set; the second processing code set includes at least one processing code that meets the preset condition in the semantic consistency verification.

[0068] In actual application, after the data processing platform determines the processing code corresponding to each candidate link, the data processing platform can simulate the execution of the processing codes to obtain the simulation execution result. Specifically, the way of simulating the execution of the processing codes can include: sending the processing codes to a sandbox cluster and simulating the execution of the codes in an isolated and resource-limited sandbox. Alternatively, other code simulation execution methods can also be used, which are not limited here.

[0069] In the embodiments of the present disclosure, if the task requirement information corresponds to a data query task, the simulation execution result obtained after the simulation execution of the processing code corresponding to each candidate link is a data query result; if the task requirement information corresponds to a data calculation task, the simulation execution result obtained after the simulation execution of the processing code corresponding to each candidate link is a data calculation result; if the task requirement information corresponds to a data analysis task, the simulation execution result obtained after the simulation execution of the processing code corresponding to each candidate link is a data analysis result.

[0070] In the embodiments of the present disclosure, the semantic consistency verification is performed on the simulation execution result of each processing code in the first processing code set, and the processing codes that meet the preset condition in the semantic consistency verification can be summarized into the second processing code set. The preset condition can be a condition preset for guaranteeing the result accuracy of the simulation execution result of the processing code. For example, the preset condition can include that the semantic similarity between the simulation execution results of the processing codes is not less than a preset similarity threshold. The preset similarity threshold can be set and adjusted according to actual needs, and no specific limitation is made. If the semantic similarity between the simulation execution results of multiple processing codes is greater than or equal to the preset similarity threshold, it can be considered that the accuracy of the simulation execution result of these processing codes is high. If the semantic similarity between the simulation execution result of a processing code and the simulation execution result of another processing code is less than the preset similarity threshold, it can be considered that the accuracy of the simulation execution result of the processing code is low. For example, the link corresponding to processing code A is a structured processing language link, the obtained simulation execution result is a, the link corresponding to processing code B is a domain-specific language link, the obtained simulation execution result is b, and the link corresponding to processing code C is a Python link, the obtained simulation execution result is c. After the semantic similarity is calculated, the semantic similarity between the simulation execution result a and the simulation execution result b is greater than the preset similarity threshold, the semantic similarity between the simulation execution result c and the simulation execution result a is less than the preset similarity threshold, and the semantic similarity between the simulation execution result c and the simulation execution result b is also less than the preset similarity threshold. It can be considered that the accuracy of the simulation execution result c is low, and the processing code C can be removed, and the processing code A and the processing code B can be added to the second processing code set.

[0071] In the embodiments of the present disclosure, the semantic consistency verification is performed on the simulation execution result of each processing code, and the processing code with low result accuracy that does not meet the semantic consistency verification standard can be screened out, thereby facilitating the improvement of the accuracy of data processing.

[0072] Step 210: determining a target processing code from the second processing code set.

[0073] In actual application, after the data processing platform determines the second processing code set, the task feature recognition model can be used to extract the task feature information in the task demand information, and based on the task feature information, the target processing code with higher execution efficiency can be determined from the second processing code set. Since the subsequent embodiments in the present specification will explain the task feature information and the process of determining the target processing code based on the task feature information in detail, the above will not be repeated here.

[0074] In actual application, after the data processing platform determines the target processing code, the target processing code can be executed to obtain the data processing result of the data processing task. Further, the data processing platform can use the client to show the data processing result to the user. For example, the user uploads the data table A at the data processing platform client, and issues the task demand information: "statistical ratio of male and female students in each class in the data table A". After the data processing platform executes the target processing code, the statistical result of the male and female students in each class in the data table A can be obtained, and the data processing platform client can show the statistical result to the user.

[0075] Figure 2 In the method, after the data table information and the task demand information for processing the data corresponding to the data table information are obtained, a plurality of candidate links for processing the data corresponding to the data table information can be determined; wherein the type of code used by any one of the plurality of candidate links is different from the type of code used by other candidate links; based on the task demand information and the plurality of candidate links, a first processing code set is determined; wherein the first processing code set includes processing codes obtained by code conversion of the task demand information according to the code types corresponding to each candidate link; then, the simulation execution results of each processing code in the first processing code set are subjected to semantic consistency verification to obtain a second processing code set, and finally, the target processing code is determined from the second processing code set. Thus, the data processing task can be executed based on the determined target processing code to obtain an accurate data processing result. Since the data processing platform determines a plurality of candidate links at the same time and then selects the optimal one from the plurality of candidate links, rather than using a fixed single processing link, the problem of limited coverage scenario and low success rate of data processing caused by the single link can be avoided, and the link with high success rate and suitable for the scenario can be selected from the plurality of candidate links for execution, which is beneficial to improve the scenario applicability and success rate of data processing. In addition, by subjecting the simulation execution results of each processing code to semantic consistency verification, the inaccurate processing codes that do not meet the semantic consistency verification standard can be filtered out, which is beneficial to improve the accuracy of data processing.

[0076] Based on Figure 2The method in the method, and the method some specific embodiments of the present application also provides the task requirement information of the data table information and the data corresponding to the data table information, which is described below.

[0077] Optionally, Figure 2 The method in the method, and the method some specific embodiments of the present application also provides the task requirement information of the data table information and the data corresponding to the data table information, which is described below.

[0078] Obtaining a data processing task; the data processing task includes the data table information and the task requirement information of the data corresponding to the data table information; the data table information includes the reference data table provided by the user or the query path information of the reference data table; the task requirement information is the information in the natural language form provided by the user.

[0079] In actual application, the user can issue a data processing task at the client of the data processing platform. The data processing platform client can transmit the data processing task to the data processing platform server, and the data processing platform server performs subsequent data processing work based on the data processing task.

[0080] Among them, the data processing task can include the data table information uploaded by the user and the task requirement information of the data corresponding to the data table information. The data table information can include but is not limited to: reference data table, query path information of the reference data table. Since the foregoing has been described in detail about the reference data table, the query path information of the reference data table and the task requirement information, it will not be repeated here.

[0081] Optionally, Figure 2 The method in the method, and the method some specific embodiments of the present application also provides the task requirement information of the data table information and the data corresponding to the data table information, which is described below.

[0082] From a plurality of alternative processing links, a plurality of candidate links for processing the data are determined; the alternative processing links include structured processing language links, domain specific language links and Python links.

[0083] Among them, the structured processing language link can refer to a link for executing data processing based on the structured processing language. Among them, the structured processing language (Structured Query Language, SQL) is a programming language specially designed for managing and operating relational databases. The most core function of SQL is to query data, and SQL can also be used to communicate with relational databases to realize data processing operations such as adding, deleting, modifying and querying.

[0084] The domain-specific language link can refer to a link for performing data processing based on a domain-specific language code. The domain-specific language (DSL) is a computer language specially designed to solve problems in a specific domain. The DSL does not pursue to solve all types of computing problems, but focuses on a specific application domain and provides high efficiency and ease of use in this domain. The DSL provides a "specific language" that enables developers and domain experts (such as business analysts, scientists, and designers) to more effectively communicate requirements, so that domain experts can also write or understand part of the code. In a specific domain, the development efficiency using the DSL is usually higher than that using a general language.

[0085] The Python link can refer to a link for performing data processing based on a Python code. The Python is a high-level, general-purpose, and interpreted programming language. The syntax of Python is simple and clear, close to natural language, easy to learn and read. The Python code is translated into machine code line by line by the interpreter at runtime and executed, without the need to compile into an executable file like C++ or Java, so that the development process becomes very fast and efficient. The application range of Python is relatively wide, and it is commonly used in web development, software development, data analysis, data visualization, data calculation, and other fields.

[0086] In actual applications, the structured processing language link, the domain-specific language link, and the Python link can be set as alternative processing links, and some other language links can also be set as alternative processing links according to actual business requirements, and no specific limitation is made.

[0087] Optionally, the determining of the multiple candidate links for processing the data from the multiple alternative processing links can specifically include:

[0088] obtaining system current load information corresponding to each of the alternative processing links;

[0089] obtaining historical success rates corresponding to each of the alternative processing links;

[0090] inputting the system current load information corresponding to each of the alternative processing links, the historical success rates corresponding to each of the alternative processing links, the data table information, and the task requirement information into a reinforcement learning strategy model to obtain candidate link information of the multiple candidate links for processing the data output by the reinforcement learning strategy model.

[0091] In actual application, each candidate processing link can correspond to a processing system, such as: the structured processing language link corresponds to the processing system 1; the domain-specific language link corresponds to the processing system 2; and the Python link corresponds to the processing system 3. Since the execution of the processing code corresponding to the processing link is performed in the processing system corresponding to the processing link, the current load condition of the processing system will also affect the execution efficiency of the processing code, and further affect the data processing efficiency. Therefore, in the process of determining the candidate link based on the reinforcement learning strategy model, the current load condition of the processing system corresponding to each candidate processing link can also be considered, and the candidate processing link with high current load can be excluded, thereby facilitating the improvement of the data processing efficiency.

[0092] In the embodiments of the present specification, each candidate processing link corresponds to a historical success rate, and the historical success rate of each candidate processing link can be determined based on the execution result of the processing code in a preset historical time period. In actual application, after the processing code of a certain link is executed, the execution result information can be stored in the memory library. Subsequently, the execution result information of a certain link in a preset historical time period can be extracted from the memory library, and the historical success rate corresponding to the link can be calculated. Since the historical success rate corresponding to the candidate processing link can reflect the data processing success rate of the candidate processing link in the preset historical time period, in the process of determining the candidate link based on the reinforcement learning strategy model, the historical success rate corresponding to each candidate processing link can also be considered, and the candidate processing link with low success rate can be excluded, thereby facilitating the improvement of the success rate of data processing.

[0093] In actual application, the base large model can be fine-tuned and trained by using the first training sample to obtain the reinforcement learning strategy model. The first training sample can include the current load information of each candidate processing link, the historical success rate corresponding to each candidate processing link, the data table information, and the task requirement information. The sample label of the first training sample can include the candidate link information of the determined multiple candidate links for processing the data corresponding to the data table information. Through fine-tuning and training, the reinforcement learning strategy model obtained after training has the ability to determine the candidate link information of the multiple candidate links for processing the data corresponding to the data table information based on the current load information of each candidate processing link, the historical success rate corresponding to each candidate processing link, the data table information, and the task requirement information.

[0094] In actual application, if a version upgrade is made to the link code generation model corresponding to a candidate processing link, the version upgrade information can also be input into the reinforcement learning strategy model as reference information for the reinforcement learning strategy model to determine the candidate link. For example, due to vulnerability B, the historical success rate of candidate processing link A in the last three days is relatively low, and a version upgrade is made to the link code generation model corresponding to candidate processing link A this time to fix vulnerability B. After the reinforcement learning strategy model is provided with the version upgrade information of the link code generation model corresponding to candidate processing link A, the reinforcement learning strategy model can comprehensively consider the historical success rate and the expected influence of the version upgrade on the success rate, so as to avoid excluding candidate processing link A simply because the historical success rate of candidate processing link A in the last three days is relatively low, thereby facilitating to improve the flexibility and accuracy of the reinforcement learning strategy model in decision-making of the candidate link.

[0095] Optionally, the obtaining of the historical success rate corresponding to each of the candidate processing links can specifically include:

[0096] obtaining, from a memory bank, link execution result information of each of the candidate processing links in a first preset time period; the memory bank is configured to store the link execution result information of each of the candidate processing links; the link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information;

[0097] based on the link execution result information of each of the candidate processing links in the first preset time period, statistically obtaining the historical success rate corresponding to each of the candidate processing links.

[0098] In the embodiments of the present disclosure, the data processing platform can be preconfigured with a memory bank for storing the link execution result information of each of the candidate processing links. Specifically, the memory bank can be a database, a data warehouse, or other software or hardware with data storage function, and no specific limitation is made thereto. The link execution result information can include execution success information, or the link execution result information can include execution failure information and failure type information. Specifically, if the processing code of a candidate processing link is executed successfully, the link execution result information includes execution success information; if the processing code of a candidate processing link is executed unsuccessfully, the link execution result information includes execution failure information and failure type information, wherein the failure type information is used to reflect the specific failure reason of this execution failure, such as running error, running timeout, data processing result error, etc.

[0099] In the embodiments of the present disclosure, after obtaining the link execution result information of each candidate processing link in the first preset time period from the memory, the historical success rate corresponding to each candidate processing link can be obtained based on the link execution result information. The first preset time period can be set and adjusted according to requirements, for example, it can be the last 24 hours, or the last 3 days, and no specific limitation is made. For example, if the candidate processing link A has been successfully executed for 7 times and failed for 3 times in the last 3 days, the historical success rate of the candidate processing link A in the last 3 days can be calculated as 7 / (3+7)=0.7=70%.

[0100] Optionally, Figure 2 In the method, the task requirement information is information in a natural language form; and determining a first processing code set based on the task requirement information and the plurality of candidate links can specifically include:

[0101] inputting the task requirement information into each link code generation model corresponding to each candidate link to obtain processing codes of each candidate link output by the link code generation model; the link code generation model is used for code conversion of the task requirement information in the natural language form to obtain processing codes of the code type of the corresponding candidate link;

[0102] obtaining the first processing code set according to the processing codes of each candidate link.

[0103] In actual application, each candidate processing link can correspond to a link code generation model, and the link code generation model can convert the task requirement information in the natural language form into processing codes of the code type of the corresponding candidate processing link. Since the candidate link is selected from the candidate processing link, each candidate link also corresponds to a link code generation model. For example, the link code generation model 1 corresponding to the structured processing language link can convert the task requirement information into processing codes in the structured processing language format; the link code generation model 2 corresponding to the domain-specific language link can convert the task requirement information into processing codes in the domain-specific language format; and the link code generation model 3 corresponding to the Python link can convert the task requirement information into processing codes in the Python format.

[0104] In actual application, the base large model can be fine-tuned and trained by using the second training sample of the target processing link to obtain the link code generation model corresponding to the target processing link. The second training sample can include task demand information in natural language form, and the sample label of the second training sample can include processing code of the code type of the target processing link converted from the task demand information. Through fine-tuning and training, the link code generation model obtained after training has the ability to convert task demand information in natural language form into processing code of the code type of the target processing link.

[0105] In actual application, after the link code generation model corresponding to each candidate link is trained, the task demand information can be input into each link code generation model corresponding to each candidate link to obtain processing code corresponding to each candidate link output by each link code generation model. The processing codes can be summarized to obtain the first processing code set. That is, the first processing code set includes processing code corresponding to each candidate link obtained by using each link code generation model corresponding to each candidate link.

[0106] Optionally, Figure 2 In the method, before the semantic consistency verification is performed on the simulation execution result of each processing code in the first processing code set to obtain the second processing code set, the method can further include:

[0107] The syntax of each processing code in the first processing code set is verified by using a static code parser to obtain a syntax verification result of the processing code.

[0108] Based on the syntax verification result, the processing code with syntax problems is filtered out from the first processing code set.

[0109] In the embodiments of the present disclosure, the static code parser (also referred to as a static analysis tool) is an automatic tool for checking the correctness, security and quality of a program by analyzing source code without actually running the program. “Static” means that the analysis is performed in the state that the code is static, that is, the code does not need to be compiled and run, which is contrary to “dynamic analysis” which needs to be checked when the program is running (for example, unit testing, performance analysis). The specific category of the static code parser is not limited in the present application.

[0110] In the embodiments of the present specification, the syntax of each processing code in the first processing code set is checked by using a static code parser. Whether each processing code in the first processing code set has a syntax problem can be checked. If the processing code has a syntax problem, it cannot be normally executed. Therefore, based on the syntax checking result of each processing code in the first processing code set, the processing code with a syntax problem can be screened out from the first processing code set. Resource waste caused by execution of the processing code with a syntax problem can be avoided. The adverse effect of the syntax error of the processing code on the link success rate can also be avoided. Therefore, the efficiency and success rate of data processing can be improved, and processing resources can be saved.

[0111] Optionally, Figure 2 In the method, before the semantic consistency checking of the simulation execution result of each processing code in the first processing code set is performed to obtain the second processing code set, the method can further include the following steps.

[0112] Simulating execution of each processing code in the first processing code set in a sandbox environment to obtain simulation execution process data of the processing code.

[0113] Screening out, from the first processing code set, a processing code with a preset execution problem based on the simulation execution process data. The preset execution problem includes at least one of a running error, resource consumption required for execution exceeding a preset resource threshold, and execution duration exceeding a preset duration.

[0114] In the embodiments of the present specification, the sandbox environment can refer to a complete, simulated, and isolated system environment constructed based on a sandbox technology, which is specially used for running, testing, or analyzing untrusted programs, codes, or files. The sandbox is a security isolation mechanism. The sandbox creates an independent and limited execution environment for the code by limiting the permissions of the internal running program, so that the code cannot access or affect the external system (such as a real operating system, hardware, network, or data) during the running process.

[0115] In the embodiments of the present specification, the simulation execution of each processing code in the first processing code set in the sandbox environment can check whether each processing code in the first processing code set has a preset execution problem such as a running error, resource consumption required for execution exceeding a preset resource threshold, or execution duration exceeding a preset duration during the simulation execution in the sandbox environment.

[0116] In the embodiments of the present specification, the preset execution problem can include at least one of a running error, resource consumption required for execution exceeding a preset resource threshold, or execution duration exceeding a preset duration. The preset resource threshold and the preset duration can be set and adjusted according to actual needs, which are not limited herein.

[0117] In practical applications, by simulating the execution of each processing code in the first processing code set in a sandbox environment, simulated execution process data for each processing code in the first processing code set can be obtained. Based on the simulated execution process data of each processing code, it can be determined whether there are pre-set execution problems during the simulated execution process. If pre-set execution problems exist, the processing codes with pre-set execution problems can be filtered out to avoid wasting resources by executing processing codes with pre-set execution problems. It can also avoid the adverse impact of processing code execution errors on the success rate of the link, thereby improving the efficiency and success rate of data processing and saving processing resources.

[0118] Optional, Figure 2 The method, before performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set, may further include:

[0119] Each processing code in the first set of processing codes is simulated and executed in a sandbox environment to obtain the simulation execution results of each processing code;

[0120] Correspondingly, Figure 2 The method described above, wherein performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain a second processing code set, may specifically include:

[0121] The simulated execution results of each processing code in the first set of processing codes are projected into the semantic vector space to obtain the semantic vector of the simulated execution result of each processing code;

[0122] Calculate the semantic similarity between the semantic vectors of the simulated execution results of each of the processing codes;

[0123] Based on the semantic similarity, the processing codes that meet the preset conditions are summarized to obtain the second set of processing codes; the preset conditions include that the semantic similarity between the semantic vectors of the simulated execution results of multiple processing codes is not less than a preset similarity threshold.

[0124] In practical applications, simulating the execution of each processing code in the first set of processing codes in a sandbox environment yields the simulated execution results for each code. For example, if the task requirement information corresponds to a data query task, the simulated execution results obtained after simulating the execution of the processing codes for each candidate link correspond to the data query results; if the task requirement information corresponds to a data analysis task, the simulated execution results obtained after simulating the execution of the processing codes for each candidate link correspond to the data analysis results.

[0125] In the embodiments of the present disclosure, the simulation execution results of the processing codes in the first processing code set can be projected into the same semantic vector space, and then the semantic vectors of the simulation execution results of the processing codes are obtained. By calculating the semantic similarity between the semantic vectors of the simulation execution results of the processing codes, the similarity between the simulation execution results of the processing codes can be determined. For example, if the semantic similarity between the semantic vector of the simulation execution result of the processing code 1 and the semantic vector of the simulation execution result of the processing code 2 is less than the preset similarity threshold, it can be indicated that the simulation execution result of the processing code 1 is greatly different from the simulation execution result of the processing code 1.

[0126] In the embodiments of the present disclosure, the preset similarity threshold can be set and adjusted according to actual needs, and no specific limitation is made. If the semantic similarity between the semantic vectors of the simulation execution results of the processing codes is greater than or equal to the preset similarity threshold, the simulation execution results of the processing codes have high similarity, and then it can be determined that the accuracy of the simulation execution results of the processing codes is high. If the semantic similarity between the semantic vector of the simulation execution result of a processing code and the semantic vector of the simulation execution result of another processing code is less than the preset similarity threshold, the accuracy of the simulation execution result of the processing code is low.

[0127] In actual application, if the first processing code set includes the processing code 1 and the processing code 2, and the semantic similarity between the semantic vector of the simulation execution result of the processing code 1 and the semantic vector of the simulation execution result of the processing code 2 is less than the preset similarity threshold, in order to ensure that the data processing result can be obtained subsequently, one of the processing code 1 and the processing code 2 can be randomly selected and added to the second processing code set, or the processing code corresponding to the smaller system load can be added to the second processing code set, or the processing code 1 and the processing code 2 can be added to the second processing code set, and no specific limitation is made. For the case that the semantic similarity between the semantic vectors of the simulation execution results of the three processing codes is less than the preset similarity threshold, the processing methods of the above two kinds can also be referred to.

[0128] In the embodiments of the present disclosure, by performing semantic consistency verification on the simulation execution results of the processing codes, the processing codes with low result accuracy that do not meet the semantic consistency verification standard can be filtered out, thereby facilitating the improvement of the accuracy of data processing.

[0129] Optionally, Figure 2 The method in the above embodiment of the present disclosure, and the method of determining the target processing code from the second processing code set can specifically include:

[0130] If the second processing code set contains one processing code, the one processing code is determined as a target processing code.

[0131] If the second processing code set contains multiple processing codes, a target processing code is determined from the multiple processing codes; the execution efficiency of the target processing code is better than that of other processing codes in the second processing code set.

[0132] In actual application, if the second processing code set contains one processing code, the processing code can be determined as a target processing code, and then the target processing code is executed to obtain a data processing result; if the second processing code set contains multiple processing codes, a processing code with execution efficiency better than that of other processing codes can be determined from the multiple processing codes as a target processing code.

[0133] In the embodiments of the present specification, since the execution efficiency of the selected target processing code is better than that of other processing codes in the second processing code set, the code execution efficiency is high in the data processing process (the execution process of the processing code), and the data processing efficiency can be effectively improved and the data processing time consumption can be shortened.

[0134] Optionally, Figure 2 The method in the above embodiment, wherein the determining the target processing code from the second processing code set can specifically include:

[0135] inputting the task demand information into a task feature recognition model to obtain task feature information output by the task feature recognition model; the task feature recognition model comprises a large language model;

[0136] determining the target processing code from the second processing code set based on the task feature information.

[0137] In the embodiments of the present specification, the task feature recognition model can comprise a large language model. In actual application, the task feature recognition model can be obtained by fine-tuning training a base large model using third training samples. The third training samples can include task demand information, and the sample labels of the third training samples can include task feature information. Through fine-tuning training, the task feature recognition model obtained after training has the ability to extract task feature information from task demand information. Therefore, after inputting the task demand information into the task feature recognition model, the task feature information output by the task feature recognition model can be obtained.

[0138] In the embodiments of the present specification, the task feature information can be used to reflect the task features existing in the task demand information. Specifically, the task feature information can include but is not limited to: the task is a complex task, the task contains a visualization requirement, the task needs to perform complex calculation on data, etc.

[0139] In actual application, after the task feature information is determined, the target processing code with higher execution efficiency than other processing codes can be determined from the second processing code set based on the task feature information. Specifically, the target processing code can be determined according to a preset feature matching rule. The preset feature matching rule can be a processing link with higher execution efficiency under a specific task feature condition preset according to expert experience for different task features. For example, if according to expert experience or actual test result, when the task needs to perform complex calculation of data, the execution efficiency of the processing code of the Python link is higher, the corresponding feature matching rule can be preset: when the task feature information contains “the task needs to perform complex calculation of data”, the processing code of the Python link is preferred as the target processing code. For another example, if according to expert experience or actual test result, when the task contains visualization requirement, the execution efficiency of the processing code of the domain-specific language link is higher, the corresponding feature matching rule can be preset: when the task feature information contains visualization requirement, the processing code of the domain-specific language link is preferred as the target processing code. Alternatively, the target processing code can be determined based on the system load condition of each link, and the processing code of the link with smaller current system load can be preferred as the target processing code, which is not limited here.

[0140] Optionally, Figure 3 In the method, after the target processing code is determined from the second processing code set, the method can further include:

[0141] executing the target processing code to obtain link execution result information of the processing link corresponding to the target processing code; the link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information;

[0142] storing the link execution result information of the processing link corresponding to the target processing code into a memory library; the memory library is used to store link execution result information of each executed processing link.

[0143] In actual application, after the target processing code is determined, the target processing code can be executed in the link system corresponding to the target processing code to obtain link execution result information of the processing link corresponding to the target processing code. For example, if the target processing code is executed successfully, the link execution result information can include execution success information; if the target processing code is executed unsuccessfully, the link execution result information can include execution failure information and failure type information, wherein the failure type information is used to reflect the specific failure reason of this execution failure, such as running error, running timeout, data processing result error, etc.

[0144] In actual application, after the link execution result information of the processing link corresponding to the target processing code is determined, the link execution result information can be stored in the memory bank, so that the related information can be obtained from the memory bank subsequently, and the historical success rate corresponding to each candidate processing link can be calculated.

[0145] In actual application, if the target processing code fails to execute, other processing codes in the second processing code set can be selected as new target processing codes for execution, so as to ensure that the user is provided with a data processing result. Specifically, when other processing codes in the second processing code set are selected as new target processing codes, the processing codes of the link with a smaller current system load can be preferably selected as new target processing codes, so as to reduce the system burden, shorten the data processing time, and improve the data processing efficiency.

[0146] Optionally, Figure 3 The method in the method can further include:

[0147] The link execution process information of the processing link corresponding to the target processing code for which the link execution result information is execution success information is stored in the data warehouse; the link execution process information includes the task demand information input in the execution process of the link code generation model corresponding to the processing link for which the execution is successful and the target processing code output by the link code generation model;

[0148] The task demand information and the target processing code in the link execution process information of the processing link for which the execution is successful and stored in the data warehouse are used as positive samples, and the link code generation model is optimized.

[0149] In the embodiments of the present specification, if the target processing code executes successfully, the link execution process information of the processing link corresponding to the target processing code in the data processing process can be stored in the data warehouse. The link execution process information includes the task demand information input in the execution process of the link code generation model corresponding to the processing link for which the execution is successful and the target processing code output by the link code generation model.

[0150] In the embodiments of the present specification, since the data warehouse stores the link execution process information of the processing link corresponding to the target processing code for which the execution is successful, the task demand information input in the execution process of the link code generation model corresponding to the processing link for which the execution is successful and the target processing code output by the link code generation model can form a positive sample pair. The positive sample pair stored in the data warehouse can be used to train and optimize the link code generation model. Specifically, the link code generation model can be trained and optimized at fixed intervals, or the link code generation model can be trained and optimized after the positive samples stored in the data warehouse reach a preset number, which is not limited in particular.

[0151] In the embodiments of the present specification, by using the task requirement information in the process information of the link corresponding to the successfully executed processing link and the target processing code as positive samples, the link code generation model is optimized, which can improve the performance of the link code generation model and enhance the accuracy of the link code generation model in generating processing code based on task requirement information.

[0152] In actual application, considering the update of the data system, it is necessary to adjust the generation logic of the code generated by the link code generation model to adapt to the update of the data system. A link adaptation module can be set, which can monitor the update of the data system and match the corresponding adaptation rule patch or adjust the code generation logic of the link code generation model according to the update of the data system, so as to quickly adapt to environmental changes without human intervention, and ensure the accuracy and stability of data processing services. The update of the data system can include but is not limited to: table structure change, field update, processing syntax upgrade, data query syntax upgrade, etc.

[0153] Figure 4 The overall flowchart of the data processing method provided by the embodiments of the present specification is shown in FIG. 1. Figure 2 As shown in FIG. 1, the process of the data processing method can specifically include the following steps:

[0154] Step 302: Obtain a data processing task; the data processing task includes data table information and task requirement information for processing data corresponding to the data table information. The data table information includes reference data table provided by a user or query path information of the reference data table; the task requirement information is information in natural language form provided by the user.

[0155] Step 304: Obtain system current load information corresponding to each candidate processing link and historical success rate corresponding to each candidate processing link.

[0156] Step 306: Input the system current load information corresponding to each candidate processing link, the historical success rate corresponding to each candidate processing link, the data table information, and the task requirement information into a reinforcement learning strategy model to obtain candidate link information of multiple candidate links for processing data output by the reinforcement learning strategy model.

[0157] Step 308: Input the task requirement information into each link code generation model corresponding to each candidate link to obtain processing code corresponding to each candidate link output by each link code generation model. The link code generation model is used for code conversion of the task requirement information in natural language form to obtain processing code of the code type of the corresponding candidate link.

[0158] Step 310: obtaining a first processing code set according to the processing codes corresponding to each of the candidate links.

[0159] Step 312: performing syntax checking on each processing code in the first processing code set by using a static code parser to obtain a syntax checking result of each processing code, and based on the syntax checking result, filtering out the processing code with syntax problems from the first processing code set.

[0160] Step 314: simulating execution of each processing code in the first processing code set in a sandbox environment to obtain simulation execution process data of each processing code, and based on the simulation execution process data, filtering out the processing code with a preset execution problem from the first processing code set. The preset execution problem includes at least one of a running error, resource consumption exceeding a preset resource threshold, and execution time exceeding a preset time length.

[0161] Step 316: simulating execution of each processing code in the first processing code set in a sandbox environment to obtain simulation execution results of each processing code, and projecting the simulation execution results of each processing code in the first processing code set to a semantic vector space to obtain semantic vectors of the simulation execution results of each processing code.

[0162] Step 318: calculating semantic similarity between the semantic vectors of the simulation execution results of each processing code, and based on the semantic similarity, collecting the processing codes meeting a preset condition to obtain a second processing code set. The preset condition includes that the semantic similarity between the semantic vectors of the simulation execution results of multiple processing codes is not less than a preset similarity threshold.

[0163] Step 320: inputting the task demand information into the task feature recognition model to obtain task feature information output by the task feature recognition model, and based on the task feature information, determining a target processing code from the second processing code set.

[0164] Step 322: executing the target processing code to obtain link execution result information of a processing link corresponding to the target processing code. The link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information.

[0165] Step 324: storing the link execution result information of the processing link corresponding to the target processing code in a memory library. The memory library is used to store link execution result information of each executed processing link.

[0166] Figure 4 A structural schematic diagram of a data processing apparatus corresponding to Figure 4 provided by an embodiment of the present specification is shown in Figure 5As shown, the apparatus can include:

[0167] The acquisition module 402 is configured to acquire data table information and task requirement information of processing data corresponding to the data table information.

[0168] The first determination module 404 is configured to determine a plurality of candidate links for processing the data; any candidate link in the plurality of candidate links adopts a code type different from a code type adopted by other candidate links.

[0169] The second determination module 406 is configured to determine a first processing code set based on the task requirement information and the plurality of candidate links; the first processing code set includes a processing code obtained by code conversion of the task requirement information according to a code type corresponding to each candidate link.

[0170] The verification module 408 is configured to perform semantic consistency verification on a simulation execution result of each processing code in the first processing code set to obtain a second processing code set; the second processing code set includes at least one processing code satisfying a preset condition in the semantic consistency verification.

[0171] The third determination module 410 is configured to determine a target processing code from the second processing code set.

[0172] Based on the apparatus, Figure 2 The embodiments of the present specification also provide some specific implementation solutions of the apparatus, which are described below.

[0173] Optionally, the acquisition module 402 can specifically include:

[0174] The acquisition sub-module is configured to acquire a data processing task; the data processing task includes the data table information and task requirement information of processing data corresponding to the data table information; the data table information includes a reference data table provided by a user or query path information of the reference data table; and the task requirement information is information in a natural language form provided by the user.

[0175] Optionally, the first determination module 404 can specifically include:

[0176] The candidate link determination sub-module is configured to determine a plurality of candidate links for processing the data from a plurality of alternative processing links; the alternative processing links include a structured processing language link, a domain-specific language link, and a Python link.

[0177] Optionally, the candidate link determination sub-module can specifically include:

[0178] The first obtaining unit is configured to obtain system current load information corresponding to each of the candidate processing links;

[0179] The second obtaining unit is configured to obtain historical success rates corresponding to each of the candidate processing links.

[0180] The candidate link information determining unit is configured to input the system current load information corresponding to each of the candidate processing links, the historical success rates corresponding to each of the candidate processing links, the data table information, and the task requirement information into a reinforcement learning strategy model, to obtain candidate link information of a plurality of candidate links for processing the data output by the reinforcement learning strategy model.

[0181] Optionally, the second obtaining unit can specifically include:

[0182] The link execution result information obtaining subunit is configured to obtain link execution result information of each of the candidate processing links in a first preset time period from a memory bank, the memory bank being configured to store the link execution result information of each of the candidate processing links, the link execution result information including execution success information, or the link execution result information including execution failure information and failure type information.

[0183] The information statistical subunit is configured to statistically obtain the historical success rates corresponding to each of the candidate processing links based on the link execution result information of each of the candidate processing links in the first preset time period.

[0184] Optionally, the task requirement information is information in a natural language form. The second determining module 406 can specifically include:

[0185] The processing code determining sub-module is configured to input the task requirement information into each link code generation model corresponding to each of the candidate links, to obtain processing codes of each of the candidate links output by the each link code generation model, the link code generation model being configured to perform code conversion on the task requirement information in the natural language form, to obtain processing codes of a code type of a corresponding candidate link.

[0186] The first processing code set determining sub-module is configured to obtain the first processing code set according to the processing codes of each of the candidate links.

[0187] Optionally, the apparatus can further include:

[0188] The syntax verification result determining module is configured to perform syntax verification on each processing code in the first processing code set by using a static code parser, to obtain syntax verification results of the each processing code.

[0189] The first screening module is configured to screen out, from the first set of processing codes, a processing code having a syntax problem based on the syntax checking result.

[0190] Optionally, the apparatus can further include:

[0191] The simulation execution process data determining module is configured to simulate execution of each processing code in the first set of processing codes in a sandbox environment to obtain simulation execution process data of the each processing code.

[0192] The second screening module is configured to screen out, from the first set of processing codes, a processing code having a preset execution problem based on the simulation execution process data; the preset execution problem includes at least one of a running error, resource consumption exceeding a preset resource threshold, and execution duration exceeding a preset duration.

[0193] Optionally, the apparatus can further include:

[0194] The simulation execution result determining module is configured to simulate execution of each processing code in the first set of processing codes in a sandbox environment to obtain simulation execution result of the each processing code.

[0195] Correspondingly, the checking module 408 can specifically include:

[0196] The semantic vector determining submodule is configured to project the simulation execution result of each processing code in the first set of processing codes to a semantic vector space to obtain a semantic vector of the simulation execution result of the each processing code.

[0197] The calculating submodule is configured to calculate semantic similarity between the semantic vectors of the simulation execution result of the each processing code.

[0198] The aggregating submodule is configured to aggregate, based on the semantic similarity, the processing codes satisfying the preset condition to obtain the second set of processing codes; the preset condition includes that the semantic similarity between the semantic vectors of the simulation execution result of a plurality of the processing codes is not less than a preset similarity threshold.

[0199] Optionally, the third determining module 410 can specifically include:

[0200] The target processing code determining submodule is configured to, if the second set of processing codes includes one processing code, determine the one processing code as a target processing code; and if the second set of processing codes includes a plurality of processing codes, determine a target processing code from the plurality of processing codes; the execution efficiency of the target processing code is higher than that of other processing codes in the second set of processing codes.

[0201] Optionally, the third determining module 410 can further include the following modules.

[0202] The task feature information determining sub-module is configured to input the task requirement information into a task feature recognition model to obtain task feature information output by the task feature recognition model; and the task feature recognition model includes a large language model.

[0203] The target processing code determining sub-module is further configured to determine a target processing code from the second processing code set based on the task feature information.

[0204] Optionally, the apparatus can further include the following modules.

[0205] The link execution result information determining module is configured to execute the target processing code to obtain link execution result information of a processing link corresponding to the target processing code; and the link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information.

[0206] The first storage module is configured to store the link execution result information of the processing link corresponding to the target processing code into a memory bank; and the memory bank is configured to store link execution result information of each executed processing link.

[0207] Optionally, the apparatus can further include the following modules.

[0208] The second storage module is configured to store link execution process information of the processing link corresponding to the link execution result information of execution success into a data warehouse; and the link execution process information includes task requirement information input in an execution process of a link code generation model corresponding to the processing link of execution success and the target processing code output by the link code generation model.

[0209] The optimization module is configured to optimize the link code generation model by taking the task requirement information and the target processing code in the link execution process information of the processing link of execution success stored in the data warehouse as positive samples.

[0210] It can be understood that each module described above refers to a computer program or program segment, and is configured to execute one or more specific functions. In addition, the division of each module described above does not mean that the actual program code must also be separated.

[0211] Based on the same idea, the embodiments of the present specification also provide a device corresponding to the above method.

[0212] Figure 5 A structure diagram of a data processing device corresponding to Figure 5 provided by the embodiments of the present specification is shown in FIG. 1. As shown in FIG. 1, the data processing device includes the following modules.Figure 1 As shown in FIG. 5, the device 500 can include:

[0213] at least one processor 510; and

[0214] a memory 530 connected with the at least one processor in communication;

[0215] wherein the memory 530 stores instructions 520 executable by the at least one processor 510, the instructions being executed by the at least one processor 510 to enable the at least one processor 510 to:

[0216] obtain data table information and task requirement information for processing data corresponding to the data table information;

[0217] determine a plurality of candidate links for processing the data, any candidate link in the plurality of candidate links adopting a code type different from a code type adopted by other candidate links;

[0218] determine a first set of processing codes based on the task requirement information and the plurality of candidate links, the first set of processing codes including a processing code obtained by code conversion of the task requirement information according to a code type corresponding to each candidate link;

[0219] perform semantic consistency verification on simulation execution results of each processing code in the first set of processing codes to obtain a second set of processing codes, the second set of processing codes including at least one processing code that satisfies a preset condition in the semantic consistency verification;

[0220] determine a target processing code from the second set of processing codes.

[0221] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device shown in the embodiments, since it is basically similar to the corresponding method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments. Figure 1

[0222] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible. ​

[0223] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code before compilation is also written in a specific programming language, which is called a hardware description language (HDL), and there are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0224] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0225] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0226] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0227] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0228] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0229] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0230] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. ​ one or more flow or blocks ​ means for functionally implementing the steps listed in the flowchart block or blocks.

[0231] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0232] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, etc. in the form of computer-readable media. The memory is an example of computer-readable media.

[0233] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0234] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0235] The specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0236] The above only describes the embodiments of the specification and is not intended to limit the specification. For those skilled in the art, the specification can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the specification shall be included in the scope of claims of the specification.

Claims

1. A data processing method, comprising: Acquire data table information, and task requirements for processing the data corresponding to the data table information; Multiple candidate links for processing the data are identified; the type of code used by any one of the multiple candidate links is different from the type of code used by the other candidate links. Based on the task requirement information and the multiple candidate links, a first set of processing codes is determined; the first set of processing codes includes processing codes obtained by converting the task requirement information into codes according to the code type corresponding to each candidate link. The semantic consistency of the simulated execution results of each processing code in the first processing code set is verified to obtain the second processing code set; The second set of processing codes includes at least one processing code that meets the preset conditions in the semantic consistency check; The target processing code is determined from the second set of processing codes.

2. The method as described in claim 1, wherein the step of obtaining data table information and the task requirement information for processing the data corresponding to the data table information specifically includes: Obtain data processing tasks; the data processing tasks include the data table information and task requirement information for processing the data corresponding to the data table information; the data table information includes a reference data table provided by the user or query path information of the reference data table; the task requirement information is information in natural language form provided by the user.

3. The method of claim 1, wherein determining multiple candidate links for processing the data specifically includes: Multiple candidate processing links for processing the data are determined from a number of alternative processing links; the alternative processing links include structured processing language links, domain-specific language links, and Python links.

4. The method of claim 3, wherein determining multiple candidate links for processing the data from multiple alternative processing links specifically includes: Obtain the current system load information corresponding to each of the candidate processing links; Obtain the historical success rate for each of the aforementioned alternative processing links; The current system load information, historical success rate, data table information, and task requirement information corresponding to each of the candidate processing links are input into the reinforcement learning policy model to obtain candidate link information for multiple candidate links used to process the data, which is output by the reinforcement learning policy model.

5. The method as described in claim 4, wherein obtaining the historical success rate corresponding to each of the candidate processing links specifically includes: Retrieve the execution result information of each of the candidate processing links within the first preset time period from the memory bank; The memory is used to store the execution result information of each of the alternative processing links; The link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information; Based on the link execution result information of each of the candidate processing links within the first preset time period, the historical success rate corresponding to each candidate processing link is statistically obtained.

6. The method as described in claim 1, wherein the task requirement information is information in natural language form; The step of determining the first set of processing codes based on the task requirement information and the multiple candidate links specifically includes: The task requirement information is input into the respective link code generation model corresponding to each of the candidate links to obtain the processing code corresponding to each of the candidate links output by the respective link code generation model; the link code generation model is used to convert the task requirement information in natural language form into code to obtain the processing code of the corresponding candidate link code type; The first set of processing codes is obtained based on the processing codes corresponding to each of the candidate links.

7. The method as described in claim 1, further comprising, before performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set: A static code parser is used to perform syntax verification on each processing code in the first processing code set to obtain the syntax verification result of each processing code. Based on the syntax verification results, processing codes with syntax problems are filtered out from the first set of processing codes.

8. The method as described in claim 1, further comprising, before performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set: Each processing code in the first set of processing codes is simulated and executed in a sandbox environment to obtain the simulated execution process data of each processing code; Based on the simulated execution process data, processing codes with preset execution problems are filtered out from the first set of processing codes; The preset execution problems include at least one of the following: runtime error, execution resource consumption exceeding a preset resource threshold, and execution time exceeding a preset time.

9. The method as described in claim 1, further comprising, before performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set: Each processing code in the first set of processing codes is simulated and executed in a sandbox environment to obtain the simulation execution results of each processing code; The step of performing semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set specifically includes: The simulated execution results of each processing code in the first set of processing codes are projected into the semantic vector space to obtain the semantic vector of the simulated execution result of each processing code; Calculate the semantic similarity between the semantic vectors of the simulated execution results of each of the processing codes; Based on the semantic similarity, the processing codes that meet the preset conditions are summarized to obtain the second set of processing codes; the preset conditions include that the semantic similarity between the semantic vectors of the simulated execution results of multiple processing codes is not less than a preset similarity threshold.

10. The method of claim 1, wherein determining the target processing code from the second set of processing codes specifically includes: If the second set of processing codes contains a processing code, then the processing code is identified as the target processing code; If the second set of processing codes contains multiple processing codes, then the target processing code is determined from the multiple processing codes; The execution efficiency of the target processing code is better than that of other processing codes in the second set of processing codes.

11. The method of claim 1, wherein determining the target processing code from the second set of processing codes specifically includes: The task requirement information is input into the task feature recognition model to obtain the task feature information output by the task feature recognition model; The task feature recognition model includes a large language model; Based on the task feature information, the target processing code is determined from the second processing code set.

12. The method of claim 1, wherein after determining the target processing code from the second set of processing codes, the method further comprises: Execute the target processing code to obtain the link execution result information of the processing link corresponding to the target processing code; The link execution result information includes execution success information, or the link execution result information includes execution failure information and failure type information; Store the link execution result information of the processing link corresponding to the target processing code into the memory bank; The memory is used to store the execution result information of each executed processing link.

13. The method of claim 12, further comprising: The link execution process information corresponding to the processing link whose execution result information is successful is stored in the data warehouse; The link execution process information includes the task requirement information input by the link code generation model during the execution process of the successfully executed processing link, and the target processing code output by the link code generation model. The task requirement information and the target processing code stored in the data warehouse corresponding to the successfully executed processing link are used as positive samples to optimize the link code generation model.

14. A data processing apparatus, comprising: The acquisition module is used to acquire data table information and task requirement information for processing the data corresponding to the data table information. The first determining module is used to determine multiple candidate links for processing the data; the type of code used by any one of the multiple candidate links is different from the type of code used by the other candidate links. The second determining module is used to determine the first set of processing codes based on the task requirement information and the multiple candidate links; The first set of processing codes includes processing codes obtained by converting the task requirement information into codes according to the code type corresponding to each candidate link; The verification module is used to perform semantic consistency verification on the simulated execution results of each processing code in the first processing code set to obtain the second processing code set. The second set of processing codes includes at least one processing code that meets the preset conditions in the semantic consistency check; The third determining module is used to determine the target processing code from the second set of processing codes.

15. A data processing apparatus, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire data table information, and task requirements for processing the data corresponding to the data table information; Multiple candidate links for processing the data are identified; the type of code used by any one of the multiple candidate links is different from the type of code used by the other candidate links. Based on the task requirement information and the multiple candidate links, a first set of processing codes is determined; the first set of processing codes includes processing codes obtained by converting the task requirement information into codes according to the code type corresponding to each candidate link. The simulated execution results of each processing code in the first processing code set are subjected to semantic consistency verification to obtain a second processing code set; the second processing code set includes at least one processing code that meets the preset conditions in the semantic consistency verification. The target processing code is determined from the second set of processing codes.

16. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the data processing method according to any one of claims 1 to 13.

Images