Data processing method and system, storage medium and electronic equipment
By automatically generating Pandas data processing code and performing multi-layer detection, the problems of low efficiency and high threshold of traditional data processing are solved, and efficient data processing by non-technical users is achieved, especially in cybercrime investigations, which significantly improves data processing efficiency.
Patent Information
- Application Number
- CN202411361598.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-07
AI Technical Summary
Traditional data processing methods are inefficient when facing large and complex data, and the use threshold of Pandas data processing database is high, making it difficult for non-technical users to process data efficiently, especially in cybercrime investigations.
By automatically generating data processing codes, using a large language model to generate the call code of the Pandas data processing library based on processing requirements and data structures, and performing multi-layer inspection to ensure the correctness and security of the code, realizing automated data processing flow.
It lowers the threshold for data processing, improves data processing efficiency, allows non-technical users to efficiently process data without writing code, and is widely used in data cleaning and analysis, especially in cybercrime investigations, which effectively improves investigation efficiency.
Smart Images
Figure CN119271192A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data processing method, system, storage medium and electronic device. Background Art
[0002] In modern data-driven application environments, data processing is a crucial step, including data cleaning and data analysis. Among them, data analysis involves extracting valuable information from data, while data cleaning is to ensure data quality and improve the accuracy of analysis results. However, with the sharp increase in data volume and the increase in data complexity, traditional data processing methods have become increasingly cumbersome and inefficient. In response to this, the Pandas data processing library in the Python programming language provides a wealth of data manipulation functions that can be widely used in data cleaning, data conversion, and data analysis. Data processing through the Pandas data processing library can become more efficient. However, although the Pandas data processing library itself is powerful, when applying the Pandas data processing library to process data, users need to have certain programming skills and a deep understanding of data operations to achieve efficient data processing. This undoubtedly raises the technical application threshold of the Pandas data processing library, resulting in many users who lack programming skills being unable to call the Pandas data processing library to efficiently process data. For example, for cybercrime, a large amount of electronic data of criminal information is stored on the Internet. At this time, criminal investigators may not be able to use the Pandas data processing library to efficiently process electronic data due to lack of programming skills, which hinders criminal investigations. Summary of the invention
[0003] The purpose of the present application is to provide a data processing method, system, storage medium and electronic device that can automatically generate data processing code that calls the Pandas data processing library to process data.
[0004] In a first aspect, the present application provides a data processing method, the method comprising:
[0005] Obtaining the data structure and processing requirements of the data to be processed; the processing requirements include processing the data to be processed using the Pandas data processing library; the data structure is a DataFrame data structure;
[0006] generating a processing generated word based on the processing requirement and the data structure;
[0007] generating a data processing code based on the processing generated word;
[0008] Performing a first detection operation on the data processing code, if the first detection passes, obtaining the execution result of the data processing code, if the first detection fails, regenerating the processing generated word;
[0009] A second detection operation is performed on the execution result. If the second detection passes, the data processing result is obtained. If the second detection fails, the processed generated word is regenerated.
[0010] In an implementation of the first aspect, generating a processing generated word based on the processing requirement and the data structure includes:
[0011] Acquire the data content of the data to be processed based on the data structure;
[0012] Acquire data attributes of the data to be processed based on the data structure; the data attributes include column name, column type, column length and row length;
[0013] The processing generated words are obtained based on the data content, the data attributes and the processing requirements.
[0014] In an implementation of the first aspect, generating a data processing code based on the processing generated word includes:
[0015] The data processing code is generated based on the processing generated words using a large language model; the data processing code is used to call the data operation function of the Pandas data processing library to process the data to be processed.
[0016] In an implementation of the first aspect, the first detection operation includes performing a dependency library detection on the data processing code; the dependency library detection determines whether a library referenced by the data processing code is a whitelist;
[0017] If it is the whitelist, obtaining the execution result of the data processing code;
[0018] If it is not in the white list, the processed generated word is regenerated.
[0019] In an implementation of the first aspect, the first detection operation includes a security detection; the security detection determines whether there are irrelevant code statements in the data processing code;
[0020] If the irrelevant code statement does not exist, obtaining the execution result of the data processing code;
[0021] If the irrelevant code statement exists, the processing generated word is regenerated.
[0022] In an implementation of the first aspect, the second detection operation includes execution exception detection; the execution exception detection determines whether the data processing code has an execution exception;
[0023] If the execution exception does not exist, taking the execution result as the data processing result;
[0024] If the execution exception exists, the processing generated word is regenerated.
[0025] In an implementation of the first aspect, the second detection operation includes an execution result content detection; the execution result content detection determines whether the execution result content meets the processing requirement;
[0026] If the processing requirement is met, the execution result is used as the data processing result;
[0027] If it does not meet the processing requirements, the processing generated words are regenerated.
[0028] In a second aspect, the present application provides a data processing system, the system comprising:
[0029] A first acquisition module is used to acquire the data structure and processing requirements of the data to be processed; the processing requirements include processing the data to be processed using the Pandas data processing library; the data structure is a DataFrame data structure;
[0030] A second acquisition module, used for generating a processing generated word based on the processing requirement and the data structure;
[0031] A generating module, used for generating a data processing code based on the processing generating word;
[0032] A first detection module, configured to perform a first detection operation on the data processing code, and if the first detection passes, obtain the execution result of the data processing code; if the first detection fails, regenerate the processing generated word;
[0033] The second detection module is used to perform a second detection operation on the execution result, and if the second detection passes, obtain the data processing result; if the second detection fails, regenerate the processed generated word.
[0034] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned data processing method.
[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned data processing method when executed by an electronic device.
[0036] As described above, the data processing method, system, storage medium and electronic device described in this application have the following beneficial effects:
[0037] (1) This application improves data processing efficiency: it can automatically generate data processing code, and call the functions of the Pandas data processing library through the data processing code to process the data, thereby realizing an automated data processing process.
[0038] (2) This application lowers the threshold for data processing: even non-technical users can directly call the Pandas data processing library through this application and perform data processing without writing code.
[0039] (3) The application scenarios of this application are broad: This application uses the generation capability of a large language model to automatically generate code, and then calls the Pandas data processing library to obtain data processing results, effectively avoiding the problem of being unable to correctly process large amounts of data due to the context length limitation of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1A Shown is a schematic diagram of an application scenario of the data processing method described in this application.
[0041] Figure 1B Shown are structural diagrams of the client-cloud interaction scenarios in these implementations.
[0042] Figure 2 Shown is a flowchart of a data processing method described in an embodiment of the present application in one embodiment.
[0043] Figure 3 Shown is a flowchart of a data processing method described in an embodiment of the present application in one embodiment.
[0044] Figure 4 Shown is a flowchart of a data processing method described in an embodiment of the present application in one embodiment.
[0045] Figure 5 Shown is a flowchart of a data processing method described in an embodiment of the present application in one embodiment.
[0046] Figure 6 Shown is a schematic diagram of a data processing method described in an embodiment of the present application in an embodiment.
[0047] Figure 7 Shown is a schematic diagram of the structure of a data processing system in one embodiment of the present application.
[0048] Figure 8 Shown is a schematic structural diagram of an electronic device of the present application in one embodiment.
[0049] Component number description
[0050] 1 Data processing device
[0051] 11 Processor
[0052] 12 Database
[0053] 2-end-cloud interactive system
[0054] 20 Terminal
[0055] 21 Cloud Server
[0056] 4 Data Processing System
[0057] 41 First acquisition module
[0058] 42 Second acquisition module
[0059] 43 Generate Module
[0060] 44 First detection module
[0061] 45 Second detection module
[0062] 51 Processing Units
[0063] 52 Memory
[0064] 521 Memory
[0065] 522 Cache memory
[0066] 523 Storage System
[0067] 524 Programs / Utilities
[0068] 5241 Program Module
[0069] 53 Bus
[0070] 54 I / O interfaces
[0071] 55 Network Adapter
[0072] Steps S1 to S5
[0073] Steps S21 to S23
[0074] Steps S41 to S42
[0075] Steps S51 to S52 DETAILED DESCRIPTION
[0076] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0077] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.
[0078] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0079] The Pandas data processing library is a powerful Python data processing library that provides a wealth of data manipulation functions and can process data efficiently. The data manipulation functions it provides include:
[0080] Data import and export: Supports reading and writing data from multiple formats such as CSV, Excel, SQL database, etc.
[0081] Data cleaning: processing missing values, duplicate data, data format conversion, etc.
[0082] Data operations: including sorting, filtering, grouping, merging, pivot tables, etc.
[0083] Data analysis: statistical analysis, data aggregation, time series analysis, etc. can be performed.
[0084] Data Visualization: Although Pandas itself does not provide charting capabilities, it is compatible with visualization libraries such as Matplotlib, which can easily generate charts and visualize data.
[0085] However, when using the Pandas data processing library to process data, it is necessary to write corresponding code to call the above data operation functions, which requires users to have certain programming skills. Obviously, this leads to a technical threshold for using the Pandas data processing library, which prevents users who lack technical foundation from using the Pandas data processing library to perform processing operations such as clearing and analyzing data.
[0086] Taking the investigation and evidence collection of cybercrime as an example, cybercrime has become an important form of new crime. In particular, new cybercrime represented by telecommunications network fraud has great social harm, and the number of cases has increased year by year, which has become the focus and difficulty of combating crime. As a non-contact crime, cybercrime does not have a direct crime scene and related trace evidence, but is recorded and retained in real time in the form of massive electronic data. At this time, in order to conduct investigation and evidence collection, it is necessary to collect massive electronic data and perform data cleaning and data analysis to extract the correlation between different data, and then clearly outline the factual context of cybercrime. There is no doubt that data processing is a particularly critical link for the investigation and evidence collection of cybercrime. At this time, because many criminal investigation investigators lack the corresponding technical foundation, they cannot call the data operation function of the above-mentioned Pandas data processing library to efficiently process electronic data, which results in criminal investigation personnel needing to spend a lot of time and manpower costs to process data. In order to at least solve the above problems, the present application embodiment provides a data processing method, so that non-technical personnel can call the Pandas data processing library to clean and analyze a large amount of electronic data without writing code, effectively reducing the technical threshold for calling the Pandas data processing library, thereby effectively improving the efficiency of data processing. The data processing method provided in the present application embodiment can be applied to Figure 1A The data processing device shown. Figure 1A As shown, the data processing device 1 includes a processor 11 and a database 12. The data processing method provided in the embodiment of the present application can be applied to the processor 11.
[0087] in, Figure 1A The processor 11 in the embodiment may be a single processor or a processor cluster or a cloud computing center composed of multiple processors, and the specific details are not limited here. Figure 1A Only one processor 11 and one database 12 are shown, but it should be understood that Figure 1A The examples are only used to understand the present solution, and the specific numbers of processors 11 and databases 12 should be flexibly determined based on actual conditions.
[0088] In some other embodiments, the data processing device 1 may not include the database 12, but only include a processor 11 with a storage function. The data processing method provided in the embodiment of the present application can be applied to the processor 11. The processor 11 with a storage function may include a tablet computer, a laptop computer, a PDA, a mobile phone, a personal computer (Personal Computer, referred to as PC), etc., which is not limited here.
[0089] In still other embodiments, the data processing method described in the present application may be applied to end-cloud interaction scenarios. Figure 1B The following is a schematic diagram of the structure of the client-cloud interaction scenario in these implementations. Figure 1B As shown, the terminal-cloud interaction system 2 includes a terminal 20 and a cloud server 21. The terminal 20 and the cloud server 21 can communicate with each other, and the communication method is not limited to wired or wireless.
[0090] Among them, the terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal. The wireless terminal can refer to a device with wireless transceiver function, which can be deployed indoors, outdoors and in industrial workshops. The terminal 20 can be a mobile phone, a tablet computer, a laptop computer, etc., which is not limited here. The cloud server 21 can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server 21 can also be called a server cluster, a management platform, a data processing center, etc., which is not limited in the embodiments of the present application.
[0091] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.
[0092] The following embodiments of the present application provide a data processing method, for example, Figure 1A The processor 11 or Figure 1B It is implemented by the cloud server 21 shown. Figure 2 The flowchart of the data processing method described in the embodiment of the present application is shown as follows: Figure 2 As shown, the data processing method includes steps S1 to S5.
[0093] S1. Obtain the data structure and processing requirements of the data to be processed.
[0094] In some embodiments, the data to be processed may be obtained from a data source, including a CSV, Excel, SQL database, etc.
[0095] In some embodiments, the processing requirement includes processing the data to be processed using a Pandas data processing library. For example, the processing requirement may be performing duplicate data cleaning and sorting on the data to be processed using a Pandas data processing library.
[0096] In some embodiments, the data structure is a DataFrame data structure. The DataFrame data structure is a two-dimensional data structure provided by the Pandas data processing library. It is similar to a table or a spreadsheet, consisting of rows and columns, can store different types of data, and can provide rich data operation functions.
[0097] In some embodiments, the data to be processed is Excel data, and the data structure of the Excel data can be obtained by converting the Excel data into a data structure DataFrame in the Pandas data processing library through the built-in method read_excel() provided by the Pandas data processing library.
[0098] S2. Generate processing generation words based on the processing requirements and the data structure.
[0099] In some embodiments, it is necessary to generate processing generated words based on the user's actual processing requirements and the DataFrame data structure. Figure 3 The flowchart of the data processing method described in the embodiment of the present application is shown as follows: Figure 3 As shown, the data processing method includes steps S21 to S23.
[0100] S21. Acquire data content of the data to be processed based on the data structure.
[0101] In some embodiments, after obtaining the data structure DataFrame of the data to be processed, that is, converting the data to be processed into DataFrame structured data, 5 data are extracted from the DataFrame as a description of the data content. The description of the data content involves basic information and features of the data to be processed, which can help understand the type and potential use of the data to be processed.
[0102] In some embodiments, the df.sample(5) method can be used to randomly extract 5 data from the DataFrame as a description of the data content, or the df.head(5) method can be used to extract the first five data from the DataFrame as a description of the data content. The extracted 5 data can be used to describe the basic situation of the data content, such as quantitative data or qualitative data, meaning, data background, etc.
[0103] S22. Acquire data attributes of the data to be processed based on the data structure; the data attributes include column name, column type, column length and row length.
[0104] In some embodiments, column names, column types, column lengths, and row lengths are extracted from a DataFrame data structure as descriptions of data attributes. The description of the data attributes refers to the detailed information and feature explanations of each column (or feature) in the data set. For example, there is a DataFrame data structure containing employee information, and the extracted column names are EmployeeID, Name, Age, Department, Salary, HireDate, and the corresponding column types are int64, object, int64, object, int64, object, and the corresponding column lengths are 5, 5, 5, 5, 5, and the corresponding row lengths are 5.
[0105] S23. Acquire the processing generated word based on the data content, the data attribute and the processing requirement.
[0106] In some embodiments, the processing generated words should include processing requirements and extracted data content and data attributes. For example, the processing requirements may be to use the Pandas data processing library to perform duplicate data cleaning and sorting on the data to be processed; the data content is described as the data to be processed is employee information, including employee ID, name, age, department, salary and employment date; the data attribute is described as the column names are EmployeeID, Name, Age, Department, Salary, HireDate, the corresponding column types are int64, object, int64, object, int64, object, the column lengths are 5, 5, 5, 5, 5, and the row lengths are 5; then the processing generated words may be: "Please generate code for using the Pandas data processing library to perform duplicate data cleaning and sorting on employee ID, the corresponding column name is EmployeeID, the column type is int64, and the column length and row length are 5".
[0107] S3. Generate a data processing code based on the processing generated words.
[0108] In some embodiments, the data processing code is generated based on the processing generated words using a large language model. The data processing code is used to call the data operation function of the Pandas data processing library to process the data to be processed.
[0109] The large language model (LLMs) is based on deep learning technology, especially a neural network-based model, for processing and generating natural language text. They are usually trained with a large amount of text data to learn the pattern and structure of the language, so as to perform a variety of tasks, such as translation, summary, code generation, text generation, etc. This application generates corresponding data processing code by inputting the processing generated words into the large language model, so as to call the data operation function of the Pandas data processing library, process the data to be processed, and then meet the processing needs of the user.
[0110] For the embodiments of the present application, an existing mature large language model, such as ChatGPT, can be used as a large language model to receive and process generated words and then output corresponding data processing codes.
[0111] S4. Perform a first detection operation on the data processing code. If the first detection passes, obtain the execution result of the data processing code. If the first detection fails, regenerate the processing generated word.
[0112] In some embodiments, the first detection operation actually performs detection on the data processing code generated by the large language model, and the first detection operation includes library dependency detection and security detection.
[0113] Figure 4 The flowchart of the first detection operation described in the embodiment of the present application is shown as follows: Figure 4 As shown, the first detection operation includes step S41 and step S42.
[0114] S41. Determine whether the library referenced by the data processing code is a whitelist.
[0115] In some embodiments, the dependency library whitelist includes Pandas data processing library and Matplotlib and other libraries added by the user. That is, the user can customize the libraries that the code can reference in advance to ensure that the data processing code reference library generated by the large language model is in the whitelist.
[0116] Specifically, when the library referenced by the data processing code is a whitelist, the execution result of the data processing code is obtained.
[0117] In some embodiments, the automatically generated data processing code is executed using eval() in Python, and the execution result is returned.
[0118] Specifically, when the library referenced by the data processing code is not in the whitelist, the processing generated words are regenerated.
[0119] In some embodiments, if the library referenced by the data processing code is not in the whitelist, the processing generated words can be regenerated based on the processing requirements and the data structure, and an accurate whitelist of referenced libraries can be added to the processing requirements. In other embodiments, the regeneration can also be performed by optimizing the processing generated words. For example, a whitelist of code reference libraries can be directly added to the original processing generated words to ensure that the data processing code reference library generated by the large language model is in the whitelist.
[0120] S42: Determine whether there are any irrelevant code statements in the data processing code.
[0121] In some embodiments, irrelevant code statements include, but are not limited to, code statements for file modification, file deletion, etc. that are irrelevant to processing requirements.
[0122] Specifically, if the irrelevant code statement does not exist in the data processing code, the execution result of the data processing code is obtained.
[0123] In some embodiments, the automatically generated data processing code is executed using eval() in Python, and the execution result is returned.
[0124] Specifically, if the irrelevant code statement exists in the data processing code, the processing generated word is regenerated.
[0125] In some embodiments, if the irrelevant code statements exist in the data processing code, the processing generated words can be regenerated based on the processing requirements and the data structure, and the processing requirements can be added with a requirement to delete the irrelevant code statements. In other embodiments, the regeneration can also be performed by optimizing the processing generated words. For example, the irrelevant code statements can be directly required to be removed from the original processing generated words.
[0126] S5. Perform a second detection operation on the execution result. If the second detection passes, obtain the data processing result. If the second detection fails, regenerate the processed generated word.
[0127] In some embodiments, the second detection operation actually detects the execution result of the data processing code, that is, after the data processing code passes the first detection operation, the automatically generated data processing code is executed using eval() in Python, and the execution result is returned, and the second detection operation is then performed on the execution result.
[0128] In some embodiments, the second detection operation includes performing anomaly detection and performing result content detection. Figure 5The flowchart of the second detection operation described in the embodiment of the present application is shown as follows: Figure 5 As shown, the second detection operation includes step S51 and step S52.
[0129] S51, determining whether there is any execution exception in the data processing code.
[0130] In some embodiments, the execution exception includes error information such as syntax error, logic error, runtime error, etc., which causes the data processing code to fail to execute correctly.
[0131] Specifically, if the data processing code does not have the execution exception, the execution result is used as the data processing result. That is, the data operation function of the Pandas data processing library can be called according to the data processing code to process the data to be processed.
[0132] Specifically, if the data processing code has the execution exception, the processing generated word is regenerated.
[0133] In some embodiments, if the execution exception exists in the data processing code, the processing generated words can be regenerated based on the processing requirements and the data structure. At this time, the error content and code examples of the execution exception can be added to the processing requirements to strengthen the cognition of the large language model so that it can generate correctly executable code based on the regenerated processing generated words.
[0134] S52: Determine whether the content of the execution result meets the processing requirements.
[0135] In some embodiments, the execution result of the data processing code is detected to determine whether the return content corresponding to the processing requirement in the processing generated word is satisfied.
[0136] Specifically, if the content of the execution result meets the processing requirement, the execution result is used as the data processing result and returned to the user.
[0137] Specifically, if the content of the execution result does not meet the processing requirement, the processing generated word is regenerated.
[0138] In some embodiments, if the execution result does not meet the processing requirements, the processing generated words can be regenerated based on the processing requirements and the data structure, so that the large language model can generate code that can meet the processing requirements based on the regenerated processing generated words.
[0139] In some embodiments, Figure 6 The diagram is a schematic diagram of the data processing method described in the embodiment of the present application. Figure 6As shown, the data processing method provided in the embodiment of the present application obtains the data to be processed from the data source, and obtains the DataFrame data structure of the data to be processed based on this, that is, after converting the data to be processed into structured data, generates processing generated words based on the user's processing requirements and data structure, and inputs them into the large language model. The large language model will automatically generate data processing codes that meet the processing requirements based on the processing generated words, that is, the generated data processing code can be used to call the data operation function of the Pandas data processing library to perform corresponding processing operations on the data to be processed. After generating the data processing code, it is necessary to perform a first detection operation on the code, including performing a dependency library detection and a security detection on the data processing code. If the first detection passes, the data processing code is executed to obtain the execution result; if the first detection fails, the processing generated words are regenerated so that the large language model generates the corresponding data processing code based on the new processing generated words. After the first detection passes and the execution result of the data processing code is obtained, the execution result will also be subjected to a second detection operation, including performing anomaly detection and execution result content detection. If the second test passes, the execution result is returned to the user as a data processing result. If the second test fails, the processing generated words are regenerated so that the large language model generates corresponding data processing codes based on the new processing generated words.
[0140] The protection scope of the data processing method described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.
[0141] An embodiment of the present application also provides a data processing system, which can implement the data processing method described in the present application. However, the implementation device of the data processing system described in the present application includes but is not limited to the structure of the data processing system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.
[0142] Figure 7 The structure diagram of the data processing system described in the embodiment of the present application is shown as follows: Figure 7 As shown, the data processing system 4 includes a first acquisition module 41 , a second acquisition module 42 , a generation module 43 , a first detection module 44 and a second detection module 45 .
[0143] The first acquisition module 41 is used to acquire the data structure and processing requirements of the data to be processed. The processing requirements include using the Pandas data processing library to process the data to be processed; and the data structure is a DataFrame data structure.
[0144] The second acquisition module 42 is used to generate a processing generated word based on the processing requirement and the data structure.
[0145] The generating module 43 is used to generate data processing codes based on the processing generating words.
[0146] The first detection module 44 is used to perform a first detection operation on the data processing code. If the first detection passes, the execution result of the data processing code is obtained. If the first detection fails, the processing generated word is regenerated.
[0147] The second detection module 45 is used to perform a second detection operation on the execution result. If the second detection passes, the data processing result is obtained. If the second detection fails, the processed generated word is regenerated.
[0148] Among them, the first acquisition module 41 can obtain the data to be processed from the data source, and convert the data to be processed into the data structure DataFrame in the Pandas data processing library through the built-in method read_excel() provided by the Pandas data processing library, that is, obtain the structured data to be processed. At the same time, the first acquisition module 41 obtains the user's processing requirements, and the processing requirements include processing the data to be processed using the Pandas data processing library. For example, the processing requirements can be to use the Pandas data processing library to perform duplicate data cleaning and sorting on the data to be processed.
[0149] The second acquisition module 42 generates processing generated words according to the actual processing requirements of the user and the DataFrame data structure. The second acquisition module 42 extracts 5 data from the DataFrame as a description of the data content, and extracts the column name, column type, column length and row length from the DataFrame data structure as a description of the data attributes. Finally, the processing generated words are obtained based on the data content, the data attributes and the processing requirements.
[0150] The generating module 43 generates the data processing code based on the processing generated words using the large language model. The data processing code is used to call the data operation function of the Pandas data processing library to process the data to be processed.
[0151] The first detection module 44 performs a first detection operation on the data processing code, including performing a dependency library detection and a security detection on the data processing code. If the first detection passes, the data processing code is executed to obtain an execution result; if the first detection fails, the processing generated words are regenerated so that the large language model generates corresponding data processing codes based on the new processing generated words.
[0152] In some embodiments, the dependent library detection determines whether the library referenced by the data processing code is a whitelist. When the library referenced by the data processing code is a whitelist, the execution result of the data processing code is obtained; when the library referenced by the data processing code is not a whitelist, the processing generated word is regenerated.
[0153] In some embodiments, the security detection determines whether there are irrelevant code statements in the data processing code. If there are no irrelevant code statements in the data processing code, the execution result of the data processing code is obtained; if there are irrelevant code statements in the data processing code, the processing generated words are regenerated.
[0154] Among them, after the first detection passes and the execution result of the data processing code is obtained, the second detection module 45 will also perform a second detection operation on the execution result, including execution anomaly detection and execution result content detection. If the second detection passes, the execution result is returned to the user as the data processing result. If the second detection fails, the processing generated words are regenerated so that the large language model generates the corresponding data processing code based on the new processing generated words.
[0155] In some embodiments, the execution exception detection determines whether the data processing code has an execution exception. If the data processing code does not have the execution exception, the execution result is used as the data processing result; if the data processing code has the execution exception, the processing generated word is regenerated.
[0156] In some embodiments, the execution result content detection determines whether the execution result content meets the processing requirements. If the execution result content meets the processing requirements, the execution result is used as the data processing result and returned to the user; if the execution result content does not meet the processing requirements, the processing generated word is regenerated.
[0157] It should be noted that the structure and principle of the data processing system 4 including the first acquisition module 41, the second acquisition module 42, the generation module 43, the first detection module 44 and the second detection module 45 correspond one to one with the steps in the above-mentioned data processing method, so they will not be described in detail here.
[0158] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.
[0159] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0160] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0161] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.
[0162] The embodiment of the present application further provides an electronic device, wherein the electronic device includes a processor and a memory.
[0163] The memory is used to store computer programs.
[0164] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.
[0165] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned data processing method.
[0166] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0167] like Figure 8As shown, the electronic device of the present application is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).
[0168] Bus 53 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.
[0169] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.
[0170] The memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 523 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 53 via one or more data medium interfaces. The memory 52 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.
[0171] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in the memory 52, such program modules 5241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 5241 generally perform the functions and / or methods of the embodiments described herein.
[0172] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 54. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 55. Figure 8 As shown, the network adapter 55 communicates with other modules of the electronic device via the bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0173] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.
[0174] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.
[0175] The present application provides a data processing method, system, storage medium and electronic device, which can automatically generate data processing code, call the function of the Pandas data processing library through the data processing code to process the data, realize the automated data processing process, and provide data processing efficiency. Secondly, even non-technical users can directly call the Pandas data processing library through this application, and can process data without writing code, which lowers the threshold for data processing. Finally, the present application uses the generation ability of the large language model to automatically generate code, and then calls the Pandas data processing library to obtain the data processing results, which effectively avoids the problem of not being able to correctly process large amounts of data due to the context length limit of the large language model, and can process various data types and quantities.
[0176] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.
[0177] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Obtaining the data structure and processing requirements of the data to be processed; the processing requirements include processing the data to be processed using the Pandas data processing library; the data structure is a DataFrame data structure; generating a processing generated word based on the processing requirement and the data structure; generating a data processing code based on the processing generated word; Performing a first detection operation on the data processing code, if the first detection passes, obtaining the execution result of the data processing code, if the first detection fails, regenerating the processing generated word; A second detection operation is performed on the execution result. If the second detection passes, the data processing result is obtained. If the second detection fails, the processed generated word is regenerated.
2. The data processing method according to claim 1, characterized in that: Generating a processing generated word based on the processing requirement and the data structure includes: Acquire the data content of the data to be processed based on the data structure; Acquire data attributes of the data to be processed based on the data structure; the data attributes include column name, column type, column length and row length; The processing generated words are obtained based on the data content, the data attributes and the processing requirements.
3. The data processing method according to claim 1, characterized in that: Generating data processing code based on the processing generated words includes: The data processing code is generated based on the processing generated words using a large language model; the data processing code is used to call the data operation function of the Pandas data processing library to process the data to be processed.
4. The data processing method according to claim 1, characterized in that: The first detection operation includes performing a dependency library detection on the data processing code; the dependency library detection determines whether the library referenced by the data processing code is a whitelist; If it is the whitelist, obtaining the execution result of the data processing code; If it is not in the white list, the processed generated word is regenerated.
5. The data processing method according to claim 1, characterized in that: The first detection operation includes a safety detection; The security detection determines whether there are irrelevant code statements in the data processing code; If the irrelevant code statement does not exist, obtaining the execution result of the data processing code; If the irrelevant code statement exists, the processing generated word is regenerated.
6. The data processing method according to claim 1, characterized in that: The second detection operation includes performing anomaly detection; The execution exception detection determines whether the data processing code has an execution exception; If the execution exception does not exist, taking the execution result as the data processing result; If the execution exception exists, the processing generated word is regenerated.
7. The data processing method according to claim 1, characterized in that: The second detection operation includes executing result content detection; the executing result content detection determines whether the executing result content meets the processing requirement; If the processing requirement is met, the execution result is used as the data processing result; If it does not meet the processing requirements, the processing generated words are regenerated.
8. A data processing system, characterized in that: The system comprises: A first acquisition module is used to acquire the data structure and processing requirements of the data to be processed; the processing requirements include processing the data to be processed using the Pandas data processing library; the data structure is a DataFrame data structure; A second acquisition module, used for generating a processing generated word based on the processing requirement and the data structure; A generating module, used for generating a data processing code based on the processing generating word; A first detection module, configured to perform a first detection operation on the data processing code, and if the first detection passes, obtain the execution result of the data processing code; if the first detection fails, regenerate the processing generated word; The second detection module is used to perform a second detection operation on the execution result, and if the second detection passes, obtain the data processing result; if the second detection fails, regenerate the processed generated word.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the data processing method according to any one of claims 1 to 7 is implemented.