Big data table creation method and apparatus, electronic device, and storage medium

Abstract

The application provides a big data table creation method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining HTTP header information and request body information after a big data platform performs a simulation table building operation, and a business requirement file of a business party; extracting and processing the business requirement information of the business party according to a file processing module in the big data platform to obtain a data set; obtaining request information according to the target data table or target field in the data set, the HTTP header information and the request body information; and controlling the big data platform to create a big data table according to the request information. The method of the application realizes an automatic table building process and improves the efficiency of establishing a big data table.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for creating large data tables. Background Technology

[0002] A big data platform is a technology stack primarily designed for handling massive data storage and computation, as well as real-time computation of streaming data. It encompasses data acquisition, data storage, data computation, and data application. In modern big data application development, a big data tool platform is typically used as the foundational platform for data management. Common functions include: creating big data tables, querying big data tables, and viewing computational tasks.

[0003] Currently, when creating big data tables, big data tool platforms typically provide a relatively complete big data resume form interface. Users can create big data tables by entering the table name, fields, and other configuration information through the WebUI, or big data developers can create them through pseudocode and SQL-like code.

[0004] However, due to the diverse and ever-changing needs of users, big data platforms often need to delete table metadata and rebuild, resulting in low table creation efficiency for large numbers of tables. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for creating large data tables, in order to solve the problem of low efficiency in creating large-scale tables.

[0006] Firstly, this application provides a method for creating a large data table, including:

[0007] Obtain the HTTP header and request body information after the big data platform performs simulated table creation operations, as well as the business requirements file of the business party. The business requirements file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirements file meets the preset data volume requirements.

[0008] Based on the file processing module in the big data platform, the business requirements information of the business party is extracted and processed to obtain a dataset. The dataset includes data table information, field information, and related information from the business requirements information. Among them, the related information is the information associated with the big data table information and field information.

[0009] The request information is obtained based on the HTTP header information, request body information, and the target data table or target field in the dataset;

[0010] Based on the requested information, control the big data platform to create big data tables.

[0011] In this embodiment of the application, the acquisition of HTTP header information and request body information after the big data platform performs a simulated table creation operation, as well as the business requirements file of the business party, includes:

[0012] Send a simulated table creation request to the big data platform so that the big data platform responds to the simulated table creation request and performs a simulated table creation operation;

[0013] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain HTTP header information and request body information.

[0014] Receive business requirements documents from the business party.

[0015] In this embodiment of the application, the NetworkMonitor tool in DevTools is invoked to monitor the process of simulating table creation operations on the big data platform, obtaining HTTP header information and request body information, including:

[0016] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain the initial HTTP header information and initial request body information.

[0017] Based on the simulated table data corresponding to the simulated table creation request, the initial HTTP header information and initial request body information are validated to obtain the HTTP header information and request body information corresponding to the simulated table data.

[0018] In this embodiment of the application, the business requirements information of the business party is extracted and processed according to the file processing module in the big data platform to obtain a dataset, including:

[0019] Based on the file processing module in the big data platform, data governance is performed on the business needs information of the business party to obtain the initial dataset file. The file processing module includes at least the fs module and the node table module.

[0020] The initial dataset file is extracted and processed using JavaScript's array API to obtain the dataset.

[0021] In this embodiment of the application, request information is obtained based on HTTP header information, request body information, and the target data table or target field in the dataset, including:

[0022] Based on the array map in JavaScript, iterate through the target data table or target field in the dataset;

[0023] Based on the HTTP header information and request body information, the target data table or target field in the dataset is integrated and processed to obtain the request information.

[0024] In this embodiment of the application, controlling the big data platform to create a big data table according to the request information includes:

[0025] The fetch module is invoked to simulate the request information and generate a fetch request;

[0026] The fetch request is transformed into a synchronous request;

[0027] Based on the synchronization request, control the big data platform to create big data tables.

[0028] In this embodiment of the application, the fetch request is transformed to obtain a synchronization request, including:

[0029] Based on the async / await mechanism in JavaScript, the fetch request is wrapped and processed to obtain a Promise object;

[0030] Input the Promise object into the synchronous / asynchronous request converter to transform the Promise object into a synchronous request.

[0031] Secondly, this application provides a large data table creation apparatus, comprising:

[0032] The acquisition module is used to acquire the HTTP header information and request body information after the big data platform performs simulated table creation operations, as well as the business requirement file of the business party. The business requirement file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirement file meets the preset data volume requirements.

[0033] The extraction module is used to extract and process the business requirements information of the business party based on the file processing module in the big data platform to obtain a dataset. The dataset includes data table information, field information, and association information in the business requirements information. Among them, the association information is the information associated with the big data table information and field information.

[0034] The Get module is used to obtain request information based on HTTP header information, request body information, and target data tables or target fields in the dataset;

[0035] The creation module is used to control the big data platform to create big data tables based on the requested information.

[0036] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0037] The memory stores instructions that the computer executes;

[0038] The processor executes computer execution instructions stored in memory to implement the big data table creation method of this application embodiment.

[0039] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the big data table creation method of the embodiments of this application.

[0040] The big data table creation method, apparatus, electronic device, and storage medium provided in this application obtain HTTP header information and request body information after a simulated table creation operation on a big data platform, as well as the business requirement file from the business party. The business requirement file is a local file or a table file, and the data volume representing the business requirements in the business requirement file meets the preset data volume requirements. Based on the file processing module in the big data platform, the business requirement information from the business party is extracted and processed to obtain a dataset. The dataset includes data table information, field information, and association information from the business requirement information, wherein the association information is information associated with the big data table information and field information. Based on the HTTP header information, request body information, and dataset... The system identifies the target data table or target field and obtains the request information. Based on this request information, it controls the methods used by the big data platform to create big data tables. By organizing and reconstructing the business requirements documents, it obtains a dataset that can be processed by the table creation module of the big data platform. Furthermore, it simulates table creation operations to determine the table creation interface and analyzes the interaction of the interface during the operation to obtain the HTTP header and request body information. Then, it integrates and transforms the data information with the HTTP header and request body information of the interface to obtain the request information. This allows for iterative request processing to create the table, reducing manual workload and preventing repetitive work during table reconstruction when requirements change, thus improving work efficiency and accuracy. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A flowchart illustrating the method for creating a large data table as provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating another method for creating a large data table provided in this application embodiment;

[0044] Figure 3 A schematic diagram of the structure of the big data table creation device provided in the embodiments of this application;

[0045] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] In existing technologies, the creation of large data tables typically involves using a comprehensive big data table interface provided by a big data tool platform. Users input the table name, fields, and other configuration information through a WebUI to complete the creation of a large data table. However, when the number of tables and fields is large, this leads to cumbersome development operations. Developers need to spend a significant amount of time and effort on data entry and verification. Furthermore, when changes in requirements lead to changes in the table structure, the big data platform usually needs to delete the table metadata and rebuild it. Alternatively, tables can be created using pseudocode or SQL-like code, requiring developers to manually align the data table, its fields, and configuration information word by word. However, this results in verbose code, and if errors are made during modification, complex debugging tasks are required, leading to a massive workload.

[0049] To address the aforementioned issues, the big data table creation method provided in this application can determine the table creation logic of the big data platform by simulating table creation operations. It also uses a browser debugger to observe the table creation operations, analyze the front-end and back-end interactions, determine the header and request body information of HTTP messages in the network, and organize the data required for table creation by the business side into a dataset that can be read by the program. The fields and tables in the dataset are parsed, and the information from the HTTP messages is integrated to form a looping request to the back-end, completing the table creation process. This eliminates the need for manual input or intervention, reducing workload and improving the efficiency and accuracy of big data table creation.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0051] The execution entity of the big data table creation method provided in this application embodiment can be a server. The server can be a mobile phone, tablet, computer, or other device. This embodiment does not impose any particular restrictions on the implementation method of the execution entity, as long as the execution entity can obtain the HTTP header information and request body information after the big data platform performs a simulated table creation operation, as well as the business requirement file from the business party. The business requirement file is a local file or a table file, and the data volume representing the business requirements in the business requirement file meets the preset data volume requirements. Based on the file processing module in the big data platform, the business requirement information from the business party is extracted and processed to obtain a dataset. The dataset includes data table information, field information, and association information from the business requirement information, where the association information is information associated with the big data table information and field information. Based on the HTTP header information, request body information, and the target data table or target field in the dataset, request information is obtained. Based on the request information, the big data platform is controlled to create the big data table.

[0052] Big data refers to information that is so large in scale that it cannot be captured, managed, processed, and organized into a more proactive business decision-making process using mainstream software tools within a reasonable timeframe.

[0053] Figure 1 This is a flowchart illustrating the method for creating a large data table provided in this embodiment. The execution entity of this method can be a server; however, this embodiment does not impose any particular limitation. Figure 1 As shown, this method may include:

[0054] S101. Obtain the HTTP header information and request body information after the big data platform performs simulated table creation operations, as well as the business requirement file of the business party. The business requirement file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirement file meets the preset data volume requirements.

[0055] In this context, HTTP header information and request body information refer to the request header and request body of an HTTP (Hypertext Transfer Protocol) request message used to transmit data between the client and server. The HTTP header information includes information about the client, the requested data, and other relevant information. The request body information refers to the data sent to the server. In this embodiment, the request body information may refer to the request parameters encapsulating the request message, including table data and field data.

[0056] Obtaining HTTP header and request body information after simulating table creation operations on a big data platform can involve simulating fake data on a browser interface, building a big data table based on this fake data, monitoring the table creation process in the browser debugger, and verifying the HTTP header and request body information of each network request one by one. If the parameters in a certain request body information are consistent with the simulated fake data, it indicates that the backend interface of that HTTP request is the interface that implements the table creation operation. The backend interface is then analyzed to obtain the HTTP header and request body information.

[0057] In this embodiment of the application, the method for obtaining the HTTP header information and request body information after the big data platform performs a simulated table creation operation, as well as the business requirement document from the business party, may include:

[0058] Send a simulated table creation request to the big data platform so that the big data platform responds to the simulated table creation request and performs a simulated table creation operation;

[0059] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain HTTP header information and request body information.

[0060] Receive business requirements documents from the business party.

[0061] Among them, DevTools can refer to Chrome developer tools, including IDE (Integrated Development Environment) functions such as Debugger, Profiler, and Logging Server for the Web platform, which can be used to debug, diagnose, and optimize browser applications.

[0062] NetworkMonitor can refer to a network protocol data analysis tool that can act as a network listener to monitor a browser, facilitating the querying or modification of information related to a specific object within the browser. In this embodiment, NetworkMonitor can be used to monitor the table creation process, recording HTTP header information and request body information within the Network header.

[0063] In this embodiment of the application, the method for monitoring the process of simulating table creation operations on a big data platform by calling NetworkMonitor in DevTools to obtain HTTP header information and request body information may include:

[0064] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain the initial HTTP header information and initial request body information.

[0065] Based on the simulated table data corresponding to the simulated table creation request, the initial HTTP header information and initial request body information are validated to obtain the HTTP header information and request body information corresponding to the simulated table data.

[0066] The verification process can refer to observing each request through NetworkMonitor, finding the parameters of a certain request among all requests and the parameters of the simulated table data filled in on the interface. If the parameters match, it indicates that the interface receiving the request is the table creation interface, which is the interface for front-end and back-end interaction. The table creation interface is then parsed to obtain the HTTP header information and request body information corresponding to the simulated table data.

[0067] The preset data volume requirement refers to the data volume that developers pre-determine based on their ability to collect, use, manage, and process data within an acceptable timeframe. Generally, massive datasets consisting of data at the terabyte (TB) level or higher are referred to as big data. Because the number of fields and the number of tables in real-world business scenarios are constantly growing, expanding, iterating, and changing, it becomes difficult to manually create tables when the data volume required by the business exceeds the preset data volume.

[0068] A business requirements document (BQV) refers to a detailed specification document provided by a business party based on its business needs. It records and defines specific business requirements and is primarily written by the business party. BQV helps developers and designers better understand business needs, ensuring the final product meets those needs. In this embodiment, the BQV may refer to a large amount of business table data provided by the business party, including the number of tables, table names, table fields, field formats, remarks, and the data to be written into the tables. This business table data is unstructured data, representing the most basic form of data. It may include duplicate entries or corrupted files and uses various formats, making it difficult for systems and algorithms to process, index, and use the data in this form. It also does not meet the table creation requirements of big data platforms. Therefore, the BQV needs to be processed, including removing illegal characters such as spaces, carriage returns, and tabs, to transform it into structured data that can be processed by the table creation module of the big data platform.

[0069] S102. Based on the file processing module in the big data platform, extract and process the business requirements information of the business party to obtain a dataset. The dataset includes data table information, field information, and related information from the business requirements information. Among them, the related information is the information associated with the big data table information and field information.

[0070] The file processing module refers to the data governance function module introduced by Node.js for business requirement files. Since Node.js is based on the CommonJS specification for modular management, file management functions can be implemented by introducing specific system modules. Node.js is a JavaScript runtime environment based on the Chrome V8 engine, which can control the reading and writing of system files and listen to network input and output.

[0071] A dataset can refer to a collection of data that can be used by a program and contains large table names, table fields, and other information.

[0072] A big data table can refer to a combination of records, representing a group of things of the same kind. Big data table information can refer to the name of the table, field information can refer to the fields of the table, that is, a characteristic or attribute of a certain thing, and related information can refer to the table format, table attributes, table notes, and other information related to the big data table information and field information.

[0073] Extraction processing refers to extracting the required data from the raw data, thereby transforming potentially erroneous data or automatically generated system errors into data that can be used by the system or program. In this embodiment, extraction processing may refer to using JavaScript's array API to search and format data in an array. The JavaScript array API includes methods such as push(), pop(), shift(), concat(), filter(), and array search APIs.

[0074] Based on the file processing module in the big data platform, the business needs information of the business party is extracted and processed to obtain a dataset. This can be achieved by reconstructing the business table using the fs module and the node table module, and then using the array API of js to perform operations and processes such as adding, deleting, truncating, merging, and transforming the data in the business table to obtain a dataset containing table creation information, database information, and other necessary information.

[0075] In this embodiment of the application, the method for extracting and processing the business needs information of the business party to obtain a dataset based on the file processing module in the big data platform may include:

[0076] Based on the file processing module in the big data platform, data governance is performed on the business needs information of the business party to obtain the initial dataset file. The file processing module includes at least the fs module and the node table module.

[0077] The initial dataset file is extracted and processed using JavaScript's array API to obtain the dataset.

[0078] The `fs` (file system) module in this context refers to the file system module provided by Node.js for accessing the local file system. This module provides file operation-related APIs (Application Programming Interfaces), such as monitoring folders and files, renaming files, reading and writing files, granting file modification permissions, and managing file read / write streams. This enables file and directory processing, including reading, writing, modifying, and deleting files, as well as creating and managing directories. It allows JavaScript running in the Node.js environment to directly read and write local files. The interface can refer to predefined functions that provide applications and developers with the ability to access a set of routines based on certain software or hardware, without needing to access the source code or understand the details of the internal workings. In this embodiment, the `fs` module can be used to process local files in a business requirements file.

[0079] The `node table` module refers to a system module provided by Node.js that supports reading and writing Excel files, including: `js-xlsx`, `node-xlsx`, `excel-parser`, `excel-export`, `node-xlrd`, etc. It is used to export data from a list to an Excel spreadsheet, making the data easy to view and archive locally. It also allows for processing of the exported Excel data. In this embodiment, the `node table` module can be used to process table files in business requirement documents.

[0080] Data governance refers to the process of processing data, including design, monitoring, and execution, to realize the value of data. It transforms unusable data into controllable and easily usable data. This can be achieved through a series of standardized processes such as collection, transmission, and storage, transforming fragmented data into standardized, structured data. Simultaneously, rigorous and comprehensive data monitoring ensures data quality. This standardized data is then further processed and analyzed to generate instructive business monitoring reports and models to support decision-making. In this embodiment, data governance may refer to introducing the `fs` module and the `node` table module to process non-standard business requirement files, reconstructing business tables to obtain the initial dataset file.

[0081] The initial dataset file can refer to the collection of all data elements provided by the business side, including information such as data items and data logic.

[0082] S103. Obtain the request information based on the HTTP header information, request body information, and the target data table or target field in the dataset.

[0083] Among them, the target data table and target field can refer to the data table information and field information that need to be created in the business requirements information.

[0084] Request information can refer to the integration of the header information and request body information of the request obtained from parsing the request in the browser, and the target data table or target field in the dataset obtained after analyzing and organizing the business requirement information, to form request information for the backend.

[0085] In this embodiment of the application, the method for obtaining request information based on HTTP header information, request body information, and target data table or target field in the dataset may include:

[0086] Based on the array map in JavaScript, iterate through the target data table or target field in the dataset;

[0087] Based on the HTTP header information and request body information, the target data table or target field in the dataset is integrated and processed to obtain the request information.

[0088] In JavaScript, the array map can refer to the array map() method, which executes a callback function for each element in the array and returns a new array whose elements are the return values ​​of the callback functions, thus forming an asynchronous loop request.

[0089] S104. Based on the requested information, control the big data platform to create a big data table.

[0090] In this context, the creation of big data tables by a big data platform can refer to the big data platform receiving request information. Since the request information contains datasets and HTTP information, the number of requests is determined based on the number of tables in the dataset, and the big data platform can respond to the request and perform automatic table creation operations.

[0091] Since the request information is asynchronous, meaning each task has one or more callback functions, the callback function is executed before the next task is executed after the previous task is completed. The next task is executed without waiting for the previous task to finish. Therefore, the execution order of the program is inconsistent with the order of the tasks and is asynchronous. Thus, asynchronous requests can be converted into synchronous requests to achieve high-concurrency table creation requests.

[0092] In this embodiment of the application, the method for controlling the big data platform to create a big data table based on the request information may include:

[0093] The fetch module is invoked to simulate the request information and generate a fetch request;

[0094] The fetch request is transformed into a synchronous request;

[0095] Based on the synchronization request, control the big data platform to create big data tables.

[0096] Calling the fetch module to simulate request information can refer to simulating a browser request using the fetch() method of an HTTP request. fetch() starts a request and returns a promise object. When the request is completed, the response will be a response object, which is a resolve object. If the request fails, it will become a reject object.

[0097] Synchronous requests can refer to multiple requests being initiated simultaneously, with a response only returned when all requests have received a response. In other words, the next task waits for the previous task to finish before it is executed, and the execution order of the program is consistent with the order in which the tasks are arranged, thus being synchronous.

[0098] In this embodiment of the application, the method for converting the fetch request to obtain a synchronization request may include:

[0099] Based on the async / await mechanism in JavaScript, the fetch request is wrapped and processed to obtain a Promise object;

[0100] Input the Promise object into the synchronous / asynchronous request converter to transform the Promise object into a synchronous request.

[0101] In this context, `async` can indicate that the function always returns a promise. If there is a `return<non-promise>` statement in the code, JavaScript will automatically wrap the returned value into a resolved promise value.

[0102] The `await` directive can make JavaScript wait until a promise is executed and returns its result before continuing execution, even if the asynchronous request is paused until the request is complete.

[0103] `async` is used to declare a function as an asynchronous function. When this function is executed, it returns a Promise object, representing the result of the current function execution. The `await` keyword can be used in the function to wait for the result of the asynchronous operation until the asynchronous operation is completed before continuing to execute the following code. `await` can only be used in `async` functions.

[0104] Based on JavaScript's async / await mechanism, wrapping a fetch request can mean using async or await to wrap the fetch request, waiting for fetch() to return a success promise to the response. After receiving the response, the response calls the response.json() method on its prototype and waits for it to return a success promise, which is then assigned to data. At this point, data is the actual data returned by the server. This maximizes code optimization, making asynchronous requests more intuitive and easier to understand, and avoiding the callback hell problem.

[0105] A synchronous / asynchronous request converter can refer to the logic set in a program that transforms the action of fulfilling a request for information from an asynchronous request into a synchronous request.

[0106] Transforming a Promise object into a synchronous request can refer to the Promise.all() helper method, such as await Promise.all(), which initiates simultaneous requests and waits for all responses to be received, thus transforming it into a looping synchronous request.

[0107] The big data table creation method provided in this application embodiment can reconstruct the business requirement file provided by the business party to obtain a dataset that can be processed by a browser plugin. This dataset includes big data table creation information, database information, and other necessary information. At the same time, by using a browser debugging and analysis engine to monitor and analyze the simulated table creation operation, the process logic of the interaction between the front-end and back-end of the big data platform can be obtained. The two are then combined to form a loop request for the big data platform, thereby realizing an automatic table creation process. This reduces the workload of manual work and prevents the problem of repeated work required for table reconstruction when requirements change, thus improving work efficiency and accuracy.

[0108] Figure 2 This is a flowchart illustrating another method for creating a large data table provided in this embodiment. The execution entity of this method can be a server; however, this embodiment does not impose any particular limitation. Figure 2 As shown, this method may include:

[0109] The big data R&D engineers determine the big data tool platform and business requirements document for table creation. They then analyze the big data tool platform using a browser debugging and analysis engine to identify the table creation module. Simultaneously, they analyze the business requirements document using a document read / write parsing engine to determine the business table data. A simulated table creation request is sent to the table creation module, and the request information is analyzed to obtain the interface call information. The business table data is then parsed to generate a usable dataset. Recursive data parsing is performed to merge the usable dataset based on the interface call information, resulting in an asynchronous loop request. Finally, an asynchronous / synchronous request converter transforms the asynchronous loop request into a synchronous request, and automated data table creation is performed based on the synchronous request.

[0110] Another method for creating big data tables provided in this application embodiment can abstract customer requirements into a usable dataset, that is, a data source that can be read by a program and is organized according to the customer requirement document, as the basic data for table creation. Then, it combines the interface call information obtained by the front-end browser debugging and analysis engine, generates asynchronous loop requests, and transforms the requests into synchronous requests to achieve automatic table creation. Thus, based on full-stack technology, it achieves complete standardization and automation of requirement table analysis, data table organization, and big data table creation.

[0111] Figure 3 This is a schematic diagram of the structure of the large data table creation device provided in an embodiment of this application. Figure 3 As shown, the large data table creation device 30 includes: an acquisition module 301, an extraction module 302, an acquisition module 303, and a creation module 304. Wherein:

[0112] The acquisition module 301 is used to acquire the HTTP header information and request body information after the big data platform performs simulated table creation operation, as well as the business requirement file of the business party. The business requirement file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirement file meets the preset data volume requirements.

[0113] The extraction module 302 is used to extract and process the business requirement information of the business party according to the file processing module in the big data platform to obtain a dataset. The dataset includes data table information, field information and association information in the business requirement information. Among them, the association information is the information associated with the big data table information and field information.

[0114] Module 303 is used to obtain request information based on HTTP header information, request body information, and target data table or target field in the dataset;

[0115] Create module 304, which is used to control the big data platform to create big data tables based on the request information.

[0116] In this embodiment of the application, the acquisition module 301 can also be specifically used for:

[0117] Send a simulated table creation request to the big data platform so that the big data platform responds to the simulated table creation request and performs a simulated table creation operation;

[0118] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain HTTP header information and request body information.

[0119] Receive business requirements documents from the business party.

[0120] In this embodiment of the application, the acquisition module 301 can also be specifically used for:

[0121] The DevTools tool NetworkMonitor is used to monitor the process of simulating table creation on the big data platform and obtain the initial HTTP header information and initial request body information.

[0122] Based on the simulated table data corresponding to the simulated table creation request, the initial HTTP header information and initial request body information are validated to obtain the HTTP header information and request body information corresponding to the simulated table data.

[0123] In this embodiment of the application, the extraction module 302 can also be specifically used for:

[0124] Based on the file processing module in the big data platform, data governance is performed on the business needs information of the business party to obtain the initial dataset file. The file processing module includes at least the fs module and the node table module.

[0125] The initial dataset file is extracted and processed using JavaScript's array API to obtain the dataset.

[0126] In this embodiment of the application, module 303 can also be specifically used for:

[0127] Based on the array map in JavaScript, iterate through the target data table or target field in the dataset;

[0128] Based on the HTTP header information and request body information, the target data table or target field in the dataset is integrated and processed to obtain the request information.

[0129] In this embodiment of the application, the creation module 304 can also be specifically used for:

[0130] The fetch module is invoked to simulate the request information and generate a fetch request;

[0131] The fetch request is transformed into a synchronous request;

[0132] Based on the synchronization request, control the big data platform to create big data tables.

[0133] In this embodiment of the application, the creation module 304 can also be specifically used for:

[0134] Based on the async / await mechanism in JavaScript, the fetch request is wrapped and processed to obtain a Promise object;

[0135] Input the Promise object into the synchronous / asynchronous request converter to transform the Promise object into a synchronous request.

[0136] The big data table creation device 30 provided in this application embodiment can execute the above method embodiment. Its specific implementation principle and technical effect can be found in the above method embodiment, and will not be repeated here.

[0137] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 includes:

[0138] The electronic device 40 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0139] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-mentioned big data table creation method.

[0140] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0141] In the above Figure 4 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0142] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0143] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0144] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described methods for creating large data tables.

[0145] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0146] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0147] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps in any of the big data table creation methods provided in embodiments of this application.

[0148] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0149] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0150] Since the instructions stored in the storage medium can execute the steps in any of the big data table creation methods provided in the embodiments of this application, the beneficial effects that any of the big data table creation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0151] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0152] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for creating a large data table, characterized in that, The method includes: The HTTP header and request body information after the big data platform performs a simulated table creation operation are obtained, as well as the business requirement file of the business party. The business requirement file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirement file meets the preset data volume requirements. According to the file processing module in the big data platform, the business requirements information of the business party is extracted and processed to obtain a dataset. The dataset includes data table information, field information and association information in the business requirements information, wherein the association information is information associated with the big data table information and the field information. The request information is obtained based on the HTTP header information, the request body information, and the target data table or target field in the dataset; The fetch module is invoked to simulate the request information and generate a fetch request; The fetch request is wrapped using the async / await mechanism in JavaScript to obtain a Promise object; The Promise object is input into the synchronous / asynchronous request converter to transform the Promise object into a synchronous request; Based on the synchronization request, control the big data platform to create a big data table.

2. The method according to claim 1, characterized in that, The acquisition of HTTP header and request body information after the big data platform performs simulated table creation operations, as well as the business requirements file from the business party, includes: Send a simulated table creation request to the big data platform so that the big data platform responds to the simulated table creation request and performs a simulated table creation operation; The DevTools tool's NetworkMonitor is invoked to monitor the process of simulating table creation on the big data platform, thereby obtaining the HTTP header information and the request body information. Receive business requirements documents from the business party.

3. The method according to claim 2, characterized in that, The process of simulating table creation operations on the big data platform by calling NetworkMonitor in DevTools is monitored to obtain the HTTP header information and the request body information, including: The DevTools tool's NetworkMonitor is invoked to monitor the process of simulating table creation on the big data platform, obtaining the initial HTTP header information and the initial request body information. Based on the simulated table data corresponding to the simulated table creation request, the initial HTTP header information and the initial request body information are validated to obtain the HTTP header information and the request body information corresponding to the simulated table data.

4. The method according to claim 1, characterized in that, The process involves extracting and processing the business requirements information from the business party using the file processing module in the big data platform to obtain a dataset, including: Based on the file processing module in the big data platform, data governance is performed on the business needs information of the business party to obtain an initial dataset file. The file processing module includes at least an fs module and a node table module. The initial dataset file is extracted and processed using JavaScript's array API to obtain the dataset.

5. The method according to claim 1, characterized in that, The step of obtaining request information based on the HTTP header information, the request body information, and the target data table or target field in the dataset includes: Based on the array map in JavaScript, iterate through the target data table or target field in the dataset; Based on the HTTP header information and the request body information, the target data table or target field in the dataset is integrated to obtain the request information.

6. A large data table creation device, characterized in that, The device includes: The acquisition module is used to acquire the HTTP header information and request body information after the big data platform performs a simulated table creation operation, as well as the business requirement file of the business party. The business requirement file is a local file or a table file, and the data volume of the data representing the business requirements in the business requirement file meets the preset data volume requirements. The extraction module is used to extract and process the business requirement information of the business party according to the file processing module in the big data platform to obtain a dataset. The dataset includes data table information, field information and association information in the business requirement information, wherein the association information is information associated with the big data table information and the field information. The module is used to obtain request information based on the HTTP header information, the request body information, and the target data table or target field in the dataset; A module is created to call the fetch module, simulate the request information, and generate a fetch request; the fetch request is wrapped using async / await in JavaScript to obtain a Promise object; the Promise object is input into a synchronous / asynchronous request converter to convert the Promise object into a synchronous request; and the big data platform is controlled to create a big data table based on the synchronous request.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the big data table creation method as described in any one of claims 1 to 5.

Images