Data exchange method and device, equipment and storage medium

Abstract

A data exchange method, apparatus and device, and a storage medium, applied to the technical field of data exchange, the method at a service side comprising: screening and merging table information according to a data exchange demand to obtain data table meta information; building built-in parameters according to the data table element information, performing rule-driven parameterized data preprocessing on the data table element information according to the built-in parameters, and generating a data exchange request; acquiring data magnitude information of the receivable data; serially updating the data exchange request according to the data magnitude information; a data exchange request is sent to the data processing center, iterative data exchange is carried out based on the asynchronous response of the data processing center, and the data exchange request is used for enabling an ETL program unit of the data processing center to carry out self-adaptive slicing and iterative data exchange on the demand data according to the data exchange request and enabling the data processing center to carry out asynchronous response. The availability and stability of data and the accuracy and freedom degree of data exchange are improved.

Description

Technical Field

[0001] This application relates to the field of data exchange technology, specifically to a data exchange method, apparatus, device, and storage medium. Background Technology

[0002] Currently, IT systems generate massive amounts of structured data daily, including business data such as transfer orders and real-time payment logs. Faced with this vast amount of business data, data developers need to research and solve problems to analyze and extract more data value from structured storage, thereby supporting strategic plans for enterprise information systems. Therefore, an efficient and stable data exchange solution is needed to provide enterprises with support for various types of business data. Existing data exchange solutions primarily rely on manually developed Extraction Transformation Loading (ETL) program units to complete core data exchange.

[0003] The above-mentioned technical solution addresses, to some extent, the business needs for data exchange, such as data outbound and inbound processing. Data requesters establish effective interaction with the data source by initiating data exchange requests, enabling stable and efficient sharing and exchange of standard data processed by ETL. However, the following issues still exist:

[0004] 1. The rules for data exchange are set by the data source. Therefore, the exchanged data usually cannot be directly applied to business scenarios and still needs to be processed by building ETL program units on the business side that requires data.

[0005] 2. Typically, when constructing data features for a business scenario, the importance of different table fields varies due to different business needs. For businesses with large amounts of data, sharing and exchanging all table field data would place enormous data analysis pressure on the business side.

[0006] 3. Since the machine performance on the business side is not known during data exchange, if a single data block that needs to be fully loaded and analyzed is too large, it can easily cause the business side to crash and go down due to insufficient memory.

[0007] Therefore, how to solve the problem that the exchanged data cannot be directly applied to the business side due to the influence of exchange rules and machine performance in the existing technology, and the huge data analysis pressure on the business side due to full sharing, has become a research direction for those skilled in the art. Summary of the Invention

[0008] At least one embodiment of this application provides a data exchange method, apparatus, device, and storage medium to solve the problem in the prior art that the exchanged data cannot be directly applied to the business side due to the influence of exchange rules and machine performance, and that the business side faces huge data analysis pressure due to full sharing.

[0009] To solve the above-mentioned technical problems, this application is implemented as follows:

[0010] In a first aspect, embodiments of this application provide a data exchange method applied to the business side, including:

[0011] Based on data exchange requirements, the table information corresponding to at least one business scenario is filtered and merged to obtain data table metadata.

[0012] Built-in parameters are constructed based on the data table metadata, and rule-driven parameterized data preprocessing is performed on the data table metadata based on the built-in parameters to generate a data exchange request.

[0013] By probing the machine's storage performance, information on the amount of data that can be received is obtained. This information includes: free memory, data block threshold, and the number of rows in the threshold data table.

[0014] Based on the data volume information, the data exchange request is serialized and updated;

[0015] The data exchange request is sent to the data processing center, and iterative data exchange is performed based on the asynchronous response of the data processing center. The data exchange request is used to enable the extraction, transformation, loading, and ETL program unit of the data processing center to adaptively divide the required data into chunks and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

[0016] Specifically, in the method described above, the step of filtering and merging the table information corresponding to each business scenario according to data exchange requirements to obtain data table metadata includes:

[0017] Based on the data exchange requirements, the table fields of the table information are filtered to obtain a pre-selected table;

[0018] The pre-selected tables under different business scenarios are merged to obtain the data table metadata.

[0019] Specifically, in the method described above, the step of constructing built-in parameters based on the data table metadata, and performing rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request includes:

[0020] Based on the data table metadata, a preprocessed Structured Query Language (SQL) statement is constructed as the built-in parameter, and placeholders and data cleaning and transformation rules are set in the SQL statement.

[0021] The preprocessing fields in the built-in parameters are preprocessed to generate the data exchange request.

[0022] Specifically, in the method described above, obtaining the data volume information of receivable data through machine storage performance probing includes:

[0023] Based on machine storage performance probing, the free memory in the data volume information is obtained;

[0024] Based on the free memory and the current business volume and scale, obtain the data block threshold in the data volume information;

[0025] Based on the data block threshold and the size and number of rows of the data table to be exchanged, obtain the threshold data table row count from the data volume information.

[0026] Specifically, as described above, the iterative data exchange based on the asynchronous response of the data processing center includes:

[0027] Based on the asynchronous response, allocate a storage space for data exchange for each iteration;

[0028] Based on the storage space and the ETL program unit of the data processing center, data exchange is performed, and the data exchange status is polled to the data processing center to obtain the corresponding asynchronous response.

[0029] Specifically, in the method described above, the iterative data exchange based on the asynchronous response of the data processing center further includes:

[0030] After confirming the completion of one iteration of data exchange, obtain the memory occupied by the received data block;

[0031] If the occupied memory is greater than a first preset ratio of the free memory and less than a second preset ratio of the free memory, then the data information and the data volume information of this data exchange are recorded, and the next iteration of data exchange begins.

[0032] If the occupied memory is less than a first preset ratio of the free memory, then the data block threshold in the data volume information is adjusted upward, and the process returns to the step of updating the data exchange request according to the data volume information;

[0033] If the occupied memory is greater than the second preset ratio of the free memory, then the data block threshold in the data volume information is adjusted downward, and the process returns to the step of updating the data exchange request according to the data volume information.

[0034] Secondly, embodiments of this application provide a data exchange method applied to an ETL program unit, comprising:

[0035] The system receives data exchange instructions, built-in parameters, and data volume information of the receivable data transmitted from the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of the receivable data are obtained by the data processing center through parsing the data exchange request.

[0036] Based on the built-in parameters, obtain the business data volume of the data to be exchanged;

[0037] The number of data blocks is determined based on the business data volume level and the data block threshold in the data volume level information;

[0038] Adaptive slicing and iterative data exchange are performed based on the number of data blocks, and data exchange status information is fed back to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information based on the data exchange status information and send an asynchronous response to the service side.

[0039] Specifically, in the method described above, determining the number of data blocks based on the business data volume level and the data block threshold in the data volume level information includes:

[0040] If the volume of the business data is less than or equal to the data block threshold, then the number of data blocks is determined to be one.

[0041] If the volume of the business data is greater than the data block threshold, then the number of data blocks is determined by further processing based on the integer quotient and remainder of the business data volume divided by the data block threshold.

[0042] Specifically, in the method described above, when there are multiple data blocks, the adaptive slicing and iterative data exchange based on the number of data blocks includes:

[0043] Obtain the time series equally divided according to the number of data blocks;

[0044] Multiple data iteration threads corresponding to the number of data blocks are created based on the time series, and the exchange status information corresponding to each data iteration thread is fed back to the data processing center, and data exchange is performed sequentially.

[0045] Specifically, the method described above also includes:

[0046] When the updated data volume information is received from the data processing center, the step of determining the number of data blocks based on the business data volume and the data block threshold in the data volume information is returned. The updated data volume information is determined by the data processing center based on the received updated data exchange request.

[0047] Thirdly, embodiments of this application provide a control device applied to the service side, including:

[0048] The first preprocessing module is used to filter and merge table information corresponding to at least one business scenario according to data exchange requirements to obtain data table metadata.

[0049] The second preprocessing module is used to construct built-in parameters based on the data table metadata, and to perform rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request.

[0050] The third preprocessing module is used to detect the machine's storage performance and obtain information on the amount of data that can be received. The information on the amount of data includes: free memory, data block threshold, and number of rows in the threshold data table.

[0051] The first update module is used to serialize and update the data exchange request according to the data volume information;

[0052] The data exchange module is used to send the data exchange request to the data processing center and perform iterative data exchange based on the asynchronous response of the data processing center. The data exchange request is used to enable the ETL program unit of the data processing center to adaptively divide the required data and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

[0053] Fourthly, embodiments of this application provide a control device applied to an ETL program unit, comprising:

[0054] The first receiving module is used to receive data exchange instructions, built-in parameters, and data volume information of receivable data transmitted by the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of receivable data are obtained by the data processing center through parsing the data exchange request.

[0055] The first processing module is used to obtain the business data volume of the data to be exchanged based on the built-in parameters.

[0056] The second processing module is used to determine the number of data blocks based on the business data volume and the data block threshold in the data volume information;

[0057] The third processing module is used to adaptively divide and iteratively exchange data according to the number of data blocks, and to feed back data exchange status information to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information according to the data exchange status information and send an asynchronous response to the service side.

[0058] Fifthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the data exchange method applied to the service side as described above, or implements the steps of the data exchange method applied to the ETL program unit as described above.

[0059] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data exchange method applied to the service side as described above, or implements the steps of the data exchange method applied to the ETL program unit as described above.

[0060] In a seventh aspect, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data exchange method applied to the service side as described above, or implement the steps of the data exchange method applied to an ETL program unit as described above.

[0061] Compared with existing technologies, the data exchange method, apparatus, device, and storage medium provided in this application can effectively ensure the accuracy of data exchange and reduce the probability of data unavailability by constructing data table metadata on the business side. Through rule-driven parameterized data preprocessing and data exchange requests based on data volume information, the ETL program unit of the data processing center can transmit data that can be directly used by the business side during data exchange, further improving business efficiency. At the same time, it ensures that the data volume during each iteration of data exchange meets the storage performance of the business side, ensuring the smooth progress of data exchange and facilitating the improvement of data exchange efficiency and adaptive adjustment of data exchange. Attached Figure Description

[0062] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0063] Figure 1 This is one of the flowcharts illustrating a data exchange method applied to the business side in this application embodiment;

[0064] Figure 2 This is a second flowchart illustrating the data exchange method applied to the business side in this application embodiment;

[0065] Figure 3 This is the third flowchart illustrating the data exchange method applied to the business side in this application embodiment;

[0066] Figure 4 This is the fourth flowchart illustrating the data exchange method applied to the business side in this application embodiment;

[0067] Figure 5 This is the fifth flowchart illustrating the data exchange method applied to the business side in this application embodiment;

[0068] Figure 6 This is one of the flowcharts illustrating the data exchange method applied to the ETL program unit in the embodiments of this application;

[0069] Figure 7 This is a second flowchart illustrating the data exchange method applied to the ETL program unit in this application embodiment;

[0070] Figure 8 This is a schematic diagram of the control device applied to the service side in the embodiments of this application;

[0071] Figure 9 This is a schematic diagram of the control device applied to the ETL program unit in the embodiments of this application;

[0072] Figure 10 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0073] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0074] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The terms “and / or” in the specification and claims indicate at least one of the connected objects.

[0075] Please refer to Figure 1 This application provides a data exchange method applied to the business side, comprising:

[0076] Step S101: Based on the data exchange requirements, filter and merge the table information corresponding to at least one business scenario to obtain data table metadata.

[0077] Step S102: Construct built-in parameters based on the data table metadata, and perform rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request;

[0078] Step S103: Obtain the data volume information of the receivable data by probing the machine storage performance. The data volume information includes: free memory, data block threshold, and number of threshold data table rows.

[0079] Step S104: Serialize and update the data exchange request according to the data volume information;

[0080] Step S105: Send the data exchange request to the data processing center, and perform iterative data exchange based on the asynchronous response of the data processing center. The data exchange request is used to enable the extraction, transformation, loading, and ETL program unit of the data processing center to adaptively segment the required data and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

[0081] This embodiment illustrates a data exchange method applied to the business side. When exchanging data, the business side first breaks down the business data requiring analysis into at least one business scenario based on user needs. Then, it filters and merges the data based on the table information corresponding to each business scenario to obtain the final data table metadata. This avoids the pressure of redundant or full data processing, improving the flexibility, accuracy, and efficiency of data exchange. Next, built-in parameters are constructed based on the obtained data table metadata, and rule-driven parameterized data preprocessing is performed on the data table metadata according to these parameters. This preprocessing converts the data table metadata into a data exchange request that can be recognized by the ETL program unit and has preset data exchange rules. This allows the ETL program unit to directly convert data extracted from the data source into data recognizable by the business side, avoiding the overhead of setting up an ETL program unit and performing data conversion on the business side, thus saving costs and improving efficiency.

[0082] To further improve data exchange efficiency and avoid situations where the business side suffers from insufficient memory or even server crashes due to excessively large single data blocks, machine storage performance is probed to obtain information on the data volume that the business side can receive. This data volume information includes at least one of the following: free memory, data block threshold, and threshold data table rows. This information is used to determine the maximum data block size and data table rows that the business side can receive in a single transaction, as well as the maximum amount of data that can be received. Subsequently, data exchange requests are updated using serialization based on this data volume information, ensuring that it carries this data volume information.

[0083] Finally, the updated data exchange request is sent to the data processing center. This allows the data processing center and the ETL program unit to extract and exchange data based on the built-in parameters and data volume information in the data exchange request, ensuring the stability, security, and applicability of data transmission. Specifically, the ETL program unit can adaptively segment and iteratively exchange the required business data according to the data volume information in the data exchange request. Each iteration of data exchange will not exceed the data block threshold limited by the data volume information, and subsequent iterations can be adaptively adjusted through iterative data exchange to ensure data exchange efficiency and accuracy. Simultaneously, the data processing center can asynchronously respond to the business side based on the data exchange progress of the ETL program unit, enabling monitoring of each data exchange and facilitating timely strategy adjustments.

[0084] In summary, the data exchange method provided in this application can effectively ensure the accuracy of data exchange and reduce the probability of data unavailability by constructing data table metadata on the business side. Through rule-driven parameterized data preprocessing and data exchange requests based on data volume information, the ETL program unit of the data processing center can transmit data that can be directly used by the business side during data exchange, further improving business efficiency. At the same time, it ensures that the data volume during each iteration of data exchange meets the storage performance of the business side, ensuring the smooth progress of data exchange and facilitating improved data exchange efficiency and adaptive adjustment of data exchange.

[0085] In one specific embodiment, the data table metadata includes, but is not limited to, at least one of the following: table sequence number, Chinese name of the table, table name, data dictionary name, fields, field types, and field descriptions.

[0086] See Figure 2 Specifically, as described above, the step of filtering and merging table information corresponding to each business scenario based on data exchange requirements to obtain data table metadata includes:

[0087] Step S201: Based on the data exchange requirements, filter the table fields of the table information to obtain a pre-selected table;

[0088] Step S202: Merge the pre-selected tables under different business scenarios to obtain the data table metadata.

[0089] In this embodiment, the steps for obtaining data table metadata are illustrated. The filtering process involves labeling table fields corresponding to all business scenarios based on user requirements and a data dictionary. Specifically, a first flag (e.g., 1) can be set for required table fields, and a second flag (e.g., 0) can be set for unnecessary table fields. The resulting tables corresponding to the first flags are the pre-selected tables for user requirements. Since the required tables may be the same in different business scenarios, resulting in potential table conflicts across multiple scenarios, the pre-selected tables with the same table fields in different business scenarios are merged to reduce the required data volume, ultimately obtaining the required data table metadata. In one embodiment, this merging process involves a logical AND operation. Taking the example that both scenario A and scenario B require table U, the operation process can be represented as follows: , where n is the number of tables U with the same fields.

[0090] See Figure 3 Specifically, in the method described above, the step of constructing built-in parameters based on the data table metadata, and performing rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request includes:

[0091] Step S301: Construct a preprocessed SQL statement based on the data table metadata as the built-in parameter, and set the placeholders and data cleaning and transformation rules in the SQL statement;

[0092] Step S302: Perform data preprocessing on the preprocessing fields in the built-in parameters and generate the data exchange request.

[0093] In this embodiment, the steps for generating a data exchange request are illustrated. First, based on the data table metadata, an SQL statement that can preprocess the fields in the data table metadata is constructed as the built-in parameter. Placeholders, basic data cleaning and transformation rules are set in the SQL statement. For easy identification, it can be set as M-SQL, where M represents the corresponding data table metadata.

[0094] In one specific implementation, the JavaScript Object Notation (JSON) template format for this M-SQL can be represented as follows: ,in, : Represents the actual table name, such as the user basic information table "db_USERINF"; : Indicates a description of the table, such as "User Basic Information Table USERINF"; : Indicates the filtering conditions used in the SQL, such as "update_time"; : Indicates a preset SQL statement constructed based on set M, which uses general SQL statements: .in, Within the statement block, placeholders are set according to the JSON template, and data cleaning and transformation rules are configured based on the database's basic functions.

[0095] Furthermore, data preprocessing is performed on the preprocessing fields in the built-in parameters, and the data exchange request is generated, wherein the data preprocessing is based on the preprocessing fields in the built-in parameters. Statement blocks perform global regular expression replacement and type conversion on the corresponding field content in the data table metadata, including preprocessing of character types, null values, newline characters, special characters, etc.

[0096] To facilitate understanding by those skilled in the art, in one embodiment, the preprocessed field is taken as the product description field as an example for explanation. In this case, the built-in parameters... The statement block is represented as: In other words, this example uses the `regexp_replace` and `COALESCE` functions from the database's basic functions. The basic logic for data preprocessing based on this statement block is as follows: First, the `COALESCE` function is used to evaluate the text field of the product description `good_desc`, returning the first non-empty expression; if all expressions are empty, it returns null. Then, the `regexp_replace` function is used to perform regular expression replacement on the text field of the product description `good_desc`, replacing newline characters, indentation characters, and other symbols in the text with empty strings. This process completes data cleaning and format conversion preprocessing during data exchange, ensuring that the business side can directly process the received data, avoiding the resource consumption caused by setting up ETL program units on the business side, and improving business efficiency.

[0097] See Figure 4 Specifically, as described above, the step of probing the machine storage performance to obtain information on the amount of data that can be received includes:

[0098] Step S401: Based on machine storage performance probing, obtain the free memory from the data volume information;

[0099] Step S402: Based on the free memory and the current business volume and scale, obtain the data block threshold in the data volume information;

[0100] Step S403: Based on the data block threshold and the size and number of rows of the data table to be exchanged, obtain the threshold data table row count from the data volume information.

[0101] In this implementation, the steps for obtaining data volume information are illustrated. First, based on machine storage performance probing, the free memory of the business-side machine is obtained. This free memory is either unused memory or partially unused memory on the business side. If it is partially unused memory, some memory will be reserved for the business side, thus helping to avoid problems such as system lag and slow response due to excessive memory usage. In a specific embodiment, the current free memory of the machine will be obtained through memory commands or programming in a programming language. Let's assume the free memory is FM (Free Memory).

[0102] Furthermore, the threshold for the amount of data loaded at one time during a single data exchange is obtained. This threshold is pre-set based on the current workload and system size, and is less than or equal to zero and less than the free memory obtained above.

[0103] Furthermore, when the transmitted data is a data table, to ensure the transmission efficiency of the data table, a threshold range for the number of rows in the data table is determined based on the data block threshold, the size of the data table to be exchanged, and the number of rows. A value is then selected from this range as the threshold number of rows. The minimum value of the threshold range is zero, and the maximum value is obtained by dividing the number of rows L by the quotient of the table size A and the data block threshold K. Specifically, this can be expressed as: .

[0104] In one embodiment, the data exchange request obtained by serializing and updating the data exchange request based on the data volume information can be represented as:

[0105]

[0106] in, Indicates free memory; Indicates the data block threshold; This indicates the number of rows in the threshold data table.

[0107] See Figure 5 Specifically, as described above, the iterative data exchange based on the asynchronous response of the data processing center includes:

[0108] Step S501: Based on the asynchronous response, allocate a storage space for data exchange for each iteration;

[0109] Step S502: Based on the storage space and the ETL program unit of the data processing center, data exchange is performed, and the data exchange status is polled to the data processing center to obtain the corresponding asynchronous response.

[0110] In this implementation, when the business side performs iterative data exchange based on asynchronous responses, it allocates a storage space for each iteration of data exchange according to the received asynchronous response. This storage space is used to receive only the data sent by the ETL program unit in the current iteration of data exchange, thereby ensuring the smooth progress of data exchange and improving the efficiency, isolation, reliability, and maintainability of data transmission. Furthermore, during the data exchange process, the data processing center is polled for the data exchange status to obtain the corresponding asynchronous response, thus promptly determining the current data exchange progress and facilitating further operations.

[0111] Specifically, in the method described above, the iterative data exchange based on the asynchronous response of the data processing center further includes:

[0112] After confirming the completion of one iteration of data exchange, obtain the memory occupied by the received data block;

[0113] If the occupied memory is greater than a first preset ratio of the free memory and less than a second preset ratio of the free memory, then the data information and the data volume information of this data exchange are recorded, and the next iteration of data exchange begins.

[0114] If the occupied memory is less than a first preset ratio of the free memory, then the data block threshold in the data volume information is adjusted upward, and the process returns to the step of updating the data exchange request according to the data volume information;

[0115] If the occupied memory is greater than the second preset ratio of the free memory, then the data block threshold in the data volume information is adjusted downward, and the process returns to the step of updating the data exchange request according to the data volume information.

[0116] In this embodiment, during data exchange, if it is determined that an iteration of data exchange has been completed based on the asynchronous response, the memory occupied by the received data block is obtained to facilitate adaptive strategy adjustment based on this memory occupied. Specifically, if the occupied memory is greater than a first preset proportion (e.g., 5%) of the idle memory on the service side and less than a second preset proportion (e.g., 90%) of the idle memory, it is determined that the current data volume information setting is reasonable. Therefore, the data information and data volume information of this data exchange are recorded, and the next iteration of data exchange begins based on this data volume information. Preferably, when starting the next iteration of data exchange, a signal indicating that no adjustment of the data volume information is needed is fed back to the data processing center. If the occupied memory is less than or equal to the first preset proportion of the idle memory, it is determined that the current data volume is too small. The data block threshold in the data volume information is then adjusted upwards, and the step of updating the data exchange request based on the data volume information is returned to adjust the data exchange request. This allows the ETL program unit to re-segment and transmit the remaining data according to the new data volume information, thereby improving the data exchange efficiency of each transmission. If the occupied memory is greater than or equal to the second preset proportion of free memory, it is determined that the current data volume is too large, which will affect the data processing efficiency of the business side. Therefore, the data block threshold in the data volume information is adjusted downward, and the execution of the step of updating the data exchange request according to the data volume information is returned to adjust the data exchange request. This allows the ETL program unit to re-cut and transmit the remaining data according to the new data volume information, which helps to ensure the normal operation of the business and ensure the exchange efficiency.

[0117] In summary, by comparing the memory occupied by the received data with a preset ratio range of free memory, and adjusting the data block threshold in the data volume information based on the comparison results, adaptive transmission of data exchange can be achieved, ensuring data exchange efficiency, memory utilization, and normal operation of business-side services.

[0118] See Figure 6 Preferably, another embodiment of this application provides a data exchange method applied to an ETL program unit, comprising:

[0119] Step S601: Receive data exchange instructions, built-in parameters, and data volume information of receivable data transmitted by the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of receivable data are obtained by the data processing center through parsing the data exchange request.

[0120] Step S602: Obtain the service data volume of the data to be exchanged based on the built-in parameters;

[0121] Step S603: Determine the number of data blocks based on the business data volume level and the data block threshold in the data volume level information;

[0122] Step S604: Adaptive slicing and iterative data exchange are performed according to the number of data blocks, and data exchange status information is fed back to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information according to the data exchange status information and send an asynchronous response to the service side.

[0123] This embodiment illustrates a data exchange method applied to an ETL program unit. Upon receiving a data exchange request from the service side, the data processing center parses it to obtain built-in parameters and the data volume information of the receivable data, and generates a corresponding data exchange instruction. Then, it sends the data exchange instruction, built-in parameters, and the data volume information of the receivable data to the ETL program unit. After receiving the data exchange instruction, built-in parameters, and the data volume information of the receivable data transmitted by the data processing center, the ETL program unit initiates the data exchange process based on the data exchange instruction and obtains the business data volume of the data to be exchanged based on the built-in parameters. In one specific embodiment, the step of obtaining the business data volume can be to obtain a data iterator, read the built-in parameter M-SQL, fill it according to the placeholders in the M-SQL, and simultaneously calculate the data volume or total amount. In one specific embodiment, `fetchDataIter` is the interface of the ETL program unit. Upon receiving a data exchange instruction, it performs two replacements on the M-SQL. The first replacement replaces the string between `select` and `from` with `count(1)` to obtain the number of data rows rather than the data itself, thus improving the efficiency of obtaining the data volume. The second replacement replaces statements such as `order by` in the data exchange instruction with empty strings, retaining only the filtering and conditional statements, thereby improving processing efficiency. After the replacement is complete, placeholders are filled with time intervals to complete the conditional statements. The time interval `t` is represented as `t∈[t_start, t_end]`. The `check_data` method is then called to execute the SQL, which retrieves the business data volume, represented as... ,in, Indicates the volume of business data. This indicates the business table identifier.

[0124] Then, based on the business data volume and the data block threshold in the data volume information, the number of data blocks corresponding to the business data can be calculated. After obtaining the number of data blocks, adaptive slicing and iterative data exchange can be performed based on this number of data blocks. It should be noted that in this embodiment, only business data with a data block count greater than 1 is sliced, so that the data volume corresponding to each sliced ​​data block is less than the data block threshold, thereby ensuring efficient data transmission.

[0125] Simultaneously, during the iterative data exchange process, the ETL program unit also feeds back data exchange status information to the data processing center. This allows the data processing center to assemble response message information based on the data exchange status information and provide an asynchronous response to the business side. This response message can be represented as follows:

[0126]

[0127] Wherein, requestid: represents the request ID; msgcode: represents the response code; action: represents the caller or requester; tasks: represents m data exchange tasks; tasks.name: represents the task name; tasks.status: represents the task status (pending: in progress, success: successful, and rejected: failed, etc.).

[0128] In summary, the data exchange method provided in this application can send data that can be directly used by the business side during data exchange, based on table-based filtering and merging, and built-in parameters after rule-driven parameterized data preprocessing. This helps reduce the probability of data unavailability and avoids the overhead of setting up ETL program units and performing data transformation on the business side, thus saving costs and improving efficiency. Furthermore, based on the received data volume information obtained from machine storage performance probing, data volume judgment, data segmentation, and / or iterative data exchange can be performed to ensure that the data volume during each iteration of data exchange meets the storage performance requirements of the business side, ensuring smooth data exchange and improving data exchange efficiency, flexibility, and accuracy.

[0129] Specifically, in the method described above, determining the number of data blocks based on the business data volume level and the data block threshold in the data volume level information includes:

[0130] If the volume of the business data is less than or equal to the data block threshold, then the number of data blocks is determined to be one.

[0131] If the volume of the business data is greater than the data block threshold, then the number of data blocks is determined by further processing based on the integer quotient and remainder of the business data volume divided by the data block threshold.

[0132] In this embodiment, an example is given on how to determine the number of data blocks. First, a judgment can be made based on the obtained business data volume and the data block threshold. If the business data volume is less than or equal to the data block threshold, the number of data blocks can be determined to be one, that is, no block processing is required, so that it can be transmitted as a single data block, ensuring the efficiency of data transmission.

[0133] If the volume of business data exceeds the data block threshold, the number of data blocks needs to be calculated. This calculation can be expressed as the integer quotient of the business data volume and the data block threshold, or the sum of the quotient and the remainder rounded up. The specific formula is as follows: .

[0134] In other words, when the volume of business data is divisible by the data block threshold, the number of data blocks is the corresponding integer quotient; when the volume of business data is not divisible by the data block threshold, the remainder will be rounded up, that is, the number of data blocks is its integer quotient plus one.

[0135] It should be noted that, in one specific embodiment, in order to improve computational efficiency, the step of comparing the volume of business data with the data block threshold can be skipped, and the number of data blocks can be determined directly according to the above calculation formula.

[0136] See Figure 7 Specifically, in the method described above, when there are multiple data blocks, the adaptive slicing and iterative data exchange based on the number of data blocks includes:

[0137] Step S701: Obtain the time series equally divided according to the number of data blocks;

[0138] Step S702: Create multiple data iteration threads corresponding to the number of data blocks according to the time series, feed back the exchange status information corresponding to each data iteration thread to the data processing center, and perform data exchange in sequence.

[0139] In this embodiment, when there are multiple data blocks, if it is determined that adaptive slicing and iterative data exchange of business data are necessary, then a time series equally divided according to the number of data blocks is obtained, which can be represented as follows: ,in, Where m is the number of data blocks. Then, multiple data iteration threads corresponding to the number of data blocks can be created based on this time series, and the exchange status information of each data iteration thread can be fed back to the data processing center. This allows the data processing center to respond asynchronously to the business side, and by exchanging data sequentially according to each data iteration thread, the business data can be transmitted to the business side.

[0140] It should be noted that, in a specific embodiment, after obtaining the time series, it is preferable to read the above-mentioned M-SQL, initialize the data iterator, traverse the time series X, fill the placeholders between the start time and end time in the time series, and create m data iteration threads in sequence, add each thread to the Daemon thread, and immediately return the data exchange status information of the m data iteration threads to the data processing center.

[0141] Specifically, the method described above also includes:

[0142] When the updated data volume information is received from the data processing center, the step of determining the number of data blocks based on the business data volume and the data block threshold in the data volume information is returned. The updated data volume information is determined by the data processing center based on the received updated data exchange request.

[0143] In this embodiment, since the service side can adjust the data block threshold and the number of rows in the threshold data table according to the relationship between the memory usage and free memory corresponding to the received data, after receiving the updated data volume information, it will re-execute the steps of determining the number of data blocks and further adaptive slicing and data exchange according to the data volume information, thereby ensuring adaptive adjustment of data exchange, ensuring data exchange efficiency and memory utilization of the service side.

[0144] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0145] See Figure 8 Preferably, another embodiment of this application provides a control device applied to the business side, comprising:

[0146] The first preprocessing module 801 is used to filter and merge table information corresponding to at least one business scenario according to data exchange requirements to obtain data table metadata.

[0147] The second preprocessing module 802 is used to construct built-in parameters based on the data table metadata, and to perform rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request.

[0148] The third preprocessing module 803 is used to obtain the data volume information of the receivable data by probing the machine's storage performance. The data volume information includes: free memory, data block threshold, and number of threshold data table rows.

[0149] The first update module 804 is used to serialize and update the data exchange request according to the data volume information;

[0150] The data exchange module 805 is used to send the data exchange request to the data processing center and perform iterative data exchange based on the asynchronous response of the data processing center. The data exchange request is used to enable the ETL program unit of the data processing center to adaptively divide the required data and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

[0151] Specifically, in the control device described above, the first preprocessing module includes:

[0152] The filtering unit is used to filter the table fields of the table information according to the data exchange requirements to obtain a pre-selected table.

[0153] The merging unit is used to merge the pre-selected tables under different business scenarios to obtain the data table metadata.

[0154] Specifically, in the control device described above, the second preprocessing module includes:

[0155] The built-in parameter construction unit is used to construct a preprocessed SQL statement as the built-in parameter based on the data table metadata, and to set the placeholders and data cleaning and transformation rules in the SQL statement;

[0156] The data preprocessing unit is used to preprocess the preprocessing fields in the built-in parameters and generate the data exchange request.

[0157] Specifically, in the control device described above, the third preprocessing module includes:

[0158] The free memory acquisition unit is used to acquire the free memory from the data volume information based on machine storage performance probing.

[0159] A data block threshold acquisition unit is used to acquire the data block threshold from the data volume information based on the free memory and the current business volume and scale.

[0160] The threshold data table row count acquisition unit is used to acquire the threshold data table row count in the data volume information based on the data block threshold and the size and number of rows of the data table to be exchanged.

[0161] It should be noted that the control device in this embodiment corresponds to the control device applied to the business side described above. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect. The device provided in this application embodiment can implement all the control device steps implemented in the above control device embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the control device embodiments and the beneficial effects will not be described in detail.

[0162] Specifically, in the control device described above, the data exchange module includes:

[0163] A space allocation unit is used to allocate a storage space for data exchange for each iteration based on the asynchronous response;

[0164] The data exchange unit is used to exchange data with the ETL program unit of the storage space and the data processing center, and to poll the data exchange status of the data processing center to obtain the corresponding asynchronous response.

[0165] Specifically, in the control device described above, the data exchange module further includes:

[0166] The data memory usage acquisition unit is used to acquire the memory usage corresponding to the received data block after determining that one iteration of data exchange has been completed.

[0167] The first processing unit is configured to, if the occupied memory is greater than a first preset proportion of the free memory and less than a second preset proportion of the free memory, record the data information and the data volume information of this data exchange, and start the next iteration of data exchange.

[0168] The second processing unit is configured to, if the occupied memory is less than a first preset ratio of the free memory, adjust the data block threshold in the data volume information upwards, and return to execute the step of updating the data exchange request according to the data volume information;

[0169] The third processing unit is configured to, if the occupied memory is greater than a second preset ratio of the free memory, adjust the data block threshold in the data volume information downwards, and return to execute the step of updating the data exchange request according to the data volume information.

[0170] See Figure 9 Preferably, another embodiment of this application provides a control device applied to an ETL program unit, comprising:

[0171] The first receiving module 901 is used to receive data exchange instructions, built-in parameters, and data volume information of receivable data transmitted by the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of receivable data are obtained by the data processing center through parsing the data exchange request.

[0172] The first processing module 902 is used to obtain the business data volume of the data to be exchanged based on the built-in parameters.

[0173] The second processing module 903 is used to determine the number of data blocks based on the business data volume and the data block threshold in the data volume information;

[0174] The third processing module 904 is used to perform adaptive slicing and iterative data exchange according to the number of data blocks, and to feed back data exchange status information to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information according to the data exchange status information and send an asynchronous response to the service side.

[0175] Specifically, in the control device described above, the second processing module includes:

[0176] The first determining unit is configured to determine the number of data blocks as one if the volume of the business data is less than or equal to the data block threshold.

[0177] The second determining unit is used to determine the number of data blocks if the volume of the business data is greater than the data block threshold, by further processing the integer quotient and remainder of the business data volume divided by the data block threshold.

[0178] Specifically, in the control device described above, when the number of data blocks is multiple, the third processing module includes:

[0179] The first processing unit is used to obtain a time series equally divided according to the number of data blocks;

[0180] The second processing unit is used to create multiple data iteration threads corresponding to the number of data blocks according to the time series, to feed back the exchange status information corresponding to each data iteration thread to the data processing center, and to perform data exchange in sequence.

[0181] Specifically, the control device described above also includes:

[0182] The fourth processing module is used to, upon receiving the updated data volume information transmitted by the data processing center, return to the step of determining the number of data blocks based on the business data volume and the data block threshold in the data volume information, wherein the updated data volume information is determined by the data processing center based on the received updated data exchange request.

[0183] It should be noted that the control device in this embodiment corresponds to the method applied to the ETL program unit described above. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect. The device provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0184] Preferably, another embodiment of this application provides an electronic device 10, including: a processor 1001, a memory 1002, and a program stored in the memory 1002 and executable on the processor 1001. When the program is executed by the processor 1001, it implements the steps of the data exchange method applied to the service side as described above, or implements the steps of the data exchange method applied to the ETL program unit as described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0185] Preferably, another embodiment of this application provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the data exchange method applied to the business side as described above, or implements the steps of the data exchange method applied to the ETL program unit as described above, achieving the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data exchange method, applied to the business side, characterized in that, include: Based on data exchange requirements, the table information corresponding to at least one business scenario is filtered and merged to obtain data table metadata. Built-in parameters are constructed based on the data table metadata, and rule-driven parameterized data preprocessing is performed on the data table metadata based on the built-in parameters to generate a data exchange request. By probing the machine's storage performance, information on the amount of data that can be received is obtained. This information includes free memory, data block threshold, and the number of rows in the threshold data table. Based on the data volume information, the data exchange request is serialized and updated; The data exchange request is sent to the data processing center, and iterative data exchange is performed based on the asynchronous response of the data processing center. The data exchange request is used to enable the extraction, transformation, loading, and ETL program unit of the data processing center to adaptively divide the required data into chunks and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

2. The method according to claim 1, characterized in that, Based on data exchange requirements, the table information corresponding to each business scenario is filtered and merged to obtain data table metadata, including: Based on the data exchange requirements, the table fields of the table information are filtered to obtain a pre-selected table; The pre-selected tables under different business scenarios are merged to obtain the data table metadata.

3. The method according to claim 1, characterized in that, The step of constructing built-in parameters based on the data table metadata, and performing rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request includes: Based on the data table metadata, a preprocessed Structured Query Language (SQL) statement is constructed as the built-in parameter, and placeholders and data cleaning and transformation rules are set in the SQL statement. The preprocessing fields in the built-in parameters are preprocessed to generate the data exchange request.

4. The method according to claim 1, characterized in that, The process of probing machine storage performance to obtain information on the amount of data that can be received includes: Based on machine storage performance probing, the free memory in the data volume information is obtained; Based on the free memory and the current business volume and scale, obtain the data block threshold in the data volume information; Based on the data block threshold and the size and number of rows of the data table to be exchanged, obtain the threshold data table row count from the data volume information.

5. The method according to claim 1, characterized in that, The iterative data exchange based on the asynchronous response of the data processing center includes: Based on the asynchronous response, allocate a storage space for data exchange for each iteration; Based on the storage space and the ETL program unit of the data processing center, data exchange is performed, and the data exchange status is polled to the data processing center to obtain the corresponding asynchronous response.

6. The method according to claim 1 or 5, characterized in that, The iterative data exchange based on the asynchronous response of the data processing center also includes: After confirming the completion of one iteration of data exchange, obtain the memory occupied by the received data block; If the occupied memory is greater than a first preset ratio of the free memory and less than a second preset ratio of the free memory, then the data information and the data volume information of this data exchange are recorded, and the next iteration of data exchange begins. If the occupied memory is less than a first preset ratio of the free memory, then the data block threshold in the data volume information is adjusted upward, and the process returns to the step of updating the data exchange request according to the data volume information; If the occupied memory is greater than the second preset ratio of the free memory, then the data block threshold in the data volume information is adjusted downward, and the process returns to the step of updating the data exchange request according to the data volume information.

7. A data exchange method applied to an ETL program unit, characterized in that, include: The system receives data exchange instructions, built-in parameters, and data volume information of the receivable data transmitted from the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of the receivable data are obtained by the data processing center through parsing the data exchange request. Based on the built-in parameters, obtain the business data volume of the data to be exchanged; The number of data blocks is determined based on the business data volume level and the data block threshold in the data volume level information; Adaptive slicing and iterative data exchange are performed based on the number of data blocks, and data exchange status information is fed back to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information based on the data exchange status information and send an asynchronous response to the service side.

8. The method according to claim 7, characterized in that, Determining the number of data blocks based on the business data volume level and the data block threshold in the data volume level information includes: If the volume of the business data is less than or equal to the data block threshold, then the number of data blocks is determined to be one. If the volume of the business data is greater than the data block threshold, then the number of data blocks is determined by further processing based on the integer quotient and remainder of the business data volume divided by the data block threshold.

9. The method according to claim 7 or 8, characterized in that, When there are multiple data blocks, the adaptive slicing and iterative data exchange based on the number of data blocks includes: Obtain the time series equally divided according to the number of data blocks; Multiple data iteration threads corresponding to the number of data blocks are created based on the time series, and the exchange status information corresponding to each data iteration thread is fed back to the data processing center, and data exchange is performed sequentially.

10. The method according to claim 7, characterized in that, Also includes: When the updated data volume information is received from the data processing center, the step of determining the number of data blocks based on the business data volume and the data block threshold in the data volume information is returned. The updated data volume information is determined by the data processing center based on the received updated data exchange request.

11. A control device applied to the business side, characterized in that, include: The first preprocessing module is used to filter and merge table information corresponding to at least one business scenario according to data exchange requirements to obtain data table metadata. The second preprocessing module is used to construct built-in parameters based on the data table metadata, and to perform rule-driven parameterized data preprocessing on the data table metadata based on the built-in parameters to generate a data exchange request. The third preprocessing module is used to detect the machine's storage performance and obtain information on the amount of data that can be received. The information on the amount of data includes: free memory, data block threshold, and number of rows in the threshold data table. The first update module is used to serialize and update the data exchange request according to the data volume information; The data exchange module is used to send the data exchange request to the data processing center and perform iterative data exchange based on the asynchronous response of the data processing center. The data exchange request is used to enable the ETL program unit of the data processing center to adaptively divide the required data and perform the iterative data exchange according to the data exchange request, and to enable the data processing center to perform the asynchronous response.

12. A control device applied to an ETL program unit, characterized in that, include: The first receiving module is used to receive data exchange instructions, built-in parameters, and data volume information of receivable data transmitted by the data processing center. The data exchange instructions are generated by the data processing center upon receiving a data exchange request sent by the service side. The built-in parameters and data volume information of receivable data are obtained by the data processing center through parsing the data exchange request. The first processing module is used to obtain the business data volume of the data to be exchanged based on the built-in parameters. The second processing module is used to determine the number of data blocks based on the business data volume and the data block threshold in the data volume information; The third processing module is used to adaptively divide and iteratively exchange data according to the number of data blocks, and to feed back data exchange status information to the data processing center. The data exchange status information is used to enable the data processing center to assemble response message information according to the data exchange status information and send an asynchronous response to the service side.

13. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data exchange method applied to the service side as described in any one of claims 1 to 6, or implements the steps of the data exchange method applied to the ETL program unit as described in any one of claims 7 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data exchange method applied to the service side as described in any one of claims 1 to 6, or implements the steps of the data exchange method applied to the ETL program unit as described in any one of claims 7 to 10.

15. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data exchange method applied to the service side as described in any one of claims 1 to 6, or implement the steps of the data exchange method applied to the ETL program unit as described in any one of claims 7 to 10.

Images