Data transmission methods, apparatus, electronic devices and readable storage media

Abstract

This application discloses a data transmission method, apparatus, electronic device, and readable storage medium. The method includes: acquiring multiple attribute information of multiple data to be transmitted, wherein the multiple attribute information includes data quantity and data type; determining a target verification rule for the multiple data to be transmitted from multiple preset verification rules based on the multiple attribute information; generating multiple data files and multiple verification files corresponding to the multiple data files based on the data quantity, the target verification rule, and the multiple data to be transmitted; and sending the multiple data files and multiple verification files to a target server based on the number of data files and a first preset quantity threshold. By generating verification files through pre-configured verification rules and uploading the packaged multiple data files and multiple verification files to the target server in a parallel manner, the efficiency of data transmission and data verification is greatly improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data transmission method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In existing technologies, when transmitting data between a sender and a receiver, the receiver typically needs to verify each received data entry to prevent errors during transmission. However, with advancements in computer hardware and software, the amount of data to be transmitted is constantly increasing. This verification method is time-consuming and inefficient. Summary of the Invention

[0003] In view of this, this application provides a data transmission method, apparatus, electronic device and readable storage medium, the main purpose of which is to solve the problem that the prior art requires a lot of time and is inefficient to verify all received data one by one.

[0004] According to a first aspect of this application, a data transmission method is provided, the method comprising:

[0005] Obtain multiple attribute information of multiple data to be transmitted, including the number of data and the data type;

[0006] Based on multiple attribute information, target verification rules for multiple data to be transmitted are determined from multiple preset verification rules;

[0007] Based on the amount of data, target verification rules, and multiple data to be transmitted, generate multiple data files and multiple verification files corresponding to the multiple data files;

[0008] Based on the number of data files and a first preset quantity threshold, multiple data files and multiple verification files are sent to the target server.

[0009] Optionally, before obtaining multiple attribute information of multiple data to be transmitted, the method further includes:

[0010] In response to a data transmission request, obtain multiple transmission information and multiple data table information included in the data transmission request;

[0011] Based on multiple transmission information and multiple data table information, multiple data to be transmitted are retrieved from a preset database.

[0012] Optionally, the step of determining the target verification rule for multiple data to be transmitted from multiple preset verification rules based on multiple attribute information specifically includes:

[0013] Determine whether multiple preset validation rules contain validation rules corresponding to multiple attribute information;

[0014] If multiple preset verification rules contain verification rules corresponding to multiple attribute information, determine the target verification rule;

[0015] If multiple preset validation rules do not contain validation rules corresponding to multiple attribute information, a prompt message will be generated based on the multiple attribute information.

[0016] The prompt message is sent to the target user's terminal so that the target user can determine the target verification rules based on the prompt message.

[0017] Optionally, the steps of generating multiple data files and multiple verification files corresponding to the multiple data files, based on the amount of data, target verification rules, and multiple data to be transmitted, specifically include:

[0018] The number of data files is determined based on the amount of data and a second preset threshold.

[0019] Based on the number of data files and the amount of data to be transferred, generate multiple data files;

[0020] Based on the multiple target data and target verification rules in each data file, generate a verification file corresponding to each data file;

[0021] Store multiple data files and multiple verification files in a preset area.

[0022] Optionally, the step of generating multiple data files based on the number of data files and multiple data to be transmitted specifically includes:

[0023] Based on the number of data files, multiple data files to be transmitted are divided into multiple data blocks, each containing multiple target data files.

[0024] Multiple target data in each data block are packaged and compressed to generate a data file.

[0025] Optionally, the step of sending multiple data files and multiple verification files to the target server based on the number of data files and a first preset quantity threshold specifically includes:

[0026] Determine whether there is a data file to be sent in the preset area according to the preset time interval;

[0027] If there are data files to be sent, determine the number of times to send multiple data files in parallel according to the first preset quantity threshold and the number of multiple data files;

[0028] Based on the number of parallel transmissions, multiple data files and multiple verification files are sent to the target server.

[0029] Optionally, the method further includes:

[0030] Obtain multiple storage information and multiple transmission information corresponding to multiple transmitted data;

[0031] Based on multiple storage information and multiple transmission information, a data transmission table is generated for multiple data to be transmitted.

[0032] According to a second aspect of this application, a data transmission apparatus is provided, the apparatus comprising:

[0033] The acquisition module is used to acquire multiple attribute information of multiple data to be transmitted, including the number of data and the data type.

[0034] The determination module is used to determine the target verification rule for multiple data to be transmitted from multiple preset verification rules based on multiple attribute information.

[0035] The generation module is used to generate multiple data files and multiple verification files corresponding to the multiple data files based on the amount of data, the target verification rules, and multiple data to be transmitted.

[0036] The sending module is used to send multiple data files and multiple verification files to the target server based on the number of data files and a first preset quantity threshold.

[0037] According to a third aspect of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods in the first aspect.

[0038] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0039] By employing the above technical solution, this application provides a data transmission method, apparatus, electronic device, and readable storage medium. During data transmission, multiple attribute information of multiple data to be transmitted is acquired. This attribute information is matched with multiple preset verification rules to determine the target verification rule with the highest compatibility with the data to be transmitted. Subsequently, the multiple data to be transmitted are divided into multiple data blocks according to the data quantity. Data files are generated based on the data to be transmitted contained in each data block. Simultaneously, a verification file is generated to verify all data in the data file according to the target verification rule. Then, according to a first preset threshold, multiple data files and multiple verification files are uploaded to a target server in batches using a parallel transmission method, so that a second terminal can download data from the target server and verify the data in each data file based on the verification files. Compared to existing technologies that require verifying all transmitted data one by one, resulting in long processing times, low verification efficiency, and limitations of XML-based message interfaces in supporting large-scale data transmission, this application addresses these issues. It uses pre-configured verification rules to determine the target verification rules for the data to be transmitted, generating verification files. Simultaneously, based on pre-set personalized parameters for data interaction, multiple concurrent tasks are initiated to upload packaged data files and verification files to the target server. This significantly improves the efficiency of large-scale data transmission while allowing the receiver to verify each data file based on the verification files, eliminating the need for sequential verification of each data item. This ensures the security and accuracy of data transmission while shortening data verification time.

[0040] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 This illustration shows a schematic flowchart of a data transmission method provided in an embodiment of this application;

[0043] Figure 2 A schematic diagram of another data transmission method provided in an embodiment of this application is shown;

[0044] Figure 3A schematic diagram of a data transmission device provided in an embodiment of this application is shown. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] This application provides a data transmission method, such as... Figure 1 As shown, the method includes:

[0048] S101. Obtain multiple attribute information of multiple data to be transmitted, including the number of data and the data type.

[0049] This application provides a data transmission method applicable to the data sender, such as a first terminal. Specifically, the first terminal refers to a device or application used for data transmission. The first terminal can be a hardware device, such as a computer, mobile phone, or smart device, or a software application, such as a web browser or communication application. The first terminal is communicatively connected to a server, and the server is communicatively connected to a second terminal, which is used to receive data transmitted by the first terminal.

[0050] In practical applications, data transmission can be used to achieve interaction and integration between different platforms. For example, data can be transferred from an application on one operating system to an application on another operating system to achieve information sharing and cross-platform interaction. In existing technologies, after receiving data transmitted by the first terminal, the second terminal needs to verify all the data to ensure its correctness. However, this method is time-consuming and inefficient when the data volume is large. To address these issues, this application proposes pre-defined verification rules for different types of data. Based on the data parameters of the data to be transmitted, the verification rules are determined, and a verification file is generated. This allows the second terminal to verify the received data based on the verification file, eliminating the need to verify each piece of data individually and effectively improving data verification efficiency.

[0051] Specifically, the data to be transmitted can be business data stored within the first terminal. Multiple attribute information includes the quantity of data and the data type of the business data. In the financial field, the first terminal can be a financial trading platform, and the data to be transmitted in the first terminal can be transaction data, including transaction information for various financial assets such as stocks, bonds, foreign exchange, futures, and options, such as buy and sell prices, data, and timestamps; it can also be market data, including stock prices, index data, exchange rates, interest rates, and futures quotes; and it can also be customer data, including customer names, addresses, contact information, bank account information, credit card information, and transaction records. To improve the flexibility and practicality of data verification, different verification rules are formulated for different business attributes of the data. Then, during data transmission, the corresponding verification rules are determined based on the quantity and business attributes of the data to be transmitted.

[0052] Optionally, the data type can be set according to its business scenario. For example, in a financial scenario, data types include transaction data, market data, customer data, risk data, financial data, etc.

[0053] S102. Based on multiple attribute information, determine the target verification rules for multiple data to be transmitted from multiple preset verification rules.

[0054] In this step, the validation rules differ for different business attributes. For example, transaction data includes transaction amounts, so the quantity of data and the transaction amount can be used as the items to be checked. Furthermore, for customer data, which does not contain monetary data, the customer name and the quantity of data can be used as the items to be checked. By pre-determining preset validation rules for different business data, during data transmission, the corresponding target validation rules can be quickly matched based on the attribute information of the data to be transmitted, improving the efficiency of subsequent data validation.

[0055] S103. Based on the amount of data, the target verification rules, and multiple data to be transmitted, generate multiple data files and multiple verification files corresponding to the multiple data files.

[0056] In this step, for data interaction between different terminals, existing technologies typically use XML interface messages. While this method is efficient for handling short transactional data transmissions, with the development of computer hardware and software, the amount of data to be transmitted is increasing, making the traditional interface interaction method inefficient and requiring significant hardware resources and costs. To address these issues, this application proposes dividing multiple data to be transmitted into data blocks of equal size. Then, based on each data block, it is compressed and packaged to generate a data file. Simultaneously, a verification file is generated for each data file; specifically, the verification file contains the verification results of multiple data items within that data file, generated based on verification rules.

[0057] By dividing a large amount of data to be transmitted into multiple data blocks, generating a data file and a checksum file for each data block, and then sending multiple data files and corresponding checksum files in batches in parallel, the data transmission efficiency is improved without affecting the data verification effect. This method is suitable for large-scale data transmissions.

[0058] S104. Based on the number of data files and the first preset quantity threshold, send the multiple data files and multiple verification files to the target server.

[0059] In this step, to improve data transmission efficiency and achieve the transmission of large amounts of data, this application proposes a data slicing method, dividing large amounts of data into multiple small transmission data blocks, and transmitting them in parallel according to a pre-set optimal number of tasks. Specifically, the first preset threshold number is the pre-set optimal number of uploaded files for each parallel transmission. By transmitting the data files and verification files of the first preset threshold number to the target server in a parallel manner, the second terminal can achieve better data transmission efficiency.

[0060] By employing the above method, data exchange between two terminals via server communication is achieved. Large datasets are divided into smaller data blocks using data slicing, and these blocks are transmitted to the server in parallel according to the optimal number of tasks. This allows the second terminal to retrieve data from the server as needed. This enables the transmission of large datasets in a distributed environment, significantly improving data transmission efficiency and saving substantial time and machine resources.

[0061] The data transmission method provided in this application involves acquiring multiple attribute information of multiple data to be transmitted during data transmission. These attribute information is then matched with multiple preset verification rules to determine the target verification rule with the highest compatibility with the data to be transmitted. Subsequently, the multiple data to be transmitted are divided into multiple data blocks based on the amount of data. Data files are generated by packaging the data to be transmitted within each data block. Simultaneously, a verification file is generated to verify all data in each data file according to the target verification rule. Then, based on a first preset threshold, the multiple data files and multiple verification files are uploaded to a target server in batches using a parallel transmission method. This allows a second terminal to download data from the target server and verify the data in each data file based on the verification files. Compared to existing technologies that require verifying all transmitted data one by one, resulting in long processing times, low verification efficiency, and limitations of XML-based message interfaces in supporting large-scale data transmission, this application addresses these issues. It uses pre-configured verification rules to determine the target verification rules for the data to be transmitted, generating verification files. Simultaneously, based on pre-set personalized parameters for data interaction, multiple concurrent tasks are initiated to upload packaged data files and verification files to the target server. This significantly improves the efficiency of large-scale data transmission while allowing the receiver to verify each data file based on the verification files, eliminating the need for sequential verification of each data item. This ensures the security and accuracy of data transmission while shortening data verification time.

[0062] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another data transmission method, such as... Figure 2 As shown, the method includes:

[0063] S201. In response to the data transmission request, obtain multiple transmission information and multiple data table information included in the data transmission request.

[0064] S202. Based on multiple transmission information and multiple data table information, retrieve multiple data to be transmitted from the preset database.

[0065] In steps S201 and S202, the system stores a large amount of business data, such as customer information and transaction data. When it is necessary to transfer some business data to other systems, business personnel initiate a data transmission request based on the selected data to be transmitted. The data transmission request includes multiple transmission information items and multiple data table information items. These multiple transmission information items and data table information items are used to determine the data to be transmitted in the system. Specifically, the multiple transmission information items may include the start and end times for obtaining the data to be transmitted, the data type of the data to be transmitted, etc. The multiple data table information items refer to the information of the data tables storing the data in the system, such as the table name, the list of fields in the data table, the acquisition time of each piece of data, etc. Based on the multiple transmission information items, the range of data to be transmitted is determined, and then, based on the data range and the multiple data table information items, multiple data items to be transmitted are retrieved from a preset database.

[0066] Optionally, some data in the system is transmitted on a fixed schedule, such as transaction data from a trading platform, which needs to be transmitted to the data system platform daily / weekly / monthly for data analysis. Therefore, a pre-defined scheduling strategy is required to automatically retrieve and transmit data according to the transmission cycle.

[0067] In practical applications, if business personnel want to transfer transaction data from the transaction platform (i.e., the first terminal) for the past month to the data platform for data analysis, the business personnel initiate a data transfer request. The system parses the request and extracts the start time (e.g., 2023-06-01), end time (e.g., 2023-06-30), data type (transaction data), and all data table information corresponding to the preset database. It then retrieves all transaction data for June 2023 from the preset database and uploads all the retrieved June transaction data to the target server, so that the data platform (i.e., the second terminal) can download this data from the target server.

[0068] S203. Obtain multiple attribute information of multiple data to be transmitted, including the number of data and the data type.

[0069] In this step, the multiple data to be transmitted can be business data stored within the first terminal. Multiple attribute information includes the quantity of data and the data type of the business data. In the financial field, the first terminal can be a financial trading platform, and the data to be transmitted in the first terminal can be transaction data, including transaction information for various financial assets such as stocks, bonds, foreign exchange, futures, and options, such as buy and sell prices, data, and timestamps; it can also be market data, including stock prices, index data, exchange rates, interest rates, and futures quotes; and it can also be customer data, including customer names, addresses, contact information, bank account information, credit card information, and transaction records. To improve the flexibility and practicality of data verification, different verification rules are formulated for different business attributes of the business data. Then, during data transmission, the corresponding verification rules are determined based on the quantity and business attributes of the data to be transmitted.

[0070] S204. Determine whether the multiple preset verification rules contain verification rules corresponding to multiple attribute information. If yes, proceed to step S205; otherwise, proceed to step S206.

[0071] In this step, the preset verification rules are the verification rules corresponding to different data in different business scenarios. After obtaining the data type, data quantity, and other attribute information of the data to be transmitted, the system determines whether there are multiple verification rules corresponding to the data to be transmitted among all the preset verification rules, based on the mapping relationship between the attribute information and the verification rules.

[0072] Optionally, multiple preset validation rules can be set separately for each data type in different business domains. For example, in the financial domain, transaction data, which involves money, can use the data quantity and total amount as validation rules; customer data, which involves information such as customer names, can use the data quantity and all customer names as validation rules. In addition, for some data with low content of important information, only the data quantity can be used as the validation rule.

[0073] S205. Determine the target verification rules.

[0074] In this step, if multiple preset verification rules contain target verification rules corresponding to multiple attribute information of the data to be transmitted, then the target verification rules will be matched according to the attribute information to be transmitted, so that a verification file of the data to be transmitted can be generated based on the rule.

[0075] S206. Generate prompt information based on multiple attribute information.

[0076] S207. Send the prompt message to the target user's terminal so that the target user can determine the target verification rules based on the prompt message.

[0077] In steps S206 and S207, if the multiple preset verification rules do not include verification rules corresponding to multiple attribute information, the data under that attribute may be data corresponding to newly added scenarios, projects, etc. in the system, and verification rules are configured for it. At this time, based on the multiple attribute information of the data to be transmitted, a prompt message is generated and sent to the target user's terminal. The target user can be the platform's operation and maintenance personnel. After receiving the prompt message sent by the first terminal, the operation and maintenance personnel's terminal determines the target verification rules based on the multiple attribute information of the multiple data to be transmitted.

[0078] Optionally, the multiple data to be transmitted may include data of one or more data types. Therefore, when generating the prompt message, the data types that do not contain the verification rules are filtered out from the multiple data to be transmitted based on multiple attribute information, and they are marked. The prompt message is generated based on the marked data types and sent to the target user's terminal, so that the target user can formulate the target verification rules based on the marked data types.

[0079] Furthermore, after the target user sets target verification rules for the data, a mapping relationship is formed between the data type and the target verification rules, and stored in multiple preset verification rules to improve the preset verification rules.

[0080] S208. Determine the number of data files based on the amount of data and the second preset threshold.

[0081] In this step, due to the large amount of data to be transmitted, packaging all the data into a single data file would result in a long transmission time and slow speed. Therefore, this application proposes a multi-task parallel transmission method. The data volume of each transmission task is predetermined, and the data to be transmitted is divided into multiple data blocks. These data blocks are then transmitted separately to the target server. Specifically, the second preset threshold is the maximum data volume that each transmission task can transmit. Based on the total amount of data to be transmitted and the maximum data volume of each transmission task, the number of data blocks to be transmitted is calculated, which is the final number of data files generated.

[0082] S209. Generate multiple data files based on the number of data files and the multiple data to be transmitted.

[0083] In this step, in order to improve data transmission efficiency and realize the transmission of large amounts of data, this application proposes a method of slicing data, dividing large amounts of data into multiple small transmission data blocks to generate multiple data files, and then transmitting them in parallel according to the most suitable number of pre-set tasks, thereby improving data transmission efficiency.

[0084] In practical applications, multiple data sets to be transmitted can be divided into data blocks of the same size based on a second preset threshold. When the data volume is too large, the data is divided based on its size. For example, if multiple data sets to be transmitted are 3.4GB in size, and the second preset threshold is 1GB, then the multiple data sets to be transmitted can be divided into 1GB data blocks. Since one data block contains only 0.4GB of data during the division, the remaining 0.6GB of data can be padded with other data (e.g., zero) to form a 1GB data block, thus forming four 1GB data blocks in total. Alternatively, the 0.4GB data block can be left unpadded.

[0085] Optionally, in step S209, which involves generating multiple data files based on the number of data files and multiple data to be transmitted, the process includes: dividing the multiple data to be transmitted into multiple data blocks based on the number of data files, wherein each data block contains multiple target data; and packaging and compressing the multiple target data in each data block to generate a data file.

[0086] In this embodiment, based on the determined number of data files, the multiple data files to be transmitted are divided equally into multiple data blocks. Each data block contains target data less than or equal to a second preset threshold number. Subsequently, the target data in each data block is packaged and compressed to generate a corresponding data file.

[0087] By using the above method, multiple data to be transmitted are divided, packaged, and compressed into multiple data files for data transmission, realizing the transmission of large amounts of data in a distributed environment, greatly improving transmission efficiency, and saving a lot of time and machine resources.

[0088] In practical applications, to ensure the uniqueness and integrity of data transmission, multiple data files to be generated can be sequentially numbered according to the order of the data to be transmitted, resulting in a sequence number for each data file. Furthermore, when packaging and compressing multiple target data to generate data files, the data is named based on the sender's terminal name, the name of the source table containing the data, the data acquisition date, the data transmission date, and the data file's sequence number. This allows the target server to verify the integrity of the data file transmission by checking the sequence number of the received data file name, preventing data transmission omissions or duplications.

[0089] S210. Generate a verification file corresponding to each data file based on multiple target data and target verification rules in each data file.

[0090] In this step, the verification file contains the correct results generated based on the verification rules, which the receiver uses to verify the integrity of the data. Specifically, multiple target data points are obtained from each data file, and a verification file for each data file is generated based on the attribute information of these target data points and the corresponding target verification rules. While each data file is being transmitted to the target server, its matching verification file is also uploaded to the target server.

[0091] In practical applications, multiple data sets to be transmitted consist of transaction data from the trading platform within the past month. These transaction data sets are divided, packaged, and compressed into 10 data files, each containing 500 transaction records. Based on the data type, the target verification rule is determined to be quantity + transaction amount. The transaction amount of the target data in each data file is then calculated. If the transaction amount of the target data in a data file is 100,000, then the corresponding verification file content for that data file is 500 records + 100,000. Subsequently, upon receiving the data file, the receiving party decompresses it to obtain the target data and verifies whether the quantity of these target data records is 500. Simultaneously, it verifies whether the total transaction amount contained in these data files is 100,000. If yes, the data in the data file is confirmed to be complete; otherwise, it is determined that the data in the data file is missing, and a transmission error warning message needs to be issued to the maintenance personnel for review and repair. It is understandable that the receiving terminal, after receiving the data files, can perform individual verification on each received data file or perform a comprehensive verification of all data, thereby effectively improving the security of data transmission.

[0092] S211. Store multiple data files and multiple verification files in a preset area.

[0093] In this step, to improve data transmission efficiency, this application employs a parallel data transmission method. Specifically, the sending terminal establishes multiple threads with the target server for data transmission. During data transmission, multiple transmission threads can be initiated to transmit multiple data files to the target server in parallel. After all the data to be transmitted is divided, packaged, and compressed to generate multiple data files and multiple checksum files, each data file and its corresponding checksum file are stored in a preset area of ​​the sending party. It should be noted that this preset area is the preset area to be transmitted. Then, the data files and checksum files within the preset area are transmitted.

[0094] Optionally, data can be transmitted in a parallel out-of-order manner, which greatly improves data transmission efficiency and enables efficient communication between the terminal and the server.

[0095] S212. According to the preset time interval, determine whether there is a data file to be sent in the preset area. If yes, proceed to step S213. If no, start timing. When the timing reaches the preset time interval, determine again whether there is a data file to be sent in the preset area.

[0096] In this step, the system periodically checks whether there is a data file to be transmitted in the preset area. If there is a data file to be uploaded, the system uses multi-threading to upload the data file to the target server in parallel. If there is no data file to be uploaded, the system continues to wait until there is a data file to be uploaded in the preset area.

[0097] S213. Determine the number of times to send multiple data files in parallel according to the first preset quantity threshold and the number of multiple data files.

[0098] S214. Based on the number of parallel transmissions, send multiple data files and multiple verification files to the target server.

[0099] In steps S213 and S214, the first preset quantity threshold is the maximum number of parallel tasks. Once it is determined that there are data files to be uploaded in the preset area, the number of parallel tasks can be determined based on the number of data files and the task threshold for each parallel task. Then, based on the number of parallel transmissions, parallel tasks are executed to upload each data file and its corresponding verification file to the target server.

[0100] Optionally, the preset time interval can be set by the operation and maintenance personnel according to business needs or the amount of data to be transmitted, such as 5 minutes, 10 minutes, etc. This application does not make specific limitations here.

[0101] In practical applications, a timed scheduling strategy is used to periodically check whether there are data files and verification files to be uploaded in a preset area at preset time intervals. If data files and verification files to be uploaded exist, the number of corresponding parallel tasks and the number of execution times of the parallel tasks are matched according to the number of data files to be transmitted and the task volume threshold of the parallel tasks. Specifically, if there are 10 data files to be uploaded in the preset area and the maximum task volume of each parallel task is 5, then the number of parallel transmissions is determined to be 2. The 10 data files and 10 verification files are divided into two batches and uploaded to the target server. Each batch selects 5 data files and 5 verification files and puts them into 5 threads for parallel transmission. Furthermore, a directory is pre-set in the target server to store the data files transmitted by the first terminal. When the target server receives the data files and verification files, it identifies the directory according to the file name of the data files and stores them in a dedicated directory.

[0102] S215. Obtain multiple storage information and multiple transmission information corresponding to multiple transmission data.

[0103] S216. Generate a data transfer table for multiple data to be transferred based on multiple storage information and multiple transmission information.

[0104] In steps S215 and S216, during data interaction between the first terminal and the second terminal, to better understand the transmission status of each transmitted data and facilitate subsequent tracing and repair of erroneous data, after determining multiple data to be transmitted, a data transmission table for this transmission is generated based on the storage information of the first terminal sending the data, the storage information of the second terminal receiving the data, and the transmission information of this transmission.

[0105] Specifically, after the first terminal retrieves the data to be transmitted from the data table, generates a data file, and uploads it to the target server, the second terminal periodically retrieves the data file from the target server and saves it to the corresponding data table based on the data in the data file. The terminal name, data table name, data field list header, start time, end time, and verification fields of the multiple data to be transmitted from the first terminal are obtained, along with the terminal name, data packet name, and corresponding field list header of the data table stored by the second terminal. These are used as multiple storage information for the multiple data to be transmitted. Simultaneously, the filename, transmission method, and transmission status of the data file generated from the division and packaging of the multiple data to be transmitted are obtained as multiple transmission information. These multiple storage and transmission information are summarized and organized to generate a data transmission table for this transmission.

[0106] In practical applications, as shown in Table 1, this is the data transfer table for the premium information of terminal A (yc_uc_md) to terminal B (uc_md) on June 30, 2023. Specifically, the data transfer table includes: multiple source data information (i.e., data storage information in terminal A); multiple target information (i.e., data storage information in terminal B); and data transfer information. The multiple source data information includes the data table name "yc_uc_md", the list of fields containing the transmitted data ("policyno", "startdate", "enddate", ...), and the field "updatetime" used for incremental retrieval. It should be noted that each data table contains a large amount of data; however, in data interaction, it is not necessary to transmit all data in the field list to other systems. Only useful data needs to be filtered out. For example, out of ten thousand data entries, only five thousand may be useful for subsequent data analysis; only these five thousand entries need to be extracted and transmitted. Therefore, retrieval rules can be set in advance according to the data processing needs, and then the fields to be transmitted can be determined from the field list based on these rules, thereby retrieving the data from these fields. The fields used for incremental acquisition are those that filter out data containing the data to be transmitted. These include the start acquisition time of the transmitted data ("2023-06-30 7:50"), the end acquisition time ("2023-06-30 18:30"), the verification field "amont" containing the data used for verification among multiple transmitted data, and the sending terminal name "A". Further, multiple target information includes the receiving terminal name "B", the target data table "uc_md" where the received data to be transmitted is stored, and the target field list "policyno, startdate, enddate, ...". Further, multiple transmission information includes the filename of each data file ("zbsff_uc_md_20230630_00001-12000"), the transmission method "parallel transmission", and the transmission status "pending transmission".

[0107] Table 1

[0108]

[0109] Furthermore, as Figure 1 To specifically implement the method, this application embodiment provides a data transmission device 300, such as... Figure 3 As shown, the device includes:

[0110] The acquisition module 301 is used to acquire multiple attribute information of multiple data to be transmitted, wherein the multiple attribute information includes the data quantity and data type;

[0111] The determination module 302 is used to determine the target verification rule for multiple data to be transmitted from multiple preset verification rules based on multiple attribute information.

[0112] The generation module 303 is used to generate multiple data files and multiple verification files corresponding to the multiple data files based on the amount of data, the target verification rules and multiple data to be transmitted.

[0113] The sending module 304 is used to send multiple data files and multiple verification files to the target server based on the number of data files and a first preset quantity threshold.

[0114] Optionally, the acquisition module 301 is also configured to, in response to a data transmission request, acquire multiple transmission information and multiple data table information included in the data transmission request.

[0115] Optionally, the device further includes:

[0116] The retrieval module 305 is used to retrieve multiple data to be transmitted from a preset database based on multiple transmission information and multiple data table information.

[0117] The judgment module 306 is used to determine whether multiple preset verification rules contain verification rules corresponding to multiple attribute information.

[0118] Optionally, module 302 is specifically used for:

[0119] Determine whether multiple preset validation rules contain validation rules corresponding to multiple attribute information;

[0120] If multiple preset verification rules contain verification rules corresponding to multiple attribute information, determine the target verification rule;

[0121] If multiple preset validation rules do not contain validation rules corresponding to multiple attribute information, a prompt message will be generated based on the multiple attribute information.

[0122] The prompt message is sent to the target user's terminal so that the target user can determine the target verification rules based on the prompt message.

[0123] Optionally, the generation module 303 is specifically used for:

[0124] The number of data files is determined based on the amount of data and a second preset threshold.

[0125] Based on the number of data files and the amount of data to be transferred, generate multiple data files;

[0126] Based on the multiple target data and target verification rules in each data file, generate a verification file corresponding to each data file;

[0127] Store multiple data files and multiple verification files in a preset area.

[0128] Optionally, the generation module 303 is further used for:

[0129] Based on the number of data files, multiple data files to be transmitted are divided into multiple data blocks, each containing multiple target data files.

[0130] Multiple target data in each data block are packaged and compressed to generate a data file.

[0131] Optionally, the sending module 304 is specifically used for:

[0132] Determine whether there is a data file to be sent in the preset area according to the preset time interval;

[0133] If there are data files to be sent, determine the number of times to send multiple data files in parallel according to the first preset quantity threshold and the number of multiple data files;

[0134] Based on the number of parallel transmissions, multiple data files and multiple verification files are sent to the target server.

[0135] Optionally, the acquisition module 301 is also used to acquire multiple storage information and multiple transmission information corresponding to multiple transmission data;

[0136] The generation module 303 is also used to generate multiple data transfer tables for data to be transmitted based on multiple storage information and multiple transmission information.

[0137] The data transmission device 300 provided in this application embodiment acquires multiple attribute information of multiple data to be transmitted during data transmission. It matches the multiple attribute information with multiple preset verification rules to determine the target verification rule with the highest compatibility with the data to be transmitted. Then, it divides the multiple data to be transmitted into multiple data blocks according to the amount of data, and packages the data to be transmitted contained in each data block to generate a data file. Simultaneously, according to the target verification rule, it generates a verification file that verifies all data in the data file. Subsequently, according to a first preset threshold, it uploads the multiple data files and multiple verification files to a target server in batches using a parallel transmission method, so that a second terminal can download data from the target server and verify the data in each data file based on the verification files. Compared to existing technologies that require verifying all transmitted data one by one, resulting in long processing times, low verification efficiency, and limitations of XML-based message interfaces in supporting large-scale data transmission, this application addresses these issues. It uses pre-configured verification rules to determine the target verification rules for the data to be transmitted, generating verification files. Simultaneously, based on pre-set personalized parameters for data interaction, multiple concurrent tasks are initiated to upload packaged data files and verification files to the target server. This significantly improves the efficiency of large-scale data transmission while allowing the receiver to verify each data file based on the verification files, eliminating the need for sequential verification of each data item. This ensures the security and accuracy of data transmission while shortening data verification time.

[0138] In an exemplary embodiment, this application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the program stored in the memory to perform the data transmission method described in the above embodiments.

[0139] In an exemplary embodiment, this application also provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data transmission method described above.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause an electronic device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0141] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0142] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0143] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0144] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A data transmission method, characterized in that, include: Obtain multiple attribute information of multiple data to be transmitted, wherein the multiple attribute information includes the number of data and the data type; Based on the multiple attribute information, the target verification rule for the multiple data to be transmitted is determined from multiple preset verification rules; Based on the data quantity, the target verification rules, and the multiple data to be transmitted, generate multiple data files and multiple verification files corresponding to the multiple data files; Based on the number of data files in the plurality of data files and a first preset quantity threshold, the plurality of data files and the plurality of verification files are sent to the target server; The step of generating multiple data files and multiple verification files corresponding to the multiple data files based on the data quantity, the target verification rule, and the multiple data to be transmitted specifically includes: The number of data files is determined based on the data quantity and the second preset quantity threshold. The plurality of data files are generated based on the number of data files and the plurality of data to be transmitted; Based on multiple target data in each data file and the target verification rules, a verification file corresponding to each data file is generated; Store the plurality of data files and the plurality of verification files in a preset area; The step of sending the plurality of data files and the plurality of verification files to the target server based on the number of data files and a first preset quantity threshold specifically includes: Determine whether there is a data file to be sent in the preset area according to the preset time interval; If there are data files to be sent, determine the number of times the multiple data files will be sent in parallel according to the first preset quantity threshold and the number of multiple data files; Based on the number of parallel transmissions, the multiple data files and the multiple verification files are sent to the target server.

2. The method according to claim 1, characterized in that, Before obtaining multiple attribute information of multiple data to be transmitted, the method further includes: In response to a data transmission request, obtain multiple transmission information and multiple data table information included in the data transmission request; Based on the multiple transmission information and the multiple data table information, the multiple data to be transmitted are retrieved from the preset database.

3. The method according to claim 1, characterized in that, The step of determining the target verification rule for the multiple data to be transmitted from multiple preset verification rules based on the multiple attribute information specifically includes: Determine whether the plurality of preset verification rules include the verification rules corresponding to the plurality of attribute information; If the plurality of preset verification rules include the verification rules corresponding to the plurality of attribute information, determine the target verification rule; If the plurality of preset verification rules do not include the verification rule corresponding to the plurality of attribute information, a prompt message is generated based on the plurality of attribute information; The prompt message is sent to the target user's terminal so that the target user can determine the target verification rule based on the prompt message.

4. The method according to claim 1, characterized in that, The step of generating the multiple data files based on the number of data files and the multiple data to be transmitted specifically includes: Based on the number of data files, the multiple data to be transmitted are divided into multiple data blocks, wherein each data block contains multiple target data. The multiple target data in each data block are packaged and compressed to generate a data file.

5. The method according to any one of claims 1 to 4, characterized in that, Also includes: Obtain multiple storage information and multiple transmission information corresponding to multiple transmitted data; Based on the multiple storage information and the multiple transmission information, a data transmission table for the multiple data to be transmitted is generated.

6. A data transmission device, characterized in that, include: The acquisition module is used to acquire multiple attribute information of multiple data to be transmitted, wherein the multiple attribute information includes the number of data and the data type; The determining module is used to determine the target verification rule for the multiple data to be transmitted from multiple preset verification rules based on the multiple attribute information. The generation module is used to generate multiple data files and multiple verification files corresponding to the multiple data files based on the data quantity, the target verification rules, and the multiple data to be transmitted; The sending module is used to send the multiple data files and the multiple verification files to the target server according to the number of data files in the multiple data files and a first preset number threshold; The generation module is specifically used for: The number of data files is determined based on the amount of data and a second preset threshold. Based on the number of data files and the amount of data to be transferred, generate multiple data files; Based on the multiple target data and target verification rules in each data file, generate a verification file corresponding to each data file; Store multiple data files and multiple verification files in a preset area; The sending module is specifically used for: Determine whether there is a data file to be sent in the preset area according to the preset time interval; If there are data files to be sent, determine the number of times to send multiple data files in parallel according to the first preset quantity threshold and the number of multiple data files; Based on the number of parallel transmissions, multiple data files and multiple verification files are sent to the target server.

7. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the steps of any one of claims 1 to 5.

8. A readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of any one of claims 1 to 5.

Images