Data transmission methods, apparatus, devices and readable storage media

Abstract

This application discloses a data transmission method, relating to the field of big data processing technology. The method includes: acquiring a data transmission request; generating data retrieval conditions based on the data transmission request and initializing a data transmission task; acquiring indicator data of the corresponding target data based on the data retrieval conditions; calculating the feature score of the target data based on the indicator data; and segmenting the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; transmitting the sub-data through an active thread and writing the sub-data to corresponding sub-files; sorting the sub-files according to the partition key number; summarizing the sub-data recorded in the sub-files to generate a data file corresponding to the target data; and returning the data file to the client. Implementing this data transmission method can improve data transmission speed, reduce waiting time consumed by data querying, provide real-time feedback on download speed and remaining data volume, and allow users to promptly understand the data transmission progress.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data transmission method, apparatus, device, and readable storage medium. Background Technology

[0002] With the rapid development of information science and technology, especially the widespread application of cloud computing and big data technologies in the financial sector, the core competitiveness of the financial industry in the future will largely depend on the speed and ability to extract information and knowledge from big data. Faced with the demand for massive data transmission, the common practice is to directly generate query statements to query the database based on data download requests initiated by clients. When the data volume reaches a certain scale, the processing speed of a single retrieval becomes very slow, resulting in lengthy waiting times and potentially causing server system errors, data file errors, and ultimately affecting the normal operation of the database. Simultaneously, while waiting for data transmission, users cannot obtain intuitive data on data transmission progress and processing speed to coordinate their work schedules related to data queries. Therefore, there is an urgent need for a data transmission method, device, equipment, and readable storage medium to improve data transmission speed and save time wasted due to waiting for data processing when dealing with large data volumes and large file sizes. Furthermore, from a user experience perspective, real-time feedback on data processing speed and query progress is also needed to facilitate user task scheduling. Summary of the Invention

[0003] To address the problem of excessively long data transmission times, especially when handling large data volumes and large file sizes, in existing technologies, this invention provides a data transmission method, apparatus, device, and readable storage medium to improve data transmission speed and reduce data query waiting time.

[0004] To solve one or more of the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] Firstly, a data transmission method is provided for use on a server side, including:

[0006] Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task, wherein the data transmission task corresponds to the data transmission request;

[0007] Based on the data retrieval criteria, obtain the corresponding target data's indicator data, calculate the target data's feature score based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; among which, the indicator data includes: total data volume, data time span, and data number span;

[0008] Sub-data is transmitted through an active thread and written to the corresponding sub-file, where the sub-file includes the partition key number corresponding to the sub-data;

[0009] The subfiles are sorted according to the partition key number, the subdata recorded in the subfiles are summarized, and a data file corresponding to the target data is generated.

[0010] Return the data file to the client.

[0011] Furthermore, the feature scores of the target data are calculated based on the indicator data, including:

[0012] The indicator data is evaluated using preset feature scoring rules to obtain the indicator scores corresponding to the indicator data.

[0013] Calculate the weighted scores for each segmentation rule in turn;

[0014] The weighted scores are sorted, the maximum weighted score is obtained, and the maximum weighted score is used as the feature score.

[0015] Furthermore, the weighted scores corresponding to all segmentation rules are calculated sequentially, including:

[0016] Poll the segmentation rules to obtain the indicator weights corresponding to the current segmentation rule, and obtain the weighted score corresponding to the current segmentation rule based on the indicator weights and indicator scores.

[0017] Furthermore, the initialization of data transmission tasks includes:

[0018] Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0019] Execute preset probe code to assess the service resources consumed by the data transmission task;

[0020] Active threads are generated based on the number of CPU cores, the amount of RAM, and service resources.

[0021] Furthermore, returning the data file to the client includes:

[0022] Data files are transmitted to the client in real time.

[0023] and / or

[0024] The data file is uploaded to the server, and the download address of the data file is sent to the client.

[0025] Furthermore, transmitting sub-data via active threads and writing the sub-data to the corresponding sub-files also includes:

[0026] At each first time interval, use the probe code to obtain the current environmental parameters;

[0027] Calculate and obtain the current server load value based on the current environmental parameters;

[0028] Adjust the number of threads based on the current server load.

[0029] Furthermore, adjusting the number of threads based on the current server load includes:

[0030] When the server load value is greater than or equal to the first threshold, add a first number of new threads;

[0031] When the server load is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0032] Furthermore, the above data transmission method also includes:

[0033] At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0034] Download speed is calculated based on the number of currently active threads;

[0035] The remaining task volume is calculated based on the amount of data already transmitted and the total amount of target data.

[0036] The download speed and remaining task amount are returned to the client.

[0037] Secondly, a data transmission method is provided for application on a client side, including:

[0038] Send a data transmission request to the server so that the server can obtain the corresponding data file according to the data transmission request. The data file includes the target data corresponding to the data transmission request.

[0039] Retrieve the data file returned by the server.

[0040] and / or

[0041] Download address for downloading data files;

[0042] Get the download speed and remaining task quantity returned by the server.

[0043] Thirdly, a data transmission device is provided, comprising:

[0044] The request acquisition module is used to acquire data transmission requests, generate data retrieval conditions based on data transmission requests, and initialize data transmission tasks, wherein the data transmission tasks correspond to the data transmission requests.

[0045] The data segmentation module is used to obtain the indicator data of the corresponding target data according to the data retrieval conditions, calculate the feature score of the target data based on the indicator data, and segment the target data according to the feature score to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span;

[0046] The data writing module is used to transfer sub-data through active threads and write the sub-data to the corresponding sub-files, wherein the sub-files include the partition key number corresponding to the sub-data;

[0047] The file generation module is used to sort sub-files according to partition key numbers, summarize the sub-data recorded in the sub-files, and generate a data file corresponding to the target data.

[0048] The file return module is used to return data files to the client.

[0049] Furthermore, the device also includes:

[0050] The task monitoring module is used to obtain the current number of active threads and the amount of data transferred at every second time interval; calculate the download speed based on the current number of active threads; calculate the remaining task volume based on the amount of data transferred and the total amount of target data; and return the download speed and remaining task volume to the client.

[0051] Fourthly, an electronic device is provided, comprising:

[0052] Processor and memory;

[0053] Memory is used to store program instructions;

[0054] The processor is used to call program instructions stored in memory to implement a data transmission method described in the first aspect.

[0055] Fifthly, a computer-readable storage medium is provided, which stores computer program instructions that, when executed, implement a data transmission method described in the first aspect.

[0056] Sixthly, a computer program product is provided, including a computer program that, when executed by a processor, implements a data transmission method described in the first aspect.

[0057] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows:

[0058] 1. By implementing the data transmission method disclosed in the embodiments of the present invention, the data transmission speed can be improved and the waiting time consumed by data query can be reduced;

[0059] 2. By periodically checking the execution status of data transmission tasks, the thread pool can be scaled up or down in a timely manner to ensure that the server executes data transmission tasks with optimal performance;

[0060] 3. Real-time feedback on download speed and remaining data volume allows users to stay informed about the data transmission progress. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of a data transmission method provided in an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of a data transmission device provided in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of a computer device implementing a data transmission method according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of another data transmission device provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The numbers in the accompanying drawings are only used to distinguish individual functional parts or modules and do not indicate logical relationships between parts or modules. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the term encompasses the element or object listed following the term and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. “Above,” “below,” “left,” “right,” etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0068] The various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that in the drawings, the same reference numerals are assigned to components having substantially the same or similar structure and function, and repeated descriptions of them will be omitted.

[0069] To address the problem of excessively long data transmission times, especially when handling large data volumes and large file sizes, in existing technologies, this invention provides a data transmission method, apparatus, device, and readable storage medium to improve data transmission speed and reduce data query waiting time.

[0070] In one embodiment, such as Figure 1 As shown, a data transmission method, applied to a server, includes:

[0071] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task, wherein the data transmission task corresponds to the data transmission request;

[0072] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0073] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0074] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0075] S150: Return the data file to the client.

[0076] In another embodiment, S110: Obtaining a data transmission request, generating data retrieval conditions based on the data transmission request, and initializing the data transmission task specifically includes:

[0077] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0078] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0079] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0080] The number of CPU cores and the amount of RAM on the server at the initialization time can be obtained by executing operating system commands. The probing code is used during the data transmission task initialization phase to estimate the resources required by the server to execute the data transmission task. Based on the resource consumption assessment, combined with the current number of CPU cores and the amount of RAM on the server, the server can create a thread pool and deploy a certain number of active threads in the thread pool to execute the data transmission task. This application does not limit the specific code representation of the probing code.

[0081] Alongside active threads, the thread pool also deploys daemon threads and thread barriers. The daemon threads monitor parameters such as server CPU utilization, remaining server memory, off-heap memory, on-heap memory, number of active threads, and number of CPU cores. They periodically calculate server load and manage the scaling of active threads to ensure high-performance data transfer on the server while balancing server performance and resources, preventing server crashes due to these data transfer tasks. The thread barriers isolate downloaded sub-files. Once all sub-files are downloaded, sorting and summarizing operations are performed based on the partition key number to ultimately generate the data file corresponding to the target data.

[0082] S120: Based on the data retrieval conditions, obtain the corresponding target data's indicator data, calculate the target data's feature score based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature score to obtain sub-data. Specifically, this includes:

[0083] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0084] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0085] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0086] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0087] The target data is retrieved based on data retrieval criteria, which are generated based on the data transmission request. The target data typically consists of hundreds of millions of data entries, occupying gigabytes of storage space. This application does not limit the size of the target data; the data transmission method disclosed in this application can also be used for smaller target data. The significant advantage of this application lies in saving transmission time when transmitting large amounts of target data.

[0088] Segmenting target data and converting it into sub-data for transmission is a crucial step in improving data transmission speed. Therefore, it is necessary to analyze the target data, extract features, and optimize the data segmentation method to improve data transmission efficiency. Typically, the target data's metrics include: total data volume, data time span, data number span, individual data record size and occupancy, total data volume and occupancy, average distribution value per day, average distribution value per hundredth of data number, current remaining memory, current number of active threads, current CPU utilization, whether data numbers are auto-incrementing, and whether data time is indexed. Preset feature scoring rules include, but are not limited to: interval scoring, veto scoring, deduction scoring, and interest rate scoring. Mapping the above metric data to feature scoring rules yields the score after evaluation by the feature scoring rules. Typically, feature scoring rules convert metric data into metric scores based on the data range in which the metric data falls.

[0089] Segmentation rules include, but are not limited to: equal data segmentation, time span segmentation, and numbered step-by-step segmentation. Each segmentation rule corresponds to a set of weighting factors for the scores of the aforementioned indicators. Under a given segmentation rule, there is a defined set of weighting factors, each corresponding to multiple indicator scores. The weighting factors can be preset and adjusted, and the corresponding score indicators can also be further optimized. Under this segmentation rule, a weighted average of the indicator scores using the weighting factors can be obtained to obtain the weighted score corresponding to that segmentation rule.

[0090] Taking the weighting factor under the equal data partitioning method as an example:

[0091] The weighting of this segmentation method is 0.8 for the total data volume score, 0.6 for the score of the size of a single data entry, 0.5 for the score of the total data volume score, 0.8 for the score of the current remaining memory, 0.8 for the score of the current number of active threads, and 0.3 for the score of the current CPU utilization.

[0092] Taking the weighted score of the equal data partitioning method as an example, the weighted score can be expressed as:

[0093]

[0094] Among them, W m The weighted scores represent the scores for equal data splitting methods. a1 represents the total data score, a2 represents the score for the size of a single data entry, a3 represents the score for the total data size and its usage, a4 represents the score for the current remaining memory, a5 represents the current number of active threads, a6 represents the current CPU utilization, k1 represents the weight of the total data score, k2 represents the weight of the score for the size of a single data entry, k3 represents the weight of the score for the total data size and its usage, k4 represents the weight of the score for the current remaining memory, k5 represents the weight of the current number of active threads, and k6 represents the weight of the current CPU utilization.

[0095] Similarly, weighted scores can be calculated for time span segmentation and numbered step-by-step segmentation. The weighted scores of each segmentation rule are then ranked sequentially, and the segmentation rule with the highest weighted score best reflects the characteristics of the target data. Segmenting the target data using this rule yields the best optimization effect for data transmission.

[0096] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0097] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0098] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0099] S133: Adjust the number of threads based on the current server load.

[0100] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0101] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0102] During sub-data transmission, dynamic scaling management of active threads can optimize server-side data transmission performance and prevent server crashes during data transmission tasks.

[0103] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data.

[0104] Due to the existence of thread barriers, after all partition subfiles have completed sub-data transfer, NIO (New IO) multiplexing is used to sort and summarize the subfiles.

[0105] S150: Return the data file to the client.

[0106] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0107] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0108] S162: Calculate the download speed based on the current number of active threads;

[0109] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0110] S164: Returns the download speed and remaining task quantity to the client.

[0111] By leveraging the atomicity and visibility among active threads, task progress-related metrics (download speed and remaining task volume) can be updated. These updated metrics are then fed back to the client, allowing users to stay informed about the data transmission progress.

[0112] In another embodiment, a data transmission method, applied to a client, includes:

[0113] S210: Send a data transmission request to the server so that the server can obtain the corresponding data file according to the data transmission request, wherein the data file includes the target data corresponding to the data transmission request;

[0114] S220: Obtain the data file returned by the server, and / or the download address for downloading the data file;

[0115] S230: Get the download speed and remaining task quantity returned by the server.

[0116] In another embodiment, such as Figure 2 As shown, a data transmission device includes:

[0117] The request acquisition module is used to acquire data transmission requests, generate data retrieval conditions based on data transmission requests, and initialize data transmission tasks, wherein the data transmission tasks correspond to the data transmission requests.

[0118] The data segmentation module is used to obtain the indicator data of the corresponding target data according to the data retrieval conditions, calculate the feature score of the target data based on the indicator data, and segment the target data according to the feature score to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span;

[0119] The data writing module is used to transfer sub-data through active threads and write the sub-data to the corresponding sub-files, wherein the sub-files include the partition key number corresponding to the sub-data;

[0120] The file generation module is used to sort sub-files according to partition key numbers, summarize the sub-data recorded in the sub-files, and generate a data file corresponding to the target data.

[0121] The file return module is used to return data files to the client.

[0122] In a preferred embodiment, the device further includes:

[0123] The task monitoring module is used to obtain the current number of active threads and the amount of data transferred at every second time interval; calculate the download speed based on the current number of active threads; calculate the remaining task volume based on the amount of data transferred and the total amount of target data; and return the download speed and remaining task volume to the client.

[0124] In another embodiment, such as Figure 3 As shown, an electronic device includes:

[0125] Processor and memory;

[0126] Memory is used to store program instructions;

[0127] The processor is used to call program instructions stored in memory to implement a data transfer method described in the first aspect, specifically including:

[0128] In another embodiment, a computer-readable storage medium stores computer program instructions that, when executed, implement a data transmission method described in the first aspect.

[0129] In another embodiment, a computer program product includes a computer program, which, when executed by a processor, describes a data transmission method in the first aspect.

[0130] By implementing the data transmission method disclosed in this embodiment of the invention, the data transmission speed can be improved and the waiting time consumed by data query can be reduced; by periodically checking the execution status of the data transmission task, the thread pool can be scaled up or down in a timely manner to ensure that the server executes the data transmission task with optimal performance; and the download speed and remaining data volume are fed back in real time so that users can understand the data transmission progress in a timely manner.

[0131] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of the present invention, and will not be described in detail here.

[0132] Example 1

[0133] like Figure 1 As shown, a data transmission method, applied to a server, includes:

[0134] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task, wherein the data transmission task corresponds to the data transmission request;

[0135] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0136] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0137] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0138] S150: Return the data file to the client.

[0139] Example 2

[0140] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task; wherein, the data transmission task corresponds to the data transmission request;

[0141] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0142] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0143] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0144] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0145] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0146] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0147] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0148] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0149] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0150] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0151] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0152] S133: Adjust the number of threads based on the current server load.

[0153] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0154] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0155] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0156] S150: Return the data file to the client.

[0157] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0158] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0159] S162: Calculate the download speed based on the current number of active threads;

[0160] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0161] S164: Returns the download speed and remaining task quantity to the client.

[0162] Example 3

[0163] S210: Send a data transmission request to the server so that the server can obtain the corresponding data file according to the data transmission request. The data file includes the target data corresponding to the data transmission request. Specifically, this includes:

[0164] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task; wherein, the data transmission task corresponds to the data transmission request;

[0165] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0166] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0167] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0168] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0169] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0170] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0171] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0172] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0173] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0174] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0175] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0176] S133: Adjust the number of threads based on the current server load.

[0177] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0178] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0179] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0180] S150: Return the data file to the client.

[0181] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0182] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0183] S162: Calculate the download speed based on the current number of active threads;

[0184] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0185] S164: Returns the download speed and remaining task quantity to the client.

[0186] S220: Obtain the data file returned by the server, and / or the download address for downloading the data file;

[0187] S230: Get the download speed and remaining task quantity returned by the server.

[0188] Example 4

[0189] like Figure 4 As shown, a data transmission device includes:

[0190] The request acquisition module is used to acquire data transmission requests, generate data retrieval conditions based on data transmission requests, and initialize data transmission tasks, wherein the data transmission tasks correspond to the data transmission requests.

[0191] The data segmentation module is used to obtain the indicator data of the corresponding target data according to the data retrieval conditions, calculate the feature score of the target data based on the indicator data, and segment the target data according to the feature score to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span;

[0192] The data writing module is used to transfer sub-data through active threads and write the sub-data to the corresponding sub-files, wherein the sub-files include the partition key number corresponding to the sub-data;

[0193] The file generation module is used to sort sub-files according to partition key numbers, summarize the sub-data recorded in the sub-files, and generate a data file corresponding to the target data.

[0194] The file return module is used to return data files to the client.

[0195] The device also includes:

[0196] The task monitoring module is used to obtain the current number of active threads and the amount of data transferred at every second time interval; calculate the download speed based on the current number of active threads; calculate the remaining task volume based on the amount of data transferred and the total amount of target data; and return the download speed and remaining task volume to the client.

[0197] Example 5

[0198] like Figure 3 As shown, an electronic device includes:

[0199] Processor and memory;

[0200] Memory is used to store program instructions;

[0201] The processor is used to call program instructions stored in memory to implement a data transfer method described in the first aspect, specifically including:

[0202] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task; wherein, the data transmission task corresponds to the data transmission request;

[0203] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0204] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0205] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0206] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0207] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0208] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0209] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0210] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0211] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0212] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0213] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0214] S133: Adjust the number of threads based on the current server load.

[0215] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0216] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0217] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0218] S150: Return the data file to the client.

[0219] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0220] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0221] S162: Calculate the download speed based on the current number of active threads;

[0222] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0223] S164: Returns the download speed and remaining task quantity to the client.

[0224] Example 6

[0225] A computer-readable storage medium stores computer program instructions, which, when executed, implement a data transmission method as described in the first aspect, specifically including:

[0226] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task; wherein, the data transmission task corresponds to the data transmission request;

[0227] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0228] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0229] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0230] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0231] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0232] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0233] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0234] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0235] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0236] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0237] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0238] S133: Adjust the number of threads based on the current server load.

[0239] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0240] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0241] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0242] S150: Return the data file to the client.

[0243] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0244] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0245] S162: Calculate the download speed based on the current number of active threads;

[0246] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0247] S164: Returns the download speed and remaining task quantity to the client.

[0248] Example 7

[0249] A computer program product includes a computer program, which, when executed by a processor, includes a data transmission method described in the first aspect, specifically comprising:

[0250] S110: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task; wherein, the data transmission task corresponds to the data transmission request;

[0251] S111: Get the number of CPU cores and the amount of running memory on the server at the initialization time;

[0252] S112: Execute the preset probe code to assess the service resources consumed by the data transmission task;

[0253] S113: Generate active threads based on the number of CPU cores, the amount of RAM, and service resources.

[0254] S120: Based on the data retrieval conditions, obtain the indicator data of the corresponding target data, calculate the feature score of the target data based on the indicator data, and segment the target data according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span.

[0255] S121: Use preset feature scoring rules to evaluate indicator data and obtain indicator scores corresponding to the indicator data;

[0256] S122: Calculate the weighted scores corresponding to all segmentation rules in sequence;

[0257] S1221: Poll the segmentation rules, obtain the indicator weights corresponding to the current segmentation rules, and obtain the weighted score corresponding to the current segmentation rules based on the indicator weights and indicator scores.

[0258] S123: Sort the weighted scores, obtain the maximum weighted score, and use the maximum weighted score as the feature score.

[0259] S130: Transmit sub-data through an active thread and write the sub-data to the corresponding sub-file, wherein the sub-file includes the partition key number corresponding to the sub-data;

[0260] S131: At the first time interval, use the probe code to obtain the current environmental parameters;

[0261] S132: Calculate and obtain the current server pressure value based on the current environment parameters;

[0262] S133: Adjust the number of threads based on the current server load.

[0263] S1331: When the server pressure value is greater than or equal to the first threshold, add a first number of new threads;

[0264] S1331′: When the server pressure value is less than or equal to the second threshold, close the second number of newly generated idle threads.

[0265] S140: Sort the subfiles according to the partition key number, summarize the subdata recorded in the subfiles, and generate a data file corresponding to the target data;

[0266] S150: Return the data file to the client.

[0267] Specifically, this includes: transmitting data files to the client in real time, and / or uploading data files to the server, and sending the download address of the data files to the client.

[0268] S161: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted;

[0269] S162: Calculate the download speed based on the current number of active threads;

[0270] S163: Calculate the remaining task volume based on the amount of data already transmitted and the total amount of target data;

[0271] S164: Returns the download speed and remaining task quantity to the client.

[0272] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program loaded on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from memory, or installed from ROM. When the computer program is executed by an external processor, it performs the functions defined in the methods of embodiments of this application.

[0273] It should be noted that the computer-readable medium in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0274] The aforementioned computer-readable medium may be included in the aforementioned server; or it may exist independently and not assembled into the server. The aforementioned computer-readable medium carries one or more programs that, when executed by the server, cause the server to: in response to detecting that the peripheral mode of the terminal is not activated, acquire the frame rate of the application on the terminal; when the frame rate meets the screen-off condition, determine whether the user is acquiring the terminal's screen information; and in response to the determination that the user is not acquiring the terminal's screen information, control the screen to enter an immediate dimming mode.

[0275] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0276] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0277] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0278] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data transmission method, characterized in that, The method is applied to the server side and includes: Obtain a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task, wherein the data transmission task corresponds to the data transmission request; According to the data retrieval conditions, the corresponding target data indicator data is obtained, the feature score of the target data is calculated based on the indicator data, and the target data is segmented according to the data segmentation rules corresponding to the feature scores to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span; The sub-data is transmitted through an active thread and written to a corresponding sub-file, wherein the sub-file includes a partition key number corresponding to the sub-data; The sub-files are sorted according to the partition key number, and the sub-files are summarized in turn to generate a data file corresponding to the target data; Return the data file to the client; The step of calculating the feature score of the target data based on the indicator data includes: The indicator data is evaluated using preset feature scoring rules to obtain the indicator score corresponding to the indicator data; the weighted score corresponding to all segmentation rules is calculated in sequence; the weighted scores are sorted to obtain the maximum weighted score, and the maximum weighted score is used as the feature score.

2. The data transmission method according to claim 1, characterized in that, The step of calculating the weighted score corresponding to all segmentation rules in sequence includes: The segmentation rules are polled to obtain the indicator weights corresponding to the current segmentation rules, and the weighted score corresponding to the current segmentation rules is obtained based on the indicator weights and the indicator scores.

3. The data transmission method according to claim 1, characterized in that, The initial data transmission task includes: Obtain the number of CPU cores and the amount of running memory of the server at the initialization time; Execute preset detection code to assess the service resources consumed by the data transmission task; Active threads are generated based on the number of CPU cores, the in-run memory, and the service resources.

4. The data transmission method according to claim 1, characterized in that, Returning the data file to the client includes: The data file is transmitted to the client in real time. and / or The data file is uploaded to the server, and the download address of the data file is sent to the client.

5. The data transmission method according to claim 1, characterized in that, The step of transmitting the sub-data through an active thread and writing the sub-data into the corresponding sub-file further includes: At each first time interval, use the probe code to obtain the current environmental parameters; Calculate and obtain the current server stress value based on the current environmental parameters; Adjust the number of threads based on the current server load value.

6. A data transmission method according to claim 5, characterized in that, The step of adjusting the number of threads based on the current server load value includes: When the server load value is greater than or equal to the first threshold, add a first number of new threads; When the server pressure value is less than or equal to the second threshold, a second number of newly generated idle threads are shut down.

7. A data transmission method according to any one of claims 1-6, characterized in that, The method further includes: At preset time intervals, obtain the current number of active threads and the amount of data already transmitted; The download speed is calculated based on the number of currently active threads. The remaining task volume is calculated based on the amount of data already transmitted and the total amount of the target data. The download speed and the remaining task amount are returned to the client.

8. A data transmission method, characterized in that, The method is applied to the client and includes: A data transmission request is sent to the server so that the server can obtain the corresponding data file according to the data transmission request, wherein the data file includes the target data corresponding to the data transmission request; Retrieve the data file returned by the server. and / or Download address for downloading the data file; Obtain the download speed and remaining task quantity returned by the server; The server-side process for obtaining the corresponding data file based on the data transmission request includes: obtaining the data transmission request; generating data retrieval conditions based on the data transmission request and initializing a data transmission task, wherein the data transmission task corresponds to the data transmission request; obtaining indicator data of the corresponding target data based on the data retrieval conditions; calculating the feature score of the target data based on the indicator data; and segmenting the target data according to the data segmentation rules corresponding to the feature score to obtain sub-data; wherein the indicator data includes: total data volume, data time span, and data number span; transmitting the sub-data through an active thread and writing the sub-data into corresponding sub-files, wherein the sub-files include partition key numbers corresponding to the sub-data; sorting the sub-files according to the partition key numbers; sequentially summarizing the sub-files to generate a data file corresponding to the target data; and returning the data file to the client. The step of calculating the feature score of the target data based on the indicator data includes: The indicator data is evaluated using preset feature scoring rules to obtain the indicator score corresponding to the indicator data; the weighted score corresponding to all segmentation rules is calculated in sequence; the weighted scores are sorted to obtain the maximum weighted score, and the maximum weighted score is used as the feature score.

9. A data transmission device, characterized in that, The device includes: The request acquisition module is used to acquire a data transmission request, generate data retrieval conditions based on the data transmission request, and initialize a data transmission task, wherein the data transmission task corresponds to the data transmission request; The data segmentation module is used to obtain the indicator data of the corresponding target data according to the data retrieval conditions, calculate the feature score of the target data according to the indicator data, and segment the target data according to the feature score to obtain sub-data; wherein, the indicator data includes: total data volume, data time span, and data number span; The data writing module is used to transmit the sub-data through an active thread and write the sub-data into a corresponding sub-file, wherein the sub-file includes a partition key number corresponding to the sub-data; The file generation module is used to sort the sub-files according to the partition key number, summarize the sub-files in sequence, and generate a data file corresponding to the target data; The file return module is used to return the data file to the client; The step of calculating the feature score of the target data based on the indicator data includes: The indicator data is evaluated using preset feature scoring rules to obtain the indicator score corresponding to the indicator data; the weighted score corresponding to all segmentation rules is calculated in sequence; the weighted scores are sorted to obtain the maximum weighted score, and the maximum weighted score is used as the feature score.

10. A data transmission device according to claim 9, characterized in that, The device further includes: The task monitoring module is used to obtain the current number of active threads and the amount of data transmitted at every second time interval; calculate the download speed based on the current number of active threads; calculate the remaining task volume based on the amount of data transmitted and the total amount of the target data; and return the download speed and the remaining task volume to the client.

11. An electronic device, characterized in that, include: Processor and memory; The memory is used to store program instructions; The processor is used to call program instructions stored in the memory to implement a data transmission method as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement a data transmission method according to any one of claims 1 to 7.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements a data transmission method according to any one of claims 1 to 7.

Images