Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of ticket processing, and provides a data processing method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a preliminary prediction step length according to a time range of a list generation task; determining a starting moment and an ending moment of the first segment according to the preliminary prediction step length and a time range of the list generation task, and performing query by taking the starting moment and the ending moment as query conditions; if the queried data size exceeds a preset number threshold value, obtaining a target prediction step length by adopting a degradation strategy, and obtaining a second segment based on the target prediction step length; and querying the data volume again according to the second segment to obtain a target data volume which does not exceed a preset quantity threshold. According to the method, the inventory data is inquired and processed in a segmented mode by configuring different step lengths, the inventory data volume is limited within the machine processing performance range, and the machine fault risk is reduced; meanwhile, the situation that prediction of the prediction model is inaccurate is compensated through a degradation strategy, and complete execution of the task is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of call bill processing, and particularly relates to a data processing method and device, electronic equipment, a storage medium and a program product. BACKGROUND

[0002] At present, in various schemes of call bill query, user requirements cannot be properly completed. For example, some schemes read and process generated call bill files in real time to generate detailed bill files, and improve query speed by establishing indexes. The schemes focus on improving query efficiency and do not involve detailed bill original data query of a distributed database. For example, some schemes pull detailed bill original data through a timing task. In a peak period, the data volume is huge, and there is a risk of machine downtime. For example, some schemes write detailed bill original data into a database after preprocessing (for example, designing rowKey), and have the problem of low universality.

[0003] The above scheme in the prior art focuses on improving query efficiency and does not involve detailed bill original data query of a distributed database. The scheme has the problem of low universality, and when the server capacity is exceeded, the scheme is prone to cause server failure. SUMMARY

[0004] The present application provides a data processing method and device, electronic equipment, a storage medium and a program product, which solve the problems of exceeding server capacity, causing server failure and low universality during list data query processing in a distributed database.

[0005] The present application provides a data processing method, which comprises: determining a preliminary prediction step length according to a time range of a list generation task; determining a starting time and an ending time of a first segment according to the preliminary prediction step length and the time range of the list generation task; querying the starting time and the ending time of the first segment as a query condition, and counting a data volume in the first segment range; if the data volume obtained by the query exceeds a preset quantity threshold, a degradation strategy is used to obtain a target prediction step length, and a second segment is obtained based on the target prediction step length; the data volume is queried again according to the second segment, and a target data volume that does not exceed the preset quantity threshold is obtained.

[0006] According to a data processing method provided by the present invention, determining a preliminary prediction step size based on the time range of a list generation task includes: forming a list generation task based on a time range selected by the user; sending a prediction request to a prediction module based on the list generation task, and the prediction module returning a preliminary prediction step size based on the prediction request; wherein the prediction module includes a model trained based on a BP neural network; after querying the data volume again based on the second segment and obtaining a target data volume that does not exceed a preset quantity threshold, the method further includes: collecting the target prediction step size as a data training set; training the BP neural network based on the data training set, and updating the old weights to new weights after training to update the BP neural network.

[0007] According to a data processing method provided by the present invention, a degradation strategy is used to obtain a target prediction step size, including: obtaining a downgraded step size by proportional degradation; if the amount of data obtained based on the downgraded step size does not exceed a preset quantity threshold, then the proportionally downgraded step size is used as the target prediction step size; if the amount of data obtained based on the downgraded step size exceeds the preset quantity threshold, then a step-down strategy is used to obtain the target prediction step size.

[0008] According to a data processing method provided by the present invention, a step-down strategy is used to obtain a target prediction step size, comprising: pre-setting a step size array; wherein the step size array includes step sizes arranged from largest to smallest; wherein each step size in the step size array is smaller than the step size after downgrading; taking the largest step size from the step size array as a candidate step size; using the candidate step size to query the data volume, stopping if the data volume does not exceed a preset data volume threshold; if the data volume exceeds the preset data volume, using the step size array to find the next candidate step size until the data volume does not exceed the preset data volume threshold or reaches the smallest step size in the step size array, so that the candidate step size when the data volume does not exceed the preset data volume threshold is used as the target prediction step size or the smallest step size is used as the target prediction step size.

[0009] According to a data processing method provided by the present invention, determining the start and end times of a first segment based on a preliminary prediction step length includes: dividing the time range of a list generation task into at least one time segment based on the preliminary prediction step length; the at least one time segment includes the first segment; taking the start time of the time range of the list generation task as the start time of the first segment, and extending the start time of the first segment backward by the preliminary prediction step length as the end time of the first segment.

[0010] According to a data processing method provided by the present invention, after querying the start time and end time of the first segment as query conditions and counting the amount of data within the first segment, the method further includes: if the amount of data obtained by query does not exceed a preset quantity threshold, the amount of data obtained by query is directly used as the target data amount.

[0011] According to a data processing method provided by the present invention, the method further includes: determining a second preliminary prediction step size based on a first segment or a second segment; determining the start and end times of a third segment based on the time range of the task generated by the second preliminary prediction step size list; querying the start and end times of the third segment as query conditions to count the amount of data within the third segment; if the amount of data within the third segment obtained by the query exceeds a preset quantity threshold, adopting a degradation strategy to obtain a second target prediction step size, obtaining a fourth segment based on the second target prediction step size, and querying the amount of data again based on the fourth segment to obtain a second target data amount that does not exceed the preset quantity threshold; if the amount of data within the third segment obtained by the query does not exceed the preset quantity threshold, directly using the amount of data within the third segment obtained by the query as the second target data amount.

[0012] The present invention also provides a data processing apparatus, comprising: a preliminary prediction step size module, used to determine a preliminary prediction step size based on the time range of the list generation task; a first segment determination module, used to determine the start and end times of the first segment based on the preliminary prediction step size and the time range of the list generation task; a data volume query module, used to query the start and end times of the first segment as query conditions and count the data volume within the first segment; a target prediction step size module, used to obtain a target prediction step size by adopting a degradation strategy if the queried data volume exceeds a preset quantity threshold, and to obtain a second segment based on the target prediction step size; and a target data volume module, used to query the data volume again based on the second segment to obtain a target data volume that does not exceed the preset quantity threshold.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.

[0016] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product. The data processing method includes: determining a preliminary prediction step size based on the time range of the inventory generation task; determining the start and end times of a first segment based on the preliminary prediction step size and the time range of the inventory generation task; querying the first segment using the start and end times as query conditions to count the amount of data within the first segment; if the amount of data obtained exceeds a preset threshold, using a degradation strategy to obtain a target prediction step size, and obtaining a second segment based on the target prediction step size; and querying the data amount again based on the second segment to obtain a target data amount that does not exceed the preset threshold. Through this method, this invention limits the amount of inventory data within the machine's processing capacity by configuring different step sizes for segmented querying and processing, reducing the risk of machine failure; simultaneously, the degradation strategy compensates for inaccurate predictions by the prediction model, ensuring the complete execution of the task. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the data processing method provided in this embodiment of the invention.

[0019] Figure 2 This is a schematic diagram of the system framework of the data processing method provided in the embodiments of the present invention.

[0020] Figure 3 This is a schematic diagram of the prediction model training process provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of a three-layer BP neural network provided in an embodiment of the present invention.

[0022] Figure 5 This is the second flowchart of the data processing method provided in the embodiments of the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the data processing device provided in an embodiment of the present invention.

[0024] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0027] This invention provides a data processing method that can reduce server load and improve data extraction efficiency; furthermore, it can query data in segments from a distributed database based on ordered fields, and further improve data processing speed while ensuring the stability of the data system through real-time prediction.

[0028] Please see Figure 1 , Figure 1 This is one of the flowcharts illustrating the data processing method provided in this embodiment of the invention. In this embodiment, the data processing method may include steps S110 to S150, each step of which is as follows: S110: Determine the initial prediction step size based on the time range of the tasks generated from the inventory.

[0029] S120: Determine the start and end times of the first segment based on the preliminary predicted step size and the time range of the inventory generation task.

[0030] S130: Use the start and end times of the first segment as query conditions to query and count the amount of data within the first segment.

[0031] S140: If the amount of data obtained from the query exceeds the preset quantity threshold, a degradation strategy is adopted to obtain the target prediction step size, and the second segment is obtained based on the target prediction step size.

[0032] S150: Based on the second segment, query the data volume again to obtain the target data volume that does not exceed the preset quantity threshold.

[0033] This embodiment proposes a data processing method. The invention queries and processes call detail record (CDR) data in segments by configuring different step sizes, limiting the amount of data processed each time to below the machine's maximum processing capacity. This reduces the cost of machine expansion, lowers the risk of machine failure, and solves the problem of server failure caused by exceeding server capacity during detailed call detail record (CDR) data query processing in distributed database scenarios. Simultaneously, a degradation strategy compensates for inaccurate predictions by the forecasting model, ensuring the complete execution of the task. Furthermore, an ordered field is used as the segmentation benchmark, addressing the issue of low versatility in related technologies.

[0034] In some embodiments, the step of determining the start and end times of the first segment based on the preliminary prediction step size may specifically include: The time range of the inventory generation task is divided into at least one time segment based on the preliminary prediction step size; the at least one time segment includes a first segment; the start time of the time range of the inventory generation task is used as the start time of the first segment, and the start time of the first segment is extended by the preliminary prediction step size to become the end time of the first segment. In some embodiments, the steps after using the start and end times of the first segment as query conditions to query and count the amount of data within the first segment may further include: If the amount of data retrieved does not exceed the preset threshold, the amount of data retrieved will be used as the target amount of data.

[0035] In some embodiments, the steps of the data processing method may further include the following: The second preliminary prediction step size is determined based on the first or second segment; the start and end times of the third segment are determined based on the time range of the task generated from the second preliminary prediction step size list; the start and end times of the third segment are used as query conditions to query and count the amount of data within the third segment; if the amount of data within the third segment obtained by the query exceeds a preset threshold, a degradation strategy is adopted to obtain the second target prediction step size, the fourth segment is obtained based on the second target prediction step size, and the amount of data is queried again based on the fourth segment to obtain the second target data amount that does not exceed the preset threshold; if the amount of data within the third segment obtained by the query does not exceed the preset threshold, the amount of data within the third segment obtained by the query is directly used as the second target data amount.

[0036] The data processing method of this invention can be applied to call detail record (CDR) queries, enabling a segmented CDR generation method based on real-time prediction. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the system framework of the data processing method provided in the embodiments of the present invention.

[0037] In this embodiment, the system framework may include a detailed transaction record extraction and processing module, a prediction module, and a monitoring and configuration module.

[0038] The detailed bill extraction and processing module can connect to a distributed database to obtain detailed bill data. The prediction module is connected to both the detailed bill extraction and processing module and the monitoring and configuration module. The detailed bill extraction and processing module can send prediction requests to the prediction module or return training data. The prediction module can send prediction results to the detailed bill extraction and processing module. The prediction module can send status data to the monitoring and configuration module. The monitoring and configuration module can send configuration requests to the prediction module or request to obtain status information.

[0039] Specifically, the detailed data extraction and processing module retrieves detailed data tasks from the task table and extracts the conditions (including start and end times) for the user's required detailed data from the tasks. Through interaction with the prediction module, it completes the function of segmented extraction and processing of the raw detailed data.

[0040] For example, the detailed order extraction and processing module can obtain the initial prediction step size by sending a prediction request to the prediction module, and segment the detailed order time range required by the user, so as to retrieve data from the database in segments (if the amount of data in the segment range exceeds the quantity threshold, a degradation strategy is further adopted for strategy compensation).

[0041] For example, the detailed order extraction and processing module can transmit the actual step size used each time and the detailed order-related information back to the prediction module as training data.

[0042] For example, the detailed order extraction and processing module can write the generated detailed order data to a file and update the task status.

[0043] The monitoring and configuration module displays the prediction module's operational data and configures parameters through a visual interface. The operational data includes recent prediction data, actual data, and prediction accuracy. Administrators can also modify the training set size of the prediction model via this interface.

[0044] It should be noted that the above process description uses a time field as the basis for segmentation. In fact, any ordered field can be used as the basis for segmentation.

[0045] The prediction module receives prediction requests from the detailed data extraction and processing module, extracts the parameters from the requests as input parameters for the input layer, and obtains the initial prediction step size.

[0046] Based on any of the above embodiments, the step of determining the preliminary prediction step size according to the time range of the inventory generation task may specifically include: A list generation task is generated based on the time range selected by the user; a prediction request is sent to the prediction module based on the list generation task, and the prediction module returns a preliminary prediction step size based on the prediction request; wherein, the prediction module includes a model trained based on a BP neural network.

[0047] The steps following the process of querying the data volume again based on the second segment to obtain the target data volume that does not exceed the preset threshold may specifically include: Collect the target prediction step size as the data training set; train the BP neural network based on the data training set, and update the old weights with the new weights after training to realize the update of the BP neural network.

[0048] Please see Figures 3-4 , Figure 3 This is a schematic diagram of the prediction model training process provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a three-layer BP neural network provided in an embodiment of the present invention.

[0049] A three-layer backpropagation (BP) neural network consists of an input layer, hidden layers, and an output layer. The input parameters of the input layer can include system TPS, user ID, application ID, quantity threshold, time point, day of the week, etc. The output parameter of the output layer is the initial prediction step size.

[0050] During the training of a three-layer BP neural network, in the initial stage, old data can be used to train the network and obtain the network model. After the prediction module interacts with the detailed data extraction and processing module, the detailed data extraction and processing module will transmit the actual data (such as system TPS, user ID, application ID, quantity threshold, time point, weekday, actual step size, etc.) to the prediction module after each data extraction for the neural network to train.

[0051] The prediction module collects the prediction step size and the actual step size used by the detailed list extraction and processing module for each prediction, thereby calculating the accuracy of the model prediction. The prediction module also collects the data transmitted from the detailed list extraction and processing module as a training set. After a certain number of training sets are collected, a separate thread is started to train the model. After training is completed, the old weights are updated to the new weights.

[0052] like Figure 3 As shown, during training, the prediction model receives prediction requests from the detailed data extraction and processing module and returns the prediction step size (i.e., the initial prediction step size). Afterward, the prediction model can receive real data from the detailed data extraction and processing module, calculate the accuracy, and add the data to the training set.

[0053] Once the training set reaches a certain size, the accuracy can be calculated and data can be added to the training set, which is the data used for training in a separate thread. After training is complete, the neural network weights can be updated.

[0054] Based on any of the above embodiments, the step of obtaining the target prediction step size using a degradation strategy may specifically include: The step size after proportional degradation is obtained; if the amount of data obtained from the step size after degradation does not exceed the preset threshold, the step size after proportional degradation is used as the target predicted step size; if the amount of data obtained from the step size after degradation exceeds the preset threshold, the step degradation strategy is adopted to obtain the target predicted step size.

[0055] Based on any of the above embodiments, the step of obtaining the target prediction step size using a step-down strategy may specifically include: A step size array is pre-set, containing step sizes arranged from largest to smallest. Each step size in the step size array is smaller than the downgraded step size. The largest step size in the step size array is taken as a candidate step size. The candidate step size is used to query the data volume. If the data volume does not exceed the preset data volume threshold, the query stops. If the data volume exceeds the preset data volume, the step size array is used to find the next candidate step size until the data volume does not exceed the preset data volume threshold or the minimum step size in the step size array is reached. The candidate step size when the data volume does not exceed the preset data volume threshold is used as the target prediction step size, or the minimum step size is used as the target prediction step size.

[0056] Furthermore, a preliminary step size array can be set up first, which includes multiple step sizes with different values. Elements with values ​​smaller than the step size after degradation can be selected from the preliminary step size array to construct the next step size array.

[0057] It should be noted that the degradation strategy is a compensation strategy for situations where the neural network's predictions are inaccurate. In this embodiment, the degradation strategy can include two types: 1. proportional degradation; 2. step degradation. The two degradation strategies can be used individually or in combination, and those skilled in the art can design them according to actual needs.

[0058] For example, when the prediction of the neural network in the prediction module is inaccurate, a proportional degradation strategy is first used, and the step size after degradation is calculated using the following formula: .

[0059] Next, the data volume is queried using this downgraded step size. If it is less than the data volume threshold, the process stops. If it is greater than the data volume threshold, a step-down strategy is adopted. The step-down strategy involves pre-setting a step size array (arranged from smallest to largest). When the step size after proportional downgrading still does not meet the data volume threshold, a new step size is taken from the pre-set step size array (this step size meets the following condition: this step size is the largest step size in the step size array that is smaller than the old step size). Then, the data volume is queried using this downgraded step size. If it is less than the data volume threshold, the process stops; otherwise, the above rules are used to find the next step size until the threshold condition is met or the minimum step size is reached.

[0060] Please see Figure 5 , Figure 5 This is the second flowchart of the data processing method provided in the embodiments of the present invention.

[0061] 1. User initiates a request to generate detailed bill and enters it into the task table: The user selects the time range for which they want to generate detailed bill in the portal and clicks the Generate Detailed Bill button. The backend receives the request and forms a detailed bill generation task, which is then entered into the task table.

[0062] 2. Task Detail Extraction and Processing Module: The task detail extraction and processing module periodically retrieves incomplete tasks from the task table.

[0063] 3. Processing of raw detailed bill data within the specified time frame: The detailed bill extraction module prepares to extract raw detailed bill data from the database based on the time frame of the user's required detailed bill data recorded in the task. If the raw detailed bill data within the user's required time frame has been processed, the process ends; otherwise, proceed to step 4.

[0064] 4. Request prediction step size: The detailed order extraction and processing module sends a prediction request to the prediction module, and the prediction module returns the prediction step size.

[0065] 5. Query the data volume from the database based on the step size: The detailed order extraction and processing module takes the start time of the detailed order required by the user as the start time of the first segment, and the predicted step size is taken as the end time of the first segment. The first segment is used as the query condition to query and count the data volume within the first segment.

[0066] 6. Data volume exceeds threshold: Determine whether the amount of data obtained from the query exceeds the quantity threshold. If the amount of data obtained from the query does not exceed the quantity threshold, skip step 7 and proceed directly to step 8; if the amount of data obtained from the query exceeds the quantity threshold, proceed to step 7.

[0067] 7. Calculate the new step size using a degradation strategy: If the amount of data obtained from the query exceeds the quantity threshold, a degradation strategy is adopted to move the end time of the first segment forward and query the data amount again. If the condition can be met, step 8 is executed; otherwise, the iteration continues until the amount of data obtained from the query does not exceed the quantity threshold.

[0068] 8. Query the original detailed bill data from the database according to the step size and process the data: The detailed bill extraction and processing module retrieves the specific original detailed bill data from the database, processes it into the detailed bill data required by the user, and appends it to the file.

[0069] In addition, the end time of the first segment can be used as the start time of the second segment. The detailed data extraction and processing module sends a prediction request to the prediction module to obtain the prediction step size. The start time of the second segment is then shifted backward by the prediction step size to become the end time of the second segment. Steps 6 to 8 are repeated until the query and processing of the detailed data within the time range required by the user is completed.

[0070] 9. Transmit the actual step size and other data to the prediction module.

[0071] 10. Output detailed data files: Display the prediction module's running data and configuration parameters through a visual page. The running data includes the latest prediction data, actual data, prediction accuracy, etc.

[0072] The solution adopted in this invention can reduce server load and improve data extraction efficiency in a distributed database scenario. Specifically, by using a segmentation approach, the amount of data processed each time is limited to below the machine's maximum processing capacity, reducing the risk of machine failure; a predictive model is introduced to predict the next segment, reducing database query operations, thereby reducing database load and data query time. Simultaneously, a degradation strategy is used to compensate for inaccurate predictions by the predictive model, ensuring the complete execution of the task.

[0073] The present invention also provides a data processing apparatus. The data processing apparatus provided by the present invention will be described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence.

[0074] Please see Figure 6 , Figure 6 This is a schematic diagram of the data processing device provided in an embodiment of the present invention. In this embodiment, the data processing device includes a preliminary prediction step size module 610, a first segmentation determination module 620, a data volume query module 630, a target prediction step size module 640, and a target data volume module 650.

[0075] The preliminary prediction step size module 610 is used to determine the preliminary prediction step size based on the time range of the tasks generated from the inventory.

[0076] The first segment determination module 620 is used to determine the start and end times of the first segment based on the preliminary predicted step size and the time range of the inventory generation task.

[0077] The data volume query module 630 is used to query the data volume within the first segment by using the start and end times of the first segment as query conditions.

[0078] The target prediction step size module 640 is used to obtain the target prediction step size by adopting a degradation strategy if the amount of data obtained by query exceeds a preset quantity threshold, and to obtain the second segment based on the target prediction step size.

[0079] The target data volume module 650 is used to query the data volume again based on the second segment to obtain the target data volume that does not exceed the preset quantity threshold.

[0080] In some embodiments, the preliminary prediction step size module 610 can be specifically used for: A list generation task is generated based on the time range selected by the user; a prediction request is sent to the prediction module based on the list generation task, and the prediction module returns a preliminary prediction step size based on the prediction request; wherein, the prediction module includes a model trained based on a BP neural network.

[0081] The data processing device also includes a model training module, which can be specifically used for: Collect the target prediction step size as the data training set; train the BP neural network based on the data training set, and update the old weights with the new weights after training to realize the update of the BP neural network.

[0082] In some embodiments, the target prediction step size module 640 can be specifically used for: The step size after proportional degradation is obtained; if the amount of data obtained from the step size after degradation does not exceed the preset threshold, the step size after proportional degradation is used as the target predicted step size; if the amount of data obtained from the step size after degradation exceeds the preset threshold, the step degradation strategy is adopted to obtain the target predicted step size.

[0083] In some embodiments, the target prediction step size module 640 can also be used for: A step size array is pre-set, containing step sizes arranged from largest to smallest. Each step size in the step size array is smaller than the downgraded step size. The largest step size in the step size array is taken as a candidate step size. The candidate step size is used to query the data volume. If the data volume does not exceed the preset data volume threshold, the query stops. If the data volume exceeds the preset data volume, the step size array is used to find the next candidate step size until the data volume does not exceed the preset data volume threshold or the minimum step size in the step size array is reached. The candidate step size when the data volume does not exceed the preset data volume threshold is used as the target prediction step size, or the minimum step size is used as the target prediction step size.

[0084] In some embodiments, the first segmentation determination module 620 may specifically be used for: The time range of the inventory generation task is divided into at least one time segment based on the preliminary prediction step size; the at least one time segment includes a first segment; the start time of the time range of the inventory generation task is taken as the start time of the first segment, and the start time of the first segment is extended backward by the preliminary prediction step size as the end time of the first segment.

[0085] In some embodiments, the data volume query module 630 can also be used for: If the amount of data retrieved does not exceed the preset threshold, the amount of data retrieved will be used as the target amount of data.

[0086] In some embodiments, the data processing module may further include a second target data volume module, which may be specifically used for: The second preliminary prediction step size is determined based on the first or second segment; the start and end times of the third segment are determined based on the time range of the task generated from the second preliminary prediction step size list; the start and end times of the third segment are used as query conditions to query and count the amount of data within the third segment; if the amount of data within the third segment obtained by the query exceeds a preset threshold, a degradation strategy is adopted to obtain the second target prediction step size, the fourth segment is obtained based on the second target prediction step size, and the amount of data is queried again based on the fourth segment to obtain the second target data amount that does not exceed the preset threshold; if the amount of data within the third segment obtained by the query does not exceed the preset threshold, the amount of data within the third segment obtained by the query is directly used as the second target data amount.

[0087] On the other hand, embodiments of the present invention also provide an electronic device, please refer to... Figure 7 , Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 7As shown, the electronic device may include: a memory 720, a processor 710, and a computer program stored in the memory 720 and executable on the processor 710. When the processor 710 executes the program, it implements the data processing methods provided by the methods described above.

[0088] Optionally, the electronic device may further include a communication bus 730 and a communication interface 740, wherein the processor 710, the communication interface 740, and the memory 720 communicate with each other via the communication bus 730. The processor 710 can call a computer program in the memory 720 to execute a data processing method, which may include: The initial prediction step size is determined based on the time range of the list generation task; the start and end times of the first segment are determined based on the initial prediction step size and the time range of the list generation task; the start and end times of the first segment are used as query conditions to query and count the amount of data within the first segment; if the amount of data obtained exceeds a preset threshold, a degradation strategy is adopted to obtain the target prediction step size, and the second segment is obtained based on the target prediction step size; the amount of data is queried again based on the second segment to obtain the target amount of data that does not exceed the preset threshold. Furthermore, the logical instructions in the aforementioned memory 720 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the data processing methods provided by the above methods. The steps and principles of these methods have been described in detail in the above methods and will not be repeated here.

[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing methods provided by the above methods. The steps and principles of the data processing methods have been described in detail in the above methods and will not be repeated here.

[0091] Non-transitory computer-readable storage media can be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized by, The method comprises the following steps: determining a preliminary prediction step according to the time range of the list generation task; determining the start time and the end time of the first segment according to the preliminary prediction step and the time range of the list generation task; querying the start time and the end time of the first segment as a query condition to count the data amount in the first segment range; if the data amount obtained by the query exceeds a preset amount threshold, a degradation strategy is used to obtain a target prediction step, and a second segment is obtained based on the target prediction step; the data amount is queried again according to the second segment to obtain a target data amount that does not exceed the preset amount threshold.

2. The data processing method according to claim 1, characterized in that, The method comprises the following steps: forming the list generation task according to the time range selected by a user; sending a prediction request to a prediction module according to the list generation task, and returning a preliminary prediction step by the prediction module according to the prediction request; wherein the prediction module comprises a model trained based on a BP neural network; after the data amount is queried again according to the second segment to obtain a target data amount that does not exceed the preset amount threshold, the method further comprises the following steps: collecting the target prediction step as a data training set; training the BP neural network according to the data training set, and updating the old weight to a new weight after the training is completed to realize the update of the BP neural network.

3. The data processing method of claim 1, wherein, The method comprises the following steps: obtaining a degraded step length by proportional degradation; if the data amount obtained according to the degraded step length does not exceed the preset amount threshold, the step length degraded by proportional degradation is taken as the target prediction step; if the data amount obtained according to the degraded step length exceeds the preset amount threshold, a step-down degradation strategy is used to obtain the target prediction step.

4. The data processing method according to claim 3, characterized in that, The method comprises the following steps: pre-setting a step length array; wherein the step length array comprises step lengths arranged from large to small; and each step length in the step length array is smaller than the degraded step length; taking the largest step length in the step length array as a candidate step length; using the candidate step length to query the data amount, and stopping if the data amount does not exceed the preset data amount threshold; if the data amount exceeds the preset data amount, the next candidate step length is found by using the step length array until the data amount does not exceed the preset data amount threshold or the minimum step length in the step length array is reached, so that the candidate step length when the data amount does not exceed the preset data amount threshold is taken as the target prediction step or the minimum step length is taken as the target prediction step.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The method comprises the following steps: dividing the time range of the list generation task into at least one time segment according to the preliminary prediction step; the at least one time segment comprises a first segment; taking the start time of the time range of the list generation task as the start time of the first segment, and taking the start time of the first segment as the end time of the first segment by extending the start time of the first segment backward by the preliminary prediction step.

6. The data processing method of claim 1, wherein, The first segment start time and end time are used as query conditions to query and count the data amount in the first segment range. If the queried data amount does not exceed the preset amount threshold, the queried data amount is directly used as the target data amount.

7. The data processing method according to claim 1 or 6, characterized by, Further comprising: determining a second preliminary prediction step length according to the first segment or the second segment; determining a third segment start time and end time according to the second preliminary prediction step length and the time range of the list generation task; using the third segment start time and end time as query conditions to query and count the data amount in the third segment range; if the queried data amount in the third segment range exceeds the preset amount threshold, using a degradation strategy to obtain a second target prediction step length, obtaining a fourth segment based on the second target prediction step length, and querying the data amount again according to the fourth segment to obtain a second target data amount that does not exceed the preset amount threshold; if the queried data amount in the third segment range does not exceed the preset amount threshold, directly using the queried data amount in the third segment range as the second target data amount.

8. A data processing apparatus, characterized by, comprising: a preliminary prediction step length module configured to determine a preliminary prediction step length according to a time range of a list generation task; a first segment determination module configured to determine a first segment start time and end time according to the preliminary prediction step length and the time range of the list generation task; a data amount query module configured to use the first segment start time and end time as query conditions to query and count the data amount in the first segment range; a target prediction step length module configured to, if the queried data amount exceeds a preset amount threshold, use a degradation strategy to obtain a target prediction step length, and obtain a second segment based on the target prediction step length; a target data amount module configured to query the data amount again according to the second segment to obtain a target data amount that does not exceed the preset amount threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the data processing method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the data processing method of any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the data processing method of any one of claims 1-7.

Images