Data processing method and device, equipment, medium and product

Abstract

The invention provides a data processing method and device, equipment, a medium and a product, and relates to the technical field of data processing. The method comprises the following steps: processing acquired to-be-processed data in a preset scene by adopting an alternative data processing strategy to obtain data operation information of the alternative data processing strategy; performing statistics on the data operation information to obtain a data statistics set; according to the data statistics set, selecting and obtaining a target data processing strategy from the at least two alternative data processing strategies; and processing the target data in the preset scene by using the target data processing strategy. According to the embodiment of the invention, the processing efficiency of the target data can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of data center technology, communication technology and Internet technology, more and more data are accumulated. In order to quickly process the accumulated big data, it is usually necessary to manually select a target data processing strategy from a variety of different types of data processing strategies, and use the target data processing strategy to process the big data to obtain valuable information.

[0003] However, the above method of selecting data processing strategies may involve subjective human judgment factors and cannot accurately measure which data processing strategy is more suitable for processing big data, thus reducing the efficiency of processing big data. Summary of the invention

[0004] The present application provides a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can automatically select a target data processing strategy suitable for processing data in a preset scenario from a plurality of alternative data processing strategies, reduce the influence of subjective factors of human selection on the data processing strategy, and improve the processing efficiency of big data.

[0005] In a first aspect, the present application provides a data processing method, which includes: using an alternative data processing strategy to process the data to be processed in a preset scene, and obtaining data operation information of the alternative data processing strategy; performing statistics on the data operation information to obtain a data statistics set; based on the data statistics set, selecting a target data processing strategy from at least two alternative data processing strategies; and using the target data processing strategy to process the target data in the preset scene.

[0006] In the second aspect, the present application provides a data processing device, which includes: a first processing module, which is used to adopt an alternative data processing strategy to process the data to be processed in the acquired preset scene, and obtain data operation information of the alternative data processing strategy; a statistical module, which is used to count the data operation information and obtain a data statistical set; a selection module, which is used to select a target data processing strategy from at least two alternative data processing strategies according to the data statistical set; a second processing module, which is used to use the target data processing strategy to process the target data in the preset scene.

[0007] In a third aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing method.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0009] In a fifth aspect, the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements the above-mentioned data processing method when executed.

[0010] In the embodiment provided by the present application, data to be processed in a preset scenario is obtained, and alternative data processing strategies are adopted to process the data to be processed, and data operation information of the alternative data processing strategies is obtained, so that the data operation information can reflect the processing methods of different alternative data processing strategies for the data to be processed, and different data processing methods can affect the processing complexity of the data to be processed; further, the data operation information is counted to obtain a data statistical set, and the values ​​in the data statistical set can reflect the processing complexity of the data to be processed by adopting the alternative data processing strategy within a preset time period, and then according to the data statistical set, a target data processing strategy is selected from at least two alternative data processing strategies to obtain. In the above screening process, since the influence of human subjective factors is eliminated, the obtained target data processing strategy can be made more suitable for processing the target data in the preset scenario, thereby improving the processing efficiency of the target data when the target data processing strategy is used to process the target data.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0013] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application is shown.

[0014] Figure 2 A block diagram showing the composition of a data processing system provided in an embodiment of the present application is shown.

[0015] Figure 3 A flow chart of a method for obtaining alternative data processing strategies and multiple sets of data to be processed provided in an embodiment of the present application is shown.

[0016] Figure 4 A schematic diagram of a process for initializing an alternative data processing strategy by a data preprocessing module provided in an embodiment of the present application is shown.

[0017] Figure 5 A schematic diagram of a flow chart of operating data to be processed using a data operation method provided in an embodiment of the present application is shown.

[0018] Figure 6 A schematic flow chart of a method for performing data processing using a data processing system provided in an embodiment of the present application is shown.

[0019] Figure 7 A block diagram showing the composition of a data processing device provided in an embodiment of the present application is shown.

[0020] Figure 8 A block diagram showing the composition of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0023] As used herein, the term "and / or" includes any and all combinations of one or more related enumerated items. The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of..." are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0024] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0025] When processing big data, a target data processing strategy is usually manually selected from a variety of different types of data processing strategies, and the target data processing strategy is used to process the big data to obtain valuable information.

[0026] However, the above method of selecting data processing strategies may involve subjective human judgment factors and cannot accurately measure which data processing strategy is more suitable for processing big data. For example, in the process of using a certain data processing strategy to process the big data to be processed, it is impossible to clearly determine whether the data processing strategy is the strategy with the best processing complexity for big data among multiple data processing strategies, thereby reducing the processing efficiency of big data.

[0027] According to the data processing method of the embodiment of the present application, it is possible to obtain the data to be processed in a preset scenario, and adopt an alternative data processing strategy to process the data to be processed, to obtain data operation information of the alternative data processing strategy, so that the data operation information can reflect the processing methods of different alternative data processing strategies for the data to be processed, and different data processing methods can affect the processing complexity of the data to be processed; further, the data operation information is counted to obtain a data statistical set, and the values ​​in the data statistical set can reflect the processing complexity of using the alternative data processing strategy to process the data to be processed within a preset time length, and then according to the data statistical set, a target data processing strategy is selected from at least two alternative data processing strategies to obtain. In the above-mentioned screening process, since the influence of human subjective factors is excluded, the obtained target data processing strategy can be made more suitable for processing the target data in the preset scenario, thereby when the target data processing strategy is used to process the target data, the processing efficiency of the target data can be improved.

[0028] The data processing method according to the embodiment of the present application can be performed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method can be executed by a server.

[0029] Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application is shown. Figure 1 , the data processing method comprises:

[0030] Step S101: Use the candidate data processing strategy to process the acquired data to be processed in the preset scenario to obtain data operation information of the candidate data processing strategy.

[0031] Among them, the preset scenarios include any one of the following: application scenarios in the financial field, application scenarios in the communication field, and application scenarios in the travel field.

[0032] In some exemplary embodiments, multiple pieces of data to be processed in a preset scenario may be obtained in the following manner: according to an identifier corresponding to the preset scenario, multiple sets of data to be processed are obtained by screening from a collected log data set.

[0033] Each set of data to be processed includes an identifier of a client and N pieces of data to be processed. Each client corresponds to at least two candidate data processing strategies, and N is an integer greater than or equal to 1.

[0034] For example, a log data set includes multiple data, each data corresponds to a different client identifier, and a preset scenario includes multiple clients, each of which can access or process multiple pieces of data to be processed by calling an alternative data processing strategy.

[0035] By filtering the data in the log data set according to the identifier corresponding to the preset scenario, and distinguishing the filtered data according to the client identifier, a set of unprocessed data corresponding to each client is obtained to clarify the set of unprocessed data that each client needs to process or access, so as to facilitate the use of at least two alternative data processing strategies to process the N pieces of unprocessed data in the set of unprocessed data corresponding to each client, so as to clarify the complexity of using each alternative data processing strategy to process the N pieces of unprocessed data for each client, so as to facilitate the measurement and screening of at least two alternative data processing strategies to obtain the target data processing strategy that is most suitable for processing the target data in the preset scenario.

[0036] Each candidate data processing strategy includes at least two data operation modes.

[0037] For example, the data operation mode includes any one of the following: query operation, deletion operation and addition operation. When the data operation mode includes the query operation, the operation of querying and obtaining the first target data from the N pieces of data to be processed is performed; when the data operation mode includes the deletion operation, the operation of deleting the preset data from the N pieces of data to be processed is performed; when the data operation mode includes the addition operation, the operation of adding the second target data to the N pieces of data to be processed is performed.

[0038] In some exemplary embodiments, the candidate data processing strategies include: an array processing strategy and a linked list processing strategy, or, a hash table processing strategy and a tree processing strategy.

[0039] Among them, the array processing strategy can be characterized as a strategy based on dynamic array implementation, which can quickly and randomly access multiple data to be processed and can automatically increase the storage space in the array to adapt to the storage needs of big data.

[0040] The linked list processing strategy can be characterized as a strategy based on a two-way linked list. Since the storage addresses in the memory corresponding to the data stored in the two-way linked list are not continuous, when the linked list processing strategy is used to process the data to be processed, the processing speed is relatively slow.

[0041] It should be noted that when an array processing strategy is used to query the data to be processed, the data stored in the array can be directly accessed; however, when a linked list processing strategy is used to query the data to be processed, it is necessary to determine the data to be queried based on the associations between the linked lists, and it may even be necessary to traverse the entire linked list to obtain the data to be queried.

[0042] In other words, the space complexity of the query operation corresponding to the data processing strategy is 1, while the space complexity of the query operation corresponding to the linked list processing strategy is a, where a is a number greater than 1.

[0043] When the array processing strategy is used to add (or delete) the data to be processed, it is necessary to first find the position to be added (or deleted) (for example, after the 5th element in the array), then move the positions of all elements after the 5th element, and then add the element to be added to the original storage position of the 5 elements (or delete the data stored on the 5th element, and then move all the data after the 5th element forward one position); when the linked list processing strategy is used to add (or delete) the data to be processed, only the front and back addresses of the linked list corresponding to the data to be added (or deleted) can be updated, and the data to be processed can be added (or deleted) to the linked list.

[0044] In other words, the space complexity of the addition operation corresponding to the data processing strategy is b, while the space complexity of the addition operation corresponding to the linked list processing strategy is 1, where b is a number greater than 1.

[0045] Therefore, the data processing complexity corresponding to the array processing strategy and the linked list processing strategy is different.

[0046] Similarly, hash table processing strategy and tree processing strategy are also data processing strategies with different processing complexity. Among them, hash table processing strategy is suitable for static data (such as blacklists, initialization parameters, and network address configuration data); tree processing strategy is suitable for dynamic data (such as task scheduling data, etc.)

[0047] Since the hash table processing strategy uses a hash algorithm to process the data to be processed, it can directly locate different array addresses without traversing the linked list; while the tree processing strategy requires traversing the linked list. Therefore, the search time complexity corresponding to the hash table processing strategy is c, while the search time complexity corresponding to the tree processing strategy is 1, where c is a number greater than 1.

[0048] By adopting the above-mentioned different data processing strategies to process the data to be processed, the data operation information corresponding to each alternative data processing strategy can be obtained, so as to clarify the data processing complexity corresponding to each alternative data processing strategy based on the data operation information, that is, clarify the advantages and disadvantages of each alternative data processing strategy in processing the data to be processed, so as to facilitate the subsequent screening of multiple alternative data processing strategies.

[0049] Step S102, statistics are collected on the data operation information to obtain a data statistics set.

[0050] The data statistics set includes data statistics corresponding to at least two alternative data processing strategies, and the data statistics corresponding to the alternative data processing strategies are used to reflect the processing complexity of processing the data to be processed using the alternative data processing strategies within a preset time period.

[0051] Among them, the processing complexity can not only characterize the time complexity corresponding to the alternative data processing strategy, but also reflect the operation space complexity corresponding to the alternative data processing strategy (for example, for a query operation on a certain data to be processed, whether the alternative data processing strategy can quickly and accurately find the required data from N data to be processed).

[0052] In some embodiments, the larger the data statistic value is, the lower the processing complexity of the corresponding candidate data processing strategy is, that is, the data to be processed can be processed conveniently and quickly.

[0053] By separately collecting statistics on the data operation information corresponding to each alternative data processing strategy, we can learn the advantages and disadvantages of each alternative data processing strategy in processing the data to be processed, and then use data statistical values ​​to characterize the advantages and disadvantages, which can more intuitively clarify whether each alternative data processing strategy is suitable for processing the data to be processed, thereby improving the screening speed of alternative data processing strategies.

[0054] Step S103: According to the data statistical set, a target data processing strategy is selected from at least two candidate data processing strategies.

[0055] Among them, at least two data statistical values ​​in the data statistical set can be sorted to obtain the alternative data processing strategy with the largest data statistical value, and the alternative data processing strategy with the largest data statistical value can be used as the target data processing strategy, so that the target data processing strategy can process the target data conveniently and quickly, thereby improving data processing efficiency.

[0056] In some embodiments, the step S103 of selecting a target data processing strategy from at least two alternative data processing strategies based on a data statistical set can be implemented in the following manner: for one alternative data processing strategy, the data statistical values ​​corresponding to multiple clients are added together to obtain a target statistical value corresponding to the alternative data processing strategy; according to a preset statistical threshold, the target statistical values ​​corresponding to each of the at least two alternative data processing strategies are screened to obtain the target data processing strategy.

[0057] The target statistical value corresponding to the target data processing strategy is a value greater than a preset statistical threshold.

[0058] It should be noted that, since an alternative data processing strategy can be called by multiple clients, and different clients process different numbers of data when using the alternative data processing strategy for data processing, therefore, for an alternative data processing strategy, the data statistics corresponding to multiple clients that call the alternative data processing strategy are summed up to obtain a target statistical value, which can make the target statistical value more comprehensively reflect the processing complexity of the alternative data processing strategy when processing the data to be processed, so as to facilitate the subsequent screening of various alternative data processing strategies based on the target statistical value.

[0059] The target statistical value corresponding to each alternative data processing strategy is compared with a preset statistical threshold (for example, 0), and the alternative data processing strategy corresponding to the target statistical value greater than the preset statistical threshold is selected as the target data processing strategy. This can make the target data processing strategy more suitable for processing the target data in the preset scenario, thereby improving the processing speed of the target data.

[0060] Step S104: Use the target data processing strategy to process the target data in the preset scenario.

[0061] Among them, the target data is the data that needs to perform specific operations in the preset scenario (such as the first target data to be queried, and / or the preset data to be deleted, and / or the second target data to be added, etc.). By using the target data processing strategy to process the target data, the processing speed of the target data can be accelerated.

[0062] According to an embodiment of the present application, N pieces of data to be processed in a preset scenario can be obtained, where N is an integer greater than 1. Then, at least two alternative data processing strategies are used to process the N pieces of data to be processed respectively, and data operation information corresponding to each alternative data processing strategy is obtained, so that the data operation information can reflect the processing methods of different alternative data processing strategies for the data to be processed, and different data processing methods can affect the processing complexity of the data to be processed; further, the data operation information corresponding to each alternative data processing strategy is counted respectively to obtain a data statistics set, which includes data statistical values ​​corresponding to at least two alternative data processing strategies, wherein the data statistical values ​​corresponding to each alternative data processing strategy can reflect the processing complexity of using the alternative data processing strategy to process the data to be processed within a preset time length, and then the data statistical values ​​corresponding to at least two alternative data processing strategies in the data statistics set are compared and screened to obtain a target data processing strategy. In the above screening process, since the influence of human subjective factors is excluded, the obtained target data processing strategy can be made more suitable for processing the target data in the preset scenario, so that when the target data processing strategy is used to process the target data, the processing efficiency of the target data can be improved.

[0063] The following is a detailed description of step S102 in the data processing method of the embodiment of the present application.

[0064] In some exemplary embodiments, the data operation information is counted in step S102 to obtain a data statistics set, including: for each data operation mode in each candidate data processing strategy, a data statistical value corresponding to the statistical operation mode, wherein the data statistical value corresponding to each data operation mode includes a change value of the number of data processed and a change value of the data processing time;

[0065] Determine a statistical value of the number of data changes according to the data processing number of data changes corresponding to at least two data operation modes;

[0066] Determine a statistical value of the time consumption change according to the data processing time consumption change values ​​corresponding to at least two data operation modes;

[0067] The difference between the number of changes and the time-consuming changes is used as the data statistic corresponding to the alternative data processing strategy;

[0068] A data statistics set is determined according to data statistics values ​​corresponding to a plurality of candidate data processing strategies.

[0069] Among them, if an alternative data processing strategy includes two data operation modes, then the statistical value of the number of changes corresponding to the alternative data processing strategy is the sum of the data processing number of changes corresponding to the two data operation modes, and the statistical value of the time consumption change corresponding to the alternative data processing strategy is the sum of the data processing time consumption change values ​​corresponding to the two data operation modes.

[0070] If an alternative data processing strategy includes three data operation modes, then the statistical value of the number of changes corresponding to the alternative data processing strategy is the sum of the data processing number of changes corresponding to the three data operation modes, and the statistical value of the time consumption change corresponding to the alternative data processing strategy is the sum of the data processing time consumption change values ​​corresponding to the three data operation modes.

[0071] By summing up the statistical values ​​of the number of changes corresponding to each data operation mode included in each candidate data processing strategy, and summing up the statistical values ​​of the time consumption changes corresponding to each data operation mode, the processing efficiency of each candidate data processing strategy on the data to be processed can be reflected from the two dimensions of the number of processed items and the processing time. Furthermore, the difference between the statistical value of the number of changes and the statistical value of the time consumption change is used as the data statistical value corresponding to the candidate data processing strategy, so that the data statistical value can more comprehensively measure the processing complexity of the candidate data processing strategy for the data to be processed, thereby facilitating the subsequent screening of each candidate data processing strategy with the data statistical value, and improving the screening speed of the strategy.

[0072] In some exemplary embodiments, the data statistical value corresponding to the statistical data operation mode includes: counting the number of data processing items of the to-be-processed data processed by the data operation mode within a preset time period, and determining the data processing item number change value according to the number of data processing items, the preset item number threshold and the operation weight corresponding to the data operation mode;

[0073] Statistics on the average time it takes to process a piece of data to be processed using data operation methods;

[0074] The change in data processing time consumption is determined based on the average processing time consumption, the preset processing time threshold, and the operation weight corresponding to the data operation method.

[0075] Among them, the preset processing time threshold is a threshold used to measure whether the average time consumption is within the preset processing time range. If the average time consumption corresponding to a certain data operation method is less than the preset processing time threshold, it indicates that the data processing method is not suitable for processing the data to be processed.

[0076] Similarly, the preset number threshold is a threshold used to measure whether the number of data processing items is within the preset range of processing items. If the number of data processing items corresponding to a certain data operation method is less than the preset number threshold, it indicates that the data processing method is not suitable for processing the data to be processed.

[0077] By using the operation weight corresponding to the data operation mode to weight the number of data processing items and the average time duration corresponding to the data operation mode, it is possible to clarify the proportion of each data operation mode in the alternative data processing strategy, that is, which data operation mode is more performed in an alternative data processing strategy. The operation weight can be set according to the usage requirements, and this application does not limit it.

[0078] For example, the data operation mode includes any one of the following: query operation, delete operation, and add operation.

[0079] In the case where the data operation method includes a query operation, the average time consumption includes the average query time consumption for querying and obtaining the first target data from N pieces of data to be processed. For example, the average time consumption corresponding to the query operation (getAvgTime) can be calculated using the formula: getAvgTime = sum (getendTime-getstartTime) / getCount; where getendTime represents the end time of the query operation, getstartTime represents the start time of the query operation, and getCount represents the number of times the query operation is executed. If getTimeRate is less than 0, set getTimeRate to 0.

[0080] In the case where the data operation method includes a deletion operation, the average time consumption includes the average deletion time consumption of deleting preset data from N pieces of data to be processed. For example, the average time consumption corresponding to the deletion operation (delAvgTime) can be calculated using the formula: delAvgTime = sum (delendTime-delstartTime) / delCount; wherein delendTime represents the end time of the deletion operation, delstartTime represents the start time of the deletion operation, and delCount represents the number of executions of the deletion operation. If delTimeRate is less than 0, set delTimeRate to 0.

[0081] In the case where the data operation mode includes an add operation, the average time duration includes the average time duration of adding the second target data to the N pieces of data to be processed. For example, the average time duration (addAvgTime) corresponding to the add operation can be calculated using the formula: addAvgTime = sum (addendTime - addstartTime) / addCount;

[0082] Where addendTime indicates the end time of the add operation, addstartTime indicates the start time of the add operation, and addCount indicates the number of times the add operation is performed. If addTimeRate is less than 0, set addTimeRate to 0.

[0083] When the data operation modes are different, the processing time corresponding to each data operation mode is used to calculate the average time consumption corresponding to each data operation mode, so as to count which data operation mode requires more time, so as to facilitate the subsequent weighting of each different data operation mode, so as to clarify the proportion of each different data operation mode in the alternative data processing strategy, and facilitate the subsequent evaluation and screening of the alternative data processing strategies.

[0084] In some exemplary embodiments, a change value of the number of data processed items is determined based on the number of data processed items, a preset number threshold, and an operation weight corresponding to the data operation method, including: determining the number growth value corresponding to the data operation method based on the preset number threshold and the number of data processed items; weighting the number growth value using the number weight corresponding to the data operation method to obtain the change value of the number of data processed items.

[0085] Among them, when the number of growth value is less than the preset number of amplitude threshold, the number of growth value is updated to the preset number of amplitude threshold; when the number of growth value is greater than or equal to the preset number of amplitude threshold, the number of growth value corresponding to the data operation mode remains unchanged.

[0086] For example, if the number of data processing items corresponding to a certain data operation mode is set to Count, the preset number threshold is Threshold, and the number weight corresponding to the data operation mode is CountWeight, then the number growth value corresponding to the data operation mode is CountRate = (Count-Threshold) / Count; correspondingly, the number change value corresponding to the data operation mode is CountRate*CountWeight.

[0087] If the preset number amplitude threshold is set to 0, if CountRate is less than 0, then CountRate is updated to 0 to ensure that the final calculated data processing time consumption change value is not a negative number.

[0088] By weighting the growth value of the number of items corresponding to the data operation mode based on the weight of the number of items corresponding to the data operation mode, the proportion of the data operation mode in the alternative data processing strategy can be reflected, so that the calculated change value of the number of data processing items corresponding to the data operation mode reflects the change of the number of data processing items corresponding to the data operation mode, so as to measure the number processing capabilities of multiple data operation modes in the alternative data processing strategy according to the change value of the number of data processing items corresponding to the data operation mode, and highlight the processing complexity (or processing performance) corresponding to the alternative data processing strategy.

[0089] In some exemplary embodiments, a data processing time change value is determined based on an average time consumption, a preset processing time threshold, and an operation weight corresponding to the data operation mode, including: determining a time amplitude value corresponding to the data operation mode based on a preset time consumption threshold and the average time consumption; and weighting the time amplitude value using the time weight corresponding to the data operation mode to obtain a data processing time change value.

[0090] Among them, when the time-consuming amplitude value is less than the preset duration threshold, the time-consuming amplitude value is updated to the preset duration threshold; when the time-consuming amplitude value is greater than or equal to the preset duration threshold, the time-consuming amplitude value remains unchanged.

[0091] For example, if the average time corresponding to a certain data operation mode is set to AvgTime, the preset time threshold is TimeThreshold, and the time weight corresponding to the data operation mode is TimeWeight, then the time amplitude value corresponding to the data operation mode is TimeRate = (AvgTime-TimeThreshold) / AvgTime; Correspondingly, the data processing time change value corresponding to the data operation mode = TimeRate*TimeWeight.

[0092] If the preset duration threshold is set to 0, if TimeRate is less than 0, then TimeRate is updated to 0 to ensure that the final calculated change in data processing time consumption is not a negative number.

[0093] By weighting the time-consuming amplitude value corresponding to the data operation mode based on the duration weight corresponding to the data operation mode, the time-consuming proportion of the data operation mode in the alternative data processing strategy can be reflected, so that the calculated data processing time consumption change value corresponding to the data operation mode reflects the change in the time dimension corresponding to the data operation mode, so as to measure the processing capabilities of multiple data operation modes in the alternative data processing strategy on the time dimension according to the data processing time consumption change value corresponding to the data operation mode, and highlight the processing complexity (or processing performance) corresponding to the alternative data processing strategy.

[0094] In some exemplary embodiments, each candidate data processing strategy includes a query operation, a delete operation, and an add operation;

[0095] Determine the number of change statistics based on the data processing number of change values ​​corresponding to at least two data operation modes, including: calculating the number difference between the data processing number of change values ​​corresponding to the query operation and the data processing number of change values ​​corresponding to the deletion operation; determine the number of change statistics based on the difference between the number of difference and the data processing number of change values ​​corresponding to the addition operation.

[0096] Among them, if the change value of the number of data processing items corresponding to the query operation is set to getCountResult, the change value of the number of data processing items corresponding to the deletion operation is delCountResult, and the change value of the number of data processing items corresponding to the addition operation is addCountResult, among them, if it is assumed that getCountResult is the largest (that is, the number of data to be processed queried is the largest), then the statistical value of the change in the number of items is: getCountResult-delCountResult-addCountResult.

[0097] By counting the difference between getCountResult-delCountResult-addCountResult, we can count the number of data to be processed that can be operated by the above three different operations in an alternative data processing strategy, so as to measure the processing complexity of the alternative data processing strategy from the dimension of the number of data processing items, so as to distinguish each alternative data processing strategy according to the statistical value of the number of changes, and improve the screening speed of multiple alternative data processing strategies.

[0098] In some exemplary embodiments, each candidate data processing strategy includes a query operation, a delete operation, and an add operation;

[0099] Determine the time consumption change statistics value based on the data processing time consumption change values ​​corresponding to at least two data operation modes, including: calculating the time consumption difference between the data processing time consumption change value corresponding to the query operation and the data processing time consumption change value corresponding to the deletion operation; determine the time consumption change statistics value based on the difference between the time consumption difference and the data processing time consumption change value corresponding to the addition operation.

[0100] Among them, if the data processing time change value corresponding to the query operation is set to getTimeResult, the data processing time change value corresponding to the deletion operation is delTimeResult, and the data processing time change value corresponding to the addition operation is addTimeResult, if it is assumed that getTimeResult is the largest (that is, the query operation consumes the longest time), then the time change statistics = getTimeResult-addTimeResult-delTimeResult.

[0101] Since each alternative data processing strategy includes three different types of operations, namely query operation, deletion operation and addition operation, and the query operation consumes the most time, the time consumed by the alternative data processing strategy after performing the above three operations on the data to be processed can be counted by subtracting addTimeResult and delTimeResult from the data processing time change value corresponding to the query operation. Then, the alternative data processing strategy can be measured according to the statistical value of the time change corresponding to the alternative data processing strategy, so as to reflect the processing efficiency of the alternative data processing strategy on the data to be processed in the time dimension.

[0102] The data processing method according to the embodiment of the present application is described below.

[0103] Figure 2 FIG. 1 is a block diagram showing a data processing system according to an embodiment of the present application. Figure 2 As shown, the data processing system 200 provided in this embodiment includes but is not limited to the following modules:

[0104] Log collection module 201 , log consumption module 202 , policy configuration module 203 , policy selection module 204 , and data preprocessing module 205 .

[0105] The log collection module 201 is used to collect log data in real time based on the log collection component (such as Filebeat component) to obtain a log data set when various parameters configured by the policy configuration module 203 take effect.

[0106] Among them, the Filebeat component is a lightweight log collection component suitable for forwarding and centralizing data. The Filebeat component can monitor the specified log file or location, collect log events, and forward the collected log data to the distributed publish-subscribe message (Kafka) system, and then store the log data in the preset database through the Kafka system.

[0107] The preset database can be implemented using a Remote Dictionary Server (Redis) database. The Redis database is an open source, network-accessible, memory-based, and persistent log-type Key-Value database. The Redis database can provide an Application Programming Interface (API) in multiple languages.

[0108] The log consumption module 202 is used to obtain the candidate data processing strategy from the strategy configuration module 203, and store the candidate data processing strategy and multiple sets of data to be processed in a preset database.

[0109] The policy configuration module 203 is used to initialize the data configuration and obtain a configuration parameter set. For example, the configuration parameter set includes the following parameters: the identifier key corresponding to the preset scenario (e.g., it can be set to getTxDetails), the preset number of thresholds (e.g., the adding number threshold addThreshold (e.g., it can be set to 100), the query number threshold getThreshold (e.g., it can be set to 0), the deletion number threshold delThreshold (e.g., it can be set to 0)), the preset detection time checkTime, the number of weights corresponding to different data operation modes (e.g., the adding number weight addCountWeight (e.g., it can be set to 60000), the query number weight getCountWeight (e.g., it can be set to 0.5), the deletion number weight delCountWeight (e.g., can be set to 1)) time weights corresponding to different data operation methods (e.g., adding time weight addTimeWeight (e.g., can be set to 0.5), querying time weight getTimeWeight (e.g., can be set to 1), deleting time weight delTimeWeight (e.g., can be set to 1)), preset time thresholds (e.g., adding time threshold addTimeThreshold (e.g., can be set to 1000 milliseconds (ms)), querying time threshold getTimeThreshold (e.g., can be set to 1000ms), deleting time threshold delTimeThreshold (e.g., can be set to 1000ms)) and data operation methods ruleType in alternative data processing strategies, etc.

[0110] Each candidate data processing strategy includes multiple data operation modes, for example, the data operation mode includes any one of the following: query (add) operation, delete (del) operation and add (get) operation.

[0111] Among them, the weight corresponding to the time complexity of the add operation is lower than the weight corresponding to the time complexity of the query operation, and lower than the weight corresponding to the time complexity of the delete operation.

[0112] For example, Table 1 shows a configuration parameter set provided in an embodiment of the present application.

[0113] Table 1 Configuration parameter set

[0114] Parameter name Parameter value The identifier (key) corresponding to the preset scene getTxDetails Add a threshold value for the number of entries (addThreshold) 100 items Query number threshold (getThreshold) 0 Deletion threshold (delThreshold) 0 Add count weight (addCountWeight) 0.5 Query number weight (getCountWeight) 1 Delete the number of weights (delCountWeight) 1 Detection time (checkTime) 60000ms Adding a time threshold (addTimeThreshold) 1000ms Query time threshold (getTimeThreshold) 1000ms Deleting the time-consuming threshold (delTimeThreshold) 1000ms Adding time weight (addTimeWeight) 0.5 Query time weight (getTimeWeight) 1 Delete time weight (delTimeWeight) 1 Data operation type (ruleType) ⊙Add operation ⊙Query operation ⊙Delete operation

[0115] Among them, the identifier corresponding to the preset scenario represents the identifier corresponding to the application scenario in the financial field, or the identifier corresponding to the user travel scenario in the travel field, etc. This application does not impose any restrictions on this.

[0116] The add count threshold indicates the count threshold for including the number of calls when adding the data to be processed into the timing strategy; if the number of calls when adding the data to be processed is greater than the add count threshold, the number of calls is determined to be obtained; if the number of calls when adding the data to be processed is less than or equal to the add count threshold, the number of calls is set to 0, that is, no statistical processing is performed on this adding operation.

[0117] The timing strategy indicates that the candidate data processing strategies are screened once every preset time interval (eg, 2 seconds, 3 seconds, etc.) to obtain the target data processing strategy.

[0118] The query count threshold indicates the count threshold for the number of calls when querying the data to be processed to be included in the timing strategy. If the number of calls when querying the data to be processed is greater than the query count threshold, the number of calls is determined to be obtained; if the number of calls when querying the data to be processed is less than or equal to the query count threshold, the number of calls is set to 0, that is, no statistical processing is performed on this query operation.

[0119] The deletion threshold indicates the number of calls for deleting data to be processed that are included in the timing strategy. If the number of calls for deleting data to be processed is greater than the deletion threshold, the number of calls is determined; if the number of calls for deleting data to be processed is less than or equal to the deletion threshold, the number of calls is set to 0, that is, no statistical processing is performed on this deletion operation.

[0120] The detection duration indicates the time range required for each execution of the timing strategy to filter data, in milliseconds (ms).

[0121] The weight of adding the number of entries indicates the weight that needs to be proportionally adjusted for the number growth value (addCountRate) corresponding to the adding operation.

[0122] The number growth value corresponding to the add operation can be calculated using the following formula: addCountRate=(addCount-addThreshold) / addCount; wherein addCount represents the total number of added items, and addThreshold represents the threshold value of the number of added items.

[0123] The query number weight indicates the weight that needs to be proportionally adjusted for the number growth value (getCountRate) corresponding to the query operation.

[0124] The growth value of the number of queries corresponding to the query operation can be calculated using the following formula: getCountRate=(getCount-getThreshold) / getCount; wherein getCount represents the total number of queries, and getThreshold represents the threshold value of the number of queries.

[0125] The deletion weight indicates the weight that needs to be proportionally adjusted for the increase in the number of deletions (delCountRate).

[0126] The growth value of the number of entries corresponding to the deletion operation can be calculated using the following formula: delCountRate=(delCount-delThreshold) / delCount; wherein delCount represents the total number of entries deleted, and delThreshold represents the threshold value of the number of entries deleted.

[0127] Adding a time-consuming threshold indicates the time threshold for including the average time consumed by the deletion operation on the data to be processed in the timing strategy. If the average time consumed by the deletion operation on the data to be processed is greater than the adding time-consuming threshold, the average time consumed by the deletion operation is determined; if the average time consumed by the deletion operation on the data to be processed is less than or equal to the adding time-consuming threshold, the average time consumed by the deletion operation is set to 0, that is, no statistical processing is performed on this adding operation.

[0128] The query time threshold indicates the time threshold for including the average time consumed by querying the data to be processed into the timing strategy. If the average time consumed by querying the data to be processed is greater than the query time threshold, the average time consumed by the query operation is determined; if the average time consumed by querying the data to be processed is less than or equal to the query time threshold, the average time consumed by the query operation is set to 0, that is, no statistical processing is performed on this query operation.

[0129] The deletion time threshold indicates the time threshold for including the average time consumed by the deletion operation on the data to be processed in the timing strategy. If the average time consumed by the deletion operation on the data to be processed is greater than the deletion time threshold, the average time consumed by the deletion operation is determined; if the average time consumed by the deletion operation on the data to be processed is less than or equal to the deletion time threshold, the average time consumed by the deletion operation is set to 0, that is, no statistical processing is performed on this deletion operation.

[0130] The add time weight indicates the weight that needs to be proportionally adjusted for the time value (addTimeRate) corresponding to the add operation.

[0131] The time-consuming amplitude value corresponding to the adding operation can be calculated using the formula: addTimeRate=(addAvgTime-addTimeThreshold) / addAvgTime.

[0132] Wherein, addTimeThreshold represents the preset time threshold corresponding to the adding operation (ie, the adding time threshold), for example, it can be set to 100;

[0133] addAvgTime indicates the average time taken by the add operation, which can be calculated using the formula: addAvgTime = sum(addendTime-addstartTime) / addCount; where addendTime indicates the end time of the add operation, addstartTime indicates the start time of the add operation, and addCount indicates the number of times the add operation is performed. If addTimeRate is less than 0, set addTimeRate to 0.

[0134] The query time weight indicates the weight that needs to be proportionally adjusted for the time value (getTimeRate) corresponding to the query operation.

[0135] The time consumption value corresponding to the query operation can be calculated using the formula: getTimeRate=(getAvgTime-getTimeThreshold) / getAvgTime.

[0136] Wherein, getTimeThreshold represents the preset time-consuming threshold corresponding to the query operation (ie, the query time-consuming threshold), for example, it can be set to 100;

[0137] getAvgTime indicates the average time taken by the query operation, which can be calculated using the formula: getAvgTime = sum(getendTime-getstartTime) / getCount; where getendTime indicates the end time of the query operation, getstartTime indicates the start time of the query operation, and getCount indicates the number of times the query operation is executed. If getTimeRate is less than 0, set getTimeRate to 0.

[0138] The deletion time weight indicates the weight that needs to be proportionally adjusted for the time value (getTimeRate) corresponding to the deletion operation.

[0139] The time-consuming amplitude value corresponding to the deletion operation can be calculated using the formula: delTimeRate=(delAvgTime-deltThreshold) / delAvgTime.

[0140] Wherein, delTimeThreshold represents a preset time-consuming threshold corresponding to the deletion operation (ie, the deletion time-consuming threshold), for example, it can be set to 100;

[0141] delAvgTime indicates the average time taken for the deletion operation, which can be calculated using the formula: delAvgTime = sum(delendTime-delstartTime) / delCount; where delendTime indicates the end time of the deletion operation, delstartTime indicates the start time of the deletion operation, and delCount indicates the number of times the deletion operation is performed. If delTimeRate is less than 0, set delTimeRate to 0.

[0142] The strategy selection module 204 is used to execute the timing strategy, that is, to screen the candidate data processing strategies once every preset time interval (such as 2 seconds, 3 seconds, etc.) to obtain the target data processing strategy and use the target data processing strategy to process the target data.

[0143] The data preprocessing module 205 is used to process the log data obtained from the log collection module 201 using the alternative data processing strategy, and use the target data processing strategy to process the log data when the strategy selection module 204 determines to update the alternative data processing strategy to the target data processing strategy.

[0144] For example, a Boolean attribute identifier useArrayList is set in the data preprocessing module 205. When useArrayList is set to false, the data preprocessing module 205 uses an array processing strategy to process log data; when useArrayList is set to true, the data preprocessing module 205 uses a linked list processing strategy to process log data.

[0145] Figure 3 A flow chart of a method for obtaining an alternative data processing strategy and multiple sets of data to be processed provided by an embodiment of the present application is shown. Figure 3 As shown, the method comprises the following steps:

[0146] Step S301 : the log collection module 201 collects log data based on the log collection component to obtain a log data set.

[0147] The log collection component may be a Filebeat component.

[0148] Step S302 : the log collection module 201 selects and obtains a plurality of to-be-processed data sets from the log data set according to the identifier corresponding to the preset scenario.

[0149] Among them, the identifier corresponding to the preset scenario is the identifier corresponding to the application scenario in the financial field (for example, set to identifier A), or the identifier corresponding to the preset scenario is the identifier corresponding to the user travel scenario in the travel field (for example, set to identifier B), etc.

[0150] For example, according to the identifier A, multiple log data in the log data set are searched to obtain multiple log data including the identifier A. Then, according to the identifier of the client included in each log data, the multiple log data are further refined and distinguished to obtain multiple data sets to be processed.

[0151] Each set of data to be processed includes N data to be processed and the identifier of the client, and each client corresponds to at least two candidate data processing strategies. The identifier of the client indicates that the N data to be processed are data corresponding to the data request initiated by the client. Each candidate data processing strategy includes at least two data operation modes (e.g., any two of the query operation, the delete operation, and the add operation).

[0152] In some embodiments, data requested by each client may be processed by one thread, and thus an identifier of one thread (eg, txId) may be used to represent an identifier of one client.

[0153] Each piece of data to be processed includes: the start time of data processing (startTime), the end time of data processing (endTime) and txId.

[0154] Step S303: the log collection module 201 stores the multiple sets of data to be processed into a distributed publish-subscribe message system.

[0155] Among them, the distributed publish-subscribe messaging system can be a Kafka system.

[0156] Step S304 : the log consumption module 202 obtains the candidate data processing strategy from the strategy configuration module 203 .

[0157] Step S305 : the log consumption module 202 stores the candidate data processing strategies and the multiple sets of data to be processed in a preset database.

[0158] The preset database may be a Redis database.

[0159] In some embodiments, the startTime, endTime and txId in each piece of data to be processed may be assembled into a json string and stored in a Redis database, so as to more quickly obtain the data to be processed from the Redis database.

[0160] In some embodiments, the at least two candidate data processing strategies include a linked list processing strategy and an array processing strategy. Therefore, during the initialization process, the data preprocessing module 205 needs to determine which candidate data processing strategy to initialize based on the initial parameters configured by the strategy configuration module 203. Figure 4 A schematic diagram of a process for initializing an alternative data processing strategy by a data preprocessing module provided in an embodiment of the present application is shown.

[0161] like Figure 4 As shown, the method for the data preprocessing module 205 to initialize the candidate data processing strategy includes the following steps:

[0162] Step S401 : obtaining the initial candidate data processing strategy configured by the strategy configuration module 203 .

[0163] Step S402: determine the type of the initial candidate data processing strategy.

[0164] Step S403: when it is determined that the initial candidate data processing strategy is the array processing strategy, an array processing instance is initialized, and the array processing instance includes a query operation, a delete operation, and an add operation.

[0165] The data preprocessing module 205 is provided with a Boolean attribute identifier useArrayList. When useArrayList is set to false, it indicates that the initial candidate data processing strategy configured by the strategy configuration module 203 is an array processing strategy. The array processing instance can be represented as ArrayList.

[0166] Step S404: when it is determined that the initial candidate data processing strategy is a linked list processing strategy, a linked list processing instance is initialized, and the linked list processing instance includes a query operation, a delete operation, and an add operation.

[0167] When useArrayList is set to true, the initial candidate data processing strategy configured by the characterization strategy configuration module 203 is a linked list processing strategy. A linked list processing instance can be represented as LinkedList.

[0168] Because each candidate data processing strategy includes at least two data operation modes, and each data operation mode has a different operation mode for processing data, therefore, Figure 5A schematic diagram of a flow chart of operating data to be processed using a data operation method provided in an embodiment of the present application is shown.

[0169] in, Figure 5 (a) in FIG. 1 shows a flow chart of using query operation to operate the data to be processed. Figure 5 As shown in (a), the operation method comprises the following steps:

[0170] Step S511, record the start time.

[0171] Step S512: query and obtain first target data from N pieces of data to be processed.

[0172] Step S513, recording the identifier of the preset scene corresponding to the first target data, the identifier of the client requesting the data query, the data query start time and the data query end time.

[0173] For example, the identifier of the preset scene corresponding to the first target data is set as key, the identifier of the client requesting data query is txId1_get; the start time of the data query is startTime1, and the end time of the data query is endTime1; the recorded data to be processed includes: key, txId1_get, startTime1, endTime1 and data text 1.

[0174] Figure 5 (b) in FIG. 1 shows a schematic diagram of a process of using a delete operation to operate on the data to be processed. Figure 5 As shown in (b), the operation method comprises the following steps:

[0175] Step S521, record the start time.

[0176] Step S522: deleting preset data from the N pieces of data to be processed.

[0177] Step S523, recording the identifier of the preset scenario corresponding to the preset data, the identifier of the client requesting data deletion, the data deletion start time and the data deletion end time.

[0178] For example, the identifier of the preset scene corresponding to the preset data is set as key, the identifier of the client requesting data deletion is txId2_del; the data query start time is startTime2, and the data query end time is endTime2; then the recorded data to be processed includes: key, txId2_del, startTime2, endTime2 and data body 2.

[0179] Figure 5(c) in FIG. 1 shows a schematic diagram of a flow chart of using an add operation to operate on the data to be processed. Figure 5 As shown in (c), the operation method includes the following steps:

[0180] Step S531, record the start time.

[0181] Step S532: adding second target data to the N pieces of data to be processed.

[0182] Step S533, recording the identifier of the preset scene corresponding to the second target data, the identifier of the client requesting to add data, the start time of adding data, and the end time of adding data.

[0183] For example, the identifier of the preset scene corresponding to the preset data is set as key, the identifier of the client requesting data deletion is txId3_add; the data query start time is startTime3, and the data query end time is endTime3; then the recorded data to be processed includes: key, txId3_add, startTime3, endTime3 and data body 3.

[0184] Figure 6 A schematic flow chart of a method for performing data processing using a data processing system provided in an embodiment of the present application is shown.

[0185] like Figure 6 As shown, the method for data processing using a data processing system includes the following steps:

[0186] Step S601, obtaining an identifier corresponding to a preset scene and performing data initialization.

[0187] The preset scenario may be an application scenario in the financial field, or a user travel scenario in the travel field, etc. This application does not impose any restrictions on this.

[0188] For example, when the preset scenario is an application scenario in the financial field, the data used therein may be the user's personal tax data (eg, the user's tax payment time, tax payment ratio data, etc.).

[0189] For another example, when the preset scenario is a user travel scenario in the travel field, the data used may be the user's appearance data (such as the travel time of the user from A to B, the travel tools used, and other data).

[0190] Among them, data initialization is that the data preprocessing module 205 initializes a processing instance according to the initial candidate data processing strategy configured by the strategy configuration module 203. For example, when it is determined that the initial candidate data processing strategy is an array processing strategy, an array processing instance is initialized, and the array processing instance includes a query operation, a delete operation, and an add operation. When it is determined that the initial candidate data processing strategy is a linked list processing strategy, a linked list processing instance is initialized, and the linked list processing instance includes a query operation, a delete operation, and an add operation.

[0191] Step S602: Filter and obtain multiple to-be-processed data sets from the collected log data sets according to the identifiers corresponding to the preset scenarios.

[0192] Each set of data to be processed includes an identifier of a client and N pieces of data to be processed, and each client corresponds to at least two candidate data processing strategies.

[0193] For example, the log collection module 201 collects log data based on the Filebeat component to obtain a log data set; then, the log collection module 201 filters and obtains multiple data sets to be processed from the log data set according to the identifier corresponding to the application scenario in the financial field (such as getTxDetails), and each data set to be processed includes the identifier of the client in the financial field and N pieces of data to be processed.

[0194] When the log consumption module 202 obtains the candidate data processing strategy (eg, array processing strategy) from the strategy configuration module 203, the log consumption module 202 stores the array processing strategy and the multiple sets of data to be processed in the Redis database.

[0195] In some embodiments, the filtering conditions for the data to be processed are set as:

[0196] 1) The time difference between the current time nowtime and the end time (endtime) of data processing of the data to be processed is less than the preset detection time checktime (for example, checktime is set to 60000ms) (that is, nowtime-endtime<60000ms);

[0197] 2) The identifier of the preset scenario is the identifier corresponding to the application scenario in the financial field, and the alternative data processing strategies include query operation, deletion operation and addition operation;

[0198] Then, the data to be processed that meets the above screening conditions is filtered out from the Redis database to obtain multiple sets of data to be processed.

[0199] Step S603: Use at least two candidate data processing strategies to process the N pieces of data to be processed respectively, and obtain data operation information corresponding to each candidate data processing strategy.

[0200] The at least two candidate data processing strategies include an array processing strategy and a linked list processing strategy, or a hash table processing strategy and a tree processing strategy. Each candidate data processing strategy includes at least two data operation modes.

[0201] For example, the data operation method includes any one of the following: query operation, deletion operation and addition operation. The data operation information includes at least two of the following information: information on processing N pieces of data to be processed by using a query operation (such as the operation duration corresponding to the query operation, and the number of pieces of data queried, etc.), information on processing N pieces of data to be processed by using a deletion operation (such as the operation duration corresponding to the deletion operation, and the number of pieces of data deleted from the N pieces of data to be processed, etc.), and information on processing N pieces of data to be processed by using an addition operation (such as the operation duration corresponding to the addition operation, and the number of pieces of data added to the N pieces of data to be processed, etc.).

[0202] When the data operation method includes a query operation, the first target data is obtained by querying from N pieces of data to be processed, and the query duration corresponding to the first target data is obtained (for example, the difference between the end time of the query operation and the start time of the query operation is used as the query duration), as well as the number of first target data queried.

[0203] In the case where the data operation mode includes a deletion operation, the preset data is deleted from the N pieces of data to be processed, and a deletion time for deleting the preset data from the N pieces of data to be processed and the number of pieces of the deleted preset data are obtained.

[0204] When the data operation method includes an adding operation, the second target data is added to the N pieces of data to be processed, and the adding duration of adding the first target data to the N pieces of data to be processed is obtained (for example, the difference between the end time of the adding operation and the start time of the adding operation is used as the adding duration), as well as the number of the second target data added.

[0205] It should be noted that the data operation mode is not limited to the above three operations, and may also include a modification operation, that is, an operation to modify a certain piece of data among the N pieces of data to be processed. The above data operation modes are only examples, and specific settings can be made according to actual needs. Other unexplained data operation modes are within the protection scope of this application and will not be repeated here.

[0206] Step S604, counting the number of data processing items of the to-be-processed data processed by the data operation method within a preset time period, and determining the change value of the number of data processing items according to the number of data processing items, the preset number threshold and the operation weight corresponding to the data operation method.

[0207] Specifically, the number growth value corresponding to the data operation method can be determined based on the preset number threshold and the number of data processed; the number growth value is weighted using the number weight corresponding to the data operation method to obtain the data processing number change value.

[0208] Among them, when the number of growth value is less than the preset number of amplitude threshold, the number of growth value is updated to the preset number of amplitude threshold; when the number of growth value is greater than or equal to the preset number of amplitude threshold, the number of growth value corresponding to the data operation mode remains unchanged.

[0209] For example, for the add operation, if the number of data to be processed by the add operation within the preset time length (also known as: the total number of added data) is set to addCount, and the preset number threshold (such as the add number threshold) addThreshold is 100, then the number growth value corresponding to the add operation addCountRate = (addCount-100) / addCount.

[0210] If addCountRate is less than a preset number amplitude threshold (eg, 0), the number growth value corresponding to the adding operation may be updated to 0; if addCountRate is greater than or equal to 0, addCountRate is kept unchanged.

[0211] Furthermore, if the number weight corresponding to the adding operation (also referred to as the adding number weight) is set to addCountWeight is 0.5, then the data processing number change value (also referred to as the adding number weight value) addCountResult = addCountRate*0.5.

[0212] For another example, for a query operation, if the number of data to be processed by the query operation within a preset time period (also referred to as the total number of queries) is set to getCount, and the preset number threshold (such as the query number threshold) getThreshold is 100, then the number growth value corresponding to the query operation getCountRate = (getCount-100) / getCount.

[0213] If getCountRate is less than a preset number of entries threshold (eg, 0), the number of entries growth value corresponding to the query operation may be updated to 0; if getCountRate is greater than or equal to 0, keep getCountRate unchanged.

[0214] Furthermore, if the number weight corresponding to the query operation (also referred to as the query number weight) is set to getCountWeight is 1, then the data processing number change value (also referred to as the query number weight value) getCountResult = getCountRate*1.

[0215] For another example, for the deletion operation, if the number of data to be processed by the deletion operation within the preset time length (also known as the total number of deletions) is set to delCount, and the preset number threshold (such as the deletion number threshold) delThreshold is 100, then the number growth value corresponding to the deletion operation delCountRate = (delCount-100) / delCount.

[0216] If delCountRate is less than a preset number amplitude threshold (eg, 0), the number growth value corresponding to the deletion operation may be updated to 0; if delCountRate is greater than or equal to 0, delCountRate is kept unchanged.

[0217] Furthermore, if the number weight corresponding to the deletion operation (also referred to as the deletion number weight) is set to delCountWeight is 1, then the data processing number change value (also referred to as the deletion number weighted value) delCountResult = delCountRate*1.

[0218] Step S605, count the time it takes to process a piece of data to be processed using the data operation method and the number of pieces of data to be processed corresponding to the data operation method within the preset time, and determine the average time consumption; determine the data processing time consumption change value based on the average time consumption, the preset processing time threshold and the operation weight corresponding to the data operation method.

[0219] Wherein, when the data operation mode is an add operation, the average time (addAvgTime) can be calculated using the formula: addAvgTime = sum (addendTime-addstartTime) / addCount; where addendTime represents the end time of the add operation, addstartTime represents the start time of the add operation, and addCount represents the number of times the add operation is performed. If addTimeRate is less than 0, addTimeRate is set to 0.

[0220] For example, when the data operation mode is a query operation, the average time (getAvgTime) can be calculated using the formula: getAvgTime = sum (getendTime-getstartTime) / getCount; where getendTime represents the end time of the query operation, getstartTime represents the start time of the query operation, and getCount represents the number of times the query operation is executed. If getTimeRate is less than 0, set getTimeRate to 0.

[0221] For another example, when the data operation mode is a delete operation, the average time (delAvgTime) can be calculated using the formula: delAvgTime = sum (delendTime-delstartTime) / delCount; where delendTime represents the end time of the delete operation, delstartTime represents the start time of the delete operation, and delCount represents the number of times the delete operation is executed. If delTimeRate is less than 0, delTimeRate is set to 0.

[0222] In some embodiments, the time amplitude value corresponding to the data operation mode is determined according to a preset time threshold and an average time duration; the time amplitude value is weighted using a duration weight corresponding to the data operation mode to obtain a data processing time change value.

[0223] Wherein, when the time-consuming amplitude value is less than the preset duration threshold, the time-consuming amplitude value is updated to the preset duration threshold; when the time-consuming amplitude value is greater than or equal to the preset duration threshold, the time-consuming amplitude value remains unchanged. The preset duration threshold can be updated once every preset threshold update duration (e.g., 1 second).

[0224] For example, for the add operation, if the preset time threshold addTimeThreshold corresponding to the add operation is set to 1000ms, the time amplitude value (addTimeRate) corresponding to the add operation can be calculated using the formula: addTimeRate = (addAvgTime-1000) / addAvgTime. If addTimeRate is less than 0, addTimeRate is set to 0.

[0225] If the duration weight corresponding to the adding operation (ie, the adding time weight addTimeWeight) is set to 0.5, the data processing time change value corresponding to the adding operation (ie, the adding time weighted value addTimeResult) can be obtained as addTimeRate*0.5.

[0226] For another example, for a query operation, if the preset time threshold getTimeThreshold corresponding to the query operation is set to 1000ms, the time amplitude value (getTimeRate) corresponding to the query operation can be calculated using the formula: getTimeRate = (getAvgTime-1000) / getAvgTime. If getTimeRate is less than 0, set getTimeRate to 0.

[0227] If the duration weight corresponding to the query operation (ie, the query time weight getTimeWeight) is set to 1, the data processing time change value corresponding to the query operation (ie, the query time weighted value getTimeResult) can be obtained as getTimeRate*1.

[0228] For another example, for a delete operation, if the preset time threshold delTimeThreshold corresponding to the delete operation is set to 1000ms, the time amplitude value (delTimeRate) corresponding to the delete operation can be calculated using the formula: delTimeRate = (delAvgTime-1000) / delAvgTime. If delTimeRate is less than 0, delTimeRate is set to 0.

[0229] If the duration weight corresponding to the deletion operation (ie, the deletion time weight delTimeWeight) is set to 1, the data processing time change value corresponding to the deletion operation (ie, the deletion time weighted value delTimeResult) can be obtained as delTimeRate*1.

[0230] Step S606: determine the data statistics corresponding to each candidate data processing strategy according to the change value of the number of data processed and the change value of the data processing time.

[0231] The change value of the number of data processed (countResult) can be calculated using the following formula: countResult=getCountResult-addCountResult-delCountResult.

[0232] The change value of the data processing time consumption (timeResult) can be calculated using the following formula: timeResult=getTimeResult-addTimeResult-delTimeResult.

[0233] The data statistics corresponding to the candidate data processing strategy include countResult and timeResult.

[0234] Step S607: for an alternative data processing strategy, the data statistical values ​​corresponding to the multiple clients are summed up to obtain a target statistical value corresponding to the alternative data processing strategy.

[0235] Among them, the target statistical value corresponding to the alternative data processing strategy can be calculated using the following formula:

[0236] Wherein, n represents the number of clients, and n is an integer greater than or equal to 1; k represents an integer greater than or equal to 1 and less than n; m represents the mth candidate data processing strategy used by the kth client;

[0237] countResult km Indicates the change in the number of data items processed when the k-th client uses the m-th candidate data processing strategy to process the data to be processed;

[0238] timeResult km It represents the change value of the data processing time consumption corresponding to the mth candidate data processing strategy used by the kth client to process the data to be processed.

[0239] Step S608: Screen the target statistical values ​​corresponding to the at least two candidate data processing strategies according to the preset statistical threshold to obtain the target data processing strategy.

[0240] Among them, if result m is greater than 0, and the target statistical values ​​corresponding to other candidate data processing strategies are all less than 0, then it can be determined that the target data processing strategy is the mth candidate data processing strategy.

[0241] In some embodiments, if the at least two candidate data processing strategies only include an array processing strategy and a linked list processing strategy, and the attribute flag useArrayList is set during initialization.

[0242] When useArrayList is set to false, the representation data preprocessing module 205 uses an array processing strategy to process the log data; when useArrayList is set to true, the representation data preprocessing module 205 uses a linked list processing strategy to process the log data.

[0243] When the target statistical value result>0 and the value of useArrayList is false, useArrayList will not be updated, that is, the target data processing strategy will remain unchanged as the array processing strategy;

[0244] If the target statistical value result>0, and the value of useArrayList is true (linked list processing strategy), useArrayList will not be updated, that is, the target data processing strategy will remain unchanged as the linked list processing strategy;

[0245] If the target statistical value result < 0, and the value of useArrayList is true, then useArrayList is updated to false, that is, the currently used target data processing strategy is updated from the linked list processing strategy to the array processing strategy;

[0246] If the target statistical value result<0 and the value of useArrayList is false, useArrayList is updated to true, that is, the currently used target data processing strategy is updated from the array processing strategy to the linked list processing strategy.

[0247] Step S609: Use the target data processing strategy to process the target data in the preset scenario.

[0248] In some embodiments, after executing step S609, multiple pieces of data to be processed corresponding to the identifiers of preset scenarios stored in the cache may be deleted, and / or the configuration information of the policy configuration module 203 may be cleared; so as to obtain more sufficient storage space to facilitate subsequent processing of data in other application scenarios.

[0249] Through the embodiments of the present application, the influence of human subjective factors can be eliminated, and the target statistical values ​​corresponding to the alternative data processing strategies obtained by calculation can be used to reflect the processing complexity of using the alternative data processing strategies to process the data to be processed in the preset scenario, and the time complexity corresponding to each alternative data processing strategy and the average time consumption for processing the data can be reflected. Based on the processing requirements of the refined data corresponding to the preset scenario, the alternative data processing strategy with the best processing complexity can be selected as the target data processing strategy, and then when the target data processing strategy is used to process the target data, the processing efficiency of the target data is improved, so that the data processing process will not be disturbed by information such as business code changes and / or device processing resources.

[0250] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0251] In addition, the present application also provides a data processing device, an electronic device, a computer-readable storage medium and a computer program product, all of which can be used to implement any data processing method provided by the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0252] Figure 7 The following is a block diagram showing the composition of a data processing device provided by an embodiment of the present application. Figure 7 , the embodiment of the present application provides a data processing device, the data processing device 700 includes:

[0253] The first processing module 701 is used to process the acquired data to be processed in the preset scenario using the alternative data processing strategy to obtain data operation information of the alternative data processing strategy;

[0254] A statistics module 702 is used to collect statistics on data operation information to obtain a data statistics set;

[0255] A selection module 703 is used to select a target data processing strategy from at least two candidate data processing strategies according to the data statistical set;

[0256] The second processing module 704 is used to process the target data in the preset scenario using the target data processing strategy.

[0257] It should be noted that the data processing device 700 in the embodiment of the present application can execute any data processing method provided in the embodiment of the present application, which will not be described in detail here.

[0258] According to the data processing device in the present application, the data to be processed in the preset scene is obtained through the first processing module, and the alternative data processing strategy is used to process the data to be processed, and data operation information of the alternative data processing strategy is obtained, so that the data operation information can reflect the processing methods of different alternative data processing strategies for the data to be processed, and different data processing methods can affect the processing complexity of the data to be processed; further, the data operation information is counted by the statistical module to obtain a data statistical set, and the values ​​in the data statistical set can reflect the processing complexity of using the alternative data processing strategy to process the data to be processed within a preset time length, and then the selection module is used to select and obtain a target data processing strategy from at least two alternative data processing strategies according to the data statistical set. In the above screening process, since the influence of human subjective factors is eliminated, the obtained target data processing strategy can be made more suitable for processing the target data in the preset scene, so that when the target data is processed by the second processing module using the target data processing strategy, the processing efficiency of the target data can be improved.

[0259] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0260] Figure 8 A block diagram showing the composition of an electronic device provided in an embodiment of the present application is shown.

[0261] Reference Figure 8 An embodiment of the present application provides an electronic device, which includes: at least one processor 801; at least one memory 802, and one or more I / O interfaces 803, connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by at least one processor 801, and the one or more computer programs are executed by at least one processor 801 so that the at least one processor 801 can execute the above-mentioned data processing method.

[0262] Among them, the data processing method includes: obtaining N pieces of data to be processed in a preset scenario; using at least two alternative data processing strategies to process the N pieces of data to be processed respectively, and obtaining data operation information corresponding to each alternative data processing strategy; respectively counting the data operation information corresponding to each alternative data processing strategy to obtain a data statistical set, the data statistical set including data statistical values ​​corresponding to at least two alternative data processing strategies, the data statistical values ​​are used to reflect the processing complexity of using the alternative data processing strategies to process the data to be processed within a preset time period; according to the data statistical set, selecting a target data processing strategy from at least two alternative data processing strategies; using the target data processing strategy to process the target data in the preset scenario.

[0263] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0264] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0265] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0266] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0267] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0268] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0269] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0270] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0271] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0272] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0273] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0274] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0275] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Using the alternative data processing strategy to process the acquired data to be processed in the preset scenario to obtain data operation information of the alternative data processing strategy; Performing statistics on the data operation information to obtain a data statistics set; According to the data statistics set, a target data processing strategy is selected from at least two candidate data processing strategies; The target data processing strategy is used to process the target data in the preset scenario.

2. The method according to claim 1, characterized in that The data statistics set includes data statistics corresponding to the at least two candidate data processing strategies; The step of selecting a target data processing strategy from at least two candidate data processing strategies according to the data statistics set includes: For one of the candidate data processing strategies, summing up the data statistics corresponding to the multiple clients to obtain a target statistics corresponding to the candidate data processing strategy; According to a preset statistical threshold, the target statistical values ​​corresponding to the at least two candidate data processing strategies are screened to obtain the target data processing strategy.

3. The method according to claim 1, characterized in that Each of the candidate data processing strategies includes at least two data operation modes; the data operation information is counted to obtain a data statistics set, including: For each of the data operation modes in each of the candidate data processing strategies, statistical data statistics corresponding to the data operation mode are collected, wherein the data statistics corresponding to each of the data operation modes include a change value of the number of data processed and a change value of the time consumed for data processing; Determine a statistical value of the number of data items changed according to the data processing number of data items changed corresponding to each of the at least two data operation modes; Determine a statistical value of the time consumption change according to the data processing time consumption change values ​​corresponding to the at least two data operation modes; The difference between the number change statistics and the time consumption change statistics is used as the data statistics corresponding to the alternative data processing strategy; The data statistics set is determined according to the data statistics values ​​corresponding to the multiple candidate data processing strategies.

4. The method according to claim 3, characterized in that The counting of data statistics corresponding to the data operation mode includes: Counting the number of data processed by the data operation method within a preset time period, and determining the change value of the number of data processed according to the number of data processed, a preset number threshold and an operation weight corresponding to the data operation method; Counting the average time taken to process a piece of the to-be-processed data using the data operation method; The data processing time consumption change value is determined according to the average time consumption, the preset processing time threshold and the operation weight corresponding to the data operation mode.

5. The method according to claim 4, characterized in that The determining the change value of the number of data processed according to the number of data processed, the preset number threshold and the operation weight corresponding to the data operation mode includes: Determine the number of entries corresponding to the data operation mode according to the preset number threshold and the number of data processed; The number of items is weighted by the number of items weight corresponding to the data operation mode to obtain the number of data processing items change value; Among them, when the number of growth value is less than the preset number of amplitude threshold, the number of growth value is updated to the preset number of amplitude threshold; when the number of growth value is greater than or equal to the preset number of amplitude threshold, the number of growth value corresponding to the data operation mode is kept unchanged.

6. The method according to claim 4, characterized in that The determining the data processing time consumption change value according to the average time consumption, the preset processing time threshold and the operation weight corresponding to the data operation mode includes: Determine the time-consuming amplitude value corresponding to the data operation mode according to the preset time-consuming threshold and the average time-consuming duration; The duration weight corresponding to the data operation mode is used to weight the duration amplitude value to obtain the data processing time consumption change value; Among them, when the time-consuming amplitude value is less than the preset duration threshold, the time-consuming amplitude value is updated to the preset duration threshold; when the time-consuming amplitude value is greater than or equal to the preset duration threshold, the time-consuming amplitude value is kept unchanged.

7. The method according to claim 4, characterized in that The data operation mode includes any one of the following: query operation, delete operation and add operation; Among them, the average time consumed includes the average query time for obtaining the first target data from N pieces of data to be processed; or, the average time consumed includes the average deletion time for deleting preset data from the N pieces of data to be processed; or, the average time consumed includes the average addition time for adding the second target data to the N pieces of data to be processed.

8. The method according to claim 3, characterized in that Each of the candidate data processing strategies includes a query operation, a delete operation, and an add operation; the determining of the number of data processing item change statistics according to the data processing item number change values ​​corresponding to each of the at least two data operation modes includes: Calculate the difference between the change value of the number of data processed items corresponding to the query operation and the change value of the number of data processed items corresponding to the delete operation; The item number change statistic value is determined according to the difference between the item number difference value and the data processing item number change value corresponding to the adding operation.

9. The method according to claim 3, characterized in that: Each of the candidate data processing strategies includes a query operation, a delete operation, and an add operation; and determining the time consumption change statistical value according to the data processing time consumption change values ​​corresponding to the at least two data operation modes, includes: Calculate the time difference between the data processing time change value corresponding to the query operation and the data processing time change value corresponding to the delete operation; The time consumption change statistic value is determined according to the difference between the time consumption difference value and the data processing time consumption change value corresponding to the adding operation.

10. The method according to claim 1, characterized in that The alternative data processing strategies include: Array processing strategy and linked list processing strategy, or, hash table processing strategy and tree processing strategy.

11. A data processing device, characterized in that: include: A first processing module is used to process the acquired data to be processed in the preset scenario using an alternative data processing strategy to obtain data operation information of the alternative data processing strategy; A statistical module, used to collect statistics on the data operation information to obtain a data statistical set; A selection module, configured to select a target data processing strategy from at least two candidate data processing strategies according to the data statistics set; The second processing module is used to process the target data in the preset scene using the target data processing strategy.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements the data processing method as described in any one of claims 1 to 10 when executing the code.

Images