Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing.The method comprises the steps that data in a data set is divided into sub buckets according to the size of a storage space; according to the machine performance, configuring the reliable concurrency capability of each sub-bucket to obtain the sub-bucket configured with the reliable concurrency capability; and performing concurrent processing on the data in the data set in the sub-buckets configured with the reliable concurrent capability. According to the scheme, the number of concurrent processing threads or processes in each sub-bucket is dynamically adjusted according to the characteristics of the data and the performance of the machine, and the processing capacity of the machine can be fully utilized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium and program product. Background Art

[0002] In the process of generating intelligent driving data sets, sometimes it is necessary to process offline large-scale data. In order to improve data processing efficiency, multi-concurrency functions are usually used to process data. However, the number of concurrency needs to be fixed before data processing, and the number of concurrency cannot be dynamically and adaptively adjusted during data processing. However, for intelligent driving data sets, they usually contain multivariate data, such as composition positioning data, perception data, planning and control data, etc. Generally, a single data occupies a large space. For machines with fixed CPU cores and memory, it is difficult to fully mobilize the efficiency of the machine by using a fixed concurrency strategy. Summary of the invention

[0003] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and program product, aiming to solve the technical problem of being unable to automatically adjust the number of concurrent data and fully utilize the machine processing capacity.

[0004] To achieve the above objectives, the present application proposes a data processing method, which includes:

[0005] Divide the data in the dataset into buckets according to the storage space size;

[0006] According to the machine performance, the reliable concurrency capability of each bucket is configured to obtain the bucket configured with the reliable concurrency capability;

[0007] In the bucket configured with reliable concurrency capability, data in the data set is processed concurrently.

[0008] In one embodiment, the step of dividing the data in the data set into respective preset buckets according to the storage space size includes:

[0009] Sort the data in the data set according to the storage space size to obtain sorted data;

[0010] The sorted data is divided into different buckets according to the storage space size of the sorted data.

[0011] The step of configuring the reliable concurrency capability of each bucket according to the machine performance to obtain the bucket configured with the reliable concurrency capability comprises:

[0012] Define the configuration dictionary to obtain reliable concurrency capabilities;

[0013] Confirm the performance of the machine;

[0014] Based on the performance of the machine, a corresponding reliable concurrency capability is configured for each bucket to obtain a bucket configured with a reliable concurrency capability.

[0015] In one embodiment, the step of defining a configuration dictionary and obtaining reliable concurrency capability includes:

[0016] According to the storage space size of the data in the bucket, a corresponding code is set for the bucket;

[0017] According to the characteristics of the data in the bucket, a corresponding reliable concurrent number of file processing and a file path list are set for the bucket;

[0018] Based on the code, the reliable concurrency number of file processing and the file path list, the definition of the configuration dictionary is completed.

[0019] In one embodiment, the step of concurrently processing the data in the data set in the preset bucket includes:

[0020] Based on the reliable concurrent number of file processing in the preset bucket, obtain the number of concurrent data processing processes;

[0021] According to the number of concurrent data processes, the data in the data set in the preset bucket is processed to obtain processed data.

[0022] In one embodiment, after the step of processing the data in the data set in the preset bucket according to the number of concurrent data processing processes to obtain processed data, the method further includes:

[0023] Information statistics are performed on the processed data to analyze the rationality of the concurrency of the processed data and to count the production status of the processed data.

[0024] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0025] The partitioning module is used to divide the data in the data set into buckets according to the storage space size;

[0026] A setting module is used to configure the reliable concurrency capability of each bucket according to the machine performance, and obtain the bucket with the configured reliable concurrency capability;

[0027] The processing module is used to concurrently process the data in the data set in the bucket configured with reliable concurrency capability.

[0028] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0029] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0031] The embodiment of the present application proposes a data processing method, specifically, dividing the data in the data set into each bucket according to the size of the storage space; setting the reliable concurrency capacity of each bucket according to the machine performance to obtain a preset bucket; and concurrently processing the data in the data set in the preset bucket. The solution dynamically adjusts the number of concurrent processing threads or processes in each bucket according to the characteristics of the data and the machine performance, and can fully utilize the machine processing capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0035] Figure 2 This is a schematic diagram of the process of dividing data into buckets involved in the first embodiment of the data processing method of the present application;

[0036] Figure 3 A schematic diagram of a machine configuration dictionary process involved in the first embodiment of the data processing method of the present application;

[0037] Figure 4 A flowchart of the second embodiment of the data processing method of this application is provided;

[0038] Figure 5 A brief flowchart diagram of the first and second embodiments of the data processing method of the present application;

[0039] Figure 6 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0040] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0041] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0043] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0044] The main solution of the embodiment of the present application is: divide the data in the data set into each bucket according to the size of the storage space; configure the reliable concurrency capability of each bucket according to the machine performance to obtain the bucket configured with reliable concurrency capability; and concurrently process the data in the data set in the bucket configured with reliable concurrency capability.

[0045] In the process of generating intelligent driving data sets, sometimes offline large-scale data needs to be processed. In order to improve data processing efficiency, multi-concurrency functions are usually used to process data. However, the number of concurrency needs to be fixed before data processing, and the number of concurrency cannot be dynamically and adaptively adjusted during data processing. However, for intelligent driving data sets, they usually contain multivariate data, such as composition positioning data, perception data, planning and control data, etc., and generally a single data occupies a large space. For machines with fixed CPU cores and memory, it is difficult to fully mobilize the efficiency of the machine by using a fixed concurrency strategy.

[0046] The present application provides a solution, which sorts the data according to the size of the storage space of the data and divides it into corresponding buckets. In the bucket, the concurrency number can be set according to the data with large storage space, for data files with small storage space; according to the machine performance, the reliable concurrency capacity of each bucket is configured to obtain the bucket with reliable concurrency capacity, so that the system can dynamically calculate the available computing resources; in the bucket with reliable concurrency capacity, the data in the data set is processed concurrently, so that the machine can process more data concurrently at the same time by configuring the bucket with reliable concurrency capacity. As the size of the data file increases, the ability of the machine to process data concurrently at the same time decreases, thereby achieving the purpose of reasonably utilizing the machine processing capacity.

[0047] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a personal computer as an example to illustrate this embodiment and the following embodiments.

[0048] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0049] In this embodiment, the data processing method includes steps S10 to S30:

[0050] Step S10, dividing the data in the data set into various preset buckets according to the storage space size;

[0051] Since the storage space size of individual data in the driving data set is inconsistent, the gap between data is large. Some data only occupy less than 1M of storage space, while some data storage space may be greater than 40M. When such a data set is processed offline on a large scale, the usual solution is to use a multi-concurrency solution, and the number of concurrencies will be designed and fixedly configured based on the largest file data. In this way, when processing data files with small storage space, it is difficult to fully utilize the computing resources of the machine. This is because loading data generally takes up a large amount of memory space. If the number of concurrencies is set according to data files with small storage space, when processing a large number of data files with large storage space at the same time, the memory space will be quickly used up, and the machine will be in an abnormally slow waiting state or in serious cases, it will be in a downtime state. Therefore, in this embodiment, before concurrent processing of the data, the data in the data set is sorted and classified into each bucket according to the storage space size, so that the buckets can be configured accordingly, and concurrent processing can be performed according to the corresponding number of concurrencies.

[0052] In a feasible implementation, step S10 may include steps S11 to S12:

[0053] Step S11, sorting the data in the data set according to the storage space size to obtain sorted data;

[0054] First, the data in the data set are arranged according to the size of the storage space they occupy, usually from small to large or from large to small. In this embodiment, the data is sorted in the order of small to large in memory.

[0055] The sorted data will be arranged in order of storage space size so that it can be subsequently divided into different buckets.

[0056] Step S12, dividing the sorted data into corresponding preset buckets according to the storage space size of the sorted data.

[0057] The sorted data is classified into the set buckets one by one according to the storage space size. For example, each bucket can be designed as 0-15M, 15M-25M, 25M-35M, 35M-45M, 45M-55M, 55M-65M, 65M-100M, greater than 100M, etc. At this time, since the storage space size of the data in the data set is 10M, the data can be divided into buckets with bucket codes of 0-15M. The specific process diagram of dividing data into buckets is as follows Figure 2 shown.

[0058] Through the above steps, the sorted data is allocated to the preset buckets one by one according to the storage space size, ensuring that the total storage space size of the data in each bucket meets the preset requirements. The data can be effectively allocated to different buckets for subsequent bucket configuration and concurrent data processing.

[0059] Step S20, configuring the reliable concurrency capability of each of the buckets according to the machine performance, and obtaining buckets configured with reliable concurrency capability;

[0060] It should be noted that reliable concurrency refers to the maximum number of concurrent requests or tasks that the system can effectively handle while ensuring system stability and data processing accuracy.

[0061] Machine performance can include the number of CPU cores and memory size of the machine.

[0062] In this embodiment, each bucket can be regarded as an independent processing unit and can be assigned to different computing nodes or threads for parallel processing.

[0063] In a feasible implementation, step S20 may include steps S21 to S23:

[0064] Step S21, define a configuration dictionary to obtain reliable concurrency capability;

[0065] It should be noted that the configuration dictionary is a mutable container model that can store key-value pairs of any type of object, including code names, reliable concurrency numbers for file processing, and file path lists.

[0066] In this embodiment, the system defines a configuration dictionary, in which the key is the code of the bucket (such as "0-15M"), and the value is an element that includes the reliable concurrency number and file path list of the bucket.

[0067] The configuration dictionary is essentially a set of predefined rules, which includes the settings of a series of key parameters in the data processing process, such as data bucketing strategy, reliable concurrency of file processing, file path list, etc. The reasonable configuration of these parameters directly affects the efficiency and quality of data processing.

[0068] In a feasible implementation manner, step 21 may further include steps A21 to A23:

[0069] Step A21, setting a corresponding code for the bucket according to the storage space size of the data in the bucket;

[0070] In this embodiment, the key value is the code of the bucket, which indicates the storage space size that the bucket can accommodate. For example, the code of the first bucket is 0-15M, indicating that the storage space size range of the data received by the bucket is from 0 to 15MB.

[0071] By setting the code, the system can quickly identify and determine which bucket the data should be placed in, thereby achieving efficient classification and management of data. In this way, when new data needs to be stored, the system can directly locate the corresponding bucket according to the size of the data, avoiding unnecessary traversal and calculation.

[0072] Step A22, according to the characteristics of the data in the bucket, setting the corresponding reliable concurrent number of file processing and file path list for the bucket;

[0073] It should be noted that the reliable concurrency number refers to the number of files that a bucket can process simultaneously while ensuring data processing quality and system stability.

[0074] The value is the reliable concurrency of file processing under the bucket, as well as the list of all file path names corresponding to the bucket. The file path name list stores the full paths of all data files belonging to the bucket, which facilitates the subsequent storage of data with the same path name.

[0075] In each bucket, usually for data files with small storage space, a higher reliable concurrency is configured; for data files with large storage space, a lower reliable concurrency is configured. Therefore, when setting the reliable concurrency of each bucket, the corresponding reliable concurrency will also be configured according to the characteristics of the data in the bucket. The reliable concurrency will decrease as the size of the data storage space increases.

[0076] Step A23, completing the definition of the configuration dictionary based on the code, the reliable concurrent number of the file processing and the file path list.

[0077] A configuration dictionary is defined through the bucket code set in steps A21 and A22, and the corresponding reliable concurrent number of file processing and the file path list, for storing the relevant configuration information of each bucket. The configuration dictionary may include the code as a key, and the corresponding reliable concurrent number of file processing and the file path list as a value.

[0078] Step S22, confirming the performance of the machine;

[0079] Since different machines have different CPU core numbers and memory spaces, the configuration dictionary needs to be set according to the specific conditions of the machine.

[0080] For example, more CPU cores means that the machine has stronger parallel processing capabilities, so you can set it higher when configuring the reliable concurrency number. A larger memory space can support more data to reside in the memory and increase the data processing speed. Therefore, for machines with larger memory, you can appropriately increase the reliable concurrency number.

[0081] Step S23: Based on the performance of the machine, a corresponding reliable concurrency capability is configured for each bucket to obtain a bucket configured with a reliable concurrency capability.

[0082] Under normal circumstances, in the same environment, there are not many types of machines used to process data. You can configure the corresponding reliable concurrency capabilities for the different existing machines. When processing data, by confirming the configuration of the good machine, automatically select the corresponding configuration dictionary to ensure that when processing data, the system can dynamically adjust the concurrency strategy according to the hardware capabilities of the current machine to achieve the best processing effect. The specific machine configuration dictionary process diagram is as follows: Figure 3 shown.

[0083] In addition, in this embodiment, the system has a certain degree of dynamic adaptability, that is, it can automatically adjust the parameters in the configuration dictionary according to the real-time monitored machine status and system load. For example, when it is detected that the CPU utilization of a machine is close to the peak, the concurrency of the bucketed file processing on the machine can be automatically reduced to avoid overload. In short, based on the performance characteristics of each machine, the concurrency of the preset bucket can be adjusted to achieve the best data processing performance.

[0084] Through the above steps, by defining the configuration dictionary and presetting the buckets based on the machine configuration, it is helpful to achieve refined management and efficient processing of data, especially in big data processing scenarios, which can significantly improve the overall performance and response speed of the system.

[0085] Step S30, concurrently processing the data in the data set in the bucket configured with reliable concurrency capability.

[0086] Based on the configuration dictionary of each bucket, each element is looped and processed in turn, where the key value represents the size range of the data file storage space in the bucket, and the corresponding value contains the reliable concurrency of the bucket and the corresponding file path list. According to the reliable concurrency in each bucket element, the number of concurrent tasks that can be processed by the bucket at the same time can be determined, and according to the number of concurrent tasks for file processing, data is processed concurrently in each preset bucket.

[0087] In a feasible implementation, step S30 may include steps S31 to S32:

[0088] Step S31, based on the preset reliable concurrency number in the bucket, obtain the number of concurrent data processing processes;

[0089] It should be noted that the number of concurrent processes refers to the number of tasks that the system can execute in parallel at the same time.

[0090] Specifically, the system reads the configuration dictionary information of the preset bucket and starts the corresponding number of concurrent processes according to the reliable concurrency of the current machine. Each process is responsible for processing a part of the data, which can effectively decompose the data processing tasks and speed up the overall data processing process.

[0091] Step S32, processing the data in the data set in the preset bucket according to the number of concurrent data processes to obtain processed data.

[0092] Once the concurrent process is started, the machine will start processing the data sets in each bucket, dividing the data sets into several data fragments according to a predefined method. These data fragments are assigned to different concurrent processes for processing. Each process is responsible for processing one or more data fragments, ensuring the concurrency of data processing, thereby speeding up the entire data processing process.

[0093] As concurrent processes start processing the data fragments assigned to them, they perform operations such as data cleaning, transformation, aggregation, and analysis. Since multiple processes work simultaneously, the overall speed of data processing is significantly accelerated.

[0094] When all concurrent processes complete their respective data processing tasks, the system collects the intermediate results generated by these processes. These results usually need to be further merged or aggregated to generate the final processed data.

[0095] Through the above-mentioned embodiment method, the data is sorted according to the size of the storage space and divided into corresponding buckets. In the bucket, the concurrency number can be set according to the data with large storage space, for data files with small storage space; the reliable concurrency number of each bucket is configured according to the machine performance, so that the system can be dynamically based on the currently available computing resources; in the bucket configured with reliable concurrency capability, the data in the data set is processed concurrently. The machine can process more data concurrently at the same time by configuring the bucket with reliable concurrency capability. As the size of the data file increases, the ability of the machine to process data concurrently at the same time decreases, thereby achieving the purpose of rationally utilizing the machine's processing power.

[0096] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 After step S32, the data processing method further comprises step B1:

[0097] Step B1, performing information statistics on the processed data to analyze the rationality of the concurrency of the processed data and to count the production status of the processed data.

[0098] After data processing is completed, key performance indicators of the data processing process, such as processing time, CPU usage, memory consumption, etc., are manually recorded and analyzed. These key performance indicators can help evaluate the efficiency of data processing and resource utilization, so as to adjust the configuration dictionary strategy of the preset bucket in real time.

[0099] By comparing the processing time and resource consumption of different concurrency levels, the rationality of concurrent processing can be evaluated, including whether the number of concurrent processes for data processing is set appropriately, whether system resources are fully utilized, and whether the expected processing efficiency and speed are achieved.

[0100] Evaluate the impact of concurrent processing on data integrity and accuracy by counting errors or anomalies that occur during data processing. A high error rate means that the current concurrency level is too high, resulting in unstable data processing.

[0101] The production status of the processed data can also be counted to help understand the production status of the data processing process, including statistical information on data output, data quality indicators, processing success rate, etc. These statistical information can be used to evaluate the overall effect and results of data processing.

[0102] Through the above-mentioned embodiment method, by performing information statistics and analysis on the processed data, the data processing process can be fully understood, the processing effect and the rationality of concurrent processing can be evaluated, and reference and basis can be provided for further optimizing the data processing flow and improving the data processing efficiency. Data resources can be better managed and utilized, and the quality and efficiency of data processing can be improved.

[0103] For example, to help understand the implementation process of the data processing method obtained by combining the first embodiment and the second embodiment, please refer to Figure 5 , Figure 5 A brief flow chart of a data processing method is provided, specifically:

[0104] First, sort the data in the data set according to the storage space size, and divide the sorted data into each set bucket.

[0105] Then, according to the machine performance and data characteristics, the corresponding reliable concurrency capability is configured for each bucket. The reliable concurrency capability can be obtained by defining a configuration dictionary.

[0106] When defining the configuration dictionary, include the code name of the configuration bucket, the reliable concurrency number of file processing under the bucket, and a list of all file paths.

[0107] When processing data files with small storage space, a higher concurrency number can be configured because the data volume is small and the processing time of a single file is short. By increasing the number of concurrent processing, system resources can be fully utilized, processing efficiency can be improved, and data processing tasks can be completed quickly. However, data files with large storage space usually contain a large amount of data and take a long time to process. When processing large data files, configuring too high a concurrency number may lead to excessive consumption of system resources, resulting in resource competition and performance degradation. Therefore, for large data files, a lower concurrency number is usually configured to avoid resource waste and system overload. And the concurrency number of the machine should also decrease appropriately as the storage size of the data increases to avoid the risk of resource exhaustion and system crash.

[0108] When data processing begins, the system extracts the configuration information for each bucket, including the bucket's storage space size (code), the number of reliable concurrent file processing, and the file path list. This information is used to guide the data processing process.

[0109] According to the configuration dictionary of each bucket extracted, the system will start the corresponding concurrent process to process the data. Single file processing and multi-file processing are two common concurrent modes. Single file processing is suitable for data sets with small storage space, and multi-file processing is suitable for data sets with small storage space. The system will select the appropriate concurrent mode according to the actual situation.

[0110] After the data processing is completed, the system will conduct a statistical analysis of the processing results, including the rationality of the concurrency of the processed data, the specific situation of data production, etc. These statistical information can be used to evaluate the efficiency and quality of data processing and to appropriately adjust the configuration dictionary.

[0111] When the data processing of a bucket is completed, the system will continue to extract the configuration information of the next bucket and repeat the above process until the data in all buckets is processed. This cyclic processing method ensures comprehensive coverage and orderly processing of the entire data set.

[0112] In addition, when new data arrives, the system first finds the corresponding bucket code according to the data size, and then checks whether the reliable concurrency under the bucket allows the addition of new processing tasks. If allowed, the file path of the data is added to the file path list and processing begins; otherwise, the data may need to wait or be reallocated to other buckets.

[0113] In summary, the embodiment of the present application proposes a data processing method, which flexibly adjusts the concurrent processing capability configuration of buckets according to data files of different storage space sizes, and optimizes the concurrent processing configuration according to actual conditions to achieve the best processing effect.

[0114] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0115] This application also provides a data processing device, please refer to Figure 6 , the data processing device comprises:

[0116] A partitioning module 10 is used to partition the data in the data set into various buckets according to the size of the storage space;

[0117] A setting module 20 is used to configure the reliable concurrency capability of each bucket according to the machine performance, and obtain the bucket configured with the reliable concurrency capability;

[0118] The processing module 30 is used to concurrently process the data in the data set in the bucket configured with reliable concurrency capability.

[0119] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the number of concurrent data cannot be automatically adjusted and the processing capacity of the machine cannot be fully utilized. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0120] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0121] Reference below Figure 7 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0122] like Figure 7 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0123] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0124] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the number of concurrent data cannot be automatically adjusted and the processing capacity of the machine cannot be fully utilized. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0127] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0128] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0129] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0130] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: divides the data in the data set into each bucket according to the size of the storage space; sets the reliable concurrency capability of each bucket according to the machine performance to obtain a preset bucket; and concurrently processes the data in the data set in the preset bucket.

[0131] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0134] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of being unable to automatically adjust the number of concurrent data and fully utilize the processing power of the machine. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be elaborated here.

[0135] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0136] The computer program product provided by the present application can solve the technical problem of being unable to automatically adjust the number of concurrent data and fully utilize the processing power of the machine. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and will not be repeated here.

[0137] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Divide the data in the dataset into buckets according to the storage space size; According to the machine performance, the reliable concurrency capability of each bucket is configured to obtain the bucket configured with the reliable concurrency capability; In the bucket configured with reliable concurrency capability, data in the data set is processed concurrently.

2. The method according to claim 1, characterized in that The step of dividing the data in the data set into each bucket according to the storage space size includes: Sort the data in the data set according to the storage space size to obtain sorted data; The sorted data is divided into different buckets according to the storage space size of the sorted data.

3. The method according to claim 1, characterized in that The step of configuring the reliable concurrency capability of each bucket according to the machine performance to obtain the bucket configured with the reliable concurrency capability comprises: Define the configuration dictionary to obtain reliable concurrency capabilities; Confirm the performance of the machine; Based on the performance of the machine, a corresponding reliable concurrency capability is configured for each bucket to obtain a bucket configured with a reliable concurrency capability.

4. The method according to claim 3, characterized in that The steps of defining a configuration dictionary and obtaining reliable concurrency capabilities include: According to the storage space size of the data in the bucket, set the corresponding code for the bucket; According to the characteristics of the data in the bucket, a corresponding reliable concurrent number of file processing and a file path list are set for the bucket; Based on the code, the reliable concurrency number of file processing and the file path list, the definition of the configuration dictionary is completed.

5. The method according to any one of claims 2 to 4, characterized in that The step of concurrently processing the data in the data set in the bucket configured with reliable concurrency capability includes: Based on the reliable concurrent number of file processing in the preset bucket, obtain the number of concurrent data processing processes; According to the number of concurrent data processing processes, the data in the data set in the preset bucket is processed to obtain processed data.

6. The method according to claim 5, characterized in that After the step of processing the data in the data set in the preset bucket according to the number of concurrent data processing processes to obtain processed data, the method further includes: Information statistics are performed on the processed data to analyze the rationality of the concurrency of the processed data and to count the production status of the processed data.

7. A data processing device, characterized in that: The data processing device comprises: The partitioning module is used to divide the data in the data set into buckets according to the storage space size; A setting module is used to configure the reliable concurrency capability of each bucket according to the machine performance, and obtain the bucket with the configured reliable concurrency capability; The processing module is used to concurrently process the data in the data set in the bucket configured with reliable concurrency capability.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to data processing are implemented.

Images