Data processing method, device, electronic device and storage medium

Abstract

The present application provides a data processing method, device, electronic device and storage medium, which relate to the field of cloud computing technology. The method includes: determining the memory size occupied by multiple fields according to the field types corresponding to the multiple fields of the data to be processed; determining the shard size of the data processing based on the memory size occupied by the multiple fields, the total memory size and the number of cores of the central processing unit; and using multiple threads to process the data to be processed in parallel according to the shard size. In this embodiment, the shard size of the data processing is adaptively determined according to the memory size occupied by the multiple fields of the data to be processed, the total memory size and the number of cores of the central processing unit, without the need to manually adjust the shard size, and can achieve reasonable application of memory when processing data in parallel, improve memory utilization, reduce the risk of memory overflow, and make parallel operations more stable and efficient.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] When reading or writing large amounts of data, concurrent processing is often used to improve efficiency. However, it's difficult to accurately determine the amount of data a single thread can handle during concurrent processing. This often results in wasted memory when processing small amounts of data, while memory overflows occur when processing large amounts of data.

[0003] Typically, the amount of data processed by a single thread (i.e., the size of the data slices processed) is manually adjusted by observing memory usage. If the memory is not fully utilized, the amount of data processed is increased; if the memory overflows, the amount of data processed is reduced. These methods are not automatic and intelligent enough, and cannot be achieved in one step. The trial and error process will always bring a bad experience to users. Summary of the Invention

[0004] Embodiments of the present application provide a data processing method, device, electronic device, and storage medium to solve the problem of being unable to automatically achieve reasonable use of memory when processing data in parallel.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Determine the memory size occupied by multiple fields according to the field types corresponding to the multiple fields of the data to be processed;

[0007] Determine the shard size for data processing based on the memory size occupied by multiple fields, the total memory size, and the number of CPU cores;

[0008] Based on the shard size, multiple threads are used to process the data in parallel.

[0009] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0010] A first determining module is used to determine the memory size occupied by multiple fields according to the field types corresponding to the multiple fields of the data to be processed;

[0011] A second determination module is used to determine the shard size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit;

[0012] The data processing module is used to process the data to be processed in parallel using multiple threads according to the size of the shards.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the methods described above is implemented.

[0015] Compared with the prior art, this application has the following advantages:

[0016] The present application provides a data processing method, device, electronic device and storage medium, which determines the memory size occupied by multiple fields of the data to be processed according to the field types corresponding to the multiple fields; determines the shard size of the data processing based on the memory size occupied by the multiple fields, the total memory size and the number of cores of the central processing unit; and processes the data to be processed in parallel using multiple threads according to the shard size. In this embodiment, the shard size of the data processing is adaptively determined according to the memory size occupied by the multiple fields of the data to be processed, the total memory size and the number of cores of the central processing unit, without the need for manual adjustment of the shard size, and can achieve reasonable application of memory when processing data in parallel, improve memory utilization, reduce the risk of memory overflow, and make parallel operations more stable and efficient.

[0017] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0019] Figure 1 This is a flow chart of a data processing method according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a data table according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a data processing method according to an embodiment of the present application;

[0022] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present application; and

[0023] Figure 5 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0024] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] The method in the embodiment of the present application can be applied in the data archiving process, in the application scenario of reading and writing the processed data in a parallel manner, and can be deployed in the computing device in the form of an application. The computing device may include: a server, a user terminal, etc.

[0028] In related technologies, partition sizes are calculated using the partitioning algorithm provided by the Spark computing engine. Spark provides partitioning functions: HashPatitioner, RangePartitioner, and CustomPartitioner. Using the HashPatitioner function to calculate partition sizes requires the user to specify the number of partitions, and the partition size is calculated modulo the hash value of the data. However, the drawback is that data is uneven, which can easily lead to data skew. In extreme cases, a few partitions may contain all the data in a Resilient Distributed Dataset (RDD). When reading partitioned data in parallel, excessively large partitions can cause memory overflows. Using the RangePartitioner function to calculate partition sizes ensures uniform data size within each partition and maintains order between partitions. That is, elements in one partition are smaller or larger than elements in another, but the order of elements within a partition is not guaranteed. Simply put, it maps data within a certain range to a specific partition. However, the specific size of each partition must be calculated based on the partition range. Otherwise, the data size of each partition may be too large, potentially leading to memory overflows when reading data in parallel. The CustomPartitioner function requires the user to specify an implementation strategy, cannot automatically determine the partition size, and does not consider the available memory situation, which can easily lead to memory waste and memory overflow problems.

[0029] In view of this, an embodiment of the present application provides a data processing method. Compared with the Spark partitioning strategy, the adaptive sharding technology provided in the embodiment of the present application takes into account the runtime memory size, automatically calculates the shard size without user specification, makes full use of memory resources, and ensures that there is no risk of memory overflow, making parallel operations more stable and efficient.

[0030] The present application embodiment provides a data processing method, which can be applied to a computing device, which may include a server, a user terminal, etc. Figure 1 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present application, including:

[0031] Step S101 : determining the memory size occupied by the multiple fields according to the field types corresponding to the multiple fields of the data to be processed.

[0032] Step S102 : determining the slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit.

[0033] Step S103: Process the data to be processed in parallel using multiple threads according to the size of the slices.

[0034] The data to be processed may be stored in the form of a data table, and the multiple fields of the data to be processed may include multiple fields in the data table. The memory size occupied by the multiple fields may include the total number of bytes occupied by the multiple fields. The total memory size may be the total memory size of the application used to execute the current parallel data processing task, and the number of cores of the central processing unit (CPU) is the number of cores of the CPU executing the current parallel data processing task. The data processing slice size is the amount of data processed by each thread in the parallel processing.

[0035] In one example, the total memory size of the application executing the current parallel data processing task is 8GB, the number of CPU cores is 2, and the calculated slice size is a bytes, taking into account the memory occupied by multiple fields in the data table of the data to be processed. Using four threads for data processing, each thread processes a bytes of data. Since there are only two cores, two threads are used to process the data in parallel. After these two threads complete data processing, two more threads are used to process the data, each thread also processing a bytes of data. This process continues in an alternating manner until all the data to be processed is complete.

[0036] The data processing method provided in the embodiment of the present application determines the memory size occupied by multiple fields of the data to be processed based on the field types corresponding to the multiple fields; determines the shard size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit; and processes the data to be processed in parallel using multiple threads based on the shard size. In this embodiment, the shard size for data processing is adaptively determined based on the memory size occupied by the multiple fields of the data to be processed, the total memory size, and the number of cores of the central processing unit. There is no need to manually adjust the shard size, and the memory can be reasonably used when processing data in parallel, thereby improving memory utilization, reducing the risk of memory overflow, and making parallel operations more stable and efficient.

[0037] How to determine the memory size occupied by multiple fields is described in detail in the following embodiment:

[0038] In one implementation, step S101, determining the memory size occupied by the multiple fields according to the field types corresponding to the multiple fields of the data to be processed, includes:

[0039] Step S1011 , determining the memory size occupied by the largest data row corresponding to each of the multiple fields according to the field types corresponding to each of the multiple fields.

[0040] Step S1012 , determining the memory sizes occupied by the multiple fields according to the memory sizes occupied by the data rows that occupy the largest amount of memory corresponding to the multiple fields.

[0041] Specifically, the field types of multiple fields in a data table of the data to be processed are obtained, and based on the field types, the number of bytes of memory space occupied by each type of field is determined. If each field corresponds to multiple data rows, the data row with the largest memory usage corresponding to each field is used as the memory size corresponding to that field. The memory sizes corresponding to the multiple fields are summed to obtain the memory sizes occupied by the multiple fields. In this embodiment, using the memory size occupied by the data row with the largest memory usage corresponding to a field type as the memory size occupied by the field of that type can avoid memory overflow and enable parallel programs to run more efficiently and stably while maintaining reasonable memory usage.

[0042] In one example, the field lengths of each field type in the data table named all_data_type-1 are as follows: Figure 2 As shown, table_name is the table name, column_name is the field name, data_type is the field type, data_length is the field length of the character type, data_percision is the field length of the numeric type, data_scale is the field precision of the numeric type, and datetime_precision is the field length of the time type.

[0043] The number of bytes occupied by each type of field is calculated below, as shown in Tables 1-4.

[0044] The byte sizes occupied by numeric fields are shown in Table 1:

[0045]

[0046] Table 1

[0047] The byte size occupied by the date and time type fields is shown in Table 2:

[0048]

[0049]

[0050] Table 2

[0051] The byte sizes occupied by string type fields are shown in Table 3:

[0052]

[0053] Table 3

[0054] The memory space occupied by the char and varchar types is the character size. In MYSQL versions below 4.0, varchar(100) occupies 100 bytes. When storing Chinese characters in UTF8 encoding, only 33 characters can be stored (each character occupies 3 bytes). In MYSQL versions 5.0 and above, varchar(100) occupies 100 characters. Whether storing numbers, letters, or Chinese characters in UTF8 encoding (each Chinese character occupies 3 bytes), 100 characters can be stored. In UTF8 encoding, one Chinese character (including numbers) occupies 3 bytes, and in GBK encoding, one Chinese character (including numbers) occupies 2 bytes.

[0055] The byte sizes occupied by binary type fields are shown in Table 4:

[0056] Type Name illustrate Storage requirements bit(M) Bit field types Approximately (M+7) / 8 bytes binary(M) Fixed-length binary string M bytes varbinary(M) Variable-length binary string M+1 bytes

[0057] Table 4

[0058] Other field types: For text, JSON, blob, and other types, the maximum occupied space needs to be queried. For example, this can be obtained through a Structured Query Language (SQL) statement.

[0059] According to the number of bytes occupied by each field type shown in Table 1-Table 4, the following statistics can be obtained: Figure 2 The number of bytes occupied by a row of data corresponding to each field is shown. The number of bytes occupied by the data row with the largest memory usage corresponding to each field is added together to obtain the number of bytes occupied by multiple fields.

[0060] In one implementation, step S103, based on the shard size, uses multiple threads to process the data in parallel, including:

[0061] Step S1031: Divide the data to be processed into multiple shard data according to the shard size.

[0062] Step S1032: Process the plurality of data slices in parallel using a plurality of threads, wherein the number of threads is greater than the number of cores of the central processing unit, and the number of threads is a multiple of the number of cores of the central processing unit.

[0063] In practical applications, multiple threads are set up, each processing a data slice. Multiple threads process their own data slices in parallel. The number of threads can be determined by the number of CPU cores. The number of threads is greater than the number of CPU cores, and the number of threads is a multiple of the number of CPU cores. For example, for non-computational tasks, the number of threads can be twice the number of CPU cores.

[0064] How to slice the data to be processed according to the slice size is described in the following example:

[0065] In one implementation, step S1031, dividing the data to be processed into multiple shards according to the shard size, includes:

[0066] Step S10311: Determine the starting position and ending position corresponding to the unique key field in the data to be processed.

[0067] Step S10312: Divide the data to be processed into multiple shard data according to the starting position, the ending position and the shard size.

[0068] The unique key field can be the field corresponding to the primary key or the field corresponding to another unique key.

[0069] In one example, in a data archiving scenario, the primary key of the user source table is used as the shard key, the starting position and ending position corresponding to the primary key field are determined, and multiple data shards are obtained based on the obtained shard size, starting position, and ending position.

[0070] In one implementation, step S10312, dividing the data to be processed into multiple shards according to the starting position, the ending position, and the shard size, includes:

[0071] Based on the fragment size, multiple fragment positions are determined between the start position and the end position.

[0072] The data to be processed is divided into multiple shard data according to the starting position, the ending position and the multiple shard positions.

[0073] In this example, assuming that the primary key of the archive table "archive" is the "id" field of type bigint, and the calculated shard size is 10,000, the method for obtaining shard data can be implemented using the SQL statement:

[0074] Get the starting id: select id from archive limit 1;

[0075] Get the termination id: select id from archive order by id desc limit 1;

[0076] Get all fragment locations between the start id and the end id based on the fragment size: select id fromarchive where id = start id limit 1 offset fragment size;

[0077] The id obtained from each query is used as the starting id for the next query, and the query ends when the current id is greater than the ending id.

[0078] The data rows corresponding to the shard positions (including the start and end positions) include: 1, 10000, 20000, 30000, 40000, so the range of data rows corresponding to the 4 shard data is: id>=1and id <10000;id> =10000and id <20000;id> =20000and id <30000;id> =30000 and id<=40000. Utilize four threads: thread1, thread2, thread3, and thread4, and assign the acquired data of different shards to different threads for concurrent reading.

[0079] In one implementation, step S102, determining the slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit, includes:

[0080] Step S1021 , determining the total number of bytes to be processed by each CPU core according to the total memory size and the number of CPU cores.

[0081] Step S1022 , determining the slice size for data processing according to the total number of bytes for data processing using the core of each central processing unit and the memory size occupied by multiple fields.

[0082] In one example, the shard size is obtained according to the following formula:

[0083] Slice size = floorNumber (total memory size * 1024 * 1024 * 1024 / number of cores * sum of bytes occupied by each field in the data table * buffer multiple) (1)

[0084] Among them, total memory size * 1024 * 1024 * 1024 / number of cores represents the total number of bytes used for data processing by each CPU core, and floorNumber is a rounding-down function. For example, if the calculated shard size is 10032, the result obtained by the rounding-down function is 10000 (unit: row). The rounding precision can be set according to specific needs. The total memory size is the memory size of the application that performs the current data processing task, in GB. The sum of the bytes occupied by each field (column) of the data table is the number of bytes occupied by all fields of the data to be processed. The cache buffer multiple is greater than 1, and a certain amount of memory space is reserved for the memory occupation of the business code. The specific value of the buffer multiple can be configured as needed.

[0085] For example, assuming that the archiving business needs to run on a machine with 8GB of memory, 2 CPU cores, and 4 parallel read threads, the calculation process is as follows:

[0086] Slice size = 8*1024*1024*1024 / 2*the sum of the bytes occupied by each field in the data table*1.5(2)

[0087] Although the number of parallel processing threads here is 4, the number of machine cores is 2. Only two threads will execute reads at the same time, so divide by 2; some memory buffer occupied by business code can be reserved by (2 * the sum of the number of bytes occupied by each field in the data table) * 1.5 to reserve a certain amount of memory space for business code memory.

[0088] In one implementation, step S103 processes the data to be processed in parallel using multiple threads according to the shard size, including: reading the data to be processed in parallel from the source database using multiple threads according to the shard size, and writing the read data to the target database, thereby realizing data archiving.

[0089] The source and target databases can be databases in multiple nodes of a distributed system. By analyzing the field types in the source database's data tables, we can determine the memory usage of the data rows corresponding to each field type. This is then adaptively calculated based on the application's memory size to determine the most appropriate shard size for parallel processing. This allows parallel programs to read and write data while maintaining reasonable memory usage.

[0090] The technical solution of this application is described in detail below through a specific embodiment.

[0091] like Figure 3 FIG. 1 is a schematic diagram of a data processing method provided by an embodiment of the present application. Figure 3 As shown, the data type of each field in the data table TableA is obtained. The fields and field types in TableA include: column1 bigint(20)primary key; column2 int(10); column3 varchar(20); column json...

[0092] Execute step 1 to count the number of bytes occupied by the data row with the largest number of bytes occupied by the json field. Figure 3The SQL statement shown is: max json column size (byte) = select max (length (column4)) from TableA. After obtaining the number of bytes occupied by the data row with the largest number of bytes occupied by the json field (max json column size), the sum of the number of bytes occupied by each field in TableA is calculated: max line size (byte) = 8 + 4 + 20 * 3+max jsoncolumn size.

[0093] Execute step 2 to calculate the shard size. Assume that the memory of the cloud server (Elastic Compute Service, ECS) for data processing is 8G and the number of cores is 2. Figure 3 For the 8G2C shown, use the SQL statement to calculate the shard size:

[0094] Partition size = floorNumber(8*1024*1024*1024 / 2*max line size*buffer), and the partition size is 10000.

[0095] Execute step 3 to split the data (split partition). Column 1 is the primary key. Based on a shard size of 10,000, multiple shards are generated: partition 1: column 1 >= 1 and column 1 < 10,000; partition 2: column 1 >= 10,000 and column 1 < 20,000; partition 3: column 1 >= 20,000 and column 1 < 30,000; partition 4: column 1 >= 30,000 and column 1 <= 40,000.

[0096] Execute step 4 to read data. Use four threads in the Java Virtual Machine (JVM): thread 1, thread 2, thread 3, and thread 4, to read data from four different shards. Since there are only two cores, two threads are used to read data in parallel. After these two threads have finished processing data, two more threads are used to read data, each processing 10,000 rows. This process continues in this alternating manner until TableA is fully read.

[0097] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing device. Figure 4 FIG. 1 is a block diagram of a data processing device according to an embodiment of the present application, wherein the device includes:

[0098] The first determining module 401 is configured to determine the memory size occupied by the multiple fields according to the field types corresponding to the multiple fields of the data to be processed.

[0099] The second determining module 402 is configured to determine a slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit.

[0100] The data processing module 403 is used to process the data to be processed in parallel using multiple threads according to the size of the slices.

[0101] The data processing device provided in an embodiment of the present application determines the memory size occupied by multiple fields of the data to be processed based on the field types corresponding to the multiple fields; determines the shard size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit; and processes the data to be processed in parallel using multiple threads based on the shard size. In this embodiment, the shard size for data processing is adaptively determined based on the memory size occupied by the multiple fields of the data to be processed, the total memory size, and the number of cores of the central processing unit. There is no need to manually adjust the shard size, and the memory can be reasonably used when processing data in parallel, thereby improving memory utilization, reducing the risk of memory overflow, and making parallel operations more stable and efficient.

[0102] In one implementation, the first determining module 401 is configured to:

[0103] According to the field types corresponding to the multiple fields, the memory size occupied by the data row with the largest memory occupation corresponding to the multiple fields is determined; according to the memory size occupied by the data row with the largest memory occupation corresponding to the multiple fields, the memory size occupied by the multiple fields is determined.

[0104] In one implementation, the data processing module 403 is configured to:

[0105] The data to be processed is divided into multiple shard data according to the shard size; multiple threads are used to process the multiple shard data in parallel, and the number of threads is greater than the number of cores of the central processing unit, and the number of threads is a multiple of the number of cores of the central processing unit.

[0106] In one implementation, when the data processing module 403 divides the data to be processed into a plurality of shard data according to the shard size, it is configured to:

[0107] Determine the starting position and ending position corresponding to the unique key field in the data to be processed; and divide the data to be processed into multiple shard data according to the starting position, the ending position, and the shard size.

[0108] In one implementation, when the data processing module 403 divides the data to be processed into a plurality of fragments according to the starting position, the ending position, and the fragment size, it is configured to:

[0109] According to the shard size, multiple shard positions are determined between the starting position and the ending position; according to the starting position, the ending position and the multiple shard positions, the data to be processed is divided into multiple shard data.

[0110] In one implementation, the second determining module 402 is configured to:

[0111] Based on the total memory size and the number of CPU cores, the total number of bytes used for data processing by each CPU core is determined; based on the total number of bytes used for data processing by each CPU core and the memory size occupied by multiple fields, the shard size for data processing is determined.

[0112] In one implementation, the data processing module 403 is configured to:

[0113] Based on the shard size, multiple threads are used to read the data to be processed from the source database in parallel and write the read data to the target database, thereby achieving data archiving.

[0114] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0115] Figure 5 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 5 As shown, the electronic device includes: a memory 510 and a processor 520. The memory 510 stores a computer program that can be run on the processor 520. When the processor 520 executes the computer program, the method in the above embodiment is implemented. The number of the memory 510 and the processor 520 can be one or more.

[0116] The electronic device also includes:

[0117] The communication interface 530 is used to communicate with external devices and perform data exchange transmission.

[0118] If the memory 510, processor 520, and communication interface 530 are implemented independently, the memory 510, processor 520, and communication interface 530 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0119] Optionally, in a specific implementation, if the memory 510, the processor 520 and the communication interface 530 are integrated on a chip, the memory 510, the processor 520 and the communication interface 530 can communicate with each other through an internal interface.

[0120] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0121] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0122] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0123] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0124] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0125] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0126] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0128] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0129] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with such instruction execution systems, apparatuses or devices.

[0130] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0131] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0132] The above is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, comprising: Determining, based on the field types corresponding to the plurality of fields of the data to be processed, the memory size occupied by the data row with the largest memory occupation corresponding to the plurality of fields; Determining the memory sizes occupied by the multiple fields according to the memory sizes occupied by the data rows with the largest memory usage corresponding to the multiple fields; Determine a shard size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit; Determining the starting position and ending position corresponding to the unique key field in the data to be processed; Dividing the data to be processed into a plurality of shard data according to the starting position, the ending position and the shard size; Based on the shard size, the data to be processed is processed in parallel using multiple threads, including: Dividing the data to be processed into a plurality of shard data according to the shard size; Processing the plurality of data slices in parallel using a plurality of threads, wherein the number of the threads is greater than the number of cores of the central processing unit and is a multiple of the number of cores of the central processing unit; The step of dividing the data to be processed into a plurality of shard data according to the starting position, the ending position and the shard size includes: Determining a plurality of shard positions between the starting position and the ending position according to the shard size; Dividing the data to be processed into a plurality of shard data according to the starting position, the ending position, and the plurality of shard positions; The determining of the slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit includes: Determining the total number of bytes of data to be processed by each core of the central processing unit according to the total memory size and the number of cores of the central processing unit; The slice size for data processing is determined according to the total number of bytes for data processing using the core of each central processing unit and the memory size occupied by the multiple fields.

2. The method according to claim 1, wherein the processing of the data to be processed in parallel using multiple threads according to the shard size comprises: According to the shard size, multiple threads are used to read the data to be processed from the source database in parallel, and the read data is written into the target database, thereby realizing data archiving.

3. A data processing device, comprising: A first determining module is configured to determine, based on field types corresponding to the plurality of fields of the data to be processed, a memory size occupied by a data row with the largest memory usage corresponding to the plurality of fields; and determining the memory sizes occupied by the multiple fields according to the memory sizes occupied by the data rows that occupy the largest amount of memory, respectively corresponding to the multiple fields; A second determining module is used to determine a slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit; a data processing module, configured to determine a starting position and an ending position corresponding to a unique key field in the data to be processed; and to divide the data to be processed into a plurality of shard data according to the starting position, the ending position, and the shard size; Based on the shard size, the data to be processed is processed in parallel using multiple threads, including: Dividing the data to be processed into a plurality of shard data according to the shard size; Processing the plurality of data slices in parallel using a plurality of threads, wherein the number of the threads is greater than the number of cores of the central processing unit and is a multiple of the number of cores of the central processing unit; The step of dividing the data to be processed into a plurality of shard data according to the starting position, the ending position and the shard size includes: Determining a plurality of shard positions between the starting position and the ending position according to the shard size; Dividing the data to be processed into a plurality of shard data according to the starting position, the ending position, and the plurality of shard positions; The determining of the slice size for data processing based on the memory size occupied by the multiple fields, the total memory size, and the number of cores of the central processing unit includes: Determining the total number of bytes of data to be processed by each core of the central processing unit according to the total memory size and the number of cores of the central processing unit; The slice size for data processing is determined according to the total number of bytes for data processing using the core of each central processing unit and the memory size occupied by the multiple fields.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to claim 1 or 2 when executing the computer program.

5. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to claim 1 or 2 is implemented.

Images