A data processing method, device and storage medium

Abstract

The present invention discloses a data processing method and device, and a storage medium, comprising: when receiving a data processing instruction from a client, slicing the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed; determining an idle application manager from an application manager queue, the application manager queue including a plurality of application managers; using the idle application manager to process the plurality of slicing data to be processed; when the idle application manager has processed the plurality of slicing data to be processed, releasing the idle application manager to the application manager queue, so as to use the idle application manager to process the data corresponding to other data processing instructions, thereby improving the processing speed of the data processing device when processing the data to be processed.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a data processing method and device, and a storage medium. Background Art

[0002] In recent years, with the development of Internet technology, a large amount of data is generated every day. These data are distributed in different databases. How to import data from one database into other databases has become a new challenge in the field of big data processing.

[0003] In the prior art, when a data processing instruction is received from the client, it is necessary to apply for allocation of resources according to the amount of data to be processed corresponding to the data processing instruction, and establish an application manager (Application Master, AM) corresponding to the data amount. Then the data processing device can use the AM to apply for allocation of resources to process the data to be processed. Since the data processing device needs to consume time in the process of establishing the AM, this increases the time it takes for the data processing device to process the data to be processed and reduces the processing speed of the data processing device when processing the data to be processed. Summary of the invention

[0004] In order to solve the above technical problems, embodiments of the present invention are intended to provide a data processing method and device, and a storage medium, which can improve the processing speed of the data processing device when processing data to be processed.

[0005] The technical solution of the present invention is achieved in this way:

[0006] The present application provides a data processing method, the method comprising:

[0007] When receiving a data processing instruction from a client, slicing the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed;

[0008] Determining an idle application manager from an application manager queue, wherein the application manager queue includes a plurality of application managers;

[0009] Using the idle application manager to process the plurality of to-be-processed slice data;

[0010] When the idle application manager has processed the plurality of to-be-processed sliced ​​data, the idle application manager is released to the application manager queue, so that the idle application manager is used to process data corresponding to other data processing instructions.

[0011] In the above solution, before determining an idle application manager from the application manager queue, the method further includes:

[0012] When the data processing device is powered on, the plurality of application managers are created;

[0013] A plurality of application managers are sequentially added to the application manager queue.

[0014] In the above solution, the processing of the plurality of to-be-processed slice data by using the idle application manager includes:

[0015] One of the idle application managers is used in sequence to process one of the multiple slice data to be processed, until the multiple slice data to be processed are processed by the idle application manager.

[0016] In the above solution, before using the idle application manager to process the plurality of to-be-processed shard data, the method further includes:

[0017] Determine a first number of the idle application managers and a second number of the plurality of shard data to be processed;

[0018] When the difference between the first number and the second number satisfies a first threshold range, determining a second number of application managers from the idle application managers;

[0019] The plurality of to-be-processed shard data are processed respectively using the second number of application managers.

[0020] In the above solution, after determining the first number of the idle application managers and the second number of the plurality of shard data to be processed, the method further includes:

[0021] When the difference between the first quantity and the second quantity satisfies a second threshold range, determining a first quantity of shard data to be processed from the plurality of shard data to be processed, wherein the first quantity of shard data to be processed is part of the shard data to be processed among the plurality of shard data to be processed;

[0022] Using the idle application manager to process the first number of to-be-processed slice data respectively;

[0023] When the idle application manager has processed the first number of shard data to be processed, the idle application manager is released, and the remaining shard data to be processed is processed using the released idle application manager, where the remaining shard data to be processed is the shard data to be processed except the first number of shard data to be processed among the multiple shard data to be processed.

[0024] In the above solution, the step of using one of the idle application managers to process one of the plurality of to-be-processed fragment data includes:

[0025] Dividing the one to-be-processed fragmented data into a plurality of data segments;

[0026] Using a read thread pool, the plurality of data segments are read into a first storage unit and / or a second storage unit in the one application manager, wherein the first storage unit is a storage unit in the one application manager for temporarily storing the plurality of data segments, and the second storage unit is a storage unit in the one application manager for temporarily storing the plurality of data segments;

[0027] The plurality of data segments in the first storage unit and / or the second storage unit are read into a target device by using a write thread pool.

[0028] In the above solution, the step of using a read thread pool to read the plurality of data segments into a first storage unit and / or a second storage unit in the one application manager includes:

[0029] When the number of data segments stored in the first storage unit meets a first preset threshold range, using a reading thread pool to read first shard data into the first storage unit, the first shard data being data read from the multiple data segments into the first storage unit;

[0030] When the number of data segments stored in the first storage unit meets a second preset threshold range, detecting the number of data segments stored in the second storage unit;

[0031] When the number of data segments stored in the second storage unit meets a third preset threshold range, the second shard data is read into the second storage unit using a read thread pool until the multiple data segments are read into the first storage unit and / or the second storage unit, and the second shard data is the data read into the second storage unit from the multiple data segments.

[0032] In the above solution, the step of using the write thread pool to read the plurality of data segments in the first storage unit and / or the second storage unit into the target device includes:

[0033] When the number of data segments stored in the first storage unit meets the second preset threshold range, using the write thread pool to read the first fragment data into the target device;

[0034] When the number of data segments stored in the first storage unit meets the first preset threshold range, detecting the number of data segments stored in the second storage unit;

[0035] When the number of data segments stored in the second storage unit meets the second preset threshold range, the second shard data is read into the target device using the write thread pool until the multiple data segments in the first storage unit and / or the second storage unit are read into the target device.

[0036] In the above solution, after using the write thread pool to read the multiple data segments in the first storage unit and / or the second storage unit into the target device, the method further includes:

[0037] Within a first preset time period, obtaining first running status information of the read thread pool and second running status information of the write thread pool;

[0038] The standby server is used to record the first operation status information and the second operation status information.

[0039] In the above solution, after the idle application manager is used to process the plurality of to-be-processed shard data, the method further comprises:

[0040] Within a second preset time period, obtaining processing progress information of the plurality of to-be-processed shard data;

[0041] Feedback the processing progress information to the client.

[0042] The present application provides a data processing device, the device comprising:

[0043] The processing unit is used for, when receiving a data processing instruction from a client, performing slicing processing on the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed; and processing the plurality of slicing data to be processed by using an idle application manager;

[0044] A determining unit, configured to determine the idle application manager from an application manager queue, wherein the application manager queue includes a plurality of application managers;

[0045] A release unit is used to release the idle application manager to the application manager queue when the idle application manager has processed the multiple sliced ​​data to be processed, so as to use the idle application manager to process the data corresponding to other data processing instructions.

[0046] The present application provides a data processing device, the device comprising:

[0047] A memory, a processor and a communication bus, wherein the memory communicates with the processor via the communication bus, and the memory stores a data processing program executable by the processor. When the data processing program is executed, the data processing method described above is executed by the processor.

[0048] An embodiment of the present application provides a storage medium having a computer program stored thereon, which is applied to a data processing device, and is characterized in that the computer program implements the above-mentioned data processing method when executed by a processor.

[0049] The embodiment of the present invention provides a data processing method and device, and a storage medium. The data processing method includes: when receiving a data processing instruction from a client, slicing the data to be processed corresponding to the data processing instruction to obtain multiple slicing data to be processed; determining an idle application manager from an application manager queue, the application manager queue including multiple application managers; using the idle application manager to process the multiple slicing data to be processed; when the idle application manager has processed the multiple slicing data to be processed, releasing the idle application manager to the application manager queue, so as to use the idle application manager to process the data corresponding to other data processing instructions. By adopting the above method implementation scheme, the data processing device sets the application manager queue, and when the data processing device receives the data processing instruction from the client, the data processing device can use the multiple application managers in the application manager queue to process the data to be processed corresponding to the data processing instruction, and the data processing device does not need to establish an application manager according to the data amount of the data to be processed corresponding to the data processing instruction, thereby reducing the processing time of the data processing device when processing the data to be processed, and improving the processing speed of the data processing device when processing the data to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of an exemplary data processing method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of a method for an exemplary data processing device provided in an embodiment of the present application to process a plurality of to-be-processed sliced ​​data using an idle application manager;

[0053] Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application Figure 1 ;

[0054] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0056] With the development of Internet technology, a large amount of data is generated every day, and these data are distributed in different databases, such as file systems, relational databases, and non-relational databases, etc. In the prior art, when it is necessary to synchronize data in two databases, big data processing technology and big data import and export tools are used to import and export the data to be synchronized. Among them, big data processing technology is: Yet Another Resource Negotiator (Yarn), and big data import and export tools include: DataX, Sqoop, Kettle and TurboDX, etc. Among the numerous data import and export tools, DataX and Sqoop are two open source tools that are widely used.

[0057] A new distributed system resource manager (Apache Hadoop Yarn) is a general resource management system that can provide unified resource management and scheduling for upper-level applications, improve cluster utilization, manage resources in a unified manner, and achieve data sharing. This general resource management system separates the resource management function of the job tracker (JobTracker) from the job scheduling / monitoring function. It does this by creating a global resource manager (ResourceManager, RM) and several resource allocation managers (ApplicationMaster, AM) for applications. Among them, the application refers to the traditional parallel program computing (MapReduce) job mode.

[0058] DataX is an open source data import and export tool that runs on a single node. The import and export functions of DataX are completed by read threads and write threads. The read thread reads data from the source end to the memory, and the write thread reads data from the memory and writes it to the destination end, importing and exporting data in a non-landing manner. Since the DataX tool only runs on a single node, when it receives data processing instructions from the client, it can only process the data corresponding to the data processing instructions on a single node, and cannot perform distributed data processing on the data. When processing large amounts of data in the big data field, the speed of data processing is reduced.

[0059] The open source tool Sqoop is mainly used for importing and exporting big data between distributed system (Hadoop) components and relational databases. Sqoop can convert the data processing instructions received from the client into parallel program calculation (MapReduce) operations to process the data corresponding to the data processing instructions. Specifically, when processing the data corresponding to the data processing instructions, it is necessary to first apply for resource allocation to start AM, and then apply for resource allocation in AM before processing the data. The two resource applications and allocations take up a lot of time, that is, the processing speed is slow when using Sqoop to process the data corresponding to the data processing instructions.

[0060] The problems existing in the prior art can be specifically solved by the method in Example 1.

[0061] Embodiment 1

[0062] The present application embodiment provides a data processing method, Figure 1 A data processing method flow provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the data processing method may include:

[0063] S101. When a data processing instruction is received from a client, slice the data to be processed corresponding to the data processing instruction to obtain a plurality of sliced ​​data to be processed.

[0064] A data processing method provided in an embodiment of the present application is suitable for a scenario in which a data processing device is used to process data to be processed corresponding to a data processing instruction of a client.

[0065] In an embodiment of the present application, the data to be processed is data that the data processing instruction instructs the data processing device to process. The data to be processed may be data carried in the data processing instruction, or may be data received by the data processing device before receiving the data processing instruction, or may be data received by the data processing device after receiving the data processing instruction. The specific data may be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0066] In an embodiment of the present application, the data processing device may perform slicing processing on the data to be processed by comparing the data to be processed with a preset data length and slicing the data to be processed according to the preset data length. The data processing device may also calculate a hash value of the data to be processed according to data features of the data to be processed and slice the data to be processed using the hash value of the data to be processed. The data processing device may also slice the data to be processed according to a preset number of bytes. The specific method may be determined according to actual conditions and is not limited to this in the embodiment of the present application.

[0067] In the embodiment of the present application, after the data processing device slices the data to be processed, the data processing device can obtain a plurality of sliced ​​data to be processed.

[0068] S102: Determine an idle application manager from an application manager queue, where the application manager queue includes a plurality of application managers.

[0069] In the embodiment of the present application, when the data processing device receives a data processing instruction, the data processing device determines an idle application manager from the application manager queue.

[0070] It should be noted that the application manager queue includes multiple application managers, and an idle application manager is an application manager in the application manager queue that does not process data.

[0071] In the embodiment of the present application, before the data processing device determines an idle application manager from the application manager queue, the data processing device creates a plurality of application managers after the data processing device is powered on.

[0072] It should be noted that the multiple application managers may have the same multiple resource spaces or may have different multiple resource spaces. The specific number may be determined based on actual conditions and is not limited in this embodiment of the present application.

[0073] In an embodiment of the present application, the number of multiple application managers can be a preset number, that is, a preset number is set in the data processing device, and when the data processing device is powered on, the data processing device creates the preset number of application managers. It can also be that after the data processing device is powered on, according to the creation instructions of the application managers received from the client, the number of multiple application managers is determined. The specific number can be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0074] In the embodiment of the present application, after the data processing device creates multiple application managers, the data processing device adds the multiple application managers to the application manager queue in sequence.

[0075] It should be noted that the data processing device may sequentially add the multiple application managers to the application manager queue in the order of their creation time, or the data processing device may randomly number the multiple application managers and sequentially add the multiple application managers to the application manager queue in the order of the random numbers. There may also be other methods of sequentially adding multiple application managers to the application manager queue. The specific method may be determined according to actual conditions, and the embodiments of the present application are not limited to this.

[0076] S103: Use an idle application manager to process a plurality of to-be-processed slice data.

[0077] In the embodiment of the present application, after the data processing device determines an idle application manager from the application manager queue, the data processing device uses the idle application manager to process multiple shard data to be processed.

[0078] In an embodiment of the present application, the process of a data processing device using an idle application manager to process multiple shard data to be processed is specifically: the data processing device sequentially uses an application manager among the idle application managers to process one of the multiple shard data to be processed, until the idle application manager is used to process multiple shard data to be processed.

[0079] In an embodiment of the present application, before the data processing device sequentially uses an idle application manager to process one of the multiple shard data to be processed, the data processing device will determine one shard data to be processed from the multiple shard data to be processed. The data processing device can determine one shard data to be processed from the multiple shard data to be processed by randomly selecting one shard data to be processed from the multiple shard data to be processed, or by selecting one shard data to be processed according to the data arrangement order of the multiple shard data to be processed. The specific method can be determined according to the actual situation, and the embodiment of the present application does not limit this.

[0080] In the embodiment of the present application, after the data processing device determines a piece of shard data to be processed from a plurality of shard data to be processed, the data processing device determines an application manager from idle application managers.

[0081] In an embodiment of the present application, the data processing device determines an application manager from the idle application managers. The data processing device may randomly select an application manager from the idle application managers, or the data processing device may select an application manager according to the storage order of the idle application managers in the application manager queue. The specific method may be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0082] In an embodiment of the present application, after the data processing device determines an application manager from the idle application managers, the data processing device uses the application manager to process a shard data to be processed, until the data processing device uses each application manager in the idle application managers to process each shard data in multiple shard data to be processed.

[0083] In an embodiment of the present application, the data processing device uses an application manager to process a piece of data to be processed, and the data processing device can read the piece of data to be processed into a target device through the application manager.

[0084] It should be noted that the target device can be the device information carried in the data processing instruction of the client. When the data processing device receives the data processing instruction of the client, the data processing device can determine the target device based on the device information carried in the data processing instruction. After the data processing device processes the data to be processed, it can transmit the data to be processed to the target device.

[0085] In an embodiment of the present application, the process in which a data processing device uses an idle application manager to process a slice of data to be processed among multiple slices of data to be processed specifically includes: the data processing device divides a slice of data to be processed into multiple data segments.

[0086] It should be noted that the data processing device divides a piece of data to be processed into multiple data segments in a manner that the data processing device compares a piece of data to be processed with a preset data segment length and divides the piece of data to be processed into multiple data segments according to the preset data segment length. The data processing device may also divide the piece of data to be processed into multiple data segments according to a preset byte length. The specific manner in which other data processing devices divide a piece of data to be processed into multiple data segments can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0087] In an embodiment of the present application, after the data processing device divides a shard data to be processed into multiple data segments, the data processing device uses a read thread pool to read the multiple data segments into a first storage unit and / or a second storage unit in an application manager.

[0088] It should be noted that the first storage unit is a storage unit for temporarily storing multiple data segments in an application manager, and the second storage unit is a storage unit for temporarily storing multiple data segments in an application manager.

[0089] It should also be noted that the storage capacity of the first storage unit may be the same as the storage capacity of the second storage unit, or the storage capacity of the first storage unit may be different from the storage capacity of the second storage unit, for example, the storage capacity of the first storage unit is greater than the storage capacity of the second storage unit, or the storage capacity of the first storage unit is less than the storage capacity of the second storage unit. The specific capacity can be determined based on actual conditions, and the embodiments of the present application are not limited to this.

[0090] In an embodiment of the present application, a process in which a data processing device uses a read thread pool to read multiple data segments into a first storage unit and / or a second storage unit in an application manager specifically includes: when the number of data segments stored in the first storage unit meets a first preset threshold range, using the read thread pool to read the first shard data into the first storage unit.

[0091] It should be noted that the first fragment data is data read into the first storage unit from multiple data segments.

[0092] It should be noted that the first preset threshold range can be the minimum storage quantity range for the first storage unit to store the first shard data. Exemplarily, the first preset threshold range can be a range of less than or equal to 0 segments, that is, the first storage unit storage can store multiple data segments less than or equal to 0 segments. The first preset threshold range can be a range of less than or equal to 3 segments, that is, the first storage unit storage can store multiple data segments less than or equal to 3 segments. The specific first preset threshold range can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0093] In an embodiment of the present application, after the data processing device uses the read thread pool to read the first shard data into the first storage unit, the data processing unit detects the amount of data segments stored in the first storage unit. When the data processing device determines that the number of data segments stored in the first storage unit meets the second preset threshold range, the data processing device detects the number of data segments stored in the second storage unit.

[0094] It should be noted that the second preset threshold range can be the maximum storage quantity range of the first shard data stored in the first storage unit. Exemplarily, the second preset threshold range can be a range greater than or equal to 8000 segments, and the second preset threshold range can be a range greater than or equal to 5000 segments. The specific second preset threshold range can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0095] In an embodiment of the present application, after the data processing device detects the number of data segments stored in the second storage unit, when the data processing device determines that the number of data segments stored in the second storage unit meets the third preset threshold range, the data processing device uses the read thread pool to read the second shard data into the second storage unit until multiple data segments are read into the first storage unit and / or the second storage unit.

[0096] It should be noted that the second fragment data is data read into the second storage unit from multiple data segments.

[0097] It should be noted that the third preset threshold range can be the minimum storage quantity range for the second storage unit to store the second fragment data. Exemplarily, the third preset threshold range can be a range of less than or equal to 2 segments, that is, the third storage unit storage can store multiple data segments less than or equal to 2 segments. The third preset threshold range can be a range of less than or equal to 1 segment, that is, the third storage unit storage can store multiple data segments less than or equal to 1 segment. The specific third preset threshold range can be determined according to actual conditions, and the embodiments of the present application are not limited to this.

[0098] In an embodiment of the present application, the data processing device may use a read thread pool to read data segments other than the first fragment among the multiple data segments into a second storage unit to obtain second fragment data.

[0099] In an embodiment of the present application, after the data processing device uses a read thread pool to read multiple data segments into a first storage unit and / or a second storage unit in an application manager, the data processing device uses a write thread pool to read the multiple data segments in the first storage unit and / or the second storage unit into a target device.

[0100] In an embodiment of the present application, a process in which a data processing device uses a write thread pool to read multiple data segments from a first storage unit and / or a second storage unit into a target device includes when the data processing device determines that the number of data segments stored in the first storage unit meets a second preset threshold range, the data processing device uses the write thread pool to read the first shard data into the target device.

[0101] In the embodiment of the present application, the data processing device may read the first fragment data into the target device in segments in the form of data segments.

[0102] In an embodiment of the present application, after the data processing device uses the write thread pool to read the first shard data into the target device, the data processing device detects the number of data segments stored in the first storage unit. When the data processing device determines that the number of data segments stored in the first storage unit meets the first preset threshold range, the data processing device detects the number of data segments stored in the second storage unit.

[0103] In an embodiment of the present application, after the data processing device detects the number of data segments stored in the second storage unit, when the data processing device determines that the number of data segments stored in the second storage unit meets the second preset threshold range, the data processing device uses the write thread pool to read the second shard data into the target device until multiple data segments in the first storage unit and / or the second storage unit are read into the target device.

[0104] In an embodiment of the present application, after using the write thread pool to read multiple data segments in the first storage unit and / or the second storage unit into the target device, the data processing device obtains first operating status information of the read thread pool and second operating status information of the write thread pool within a first preset time period.

[0105] It should be noted that the first preset time period may be the time interval for the data processing device to obtain the first operating status information and the second operating status information. The first preset time period may be 2ms, the first preset time period may be 0.5ns, or the first preset time period may be 3ns. The specific first preset time period may be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0106] It should be noted that the first running status information includes the reading data speed information of the reading thread pool, the length of time the reading thread pool reads data, the number of multiple data segments read by the reading thread pool, and the time point information when the reading thread pool reads data, etc., and the second running status information includes the reading data speed information of the writing thread pool, the length of time the writing thread pool reads data, the number of multiple data segments read by the writing thread pool, and the time point information when the writing thread pool reads data, etc.

[0107] In an embodiment of the present application, after the data processing device obtains the first running status information of the read thread pool and the second running status information of the write thread pool, the data processing device uses the backup server to record the first running status information and the second running status information.

[0108] It should be noted that the backup server may be a distributed coordination server (Zookeeper) or a Hadoop cluster server, and the specific one may be determined according to the actual situation, and the embodiment of the present application does not limit this.

[0109] It can be understood that the data processing device uses a backup server to record the first operating status information and the second operating status information. When the data processing device fails and the data processing device is restarted, the data processing device can also obtain processing information on the data to be processed through the backup server, thereby improving the intelligence of the data processing device in processing the data to be processed.

[0110] In an embodiment of the present application, before the data processing device uses an idle application manager to process a plurality of to-be-processed shard data, the data processing device determines a first number of idle application managers and a second number of the plurality of to-be-processed shard data.

[0111] In an embodiment of the present application, after the data processing device determines a first number of idle application managers and a second number of multiple shard data to be processed, when the data processing device determines that the difference between the first number and the second number satisfies a first threshold range, the data processing device determines a second number of application managers from the idle application managers.

[0112] It should be noted that the first number is the number of multiple shard data to be processed, and the second number is the number of idle application managers.

[0113] In the embodiment of the present application, the first threshold range is a range of a quantity difference obtained by subtracting the second quantity from the first quantity, and the first threshold range may be a range greater than or equal to 0.

[0114] In the embodiment of the present application, after the data processing device determines a second number of application managers from the idle application managers, the data processing device uses the second number of application managers to process the multiple shard data to be processed respectively.

[0115] In an embodiment of the present application, the data processing device uses a second number of application managers to process multiple shard data to be processed respectively, so that the data processing device can read the multiple shard data to be processed into the target device respectively through the second number of application managers.

[0116] In an embodiment of the present application, after the data processing device determines a first number of idle application managers and a second number of multiple shard data to be processed, when the data processing device determines that the difference between the first number and the second number satisfies a second threshold range, the data processing device determines a first number of shard data to be processed from the multiple shard data to be processed.

[0117] It should be noted that the first number of shard data to be processed is part of the shard data to be processed among the multiple shard data to be processed.

[0118] In the embodiment of the present application, the second threshold range is a range of a quantity difference obtained by subtracting the second quantity from the first quantity, and the second threshold range may be a range less than 0.

[0119] In an embodiment of the present application, after the data processing device determines a first number of shard data to be processed from a plurality of shard data to be processed, the data processing device uses an idle application manager to process the first number of shard data to be processed respectively.

[0120] In an embodiment of the present application, the data processing device uses an idle application manager to process the first number of shard data to be processed respectively, and the data processing device can read the first number of shard data to be processed into the target device respectively through the idle application manager.

[0121] In an embodiment of the present application, after the idle application manager is used to process the first number of shard data to be processed respectively, the data processing device obtains the progress information of the idle application manager on the processing of the first number of shard data to be processed within a preset time period. When the data processing device determines that the idle application manager has processed the first number of shard data to be processed, the data processing device releases the idle application manager and uses the released idle application manager to process the remaining shard data to be processed.

[0122] It should be noted that the preset time period can be the time interval for the data processing device to obtain the progress information of the first number of shard data to be processed. The preset time period can be 2ms, the preset time period can also be 0.5ns, and the preset time period can also be 3ns. The specific preset time period can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0123] It should be noted that the remaining shard data to be processed is the shard data to be processed except the first number of shard data to be processed among the multiple shard data to be processed.

[0124] In an embodiment of the present application, the data processing device uses the released idle application manager to process the remaining slice data to be processed, and the data processing device can read the remaining slice data to be processed into the target device through the released idle application manager.

[0125] In the embodiment of the present application, after the data processing device processes a plurality of to-be-processed shard data using an idle application manager, the data processing device obtains processing progress information of the plurality of to-be-processed shard data within a second preset time period.

[0126] It should be noted that the processing progress information includes the remaining processing time, the remaining data volume, or the ratio of the remaining data volume to the total data volume of multiple shard data to be processed, etc. The specific processing progress information can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0127] In the embodiment of the present application, the data processing device may determine the processing progress information of the plurality of shard data to be processed through the first operation status information, the second operation status information and the plurality of shard data to be processed.

[0128] It should be noted that when the processing progress information includes the remaining processing time, the data processing device can determine the processing time length of the multiple processed shard data to be processed and the data volume of the multiple processed shard data to be processed based on the first operating status information and the second operating status information, and predict the remaining processing time. When the processing progress information includes the remaining data volume, the data processing device can determine the data volume of the multiple processed shard data to be processed based on the first operating status information and the second operating status information, and determine the remaining data volume based on the total data volume of the multiple shard data to be processed and the data volume of the multiple processed shard data to be processed.

[0129] It should be noted that the second preset time period is the time interval for obtaining the processing progress information of multiple shard data to be processed. The second preset time period can be 2ms, the second preset time period can also be 0.5ns, and the second preset time period can also be 3ns. The specific second preset time period can be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0130] In an embodiment of the present application, after the data processing device obtains processing progress information of a plurality of shard data to be processed within a second preset time period, the data processing device feeds back the processing progress information to the client.

[0131] It can be understood that the data processing device feeds back the processing progress information to the client, so that the client can understand information such as the processing speed of the data processing device on the data to be processed corresponding to the data processing instruction, thereby improving the intelligence of the data processing device in processing the data to be processed corresponding to the data processing instruction.

[0132] S104. When the idle application manager has processed a plurality of to-be-processed sliced ​​data, the idle application manager is released to the application manager queue, so that the idle application manager can be used to process data corresponding to other data processing instructions.

[0133] In an embodiment of the present application, when a data processing device uses an idle application manager to process multiple shard data to be processed, the data processing device obtains progress information of the idle application manager processing the multiple shard data to be processed within a preset time period. When the data processing device determines that the idle application manager has completed processing the multiple shard data to be processed, the data processing device releases the idle application manager to the application manager queue.

[0134] It should be noted that the preset time period can be the time interval for the data processing device to obtain the progress information of multiple shard data to be processed. The preset time period can be 2ms, the preset time period can also be 0.5ns, and the preset time period can also be 3ns. The specific preset time period can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0135] In an embodiment of the present application, other data processing instructions may be data processing instructions received by the data processing device after processing the data processing instructions of the client, or may be data processing instructions received by the data processing device after receiving the data processing instructions of the client. The specific instructions may be determined based on actual conditions, and the embodiment of the present application does not limit this.

[0136] It can be understood that the data processing device releases the idle application manager to the application manager queue, so that when the data processing device receives data corresponding to other data processing instructions, it can use the released idle application manager to process the data corresponding to the other data processing instructions. The data processing device does not need to apply for resource allocation based on the data volume corresponding to other data processing instructions and establish an application manager corresponding to the data volume, thereby improving the processing speed of the data processing device when processing the data to be processed.

[0137] Exemplarily, the flow chart of the method for a data processing device to process data to be processed is as follows: Figure 2 As shown:

[0138] 1. The data processing device creates an application manager queue.

[0139] It should be noted that the data processor can create an application manager queue when the data processor is started, that is, when the data processing device is powered on.

[0140] 2. When the data processing device receives the data processing instruction from the client, the data processing device segments the data to be processed to obtain a plurality of segmented data to be processed.

[0141] 3. The data processing device determines a piece of shard data to be processed from a plurality of shard data to be processed.

[0142] 4. When there are idle application managers in the application manager queue, the data processing device uses the idle application managers to process each of the multiple slice data to be processed in turn.

[0143] 5. When an idle application manager has processed multiple pending slice data, the idle application manager is released to the application manager queue.

[0144] 6. The data processing device sends the operation status information of the idle application manager when processing the plurality of data to be processed to the standby server.

[0145] In an embodiment of the present application, after the data processing device executes step 4, the data processing device executes step 5; or after the data processing device executes step 4, the data processing device executes step 6. The specific details can be determined according to actual conditions, and this application is an embodiment and does not limit this.

[0146] 7. The data processing device obtains the data to be processed and the operation status information, and obtains the processing progress information according to the data to be processed and the operation status information.

[0147] 8. When the data processing device determines that the processing of the data to be processed is completed based on the data to be processed and the operation status information, the data processing device outputs the processing progress information.

[0148] 9. When the data processing device determines, based on the data to be processed and the running status information, that the data to be processed has not been completely processed, the data processing device continues to obtain the running status information.

[0149] In an embodiment of the present application, after the data processing device executes step 7, the data processing device executes step 8; or after the data processing device executes step 7, the data processing device executes step 9. The specific details can be determined according to actual conditions, and this application is an embodiment and does not limit this.

[0150] Exemplarily, the process of the data processing device using the idle application manager to process multiple shard data to be processed is as follows: Figure 3 As shown:

[0151] 11. The data processing device creates a first storage unit and a second storage unit.

[0152] 12. The data processing device creates a read thread pool and runs the read thread pool.

[0153] 13. The data processing device creates a write thread pool and runs the write thread pool.

[0154] 14. The data processing device obtains first running status information of the read thread pool and second running status information of the write thread pool, and writes the first running status information and the second running status information to the standby server.

[0155] 15. When the data processing device determines, based on the first running status information and the second running status information, that both the read thread pool and the write thread pool have finished running, the data processing device ends running.

[0156] 16. When the data processing device determines, based on the first running status information and the second running status information, that both the reading thread pool and the writing thread have not finished running, the data processing device continues to obtain the first running status information and the second running status information.

[0157] In an embodiment of the present application, after the data processing device executes step 14, the data processing device executes step 15; or after the data processing device executes step 14, the data processing device executes step 16. The specific details can be determined according to actual conditions, and this application is an embodiment and does not limit this.

[0158] It can be understood that by setting up an application manager queue, when the data processing device receives a data processing instruction from the client, the data processing device can use multiple application managers in the application management queue to process the pending data corresponding to the data processing instruction. The data processing device does not need to establish an application manager based on the data volume of the pending data corresponding to the data processing instruction, thereby reducing the processing time of the data processing device for processing the pending data and improving the processing speed of the data processing device for processing the pending data.

[0159] Embodiment 2

[0160] Based on the same inventive concept of the first embodiment, the present application embodiment provides a data processing device 1, corresponding to a data processing method; Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application Figure 1 , the data processing device 1 may include:

[0161] The processing unit 11 is used for, when receiving a data processing instruction from a client, performing slicing processing on the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed; and processing the plurality of slicing data to be processed by using an idle application manager;

[0162] A determining unit 12, configured to determine the idle application manager from an application manager queue, wherein the application manager queue includes a plurality of application managers;

[0163] The release unit 13 is used to release the idle application manager to the application manager queue when the idle application manager has processed the multiple to-be-processed slice data, so as to use the idle application manager to process the data corresponding to other data processing instructions.

[0164] In some embodiments of the present application, the apparatus further comprises a creating unit and an adding unit;

[0165] The creation unit is used to create the multiple application managers when the data processing device is powered on;

[0166] The adding unit is used to add multiple application managers to the application manager queue in sequence.

[0167] In some embodiments of the present application, the processing unit 11 is used to sequentially use one of the idle application managers to process one of the multiple shard data to be processed until the multiple shard data to be processed are processed using the idle application manager.

[0168] In some embodiments of the present application, the determining unit 12 is used to determine a first number of the idle application managers and a second number of the plurality of shard data to be processed; when a difference between the first number and the second number satisfies a first threshold range, determine a second number of application managers from the idle application managers;

[0169] The processing unit 11 is used to use the second number of application managers to process the multiple shard data to be processed respectively.

[0170] In some embodiments of the present application,

[0171] The determining unit 12 is configured to determine a first number of shard data to be processed from the plurality of shard data to be processed when the difference between the first number and the second number satisfies a second threshold range, wherein the first number of shard data to be processed is part of the shard data to be processed among the plurality of shard data to be processed;

[0172] The processing unit 11 uses the idle application manager to process the first number of to-be-processed shard data respectively; uses the released idle application manager to process the remaining to-be-processed shard data, wherein the remaining to-be-processed shard data is the to-be-processed shard data among the multiple to-be-processed shard data except the first number of to-be-processed shard data;

[0173] The releasing unit 13 is configured to release the idle application manager when the idle application manager has processed the first amount of to-be-processed slice data.

[0174] In some embodiments of the present application, the device further includes a slicing unit and a data reading unit.

[0175] The slicing unit is used to divide the one slicing data to be processed into multiple data segments;

[0176] The data reading unit is used to use a reading thread pool to read the multiple data segments into a first storage unit and / or a second storage unit in the application manager, wherein the first storage unit is a storage unit in the application manager for temporarily storing the multiple data segments, and the second storage unit is a storage unit in the application manager for temporarily storing the multiple data segments; and use a writing thread pool to read the multiple data segments in the first storage unit and / or the second storage unit into a target device.

[0177] In some embodiments of the present application, the device further comprises a detection unit;

[0178] The data reading unit is used to read first shard data into the first storage unit by using a reading thread pool when the number of data segments stored in the first storage unit meets a first preset threshold range, wherein the first shard data is data read into the first storage unit from the multiple data segments; and to read second shard data into the second storage unit by using a reading thread pool when the number of data segments stored in the second storage unit meets a third preset threshold range, until the multiple data segments are read into the first storage unit and / or the second storage unit, wherein the second shard data is data read into the second storage unit from the multiple data segments;

[0179] The detection unit is used to detect the number of data segments stored in the second storage unit when the number of data segments stored in the first storage unit meets a second preset threshold range.

[0180] In some embodiments of the present application, the data reading unit is used to read the first shard data into the target device using the write thread pool when the number of data segments stored in the first storage unit meets the second preset threshold range; and read the second shard data into the target device using the write thread pool when the number of data segments stored in the second storage unit meets the second preset threshold range, until the multiple data segments in the first storage unit and / or the second storage unit are read into the target device;

[0181] The detection unit is used to detect the number of data segments stored in the second storage unit when the number of data segments stored in the first storage unit meets the first preset threshold range.

[0182] In some embodiments of the present application, the device further includes an acquisition unit and a recording unit;

[0183] The acquisition unit is used to acquire the first running status information of the read thread pool and the second running status information of the write thread pool within a first preset time period;

[0184] The recording unit is used to record the first operating status information and the second operating status information using the standby server.

[0185] In some embodiments of the present application, the device further comprises a feedback unit;

[0186] The acquisition unit is used to acquire the processing progress information of the plurality of to-be-processed slice data within a second preset time period;

[0187] The feedback unit is used to feed back the processing progress information to the client.

[0188] It should be noted that, in actual applications, the above-mentioned processing unit 11, determination unit 12, release unit 13, creation unit, addition unit, slicing unit, data reading unit, detection unit, acquisition unit, recording unit and feedback unit can be implemented by a processor 14 on the data processing device 1, specifically a CPU (Central Processing Unit), MPU (Microprocessor Unit), DSP (Digital Signal Processing) or a field programmable gate array (FPGA); the above-mentioned data storage can be implemented by a memory 15 on the data processing device 1.

[0189] The embodiment of the present invention further provides a data processing device 1, such as Figure 5 As shown, the data processing device 1 includes: a processor 14, a memory 15 and a communication bus 16. The memory 15 communicates with the processor 14 via the communication bus 16. The memory 15 stores a program executable by the processor 14. When the program is executed, the data processing method described above is executed by the processor 14.

[0190] In practical applications, the memory 15 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0191] An embodiment of the present invention provides a computer-readable storage medium having a computer program thereon, and when the program is executed by the processor 14, the data processing method as described above is implemented.

[0192] It can be understood that by setting up an application manager queue, when the data processing device receives a data processing instruction from the client, the data processing device can use multiple application managers in the application management queue to process the pending data corresponding to the data processing instruction. The data processing device does not need to establish an application manager based on the data volume of the pending data corresponding to the data processing instruction, thereby reducing the processing time of the data processing device for processing the pending data and improving the processing speed of the data processing device for processing the pending data.

[0193] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0194] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a data processing device, the method comprises: When receiving a data processing instruction from a client, slicing the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed; Determining an idle application manager from an application manager queue, wherein the application manager queue includes a plurality of application managers; Using the idle application manager to process the plurality of to-be-processed slice data; When the idle application manager has processed the plurality of to-be-processed sliced ​​data, the idle application manager is released to the application manager queue, so as to use the idle application manager to process data corresponding to other data processing instructions; The step of processing the plurality of to-be-processed slice data by using the idle application manager includes: sequentially using one of the idle application managers to process one of the plurality of to-be-processed fragmented data until the plurality of to-be-processed fragmented data are processed by using the idle application manager; The using one of the idle application managers to process one of the plurality of to-be-processed fragment data comprises: Dividing the one to-be-processed fragment data into a plurality of data segments; When the number of data segments stored in the first storage unit meets a first preset threshold range, using a read thread pool to read first shard data into the first storage unit, the first shard data being data read from the multiple data segments into the first storage unit; When the number of data segments stored in the first storage unit meets a second preset threshold range, detecting the number of data segments stored in the second storage unit; When the number of data segments stored in the second storage unit meets a third preset threshold range, the second shard data is read into the second storage unit by using a read thread pool until the multiple data segments are read into the first storage unit and / or the second storage unit, the second shard data is data read into the second storage unit from the multiple data segments, the first storage unit is a storage unit in the one application manager that temporarily stores the multiple data segments, and the second storage unit is a storage unit in the one application manager that temporarily stores the multiple data segments; The plurality of data segments in the first storage unit and / or the second storage unit are read into a target device by using a write thread pool.

2. The method according to claim 1, characterized in that Before determining an idle application manager from the application manager queue, the method further includes: When the data processing device is powered on, the plurality of application managers are created; A plurality of application managers are sequentially added to the application manager queue.

3. The method according to claim 1, characterized in that Before using the idle application manager to process the plurality of to-be-processed shard data, the method further includes: Determine a first number of the idle application managers and a second number of the plurality of shard data to be processed; When the difference between the first number and the second number satisfies a first threshold range, determining a second number of application managers from the idle application managers; The plurality of to-be-processed shard data are processed respectively using the second number of application managers.

4. The method according to claim 3, characterized in that After determining the first number of idle application managers and the second number of the plurality of to-be-processed shard data, the method further comprises: When the difference between the first quantity and the second quantity satisfies a second threshold range, determining a first quantity of shard data to be processed from the plurality of shard data to be processed, wherein the first quantity of shard data to be processed is part of the shard data to be processed among the plurality of shard data to be processed; Using the idle application manager to process the first number of to-be-processed slice data respectively; When the idle application manager has processed the first number of shard data to be processed, the idle application manager is released, and the remaining shard data to be processed is processed using the released idle application manager, where the remaining shard data to be processed is the shard data to be processed except the first number of shard data to be processed among the multiple shard data to be processed.

5. The method according to claim 1, characterized in that The step of using the write thread pool to read the plurality of data segments in the first storage unit and / or the second storage unit into the target device includes: When the number of data segments stored in the first storage unit meets a second preset threshold range, using the write thread pool to read the first shard data into the target device; When the number of data segments stored in the first storage unit meets a first preset threshold range, detecting the number of data segments stored in the second storage unit; When the number of data segments stored in the second storage unit meets the second preset threshold range, the second shard data is read into the target device using the write thread pool until the multiple data segments in the first storage unit and / or the second storage unit are read into the target device.

6. The method according to claim 1, characterized in that After using the write thread pool to read the multiple data segments in the first storage unit and / or the second storage unit into the target device, the method further includes: Within a first preset time period, obtaining first running status information of the read thread pool and second running status information of the write thread pool; The first operation status information and the second operation status information are recorded by using a standby server.

7. The method according to claim 1, characterized in that After the idle application manager is used to process the plurality of to-be-processed shard data, the method further comprises: Within a second preset time period, obtaining processing progress information of the plurality of to-be-processed shard data; Feedback the processing progress information to the client.

8. A data processing device, characterized in that: The device comprises: The processing unit is used for, when receiving a data processing instruction from a client, performing slicing processing on the data to be processed corresponding to the data processing instruction to obtain a plurality of slicing data to be processed; and processing the plurality of slicing data to be processed by using an idle application manager; A determining unit, configured to determine the idle application manager from an application manager queue, wherein the application manager queue includes a plurality of application managers; A releasing unit, configured to release the idle application manager to the application manager queue when the idle application manager has processed the plurality of to-be-processed slice data, so as to utilize the idle application manager to process data corresponding to other data processing instructions; The processing unit is used to sequentially use one of the idle application managers to process one of the multiple slice data to be processed, until the idle application manager is used to process the multiple slice data to be processed; The device further comprises a slicing unit, a data reading unit and a detection unit. The slicing unit is used to divide the one slicing data to be processed into multiple data segments; The data reading unit is used to read the first slice data into the first storage unit by using the reading thread pool when the number of data segments stored in the first storage unit meets the first preset threshold range, and the first slice data is the data read into the first storage unit from the multiple data segments; when the number of data segments stored in the second storage unit meets the third preset threshold range, read the second slice data into the second storage unit by using the reading thread pool until the multiple data segments are read into the first storage unit and / or the second storage unit, and the second slice data is the data read into the second storage unit from the multiple data segments, the first storage unit is a storage unit for temporarily storing the multiple data segments in the one application manager, and the second storage unit is a storage unit for temporarily storing the multiple data segments in the one application manager; and read the multiple data segments in the first storage unit and / or the second storage unit into the target device by using the writing thread pool; The detection unit is used to detect the number of data segments stored in the second storage unit when the number of data segments stored in the first storage unit meets a second preset threshold range.

9. A data processing device, characterized in that: The device comprises: A memory, a processor and a communication bus, wherein the memory communicates with the processor via the communication bus, the memory stores a data processing program executable by the processor, and when the data processing program is executed, the method according to any one of claims 1 to 7 is performed by the processor.

10. A storage medium having a computer program stored thereon, applied to a data processing device, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images