Data processing method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a data processing method, device, storage medium and electronic device, wherein the method comprises: determining a target operator required for processing task data based on acquired task data; determining the data volume of the task data included in the target operator when the target operator processes the task data; in the case where the number of threads included in the target operator does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume; and processing the task data through the target number of threads. Through the present invention, the problem of slow data processing speed existing in the related art is solved, and the effect of improving data processing speed and saving resources is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a data processing method, device, storage medium and electronic device. Background Art

[0002] With the widespread application of big data technology in all walks of life, there is an increasing demand for real-time processing of massive data. At the same time, the business logic of data processing is becoming more and more complex. Traditional batch processing methods and early stream processing frameworks are increasingly difficult to meet the increasingly demanding business requirements in terms of latency, throughput, fault tolerance, and ease of use. There are many application scenarios for big data streaming computing. According to the three different dimensions of data generation method, data size, and technology maturity, there are financial banking applications, Internet applications, and Internet of Things applications. At present, there are many mature real-time streaming processing systems in the industry. There are mature commercial products, as well as products that are open sourced by companies, organizations, or individuals. Some of these products have been verified by production, while some are just prototypes, and there are too many open interfaces for calling. This product status has led to many difficulties and pressures for front-line code developers in the face of so many dazzling products in the early product selection and trial, and later product learning and use, which increases the learning curve of developers and the calling process is too complicated. Therefore, the industry is faced with the problem of how to simplify existing streaming processing products.

[0003] In related technologies, business systems generate events (messages / calls) in real time, and after a series of processing and analysis, this process is completed in parallel on multiple computers, which looks like events continuously flowing through multiple computing nodes to form a streaming computing process. However, the nodes included in the multiple computing nodes are prone to performance bottlenecks, affecting the processing data of other nodes, resulting in slow system data processing speed.

[0004] It can be seen from this that the related technology has the problem of slow data processing speed.

[0005] With respect to the above-mentioned problems existing in the related technologies, no effective solution has been proposed yet. Summary of the invention

[0006] Embodiments of the present invention provide a data processing method, device, storage medium and electronic device to at least solve the problem of slow data processing speed existing in the related art.

[0007] According to one embodiment of the present invention, there is provided a data processing method, comprising: determining a target operator required to process the task data based on acquired task data; determining the data volume of the task data included in the target operator when the target operator processes the task data; if the number of threads included in the target operator does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume; and processing the task data by means of the target number of threads.

[0008] According to another embodiment of the present invention, there is provided a data processing device, comprising: an acquisition module, for determining a target operator required to process the task data based on acquired task data; a determination module, for determining the data volume of the task data included in the target operator when the target operator processes the task data; an adjustment module, for adjusting the number of threads included in the target operator to a target number matching the data volume when the number of threads included in the target operator does not match the data volume; and a processing module, for processing the task data through the target number of threads.

[0009] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0010] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0011] Through the present invention, the target operator required for processing the task data is determined according to the acquired task data, the data amount of the task data included in the target operator when the target operator processes the task data is determined, and when the number of threads included in the target operator does not match the data amount, the number included in the target operator is adjusted to the target number that matches the data amount, and the task data is processed by the target number of threads. Since the data amount in the target operator does not match the number of threads, the number of threads can be adjusted to the target number that matches the data amount, and the task data is processed by the target number of threads, that is, when a performance bottleneck occurs in the processing node, the data can be processed by adjusting the number of threads to increase the speed of data processing, and when the number of threads is large, the number of threads can be reduced to save resources. Therefore, the problem of slow data processing speed existing in the related art can be solved, and the effect of increasing the data processing speed and saving resources can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a hardware structure block diagram of a mobile terminal of a data processing method according to an embodiment of the present invention;

[0013] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0014] Figure 3 is a flow chart of a data processing method according to an exemplary embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of SQL operator processing task data according to an exemplary embodiment of the present invention;

[0016] Figure 5 is a structural block diagram of a data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a data processing method is provided. Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0023] Step S202, determining a target operator required to process the task data based on the acquired task data;

[0024] Step S204, determining the data volume of the task data included in the target operator when the target operator processes the task data;

[0025] Step S206, when the number of threads included in the target operator does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume;

[0026] Step S208: Processing the task data through the target number of threads.

[0027] In the above embodiment, the target operator can be determined by the type of task data or by the user's configuration. The task data can be stored in the server, and after the user subscribes to the task data, the user can obtain the data through http / https / rmq / kafka and the like.

[0028] In the above embodiment, the target operator includes a certain number of threads, which can be manually set at the beginning. Each thread can process a fixed amount of data. When the target operator processes data, it can obtain data through http / https / rmq / kafka and other methods. If the amount of data obtained exceeds the total amount of data processed by the total number of threads, or is less than the total amount of data processed by a single thread, the number of threads can be adjusted to match the number of threads with the amount of data, and the task data can be processed by the adjusted number of threads.

[0029] Optionally, the execution entity of the above steps can be a server, a background processor, or other devices with similar processing capabilities, or a machine that at least integrates a data processing device, where the data processing device may include terminals such as computers and mobile phones, but is not limited to this.

[0030] Through the present invention, the target operator required for processing the task data is determined according to the acquired task data, the data amount of the task data included in the target operator when the target operator processes the task data is determined, and when the number of threads included in the target operator does not match the data amount, the number included in the target operator is adjusted to the target number that matches the data amount, and the task data is processed by the target number of threads. Since the data amount in the target operator does not match the number of threads, the number of threads can be adjusted to the target number that matches the data amount, and the task data is processed by the target number of threads, that is, when a performance bottleneck occurs in the processing node, the data can be processed by adjusting the number of threads to increase the speed of data processing, and when the number of threads is large, the number of threads can be reduced to save resources. Therefore, the problem of slow data processing speed existing in the related art can be solved, and the effect of increasing the data processing speed and saving resources can be achieved.

[0031] In an exemplary embodiment, when the number of threads included in the target operator does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume includes: determining the data processing volume of each of the threads included in the target operator; determining the total data processing volume of all the threads included in the target operator based on the data processing volume; determining whether the number of threads matches the data volume based on the total data processing volume; and when determining that the number of threads does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume. In this embodiment, the target operator may include multiple threads, determine the data processing volume of each thread in the target operator, determine the total data processing volume of all the threads included in the target operator based on the data processing volume of each thread, determine whether the number of threads in the target operator matches the data volume based on the total data processing volume, and when they do not match, adjust the number of threads in the target operator to match the data volume.

[0032] In an exemplary embodiment, when it is determined that the number of threads does not match the amount of data, adjusting the number of threads included in the target operator to a target number that matches the amount of data includes: when it is determined that the amount of data exceeds the total data processing amount, determining that the number of threads included in the target operator does not match the amount of data, determining a first number of threads that need to be added in the target operator based on the amount of data and thread attributes, and adding the first number of threads in the target operator, wherein the thread attributes are used to indicate the ability of threads to process data. In this embodiment, when the amount of data exceeds the total amount of data processing, the number of threads can be adjusted by adding threads. Determine the thread attributes, and determine the ability of the thread to process data based on the thread attributes. For example, a single thread can process 10W data. The current target operator includes two threads, and the amount of data is 40W. Therefore, two threads can be added, and four threads can be used to process task data.

[0033] In an exemplary embodiment, in the case where it is determined based on the total data processing amount that the number of threads does not match the data volume, adjusting the number of threads included in the target operator to a target number that matches the data volume includes: in the case where it is determined that the data volume is less than the total data processing volume, determining that the number of threads included in the target operator does not match the data volume, determining a second number of threads that need to be reduced in the target operator based on the data volume and thread attributes, and reducing the second number of threads in the target operator, wherein the thread attributes are used to indicate the ability of threads to process data. In this embodiment, when the data volume is less than the total data processing volume, the number of threads can be adjusted by reducing the number of threads. Determine the thread attributes, and determine the ability of the threads to process data based on the thread attributes. For example, the current target operator includes four threads, the data volume is 20W, and each thread can process 10W data. Therefore, two threads can be reduced and two threads can be used to process task data.

[0034] In an exemplary embodiment, after adding the first number of threads to the target operator, the method further includes: determining the amount of change in the amount of data within a predetermined time; when the amount of change is less than a first threshold, deleting the first number of threads in the target operator; and when the amount of change is greater than a second threshold, pushing the log information of the target operator processing the task data. In this embodiment, after adding the first number of threads to the target operator, the amount of change in the amount of data in the target operator within a predetermined time can also be determined. When the amount of change is less than the first threshold, the added threads can be deleted because the added threads have little effect on the performance improvement of the node. The predetermined time can be 1 minute (this value is only an exemplary description, and the present invention does not limit the predetermined time. For example, 30 seconds, 1.5 minutes, etc. can also be taken). The first threshold can be determined according to the speed of thread processing data and the predetermined time. For example, the first threshold can be equal to half of the product of the speed of thread processing data and the predetermined time. The present invention does not limit the first threshold. When the amount of change is greater than the second threshold, the log of processing task data and the current number of threads can be pushed. The second threshold may be determined according to the speed at which the thread processes data and the predetermined time. For example, the second threshold may be equal to the product of the speed at which the thread processes data and the predetermined time.

[0035] In the above embodiment, the user can configure multiple intermediate processing processes and multiple destinations. After the data flow is imported from the source, it is finally written to the destination after the intermediate processing operator. By adjusting the number of threads, the problem that the poor performance of a single link will become the performance bottleneck of the entire task flow is solved. The server can dynamically adjust the number of threads in each node to adapt to the amount of task data. Among them, the data processing method flow chart can be found in the attached Figure 3 Each operator in the stream processing framework is independent of each other, and the corresponding concurrency can be configured according to the complexity of each operator to ensure that a single operator will not become a performance bottleneck in the stream processing process.

[0036] In an exemplary embodiment, when the amount of change is less than the first threshold, after deleting the first number of threads in the target operator, the method further includes: executing an alarm operation, and displaying the status information of processing the task data in the target display interface. In this embodiment, after deleting the added threads, an alarm operation may also be executed, and the status information of processing the task data may be displayed in the target display interface. Among them, the status information may include the processing status of the current task, consumption statistics, log information, etc. The user can monitor the execution of the task in real time, and can also monitor the information of each link in the task execution process, so as to facilitate the task configuration personnel to quickly locate the problem and make timely adjustments. For example, changing the order of nodes for processing tasks, changing the target operator, etc.

[0037] In an exemplary embodiment, determining the target operator required to process the task data based on the acquired task data includes: determining the target type of the task data; determining the target operator based on the target type. In this embodiment, a single task can be composed of multiple operators (source-end operators, intermediate processing operators, destination-end operators, etc.) according to the user's configuration. The source-end operator can support obtaining data from rabbitmq / kafka / mysql / oracle / mpp / es / hive, etc. That is, the rule engine can pull data from the server according to the user configuration. The target operator can be determined according to the type of task data, that is, the target operator can be an intermediate processing operator configured by the user, and the intermediate processing operator can include an SQL processing operator / formatting processing operator / UDF operator. SQL processing, that is, expressing streaming calculations through SQL expressions, analyzing, filtering, and transforming the incoming data; formatting operators, that is, transforming the incoming messages according to user-defined content; UDF operators, that is, user-defined operators. The destination operator can store the data obtained after the intermediate processing operator is processed in rabbitmq / kafka / mysql / oracle / mpp / es / hive / ftp / neo4j, etc., that is, the rule engine writes the accessed data stream to the corresponding destination. In this embodiment, the server can determine the source operator, the intermediate processing operator and the target operator according to the type of task data. When configuring the target operator, a memory pool can be allocated for the target operator, that is, a memory pool is used to avoid frequent memory applications. At the same time, the metadata of the data source is unchanged. After the first message is processed, the metadata information of the message is stored in the storage address specified by the destination operator, thereby improving the performance of data structure transformation.

[0038] In an exemplary embodiment, determining the target operator based on the target type includes: determining the target operator as a structured query language SQL processing operator when the target type is determined to be a single message data type; and determining the target operator as a join operator when the target type is determined to be a message table data type. In this embodiment, when the type of task data is a single message data type, the target operator may be an SQL operator, and when the type of task data is a message table data type, the target operator may be a join operator, i.e., a join operator.

[0039] In an exemplary embodiment, when the target operator is a structured query language SQL processing operator, processing the task data through the target number of threads includes: parsing the task data processed in each thread through each thread included in the target number of threads to determine the target field of the task data; performing structural transformation on the target field; and storing the target field after structural transformation in the target storage space. In this embodiment, the complete SQL filtering transformation involves select and where parts. In this process, there may be only a few or no fields involved in the where part or function calculation. If the entire message is structurally transformed, the performance will be very poor. For actual scenarios, SQL can be parsed and analyzed in advance to extract the fields involved in filtering and function transformation. Only this part is structurally transformed. For other projections, constants, etc., the structure mapping is directly performed using the stored metadata. In the actual test process, this method can improve the performance by about 3 times (datum is the internal data structure when SQL is matched). Among them, the schematic diagram of SQL operator processing task data can be found in the attached figure. Figure 4 .

[0040] In an exemplary embodiment, when the target operator is a connection operator, processing the task data through the target number of threads includes: performing the following operations through each thread included in the target number of threads: determining that the first message table included in the task data to be processed by each thread includes corresponding data, determining the first data with access volume greater than a third threshold from the corresponding data, storing the first data in the main memory, and storing the data other than the first data included in the corresponding data in the auxiliary memory; processing the first data stored in the main memory according to a predetermined rule to obtain the second data, and processing the second data and the third data in the second message table included in the task data to be processed by each thread to determine the first target data. In this embodiment, the message table may include a first message table and a second message table, the first message table may be an external table, and the second message table may be a small table. Usually, the first message table is a large table ranging from a few rows to hundreds of rows, and the second message table is a small table. When the amount of external data is greater than 10W, the hotspot data (corresponding to the first data) can be stored in the memory (corresponding to the main memory), and other data is stored in the local disk (corresponding to the auxiliary memory), and the hotspot data is updated in real time to reduce memory pressure. The memory data of the outer table is pre-sorted according to the join condition to obtain the second data, and the second data is compared with the third data to be processed in each thread, and the first target data is determined according to the predetermined processing requirements. The processing requirements may be to find the data that is repeated in the first message table, or the data that has no intersection with the first message table, in the second data stored in the memory. During the processing, the data in the two tables need to be collided (i.e., compared). Since the first data is processed according to the predetermined processing rules, when joining with the small table, the collision range can be quickly reduced, greatly improving the performance of the join. The predetermined rules may be sorting in ascending order, sorting in descending order, etc.

[0041] In an exemplary embodiment, when the target operator is a connection operator, processing the task data through the target number of threads includes: performing the following operations through each thread included in the target number of threads: determining a filtering condition based on the second message table included in the task data; filtering fourth data from the data in the first message table included in the task data that each thread needs to process based on the filtering condition; processing the fourth data and the third data in the second message table to determine the second target data. In this embodiment, the uncertain condition can be pushed down to the first message table according to the value of the message table. That is, the server can determine the filtering condition of the data according to the second message table, and only retrieve the data that meets the condition, which greatly reduces the amount of data calculated by the first message table.

[0042] In an exemplary embodiment, before determining the target operator required to process the task data based on the acquired task data, the method further includes: determining a directed acyclic graph of a pre-configured task; determining a first storage address of the task data based on the directed acyclic graph, and obtaining the task data in the storage address. In this embodiment, the user can configure the data flow to DAG (directed acyclic graph) of the task through a graphical interface to complete the task flow processing logic. The user only needs to configure the DAG of the flow processing on the graphical interface to define the flow processing task, which has low learning cost and convenient modification. In addition, the user can choose to subscribe to single or multiple task operation and maintenance information as needed. After receiving the subscription request, the server can push the task data to the user through http / https / rmq / kafka and other methods. The user can view the status, consumption statistics, logs and other information of the current task through the operation and maintenance interface, and can monitor the execution of the task in real time; at the same time, the information of each link in the task execution process can be monitored, so that the task configuration personnel can quickly prepare the positioning problem and make timely adjustments.

[0043] In an exemplary embodiment, after processing the task data by the target number of threads, the method further includes: determining a second storage address of the processed data obtained after processing the task data; and storing the processed data in the second storage address. In this embodiment, the second storage address can be determined by the DAG pre-configured by the user, and the second storage address can also be determined by the destination operator. Among them, the destination operator can store the data obtained after processing by the intermediate processing operator in rabbitmq / kafka / mysql / oracle / mpp / es / hive / ftp / neo4j, etc., that is, the rule engine writes the accessed data stream to the corresponding destination.

[0044] In the above embodiment, by splitting the join conditional statements, storing the large table in memory table and sorting the large table according to the filter condition key (corresponding to the above filter condition), the number of table collisions is reduced when two tables are joined, and the processing performance is improved. The separate message table is processed, and only the fields involved in function transformation or filter conditions are processed, and other fields are mapped by structure to improve the processing performance of large message bodies. The storage and search methods of join are optimized, and the sorting and condition pushdown methods are used to greatly improve the performance of double-table join. For SQL expressions, select, where, and expression are distinguished during the preprocessing process. During matching, only the where and expression parts are converted and calculated. In actual scenarios, most of the projection parts are not converted, which greatly improves the matching performance of SQL expressions. For operators with independent tasks, the concurrency within the operator is automatically adjusted according to the size of the inbound and outbound flows within the operator to prevent a certain operator from becoming the performance bottleneck of the task chain. For performance problems that cannot be solved by concurrency, alarm logs are pushed to users, who can check the service performance of the source and destination in a timely manner based on the logs. Based on the characteristics that the metadata of a single task message remains unchanged and the number of messages is the same, a memory pool is used within a single task to greatly reduce the number of memory requests, while storing message metadata, which greatly improves processing performance.

[0045] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0046] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0047] Figure 5 is a structural block diagram of a data processing device according to an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0048] An acquisition module 52, configured to determine a target operator required for processing the task data based on the acquired task data;

[0049] A determination module 54, used to determine the data volume of the task data included in the target operator when the target operator processes the task data;

[0050] An adjustment module 56, configured to adjust the number of threads included in the target operator to a target number matching the data volume when the number of threads included in the target operator does not match the data volume;

[0051] The processing module 58 is configured to process the task data through the target number of threads.

[0052] In an exemplary embodiment, the adjustment module 56 can adjust the number of threads included in the target operator to a target number that matches the data volume when the number of threads included in the target operator does not match the data volume in the following manner: determine the data processing volume of each of the threads included in the target operator; determine the total data processing volume of all the threads included in the target operator based on the data processing volume; determine whether the number of threads matches the data volume based on the total data processing volume; and if it is determined that the number of threads does not match the data volume, adjust the number of threads included in the target operator to a target number that matches the data volume.

[0053] In an exemplary embodiment, the adjustment module 56 can adjust the number of threads included in the target operator to a target number that matches the data volume when it is determined that the number of threads does not match the data volume in the following manner: when it is determined that the data volume exceeds the data processing volume of all threads included in the target operator, determine that the number of threads included in the target operator does not match the data volume, determine a first number of threads that need to be added in the target operator based on the data volume and thread attributes, and add the first number of threads to the target operator, wherein the thread attributes are used to indicate the ability of a thread to process data.

[0054] In an exemplary embodiment, the adjustment module 56 can adjust the number of threads included in the target operator to a target number that matches the data volume when it is determined that the number of threads does not match the data volume in the following manner: when it is determined that the data volume is less than the total data processing volume, determine that the number of threads included in the target operator does not match the data volume, determine a second number of threads that need to be reduced in the target operator based on the data volume and thread attributes, and reduce the second number of threads in the target operator, wherein the thread attributes are used to indicate the ability of a thread to process data.

[0055] In an exemplary embodiment, the device can be used to determine the change in the amount of data within a predetermined time after adding the first number of threads to the target operator; when the change is less than a first threshold, delete the first number of threads in the target operator; when the change is greater than a second threshold, push log information of the target operator processing the task data.

[0056] In an exemplary embodiment, the device may be configured to, when the change amount is less than a first threshold, perform an alarm operation after deleting the first number of threads in the target operator, and display status information of processing the task data on a target display interface.

[0057] In an exemplary embodiment, the determination module 54 may determine the target operator required for processing the task data based on the acquired task data in the following manner: determine the target type of the task data; and determine the target operator based on the target type.

[0058] In an exemplary embodiment, the determination module 54 can determine the target operator based on the target type in the following manner: when the target type is determined to be a single message data type, the target operator is determined to be a structured query language SQL processing operator; when the target type is determined to be a message table data type, the target operator is determined to be a join operator.

[0059] In an exemplary embodiment, the determination module 54 can be implemented in the following manner to process the task data through the target number of threads when the target operator is a structured query language SQL processing operator: parse the task data processed in each thread by each thread included in the target number of threads to determine the target field of the task data; perform structural transformation on the target field; and store the target field after structural transformation in the target storage space.

[0060] In an exemplary embodiment, the processing module 58 can be implemented in the following manner to process the task data through the target number of threads when the target operator is a connection operator: each thread included in the target number of threads performs the following operations: determines that the first message table included in the task data that each thread needs to process includes corresponding data, determines from the corresponding data the first data whose access volume is greater than a third threshold, stores the first data in a main memory, and stores the data included in the corresponding data except the first data in an auxiliary memory; processes the first data stored in the main memory according to a predetermined rule to obtain second data, and processes the second data and the third data in the second message table included in the task data that each thread needs to process to determine the first target data.

[0061] In an exemplary embodiment, the processing module 58 can be implemented in the following manner, when the target operator is a connection operator, to process the task data through the target number of threads: each thread included in the target number of threads performs the following operations: determining a filtering condition based on a second message table included in the task data; filtering fourth data from the data in the first message table included in the task data that each thread needs to process based on the filtering condition; processing the fourth data and the third data in the second message table to determine the second target data.

[0062] In an exemplary embodiment, the device can be used to determine a directed acyclic graph of pre-configured tasks before determining a target operator required to process the task data based on the acquired task data; determine a first storage address of the task data based on the directed acyclic graph, and obtain the task data at the storage address.

[0063] In an exemplary embodiment, the apparatus may also be configured to determine a second storage address of processed data obtained after processing the task data by the target number of threads; and store the processed data in the second storage address.

[0064] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0065] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0066] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0067] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0068] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0069] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0070] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Determine a target operator required to process the task data based on the acquired task data; Determining the data volume of the task data included in the target operator when the target operator processes the task data; When the number of threads included in the target operator does not match the data amount, adjusting the number of threads included in the target operator to a target number that matches the data amount; Processing the task data by the target number of threads; Determine a target operator required to process the task data based on the acquired task data: determine a target type of the task data; determining the target operator based on the target type; Determining the target operator based on the target type of the task data, and determining the target operator as a connection operator when the target type is determined to be a message table data type; In the case where the target operator is a connection operator, processing the task data through the target number of threads includes: performing the following operations through each thread included in the target number of threads: determining that a first message table included in the task data to be processed by each thread includes corresponding data, determining first data having an access amount greater than a third threshold from the corresponding data, storing the first data in a main memory, and storing data included in the corresponding data except the first data in an auxiliary memory; The first data stored in the main memory is processed according to a predetermined rule to obtain second data, and the second data and third data in the second message table included in the task data to be processed by each thread are processed to determine the first target data.

2. The method according to claim 1, characterized in that: In a case where the number of threads included in the target operator does not match the data amount, adjusting the number of threads included in the target operator to a target number matching the data amount includes: Determining a data processing amount of each of the threads included in the target operator; Determine the total data processing volume of all threads included in the target operator based on the data processing volume; Determining whether the number of threads matches the data volume based on the total data processing volume; In the case where it is determined that the number of threads does not match the data amount, the number of threads included in the target operator is adjusted to a target number that matches the data amount.

3. The method according to claim 2, characterized in that In the case where it is determined that the number of threads does not match the amount of data, adjusting the number of threads included in the target operator to a target number matching the amount of data includes: In the case where it is determined that the data volume exceeds the total data processing volume, it is determined that the number of threads included in the target operator does not match the data volume, a first number of threads that need to be added to the target operator is determined based on the data volume and thread attributes, and the first number of threads is added to the target operator, where the thread attributes are used to indicate a capability of a thread to process data.

4. The method according to claim 2, characterized in that: In a case where it is determined based on the total data processing amount that the number of threads does not match the data amount, adjusting the number of threads included in the target operator to a target number matching the data amount includes: In a case where it is determined that the data volume is less than the total data processing volume, it is determined that the number of threads included in the target operator does not match the data volume, a second number of threads that need to be reduced in the target operator is determined based on the data volume and thread attributes, and the second number of threads is reduced in the target operator, where the thread attributes are used to indicate a capability of a thread to process data.

5. The method according to claim 3, characterized in that: After adding the first number of threads in the target operator, the method further includes: Determining a change in the amount of data within a predetermined time; When the change amount is less than a first threshold, deleting the first number of threads in the target operator; When the change amount is greater than a second threshold, log information of the target operator processing the task data is pushed.

6. The method according to claim 5, characterized in that When the change amount is less than a first threshold, after deleting the first number of threads in the target operator, the method further includes: Execute an alarm operation and display status information of processing the task data on the target display interface.

7. The method according to claim 1, characterized in that Determining the target operator based on the target type further includes: When it is determined that the target type is a single message data type, the target operator is determined to be a structured query language SQL processing operator.

8. The method according to claim 7, characterized in that In a case where the target operator is a structured query language SQL processing operator, processing the task data by using the target number of threads includes: Parsing the task data processed in each thread by each thread included in the target number of threads to determine a target field of the task data; Performing structural transformation on the target field; and storing the target field after structural transformation in a target storage space.

9. The method according to claim 7, characterized in that: When the target operator is a join operator, processing the task data by using the target number of threads includes: Perform the following operations by each thread included in the target number of threads: determining a screening condition based on a second message table included in the task data; Filter fourth data from the data in the first message table included in the task data that each thread needs to process based on the filtering condition; The fourth data and the third data in the second message table are processed to determine second target data.

10. The method according to claim 1, characterized in that Before determining a target operator required for processing the task data based on the acquired task data, the method further includes: Determine a directed acyclic graph of pre-configured tasks; A first storage address of the task data is determined based on the directed acyclic graph, and the task data is obtained from the storage address.

11. The method according to claim 1, characterized in that: After processing the task data by the target number of threads, the method further includes: Determining a second storage address of processed data obtained after processing the task data; The processed data is stored in the second storage address.

12. A data processing device, characterized in that: include: An acquisition module, used to determine a target operator required to process the task data based on the acquired task data; A determination module, used to determine the data volume of the task data included in the target operator when the target operator processes the task data; an adjustment module, configured to adjust the number of threads included in the target operator to a target number matching the data amount when the number of threads included in the target operator does not match the data amount; A processing module, configured to process the task data through the target number of threads; The acquisition module may determine the target operator required for processing the task data based on the acquired task data in the following manner: determining the target type of the task data; determining the target operator based on the target type; The target operator is determined based on the target type of the task data, and when it is determined that the target type is a message table data type, the target operator is determined to be a connection operator; the device can process the task data through the target number of threads in the case where the target operator is a connection operator in the following manner: each thread included in the target number of threads performs the following operations: determining that the first message table included in the task data to be processed by each thread includes corresponding data, determining first data with an access volume greater than a third threshold from the corresponding data, storing the first data in a main memory, and storing data included in the corresponding data except the first data in an auxiliary memory; The first data stored in the main memory is processed according to a predetermined rule to obtain second data, and the second data and third data in the second message table included in the task data to be processed by each thread are processed to determine the first target data.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 11 when executed.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 11.

Images