A data processing method and device, computer equipment and readable storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, computer equipment and a readable storage medium. The data processing method comprises: obtaining state information of a first stream data processing engine when processing data stored in a message middleware; if the state information satisfies a parallel degree adjustment condition, determining a target parallel degree according to the state information; adjusting the parallel degree of the message middleware according to the target parallel degree, and determining a second stream data processing engine according to the target parallel degree, the second stream data processing engine being used for processing data stored in the message middleware after the parallel degree adjustment. Through the embodiments of the present application, the fault in the data processing process can be detected and monitored in real time, so that the parallel degree is reasonably adjusted, and the elastic scaling of the parallel degree is realized by avoiding stopping the stream data processing engine, thereby ensuring the losslessness and real-time performance of data processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] In real-time stream computing systems, when stream processing encounters a sudden surge in data volume, i.e., a significant increase in data reporting traffic, the stream processing engine may face the problem of being unable to handle the load. For example, when using Kafka as a message queue, data backlog may occur, leading to backpressure on Flink (the data stream execution engine). In such cases, it is necessary to increase the parallelism of Flink and Kafka to improve the real-time data processing capabilities.

[0003] However, current solutions often adjust parallelism in a rather general way, which may lead to resource waste or insufficient resources. Furthermore, they typically involve stopping data stream processing to reallocate resources to expand or reduce parallelism, which can result in data processing delays or data loss. Therefore, it is necessary to design a solution that can not only appropriately adjust parallelism but also ensure data processing quality. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, and readable storage medium. By detecting and monitoring faults in the data processing process in real time, the parallelism can be reasonably adjusted, while avoiding stopping the streaming data processing engine to achieve elastic scaling of the parallelism, thus ensuring the lossless and real-time nature of data processing.

[0005] One embodiment of this application provides a data processing method, including:

[0006] Obtain the status information of the first-class data processing engine when processing data stored in the message middleware;

[0007] If the state information meets the parallelism adjustment condition, then the target parallelism is determined based on the state information;

[0008] The parallelism of the message middleware is adjusted according to the target parallelism, and a second-stream data processing engine is determined based on the target parallelism. The second-stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment.

[0009] One embodiment of this application provides a data processing apparatus, including:

[0010] The acquisition module is used to acquire the status information of the first-line data processing engine when processing the data stored in the message middleware;

[0011] The determination module is used to determine the target parallelism based on the state information if the state information meets the parallelism adjustment conditions.

[0012] The determination module is also used to adjust the parallelism of the message middleware according to the target parallelism, and to determine the second-stream data processing engine according to the target parallelism. The second-stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment.

[0013] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the data processing method of this application embodiment.

[0014] One aspect of this application provides a computer-readable storage medium storing a computer program, which includes program instructions. When executed by a processor, the program instructions perform the data processing method described in this application.

[0015] Accordingly, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method provided in one aspect of embodiments of this application.

[0016] In this embodiment, the target parallelism is determined by the state information of the data processed by the first-stream data processing engine based on the data stored in the message middleware. Parallelism can be acquired on demand, ensuring that it matches the specific data processing requirements and avoiding resource waste. Furthermore, a new streaming data processing engine, namely a second-stream data processing engine, is determined based on the target parallelism. This second-stream data processing engine is used to process the data in the message middleware after the parallelism adjustment in a timely manner. It can process data in the message middleware simultaneously without stopping the first-stream data processing engine, ensuring that data processing is not delayed or lost, thereby improving the real-time performance and accuracy of data processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a network architecture diagram of a data processing system provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the effect of applying a data processing method provided in this application embodiment to a real environment;

[0021] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0022] Figure 5 This is a functional block diagram of a data processing method provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of a process for expanding parallelism according to an embodiment of this application;

[0024] Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The solution provided in this application belongs to the cloud computing field. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. In this solution, the use of a streaming data processing engine to process data stored in the message middleware is implemented through cloud computing.

[0029] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0030] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed within this pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices. Logically, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be directly deployed on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally, SaaS and PaaS are upper layers compared to IaaS.

[0031] Please see Figure 1This is a network architecture diagram of a data processing system provided in this application embodiment, including multiple terminal devices 100 and servers with different functions, wherein the servers include a data source access server 101, a monitoring server 102, a message middleware server 103, a data processing server 104, an ingestion server 105, and a database 106.

[0032] Terminal device 100 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. As a data source, it can generate various types of data to be processed, such as image data, text data, audio data, etc. Classifying data sources from different perspectives, it can also be log data, structured data, IoT data (a type of highly unstructured data), file data, etc. To a certain extent, the data source can also be data stored in other databases. Data processing in specific scenarios is often massive. For example, in an e-commerce system, the data generated by the terminal device can be user browsing history, order count, transaction volume, etc., or the sales volume, access frequency, and collection count of a product within a fixed time period. The specific data objects processed can differ in different application scenarios. Therefore, no restrictions are placed on the volume or type of data processed by terminal device 100 or this network architecture diagram.

[0033] The data source access server 101 can act as an access layer, processing data sources from terminal devices or other computer devices. Data access enables unified management of data from different sources, primarily by reducing access costs through standardized access methods. Furthermore, as a fundamental and crucial component of the data processing platform, a robust underlying data access infrastructure provides more stable and reliable transmission services to the upper layers, fully maximizing the value of the data. In this embodiment, massive amounts of data can be accessed to the data processing platform via a message middleware cluster (such as a Kafka cluster) deployed in the message middleware server 103. The Kafka cluster, acting as a message passing system, serves as a data transmission pipeline. The data processing platform refers to a streaming data processing engine (such as Flink) deployed in the data processing server 104.

[0034] Data processing server 104 obtains standardized data from message middleware server 103 and uses its streaming data processing engine to perform real-time or offline processing of massive amounts of data, such as sorting, filtering, aggregating, and calculating millions of data points, to obtain the data processing results. These results are then stored in database 106 via a message middleware cluster deployed on inbound server 105. This means that inbound server 105, similar to message middleware server 103, also has data transfer capabilities, ensuring no data loss should the database 106 or data processing server 104 fail (e.g., crash or message congestion). Database 106 can be DLL, Elasticsearch, etc., without limitation.

[0035] The monitoring server 102 plays a crucial role in the data processing system. It monitors in real-time changes in the consumption lag of Kafka data skew in the message middleware server 103. When excessive data backlog is detected, the monitoring server 102 determines the concurrency level based on the specific changes and issues instructions to the message middleware server 103 to expand its capacity according to the determined concurrency level. Simultaneously, it issues instructions to the data processing server 104 to start a Flink streaming data processing engine with increased parallelism. The original Flink stream in the data processing server 104 is then stopped because its parallelism cannot support the real-time processing of the data stream. This allows for timely expansion of the data transmission pipeline in the message middleware server 103 and the streaming data processing engine in the data processing server 104 when faced with a sudden increase in data volume, ensuring timely data processing and preventing data loss or delayed processing. Furthermore, when processing data with high concurrency, if the amount of data decreases sharply within a certain period of time, the degree of parallelism can be adjusted according to the specific data processing requirements. The concurrency of the message middleware and the streaming data processing engine can be adjusted in the same way as described above, which will not be elaborated here.

[0036] It can be observed that by monitoring the message middleware server 103 in real time through the monitoring server 102, when data processing is required, the monitoring server 102 issues a command to the data processing server 104 to start a new Flink stream with adjusted parallelism. This allows data processing to use a new streaming data processing engine to process data in the message middleware without pausing the original Flink stream. At appropriate times, the monitoring server 102 can issue a command to the data processing server 104 to stop the original streaming data processing engine, achieving seamless data processing and ensuring no data loss or delay. Determining the parallelism based on data lag allows for more reasonable and accurate adjustments. The monitoring service in the monitoring server 102, as a management tool, can monitor and manage faults such as data backpressure in real time, thereby enabling flexible adjustments to the parallelism.

[0037] It should be noted that the aforementioned server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device 100 and the server (here referring to the data source access server 101) can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0038] Furthermore, for ease of understanding, the methods provided in the following embodiments of this application are all described using the monitoring server 102 described above as an example. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method may include at least the following steps S101 to S103:

[0039] S101, obtain the status information of the first-class data processing engine when processing the data stored in the message middleware.

[0040] In one embodiment, the streaming data processing engine can be Flink, an open-source distributed streaming processing framework that provides data distribution, data communication, and fault tolerance mechanisms for distributed computing of data streams. The message middleware can be Kafka, a message passing system specifically designed for distributed high-throughput systems that can transmit messages from one endpoint to another. For ease of understanding, this application uses Flink as the streaming data processing engine and Kafka as the message middleware in its embodiments. Accordingly, the first streaming data processing engine refers to Flink, the data stream execution engine that currently acquires and processes Kafka data. Based on such... Figure 1 The provided data processing system architecture diagram shows how data from different sources is processed. Figure 1 After the data source receiving server 101 normalizes the data, it is stored in the message middleware Kafka. The streaming data processing engine in the data processing server 104 then processes the stored data. Essentially, the processing involves performing one or more operations—aggregation, transformation, or computation—on data read in real-time from a Kafka topic, and then writing the results to a new topic. Therefore, from a certain perspective, the streaming data processing engine can be seen as a consumer, actively pulling data from the message middleware for processing (or consumption). Kafka, as the message middleware, has producers writing messages into it, meaning that collected data is stored in the message middleware. Through the message middleware, a real-time streaming data pipeline is established, reliably obtaining data between systems or applications.

[0041] During data processing, objective or subjective factors, such as a sudden surge in data volume or a crash in the streaming data processing engine, may cause consumers' consumption speed to lag behind producers' data production speed, resulting in data backlog. Consequently, data received by upstream nodes cannot be processed promptly by downstream nodes, leading to data loss or processing delays. Therefore, to better address these issues, real-time monitoring of data processing is necessary. This application provides a solution that uses a monitoring server 102 to collect real-time status information of the streaming data processing engine—a crucial indicator—to control data consumption progress. This status information indicates whether the streaming data processing engine can process data stored in the message middleware in a timely manner. The options for obtaining status information include using the command-line tool scripts provided with the message middleware, API programming, or other methods. No restrictions are placed on the data representation used for the status information.

[0042] S102, if the state information meets the parallelism adjustment condition, then determine the target parallelism based on the state information.

[0043] In one embodiment, when the status information obtained by the monitoring server 102 indicates that the data processing server 104 is processing data in the message middleware server 103 in real time, a phenomenon occurs where the rate at which data is generated upstream of a certain node in the data pipeline is greater than the rate at which that node processes data (i.e., back pressure). In this case, it is necessary to increase the parallelism of the currently processed data. Alternatively, after the streaming data processing engine has been processing data at high saturation for a period of time, the status information indicates that the amount of data collected has decreased significantly. To save computing resources, the parallelism required for data processing can also be reduced. In summary, whether the parallelism needs to be adjusted can be measured by the status information. Furthermore, another function of the status information is to determine the target parallelism required for the final data processing.

[0044] Optionally, the status information includes the data consumption lag. Before this step, it further includes: if the data consumption lag continues to increase for a duration reaching a first duration and the increase per unit time is greater than or equal to a first quantity threshold, or if the data consumption lag is greater than or equal to a second quantity threshold, then the status information is determined to meet the parallelism adjustment condition. Here, the data consumption lag is denoted as the Lag value, which represents the amount of data the consumer lags behind the producer. That is, the Lag value is the difference between the offset value of the latest consumed message and the offset value of the latest produced message (or the difference between the number of topic records and the consumption progress (offset) of the consumer group). Simply put, it is the lag in the processing of data in the message middleware by the streaming data processing engine compared to the data stored in the message middleware. Under normal processing conditions, the lag value should be close to 0, indicating that consumers can consume messages produced by producers in a timely manner with minimal lag. However, if the lag value is large or continues to increase to a certain threshold, it means that consumers cannot keep up with the producer's speed, which will slow down the processing speed of downstream messages. If the lag value is within the range that the streaming data processing engine can handle, that is, if the streaming data processing engine and message middleware can handle small lag values ​​on their own, then no adjustment to the parallelism is needed. Conversely, if the lag value significantly increases to a level that exceeds the processing capacity of the streaming data processing engine and the receiving capacity of the message middleware, then necessary measures need to be taken to expand the capacity of the relevant data tools. The specific details are not elaborated here. This boundary is the judgment content of whether the parallelism adjustment conditions are met, as mentioned above. It includes two types: one is a joint judgment based on the duration of the continuous increase in lag value and the increase per unit time; the other is a judgment based on the comparison of the increase in lag value with a second quantity threshold. When the lag value meets either of these conditions, it can be determined that the status information meets the parallelism adjustment conditions. For example, the first duration is set to 1 hour, the first quantity threshold is set to 20,000, and the second quantity threshold is set to 2,000,000. The monitoring service in monitoring server 102 monitors the latency of the streaming data processing engine every minute. If the lag value of the message middleware increases continuously for 1 hour and increases by 20,000 per minute, it can be determined that the lag meets the parallelism adjustment condition. Alternatively, if the lag value is greater than 2,000,000, it can also be determined that the lag meets the parallelism adjustment condition. Conversely, if neither of these conditions is met, the parallelism adjustment condition is not met. In this case, due to the increasing trend of the lag value, the corresponding parallelism adjustment refers to expanding the parallelism.

[0045] Optionally, the determination of whether the status information meets the parallelism adjustment conditions can also be made in the following way: if the duration for which the data consumption lag continuously decreases reaches a second duration and the decrease per unit time is greater than or equal to a third quantity threshold, or if the data consumption lag is less than or equal to a fourth quantity threshold, then the status information is determined to meet the parallelism adjustment conditions. Corresponding to the adjustment conditions for expanding parallelism mentioned above, this determination method is a judgment condition for parallelism reduction adjustment. Typically, when both the streaming data processing engine and the message middleware are processing data at high concurrency, the data consumption lag will inevitably continue to decrease. When the lag decreases to near 0, it indicates that the streaming data processing engine can meet the data processing load. However, there is another scenario where the data volume decreases at the source, and the corresponding lag also gradually decreases. In this case, the data volume does not require such a high degree of parallelism to support the data processing function. Therefore, if the monitoring server 102 detects that the data consumption lag in the status information continuously decreases, and the duration of the decrease and the amount of decrease per unit time reach the thresholds (corresponding to the second duration and the third quantity threshold, respectively), or the Lag value is directly less than or equal to the fourth quantity threshold, then it can be determined that the status information meets the parallelism adjustment conditions. For example, if the monitoring server 102 detects that the current Lag value decreases from 5 million at a rate of 50,000 per minute (the third quantity threshold) for 30 minutes (the second duration), or the Lag value decreases directly from 5 million to 2 million (the fourth quantity threshold), it can be determined that the status information meets the parallelism adjustment conditions. After determining the parallelism based on the status information, the current parallelism is adjusted.

[0046] It should be noted that the second duration can be the same as or different from the first duration. Similarly, the third quantity threshold can be the same as or different from the first quantity threshold, and the fourth quantity threshold can be the same as or different from the second quantity threshold. No restrictions are imposed here.

[0047] In one embodiment, an optional implementation of determining the target parallelism based on state information can be: obtaining the reference data processing volume corresponding to a single parallelism and the current parallelism of the message middleware; determining the parallelism adjustment amount based on the reference data processing volume and the data consumption lag; and determining the target parallelism based on the current parallelism and the parallelism adjustment amount. In this embodiment, the maximum data volume consumed by a single parallelism of the streaming data processing engine in the message middleware is called the reference data processing volume. The current parallelism of the message middleware is the parallelism before adjustment. The ratio of the data consumption lag to the reference data volume is used as the parallelism adjustment amount. For example, if 1 million is used as the reference data volume, corresponding to one parallelism, and the data consumption lag is 2 million, then 2 million / 1 million = 2 parallelisms, which is the parallelism adjustment amount used to address the lag. Depending on the situation, the implementation of determining the target parallelism based on the parallelism adjustment amount and the current parallelism includes adding or subtracting the parallelism adjustment amount to the current parallelism. The adjustment based on the target parallelism corresponds to scaling up and scaling down, respectively. Optionally, users can also configure parameters in the monitoring service to control whether automatic scaling down occurs after automatic scaling up, or whether automatic scaling up occurs after automatic scaling down, based on business characteristics and the actual situation of current physical resources. This can enhance the flexibility of the system.

[0048] In short, adjusting the parallelism requires considering the current parallelism, the reference data processing volume, and the lag at different times. For example, if the lag increases from 0 to 200w, the parallelism needs to be increased based on the current parallelism. Here, 0 and 200w correspond to the lag in data consumption at different times. Following the rule of 100w corresponding to one parallelism, this specifically means adding two parallelisms. Conversely, if the lag decreases from 200w to 0, the parallelism needs to be reduced based on the current parallelism, again requiring a reduction of two parallelisms. It should be noted that the above parallelism adjustment amount and target parallelism calculation can be implemented through the monitoring service application in monitoring server 102. The parallelism adjustment amount when expanding parallelism can be determined by referring to the lag at the final time and the reference data processing volume. However, the parallelism adjustment when reducing parallelism needs to be determined using the lag at different times and the reference data processing volume. This is because the current parallelism is fully sufficient and even redundant for data processing. By reflecting the parallelism adjustment amount through changes in lag, reducing the parallelism can fully utilize existing data resources to complete the data processing function.

[0049] It can be seen that whether expanding or reducing the parallelism of streaming data processing engines or message middleware, it can be determined by the data consumption lag and the maximum data processing volume that the parallelism can support. This makes the adjustment of the parallelism reasonable and more applicable.

[0050] S103, adjust the parallelism of the message middleware according to the target parallelism, and determine the second-stream data processing engine according to the target parallelism. The second-stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment.

[0051] In one embodiment, since the message middleware can establish real-time streaming data pipelines, the parallelism of the message middleware can be the same as the number of real-time streaming data pipelines; simply put, one real-time data pipeline corresponds to one degree of parallelism. Adjusting the parallelism of the message middleware according to the target parallelism means adjusting the current parallelism of the message middleware to the target parallelism, including either expanding or reducing the parallelism. Optionally, the parallelism of the message middleware can be expanded or reduced by calling the API interface of the Kafka message middleware, so that after the parallelism of the message middleware is adjusted, it can store the messages (i.e., data) produced by the producer in a timely manner or make full use of the existing message middleware to store data.

[0052] Furthermore, corresponding adjustments have been made to the streaming data processing engine. A second streaming data processing engine is determined based on the target parallelism. This second engine can be a Flink stream restarted according to the target parallelism. Unlike the first engine, the parallelism of the second engine matches the target parallelism. This ensures that the processing capacity of the second engine is adapted to the data volume, fully utilizing resources to consume data stored in the message middleware. In this process, because the second engine is determined separately, its parallelism is expanded or reduced compared to the first engine. This expansion or reduction is not achieved by stopping the currently processing first engine, thus avoiding system pauses and data processing delays or loss.

[0053] Based on the above solution, when the data reporting volume increases and the streaming data processing engine cannot keep up, lossless scaling of Kafka and Flink can achieve system uptime-free operation and no data loss, while also reasonably increasing the parallelism and making efficient use of resources. For the effects of this solution in a real-world environment, please refer to [link to relevant documentation / reference]. Figure 3 The content shown is as follows: Figure 3As shown, the Flink stream's single-parallel processing capacity is 1 million / min. Around 20:00, Flink couldn't keep up with the computation, resulting in backpressure. After the monitoring server detected this, it reasonably increased the parallelism of the upstream Kafka to 2 and the Flink parallelism to 2. At this point, the Flink processing volume naturally increased, reaching a peak of 1.6 million / min by 22:38, with a data processing volume of 1,609,189. The different curves represent the amount of Kafka data processed by the Flink stream on different dates.

[0054] In summary, the embodiments of this application have at least the following advantages:

[0055] By monitoring the status information of the data consumed by the streaming data engine in real time through the monitoring server, the system can monitor the data processing status in real time and make timely adjustments based on the status information to ensure that problems arising during data processing are addressed promptly. Under appropriate parallelism adjustment conditions, the system can accurately determine the amount of parallelism expansion or contraction (i.e., the target parallelism) based on the status information, thereby reasonably adjusting the parallelism of the message middleware and avoiding resource waste or insufficient resources. In addition, a new streaming data processing engine is determined according to the target parallelism to process the data in the message middleware after the parallelism adjustment, avoiding sacrificing the real-time performance of data processing and ensuring that there is no data loss or delay during expansion or contraction, thus achieving lossless expansion.

[0056] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method may include at least the following steps S201 to S204:

[0057] S201, obtain the status information of the first-level data processing engine when processing the data stored in the message middleware.

[0058] S202, if the state information meets the parallelism adjustment condition, then determine the target parallelism based on the state information.

[0059] For details on the implementation of steps S201 to S202, please refer to the above. Figure 2 The steps S101 to S102 in the corresponding embodiments will not be described again here.

[0060] S203 adjusts the parallelism of the message middleware according to the target parallelism and creates a second-stream data processing engine on the data processing server.

[0061] In one embodiment, adjusting the parallelism of the message middleware can be found in the foregoing embodiments and will not be repeated here. Optionally, the data processing server can refer to, for example, Figure 1The data processing server 104 can be equipped with a streaming data processing engine (such as Flink) for real-time computing. Therefore, a second streaming data processing engine can be created within the data processing server. Specifically, the monitoring server can issue instructions to the data processing server to start a new Flink stream, i.e., a new streaming data processing engine. This streaming data processing engine has the same function as the first streaming data processing engine, namely, processing the data stored in the message middleware with adjusted parallelism. The only difference is the size of the data processing capacity. The principle for creating the second streaming data processing engine is to match it with the real-time requirements of data processing. Specific matching rules will be explained in subsequent steps.

[0062] S204, Initiate one or more data processing threads in the second-stream data processing engine that match the target parallelism.

[0063] In one embodiment, the streaming data processing engine allows the system to provide high concurrency while offering strong consistency guarantees at the same time, satisfying both high concurrency and low latency. This means that large amounts of data can be computed very quickly, primarily supported by parallel data processing threads. Therefore, data processing threads need to be started within the streaming data processing engine. Correspondingly, one or more data processing threads matching the target parallelism are started in the second streaming data processing engine. This can be the number of data processing threads required for the target parallelism; for example, if the target parallelism is 5, then 5 data processing threads can be started. These multiple data processing threads can concurrently process data in the message middleware, enabling rapid computation of massive amounts of data. It's important to note that data processing threads are started within distributed data processing servers. Each data processing server can host one or more data processing threads. The target parallelism can be greater than or less than the parallelism of the first-level data processing engine. For example, if the parallelism of the data processing threads started in the first-level data processing engine is 2, it cannot handle the current data processing. Based on the latency, the target parallelism is determined to be 4, meaning that two more parallelisms are needed to meet the real-time requirements of data processing. Therefore, the parallelism of the data processing threads started in the second-level data processing engine created on the data processing server is also 4. Of course, the above example describes the case of scaling up the parallelism of data processing threads. In the case of scaling down, a second-level data processing engine with a parallelism less than that of the first-level data processing engine can also be created, which will not be elaborated upon here.

[0064] In one embodiment, the second-stream data processing engine includes a receiving unit, a computing unit, and an input unit. An optional implementation of step S204 may be: starting a target parallelism number of data processing threads in the receiving unit; starting one or more data processing threads in the computing unit and the input unit proportionally to the amount of data received by the receiving unit; setting a flexible maximum parallelism for the computing unit and the input unit. This flexible maximum parallelism is used by the second-stream data processing engine to adjust the parallelism of the data processing threads running in the computing unit and the input unit based on the amount of data transmitted from the receiving unit to the computing unit and the computational complexity during data processing.

[0065] Here, the aforementioned receiving unit, computing unit, and storage unit are collectively referred to as data processing units. All of these units can be deployed in the data processing server via program code. While each data processing unit and its parallelism can be independent, they are interconnected when building the streaming data processing engine. The receiving unit retrieves data from the message middleware, the computing unit performs operations such as aggregation, transformation, and calculation on the data retrieved by the receiving unit, and the storage unit receives the results of the computing unit's data processing and writes them into the message middleware. Since different data processing units have different functions, the required parallelism of the data processing threads will be supplied as needed. Optionally, the number of data processing threads started in the receiving unit is matched with the target parallelism. This matching means that the number of data processing threads started in the receiving unit is the same as the target parallelism, thus ensuring the parallelism of the data processing threads in the receiving unit is equivalent to the parallelism of the message middleware, which can meet the requirement of real-time data retrieval. Furthermore, the number of data processing threads launched in the computing unit and the storage unit is adjusted proportionally. This proportional adjustment includes proportional expansion or contraction, which can be based on the amount of data received by the data processing threads of the receiving unit at the target parallelism and the maximum amount of data that the data processing threads launched in the computing unit can process. That is, if the received data volume increases or decreases by a certain factor, the parallelism is determined based on the increased or decreased data volume and the unit data processing volume. Since the core of the streaming data processing engine's data processing capability is reflected in the computing power of the computing unit, generally, the parallelism of the data processing threads launched in the computing unit and the storage unit is greater than or equal to the parallelism of the data processing threads launched in the receiving unit, which in this case is higher than the target parallelism. This allows the data processing threads to process data concurrently, enabling the computing unit to handle complex data processing and quickly obtain processing results, while also further improving the efficiency of the storage unit in storing the processing results. It should be noted that the number of data processing threads started in the computing unit and the data entry unit can be the same or different, and the first-level data processing engine can also include the aforementioned data processing units.

[0066] Furthermore, the monitoring service on the monitoring server can set an elastic maximum parallelism when launching a new Flink stream on the data processing server. This elastic maximum parallelism is set for both the computing unit and the data ingestion unit. The purpose is to allow one or more data processing threads in the computing and data ingestion units to be shut down or stopped when the amount of data transmitted from the receiving unit to the computing unit decreases or when the computational complexity decreases due to business needs. This enables elastic scaling down and saves computing resources. It's important to note that when launching a new Flink stream on the data processing server, two streams will simultaneously consume Kafka data. The new Flink stream (i.e., the second data processing engine) and the old Flink stream (i.e., the first data processing engine) will be in the same Kafka consumer group, consuming data from the Kafka with increased parallelism. This avoids duplicate or under-consumed data. However, processing too many data simultaneously is wasteful. Therefore, the monitoring service on the monitoring server will stop the old Flink stream at an appropriate time, allowing the new Flink stream to consume Kafka data, thus completing lossless scaling up or down. It should be noted that the data processing threads in different data processing units have different functions, and they correspond one-to-one with the functions of the data processing units.

[0067] Applying this solution to a data processing approach using Flink as the real-time computing framework enables lossless and uninterrupted scaling of Flink under sudden increases in data volume, thereby improving data processing capabilities, avoiding data processing latency, and enhancing the user experience. The functional block diagram of the data processing method can be shown as follows: Figure 5The content shown includes data source 501, access layer 502, message middleware A (Kafka-A) 503, Flink stream 504, message middleware B (Kafka-B) 505, database (e.g., Driud) 506, monitoring service 507, and a new Flink stream 508. Among them, access layer 502 is the receiving node, Flink stream 504 is the computing node, and database 506 is the ingestion node. Access layer 502 is mainly responsible for receiving data from different sources included in data source 501, unifying it, and standardizing the data format. Flink stream 504 is responsible for obtaining massive amounts of data from message middleware A (Kafka-A) 503 (Kafka-A), performing aggregation calculations, sorting, and other processing, and writing the processing results to message middleware B (Kafka-B) 505. Database 506 is responsible for obtaining the data processing results from Kafka-B and storing them uniformly. Monitoring service 507 monitors the growth of Kafka consumption lag (offset deviation) in message middleware A (Kafka-A) 503 in real time. When the consumption lag growth reaches a specified condition, it expands Kafka parallelism via API interface. Specifically, it expands the parallelism of Kafka-A and starts a new Flink stream 508 with expanded parallelism to consume Kafka data. That is, it consumes data from message middleware A (Kafka-A) 503 with expanded parallelism. At the same time, the new Flink stream 508 is set to elastic maximum parallelism, which can release resources when the data volume decreases. In addition, monitoring service 507 stops the old Flink stream 504 to release resources, thus achieving lossless scaling. It should be noted that the results of data consumption by both the old Flink stream and the newly started Flink stream are written to message middleware B (Kafka-B) 505, which can be used by other applications, and also stored in database 506 for data backup.

[0068] For a detailed flowchart of the corresponding capacity expansion process, please refer to [link / reference]. Figure 6First, step 601 is executed to monitor the growth of Kafka consumption lag. Specifically, the monitoring service periodically monitors Flink latency every 1 minute. Based on the Flink latency, it determines whether to increase parallelism. That is, after obtaining the lag growth data, step 602 is executed to determine whether the Kafka lag has been continuously increasing for 1 hour, with 20,000 data entries per minute, or whether the lag value is greater than 2 million. If so, the following scaling steps are executed in sequence: Step 603: Increase the parallelism of the message middleware. At this time, the monitoring service needs to calculate the required increase in parallelism. Currently, it is calculated that 1 million data entries correspond to one parallelism. The Kafka interface is called to increase the Kafka parallelism. Step 604: The monitoring service starts a Flink stream with increased parallelism (the parallelism of the receiving unit is equal to the parallelism of Kafka, and the parallelism of the computing unit and the storage unit are increased proportionally). Flink is set to elastic maximum parallelism so that it can elastically shrink when the data volume decreases. At this time, two streams will consume Kafka data simultaneously. Step 605: The monitoring service stops the old Flink stream, and the new Flink stream begins consuming Kafka data. At this point, lossless scaling is complete.

[0069] In summary, the embodiments of this application have at least the following advantages:

[0070] By setting a flexible maximum parallelism for the computing and data entry units, the parallelism can be dynamically adjusted as the data volume changes, enabling dynamic management of the parallelism of different data processing units. Furthermore, under different business scenarios, the parallelism of the data processing threads included in the created streaming data processing engine can be configured on demand, ensuring that the data processing capabilities of the streaming data processing engine match the real-time requirements of specific business scenarios, rationally scheduling resources, and improving the resource utilization and data processing capabilities of the computing system.

[0071] Please see Figure 7 , Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method may include at least the following steps S301 to S305:

[0072] S301, obtain the status information of the first-line data processing engine when processing the data stored in the message middleware.

[0073] S302, if the state information meets the parallelism adjustment condition, then determine the target parallelism based on the state information.

[0074] S303, adjust the parallelism of the message middleware according to the target parallelism, and determine the second-stream data processing engine according to the target parallelism. The second-stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment.

[0075] For details on the implementation of steps S301 to S303, please refer to the above. Figure 2 The steps S101 to S103 in the corresponding embodiments will not be described again here.

[0076] S304: Obtain the location information of the data currently being processed by the first-level data processing engine.

[0077] In one embodiment, after the second-stream data processing engine is determined based on the target parallelism, both the first-stream and second-stream data processing engines process the data stored in the message middleware simultaneously. However, since the second-stream data processing engine is better suited to the data processing requirements than the first-stream data processing engine, the monitoring server needs to stop the first-stream data processing engine at an appropriate time, allowing the second-stream data processing engine to take over and continue processing the data stored in the message middleware. Therefore, the monitoring service on the monitoring server first needs to obtain the location information of the data currently being processed, which is actively reported by the first-stream data processing engine to the monitoring server. This location information is then used to determine whether the first-stream data processing engine can be stopped.

[0078] S305, if the location information indicates that the currently processed data is the last data in the data received by the first-stream data processing engine, then after the first-stream data processing engine successfully processes the currently processed data, the first-stream data processing engine is closed and the location information is sent to the second-stream data processing engine.

[0079] In one embodiment, location information is used by the second-stream data processing engine to retrieve data from the message middleware with adjusted parallelism. The location information indicates the offset of the latest consumed message. When the latest consumed data is the last data received by the receiving unit of the first-stream data processing engine, after the first-stream data processing engine successfully processes the currently processed data (i.e., outputs the processing result), it can be shut down. Simultaneously, the monitoring service sends this location information to the second-stream data processing engine, informing it to start processing from the next position of the data currently being processed by the first-stream data processing engine, thus achieving seamless transition without system interruption. The data processed by the second-stream data processing engine is stored in the message middleware with adjusted parallelism. Unlike the first-stream data processing engine, if the message middleware has increased parallelism, the amount of data processed in parallel by the second-stream data processing engine can be significantly increased, resulting in a substantial improvement in data processing capabilities.

[0080] It's important to note that shutting down the first-stream data processing engine can be done by directly destroying it, or by severing the communication connection with the message middleware without destroying it, or by backing up the first-stream data processing engine before destruction. This way, after the second-stream data processing engine has processed a massive amount of data, the amount of data received is reduced. When certain conditions are met, the second-stream data processing engine can be shut down, and the backup first-stream data processing engine can be reactivated. After the streaming data engine finishes processing the data, the processed data can be written to a new Kafka topic for users and applications to access, while simultaneously being backed up to the database.

[0081] In summary, the embodiments of this application have at least the following advantages:

[0082] The monitoring service monitors the location of the latest data processed by the first-stream data processing engine and informs the new streaming data processing engine (i.e., the second-stream data processing engine) of the location information. This allows the second-stream data processing engine to seamlessly connect with the first-stream data processing engine to process the data in the message middleware. This approach avoids short-term system interruptions, reduces data processing latency, and ensures that no data is lost during the processing.

[0083] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0084] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a data processing device 80 provided in an embodiment of this application. The data processing device 80 can be a standalone device or a component within a standalone device, such as a chip or integrated circuit. The data processing device 80 includes components for implementing the aforementioned... Figure 2 , Figure 4 , Figure 7 The functional modules of the embodiment shown.

[0085] In one possible implementation, the data processing device 80 may include an acquisition module 801 and a determination module 802. Optionally, it may also include a stop module 803 and a transmission module 804.

[0086] The acquisition module 801 is used to acquire the status information of the first-level data processing engine when processing the data stored in the message middleware.

[0087] The determination module 802 is used to determine the target parallelism based on the state information if the state information meets the parallelism adjustment conditions.

[0088] The determination module 802 is also used to adjust the parallelism of the message middleware according to the target parallelism, and to determine the second stream data processing engine according to the target parallelism. The second stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment.

[0089] In one embodiment, the determining module 802 is specifically used to: determine that the state information meets the parallelism adjustment condition if the duration of the continuous increase in the data consumption lag reaches a first duration and the increase per unit time is greater than or equal to a first quantity threshold, or if the data consumption lag is greater than or equal to a second quantity threshold.

[0090] In one embodiment, the status information includes the lag in data consumption. The determination module 802 is further configured to: determine that the status information satisfies the parallelism adjustment condition if the duration of the continuous decrease in the lag in data consumption reaches a second duration and the decrease per unit time is greater than or equal to a third quantity threshold, or if the lag in data consumption is less than or equal to a fourth quantity threshold.

[0091] In one embodiment, the status information includes the data consumption lag. The determination module 802 is further configured to: obtain the reference data processing volume corresponding to a single parallelism and the current parallelism of the message middleware; determine the parallelism adjustment amount based on the reference data processing volume and the data consumption lag; and determine the target parallelism based on the current parallelism and the parallelism adjustment amount.

[0092] In one embodiment, the determining module 802 is specifically used to: create a second-stream data processing engine on the data processing server; and start one or more data processing threads in the second-stream data processing engine that match the target parallelism.

[0093] In one embodiment, the determining module 802 is specifically used for: the second-stream data processing engine includes a receiving unit, a computing unit, and an input unit; the determining module 802 is specifically used for: starting a target parallelism number of data processing threads in the receiving unit; starting one or more data processing threads in the computing unit and the input unit proportionally to the amount of data received by the receiving unit; setting the elastic maximum parallelism of the computing unit and the input unit, the elastic maximum parallelism being used by the second-stream data processing engine to adjust the parallelism of the data processing threads running in the computing unit and the input unit according to the amount of data transmitted from the receiving unit to the computing unit and the computational complexity during the data processing process.

[0094] In one embodiment, the acquisition module 801 is further configured to acquire the location information of the data currently being processed by the first-stream data processing engine;

[0095] The stop module 803 is used to shut down the first-stream data processing engine after the first-stream data processing engine has successfully processed the data if the position information indicates that the currently processed data is the last data in the data received by the first-stream data processing engine.

[0096] The sending module 804 is used to send location information to the second-stream data processing engine; wherein, the location information is used by the second-stream data processing engine to obtain data from the message middleware with adjusted parallelism.

[0097] It is understood that in the various device embodiments of this application, the division of multiple units or modules is merely a logical division based on function and does not constitute a limitation on the specific structure of the device. In specific implementations, some functional modules may be subdivided into more smaller functional modules, and some functional modules may be combined into a single functional module. However, regardless of whether these functional modules are subdivided or combined, the general flow executed by the device during data processing is the same. Typically, each unit (or module) corresponds to its own program code (or program instructions). When the program code corresponding to each unit runs on the processor, it causes the unit to be controlled by the processor to execute the corresponding flow and thus achieve the corresponding function.

[0098] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device 90 provided in an embodiment of this application. The computer device 90 may include one or more independent devices (such as servers, nodes, terminals, etc.), or it may include components within independent devices (such as chips, software modules, or hardware modules). The computer device 90 may include at least one processor 901 and a communication interface 902. Optionally, the computer device 90 may also include at least one memory 903 and a bus 904. The processor 901, communication interface 902, and memory 903 are connected via the bus 904.

[0099] The processor 901 is a module that performs arithmetic and / or logical operations. Specifically, it can be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a microcontroller unit (MCU).

[0100] The communication interface 902 can be used to provide information input or output to the at least one processor. And / or, the communication interface 902 can be used to receive data sent externally and / or send data externally, and can be a wired link interface including an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, etc.).

[0101] The memory 903 provides storage space, in which data such as the operating system and computer programs can be stored. The memory 903 can be one or a combination of several of the following: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0102] At least one processor 901 in the computer device 90 is used to invoke a computer program stored in at least one memory 903 to execute the aforementioned data processing method, such as the one described above. Figure 2 , Figure 4 , Figure 7 The data processing method described in the illustrated embodiment.

[0103] In one possible implementation, the processor 901 in the computer device 90 is used to call a computer program stored in at least one memory 903 to perform the following operations: obtain the status information of a first-stream data processing engine when processing data stored in the message middleware through a communication interface 902; if the status information meets the parallelism adjustment conditions, determine the target parallelism based on the status information; adjust the parallelism of the message middleware according to the target parallelism, and determine a second-stream data processing engine according to the target parallelism, the second-stream data processing engine being used to process the data stored in the message middleware after the parallelism adjustment.

[0104] In one embodiment, the status information includes the lag in data consumption, and the processor 901 is further configured to: determine that the status information satisfies the parallelism adjustment condition if the duration of the continuous increase in the lag in data consumption reaches a first duration and the increase per unit time is greater than or equal to a first quantity threshold, or if the lag in data consumption is greater than or equal to a second quantity threshold.

[0105] In one embodiment, the status information includes the lag in data consumption, and the processor 901 is further configured to: determine that the status information satisfies the parallelism adjustment condition if the duration of the continuous decrease in the lag in data consumption reaches a second duration and the decrease per unit time is greater than or equal to a third quantity threshold, or if the lag in data consumption is less than or equal to a fourth quantity threshold.

[0106] In one embodiment, the status information includes the data consumption lag, and the processor 901 is further configured to: obtain the reference data processing volume corresponding to a single degree of parallelism and the current degree of parallelism of the message middleware through the communication interface 902; determine the parallelism adjustment amount based on the reference data processing volume and the data consumption lag; and determine the target parallelism based on the current parallelism and the parallelism adjustment amount.

[0107] In one embodiment, the processor 901 is specifically configured to: create a second-stream data processing engine on a data processing server; and launch one or more data processing threads in the second-stream data processing engine that match the target degree of parallelism.

[0108] In one embodiment, the second-stream data processing engine includes a receiving unit, a computing unit, and an input unit. The processor 901 is specifically configured to: start a target parallelism number of data processing threads in the receiving unit; start one or more data processing threads in the computing unit and the input unit proportionally to the amount of data received by the receiving unit; and set the elastic maximum parallelism of the computing unit and the input unit. The elastic maximum parallelism is used by the second-stream data processing engine to adjust the parallelism of the data processing threads running in the computing unit and the input unit according to the amount of data transmitted from the receiving unit to the computing unit and the computational complexity during the data processing process.

[0109] In one embodiment, the processor 901 is further configured to: obtain the location information of the data currently being processed by the first streaming data processing engine through the communication interface 902; if the location information indicates that the currently processed data is the last data in the data received by the first streaming data processing engine, then after the first streaming data processing engine successfully processes the currently processed data, the first streaming data processing engine is shut down, and the location information is sent to the second streaming data processing engine through the communication interface 902; wherein, the location information is used by the second streaming data processing engine to obtain data from the message middleware with adjusted parallelism.

[0110] It should be understood that the computer device 90 described in the embodiments of this application can perform the foregoing... Figure 2 or Figure 4 or Figure 7 The description of the data processing method in the corresponding embodiment can also be performed as described above. Figure 8 The description of the data processing apparatus 80 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0111] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned computer device 90. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 2 , Figure 4 , Figure 7 The description of the data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0112] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0113] One aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of this application.

[0114] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the status information of the first-class data processing engine when processing data stored in the message middleware; If the state information satisfies the parallelism adjustment condition, then the target parallelism is determined based on the state information; The parallelism of the message middleware is adjusted according to the target parallelism, and a second stream data processing engine is created on the data processing server. The second stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment. The second stream data processing engine includes a receiving unit, a computing unit, and an ingestion unit. The target parallelism of data processing threads is started in the receiving unit; In the computing unit and the data entry unit, one or more data processing threads are started in proportion to the amount of data received by the receiving unit, and the maximum elastic parallelism of the computing unit and the data entry unit is set. Obtain the location information of the data currently being processed by the first stream data processing engine; If the location information indicates that the currently processed data is the last data in the data received by the first streaming data processing engine, then the first streaming data processing engine will be shut down after it has successfully processed the currently processed data. The location information is sent to the second streaming data processing engine; wherein the location information is used by the second streaming data processing engine to obtain the next position data of the data currently being processed by the first streaming data processing engine from the message middleware with the parallelism adjusted. During the data processing process of the second-stream data processing engine, the parallelism of the data processing threads running in the computing unit and the data storage unit is adjusted according to the elastic maximum parallelism, the amount of data transmitted from the receiving unit to the computing unit, and the computational complexity.

2. The method as described in claim 1, characterized in that, The status information includes the data consumption lag. Before determining the target parallelism based on the status information, if the status information satisfies the parallelism adjustment condition, the method further includes: If the duration of the continuous increase in the lag of data consumption reaches a first duration and the increase per unit time is greater than or equal to a first quantity threshold, or if the lag of data consumption is greater than or equal to a second quantity threshold, then the state information is determined to meet the parallelism adjustment condition.

3. The method as described in claim 1, characterized in that, The status information includes the data consumption lag. Before determining the target parallelism based on the status information, if the status information satisfies the parallelism adjustment condition, the method further includes: If the duration of the continuous decrease in the lag of data consumption reaches a second duration and the decrease per unit time is greater than or equal to a third quantity threshold, or if the lag of data consumption is less than or equal to a fourth quantity threshold, then the state information is determined to meet the parallelism adjustment condition.

4. The method according to any one of claims 1 to 3, characterized in that, The status information includes the data consumption lag, and determining the target parallelism based on the status information includes: Obtain the reference data processing volume corresponding to a single degree of parallelism and the current degree of parallelism of the message middleware; The parallelism adjustment amount is determined based on the reference data processing volume and the data consumption lag. The target parallelism is determined based on the current parallelism and the parallelism adjustment amount.

5. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the status information of the first-line data processing engine when processing the data stored in the message middleware; The determination module is used to determine the target parallelism based on the state information if the state information satisfies the parallelism adjustment condition. The determining module is further configured to adjust the parallelism of the message middleware according to the target parallelism, and create a second stream data processing engine on the data processing server. The second stream data processing engine is used to process the data stored in the message middleware after the parallelism adjustment. The second stream data processing engine includes a receiving unit, a calculation unit, and an input unit. The determining module is further configured to start the target parallelism of data processing threads in the receiving unit; start one or more data processing threads in the computing unit and the storage unit in proportion to the amount of data received by the receiving unit; and set the elastic maximum parallelism of the computing unit and the storage unit. The acquisition module is also used to acquire the location information of the data currently being processed by the first stream data processing engine; The stop module is used to shut down the first streaming data processing engine after it has successfully processed the currently processed data if the location information indicates that the currently processed data is the last data in the data received by the first streaming data processing engine. A sending module is used to send the location information to the second streaming data processing engine; wherein the location information is used by the second streaming data processing engine to obtain the next position data of the data currently being processed by the first streaming data processing engine from the message middleware with adjusted parallelism; The determining module is further configured to adjust the parallelism of the data processing threads running in the computing unit and the storage unit based on the elastic maximum parallelism, the amount of data transmitted from the receiving unit to the computing unit, and the computational complexity during the data processing process of the second stream data processing engine.

6. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the data processing method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the data processing method according to any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the data processing method according to any one of claims 1-4.

Images