Stream computing performance optimization method and device, electronic equipment and storage medium

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for optimizing stream computing performance. The method includes: acquiring runtime parameters of a target object during the execution of a stream computing job, wherein the target object is an object reflecting the real-time runtime performance of the stream computing job; determining the runtime state of the stream computing job based on the runtime parameters, wherein the runtime state represents the strength of the message processing capability of the stream computing job; and adjusting the runtime parameters during the execution of the stream computing job using a preset adjustment strategy rule corresponding to the target object, based on the runtime state, to optimize the stream computing performance of the stream computing job. Embodiments of this disclosure can conveniently and efficiently optimize the stream computing performance of stream computing jobs.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for optimizing stream computing performance, electronic equipment, and storage medium. Background Technology

[0002] With the continuous development of internet technology, various business scenarios generate massive amounts of data. Therefore, Flink jobs (or simply Flink jobs) based on the Apache Flink distributed processing framework have been widely adopted for convenient and efficient real-time data processing. To achieve elastic scaling of task resources and reduce operational costs, Flink jobs typically run on container clusters. These container clusters can be, for example, a general-purpose distributed resource management system and scheduling platform (YARN, Yet Another Resource Negotiator) cluster or a Kubernetes (K8S) cluster.

[0003] Currently, when deploying Flink jobs in container clusters, users typically need to manually configure task resource parameters based on experience. This can lead to either excessive or insufficient stream processing performance due to improper configuration of task resource parameters. Therefore, during the execution of a Flink job, users usually need to optimize the stream processing performance by repeatedly executing the following steps: 1. Stop the Flink job; 2. Adjust the task resource parameters; 3. Restart the Flink job.

[0004] Therefore, it can be seen that existing methods for optimizing the stream computing performance of Flink jobs are at least cumbersome and inconvenient. Summary of the Invention

[0005] This disclosure provides a method and apparatus for optimizing stream computing performance, an electronic device, and a storage medium.

[0006] In a first aspect, this disclosure provides a method for optimizing the performance of stream computing, which includes:

[0007] Obtain the running parameters of the target object during the execution of the stream computing job, wherein the target object is an object that reflects the real-time running performance of the stream computing job;

[0008] Based on the operating parameters, the operating status of the stream computing job is determined, wherein the operating status is used to represent the strength of the message processing capability of the stream computing job;

[0009] Based on the running status, the running parameters are adjusted during the running of the stream computing job using a preset adjustment strategy rule corresponding to the target object, so as to optimize the stream computing performance of the stream computing job.

[0010] Secondly, this disclosure provides a stream computing performance optimization device, which includes:

[0011] The acquisition unit is used to acquire the running parameters of the target object during the execution of the stream computing job, wherein the target object is an object that reflects the real-time running performance of the stream computing job;

[0012] The determining unit is used to determine the running state of the stream computing job based on the running parameters, wherein the running state is used to represent the strength of the message processing capability of the stream computing job;

[0013] The adjustment unit is used to adjust the running parameters during the running of the stream computing job according to the running state and using a preset adjustment strategy rule corresponding to the target object, so as to optimize the stream computing performance of the stream computing job.

[0014] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described stream computing performance optimization method.

[0015] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described stream computing performance optimization method.

[0016] The embodiments provided in this disclosure acquire the running parameters of a target object that reflects the real-time performance of a Flink job during its execution, and determine the running state of the Flink job based on these running parameters. This allows electronic devices implementing the embodiments of this disclosure to adaptively adjust the running parameters based on the running state during the execution of the Flink job without human intervention, using preset adjustment strategy rules corresponding to the target object, without stopping the Flink job. This facilitates efficient optimization of the streaming performance of the Flink job.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a stream computing performance optimization method provided in this embodiment of the disclosure;

[0020] Figure 2 This is a schematic diagram of the first operating state determination process provided in an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram of the second operating state determination process provided in an embodiment of this disclosure;

[0022] Figure 4 This is a schematic diagram illustrating an application scenario of the stream computing performance optimization method provided in the embodiments of this disclosure;

[0023] Figure 5 A block diagram of a stream computing performance optimization device provided in an embodiment of this disclosure;

[0024] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0027] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0029] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0030] In related technologies, to process large amounts of data generated in real time under different business scenarios, users typically develop targeted Flink streaming computing jobs and configure task resource parameters for the Flink jobs based on their personal experience. The Flink jobs are then deployed in container clusters, such as Kubernetes clusters, to achieve real-time data processing. The business scenario can be a business scenario for performing at least one task, such as image processing, speech processing, text processing, and video processing. Under different business scenarios, the data processed by the user-developed Flink job can be at least one of image data, speech data, text data, and video data.

[0031] For example, in a face recognition scenario, a container cluster with Flink jobs deployed can consume task messages containing face image data to be recognized from the message queue, and perform streaming processing such as feature extraction and feature matching on the face image data in the task message through its internal real-time computing tasks.

[0032] For example, in e-commerce sales scenarios, in order to quickly and accurately display real-time product sales data to users, a container cluster with Flink jobs can consume task messages containing real-time order data from the message queue, and through its internal real-time computing tasks, perform streaming processing such as cleaning, filtering, and data transformation of the order data in the task messages to provide users with product sales data that meets their needs.

[0033] However, when deploying Flink jobs in container clusters, users typically configure task resource parameters manually based on their personal experience. This can lead to situations where the performance of stream computing is excessive or insufficient due to unreasonable configuration of task resource parameters during the execution of Flink jobs.

[0034] For example, the task resource parameter can be the concurrency level (Parallel). A Flink job may only need to be configured with 5 concurrency levels to meet the business processing requirements, but if the user configures it with 10 concurrency levels, it may lead to a waste of real-time computing resources, that is, the problem of excessive stream computing performance.

[0035] For example, task resource parameters can be memory resources. A user allocates 2GB of memory resources to the container where a Flink job resides. However, due to the high data throughput of Flink, i.e., the strong message processing capability, it may cause frequent memory overflows (OOM) in the container, thus leading to instability of the container cluster.

[0036] To optimize stream computing performance, related technologies typically involve users actively monitoring and identifying potential problems during Flink job execution. The performance is then tuned by manually stopping the Flink job, adjusting task resource parameters, and restarting the job to ensure it meets the real-time task processing requirements of the business.

[0037] In the process of realizing this disclosure, the inventors discovered that the stream computing performance optimization method in the related technology has the problems of being cumbersome and inconvenient in terms of steps; on the other hand, since the method relies on manual methods to discover problems, it may also lead to instability of the entire container cluster due to the untimely discovery or resolution of problems.

[0038] To address the above issues, the inventors discovered that by deploying a detection application within a Flink job, the runtime parameters of task resources during Flink job execution can be automatically detected. Combined with the startup parameters configured when starting the Flink job, this can optimize the stream processing performance of the Flink job. Alternatively, by deploying a detection application within a Flink job and sending alarm messages to remind the user to adjust task resource parameters when adjustments are detected, the stream processing performance of the Flink job can also be optimized.

[0039] While the two methods mentioned above can improve the ease of stream computing performance optimization to some extent, they either require combining the runtime parameters of the Flink job with its startup parameters for optimization, resulting in high algorithm complexity and inaccuracy, or they still rely on manual optimization, which remains inconvenient.

[0040] Therefore, to address the potential issues of cumbersome and inconvenient steps in optimizing the streaming performance of Flink jobs, this disclosure provides a streaming performance optimization method. Please refer to... Figure 1 This is a flowchart illustrating a stream computing performance optimization method provided in an embodiment of this disclosure. The method can be applied to an electronic device, which can be a server, either a physical server or a virtual server. For example, the electronic device can be a server with a detection container deployed on it. This detection container is used to detect the running parameters of a target object during the execution of a Flink job, thereby implementing the stream computing performance optimization method of this disclosure and optimizing the stream computing performance of the Flink job. Alternatively, the electronic device can be another device that communicates with the container deploying the Flink job, such as an edge terminal device in an edge computing scenario, like a mobile phone or tablet computer; no special limitations are made here.

[0041] like Figure 1 As shown, the stream computing performance optimization method provided in this embodiment may include the following steps S101-S103, which will be described in detail below.

[0042] Step S101: Obtain the running parameters of the target object during the execution of the stream computing job, wherein the target object is an object that reflects the real-time running performance of the stream computing job.

[0043] In this embodiment of the disclosure, the target object can be at least one of the following: a target job vertex that meets specific conditions in a Flink job, memory resources allocated to the container where the Flink job is located, and central processing unit (CPU) resources used by the container where the Flink job is located.

[0044] A JobVertex is a logical vertex in the JobGraph, which is the execution graph of a Flink job.

[0045] Typically, Flink's four execution graphs, in their execution order, are: StreamGraph, JobGraph, ExecutionGraph, and PhysicalGraph. For ease of understanding, the StreamGraph and JobGraph in these four execution graphs will be briefly explained below.

[0046] A flow graph is an initial execution graph generated from code written by the user through the Stream API. It includes flow nodes (StreamNode) and flow edges (StreamEdge). Stream nodes are classes used to represent operators, and flow edges represent the edges connecting two flow nodes.

[0047] A job graph is a graph structure generated after optimizing a flow graph. In a job graph, multiple flow nodes that meet the optimization conditions may be chained together to generate a job vertex. That is, a job vertex contains one or more operators. The one or more operators linked in a job vertex are scheduled for execution as computational tasks. The input of a job vertex is a job edge, and the output is an intermediate dataset.

[0048] As explained above, the message processing volume of a computation task in a Flink job per unit time is affected by the slowest operator in its operator chain. For example, if a Flink job is set to consume a maximum of 2000 messages from the message queue per unit time (1 second), but one of its computation tasks, a Map operator with a concurrency of 10, calls the open-source distributed service framework Dubbo from the business side, and a Dubbo interface typically takes 10ms from request to response, then the message processing volume of this Map operator per unit time may only be: 1000ms / 10ms*10 = 1000 messages. Consequently, the message processing volume of the computation task containing this Map operator per unit time also drops to 1000 messages, which is far lower than its message input volume.

[0049] Therefore, it can be seen that the message processing capability of a Flink job is limited by the slowest computation task among all its computation tasks. The message processing capability of the slowest computation task is limited by the message processing capability of the operators it contains. Therefore, by finding the logic of the operator with the lowest message processing capability, and then determining the computation task with the lowest message processing capability based on that operator, the job vertex containing that computation task can be determined as the target job vertex, that is, the logical job vertex that can reflect the real-time running performance of the Flink job.

[0050] As can be seen from the above description, the target job vertex can be a logical job vertex in a Flink job that includes the target computation task. The target computation task is the computation task with the lowest message processing volume per unit time among the multiple computation tasks of the Flink job.

[0051] In this embodiment of the disclosure, the target job vertex can be determined by obtaining the processing time of a single message record of each computation task and operator in the Flink job. The processing time of a single message record of each computation task and operator can be obtained by adding a custom monitoring metric (Mertic) to the code of the Flink job.

[0052] The above explains how to determine the target job vertex when the target object is the target job vertex. It should be noted that, when the target object is the target job vertex, the target object's runtime parameters can include the target job vertex's concurrency, the target job vertex's message output per unit time, the target job vertex's message input per unit time, and the target computation task's maximum message processing time per unit time.

[0053] Since the data source for Flink job tasks is generally a message queue, such as a Kafka message queue, the amount of message input to the target job vertex per unit time can be obtained by calling specific metrics interfaces of the Kafka message queue.

[0054] For example, the message input volume can be obtained by calling the Kafka Broker JMX metrics interface, or by calling the Flink Rest Monitoring application interface; no special restrictions are imposed here.

[0055] To avoid potential impact on data source input due to backpressure when using the Flink Rest Monitoring application interface, in some embodiments, it is preferable to use the Kafka Broker JMX metrics interface to obtain the message input volume of the target job vertex per unit time, thereby avoiding potential impact on the data source input of the Flink job.

[0056] The number of messages output by the target job vertex per unit time, and the maximum message processing time of the target computation task per unit time, can be obtained by adding custom monitoring metrics in the code of the Flink job. The detailed processing process will not be elaborated here.

[0057] The above explains how to obtain the runtime parameters of the target object when the target object is the vertex of the target job. Additionally, in some embodiments, when the target object is memory resources allocated to the container where the Flink job resides, the runtime parameters of the target object can be the memory occupied by the old generation. This old generation memory occupied value represents the size of the remaining old generation (oldGen) memory resources after the virtual machine (JVM) hosting the Flink job has performed multiple heap memory reclamation (Full GC) processes.

[0058] In this implementation, the memory usage of the old generation can be obtained by parsing the garbage collection (GC) log file of the Flink job.

[0059] GC logs include the changes and usage of memory resources in different regions after each garbage collection. By parsing GC logs, we can obtain the size of the remaining old generation memory resources occupied by the task scheduler in the Flink job after each Full GC, that is, the size of the remaining old generation space.

[0060] Of course, when the target object is the memory resource allocated for the container where the Flink job is located, its running parameters can also include other parameters, such as the value of the memory resource occupied by the Young Generation, which is not specifically limited here.

[0061] In addition, in some embodiments, when the target object is the CPU resources used by the container where the Flink job is located, the running parameters of the target object can also be the average CPU resource utilization of the container within a preset time period, wherein the preset time period can be, for example, 10 minutes, without any special limitation.

[0062] The above explains how to obtain the running parameters corresponding to the target objects when they are different objects. The following explains how to determine the running status of the Flink job based on the running parameters of the target objects after obtaining them in different implementations.

[0063] Step S102: Determine the running status of the Flink job based on the running parameters, where the running status is used to represent the strength of the Flink job's message processing capability.

[0064] As can be seen from the above description, during the execution of a Flink job, its running status can be determined through different dimensions such as the real-time message processing capability, memory resource usage, and CPU resource utilization of the Flink job. In other words, the stream computing performance of a Flink job can be determined by the running parameters of the target object that reflect its real-time running performance during its execution.

[0065] For example, if the real-time message processing capability of a Flink job is much greater than or much less than its message input and message output capabilities, it indicates that the real-time stream computing performance of the Flink job may be too strong or too weak.

[0066] For example, if a Flink job is found to frequently run out of memory during its execution, it indicates that its stream processing performance may be too weak.

[0067] For example, if the CPU resource utilization of the container in which the Flink job resides remains low for an extended period, the streaming performance of that Flink job may be too weak.

[0068] Based on the above findings, in order to conveniently and efficiently optimize the performance of stream computing, in this embodiment of the disclosure, the running parameters of the target object during the execution of the Flink job can be obtained, the running status of the Flink job can be determined based on the running parameters, and then it can be determined whether the running parameters need to be optimized based on the running status, so as to optimize the stream computing performance of the Flink job.

[0069] Step S103: Based on the running status, use the preset adjustment strategy rules corresponding to the target object to adjust the running parameters during the running of the stream computing job in order to optimize the stream computing performance of the stream computing job.

[0070] Compared to related technologies that require manually stopping the Flink job, adjusting task resource parameters, and restarting the Flink job when optimizing stream computing performance, which is inconvenient, in this embodiment of the present disclosure, after determining the running state of the Flink job based on the running parameters of the target object, the electronic device implementing the method of this embodiment can adaptively adjust the running parameters during the running of the Flink job without human intervention, using a preset adjustment strategy rule corresponding to the target object, so as to conveniently and efficiently optimize the stream computing performance of the Flink job.

[0071] Please refer to Figure 2This is a schematic diagram of the first running state determination process provided in the embodiments of this disclosure. As can be seen from the above description, in the implementation where the target object of this disclosure can be a target job vertex, the running parameters may include the concurrency of the target job vertex (hereinafter referred to as P), the message output of the target job vertex per unit time (hereinafter referred to as O), the message input of the target job vertex per unit time (hereinafter referred to as S), and the maximum message processing time of the target computing task per unit time (hereinafter referred to as T).

[0072] In this implementation, such as Figure 2 As shown, determining the running status of the stream computing job based on the running parameters includes: step S201, calculating the message processing volume of the target job vertex per unit time based on the concurrency and the maximum message processing time; step S202, determining the running status of the stream computing job based on the message input volume, message output volume, and message processing volume.

[0073] Let D represent the message processing volume of the target job vertex per unit time. Then, when the unit time is 1 second, the message processing volume can be calculated by the formula: D = 1 / T * P.

[0074] After calculating the message processing volume, the running status of the Flink job can be determined by comparing the relationship between the message input volume, message output volume, and message processing volume.

[0075] In this embodiment, determining the running state of the stream computing job based on the message input volume, message output volume, and message processing volume can be as follows: obtaining the absolute value of the difference between the message input volume and the message output volume as a first difference; if the first difference is less than a first preset threshold, obtaining the absolute value of the difference between the message processing volume and the message input volume as a second difference; if the message processing volume is greater than the message input volume and the second difference is greater than or equal to a second preset threshold, determining the running state as a strong message processing capability state; if the message output volume is less than the message input volume, the message processing volume is less than the message input volume, and the second difference is greater than or equal to a second preset threshold, determining the running state as a weak message processing capability state.

[0076] The first preset threshold can be a value used to determine whether the message input and message output are approximately equal; the second preset threshold can be a value used to determine whether the difference between the message processing volume and the message input volume is large. The specific values ​​of the first and second preset thresholds can be set as needed, and no special limitation is made here. For example, the first preset threshold can be 0.1, and the second preset threshold can be 1000.

[0077] That is, in this implementation, if the message input and message output of the target job vertex are approximately equal based on the first difference, the size of its message processing volume and its message input volume can be compared. If its message processing volume is much greater than its message input volume, it indicates that the Flink job is in a state of strong message processing capability; while if the message output volume of the target job vertex is less than its message input volume, and its message processing volume is also much less than its message input volume, it indicates that the Flink job is in a state of weak message processing capability.

[0078] As can be seen from the above description, in this embodiment, compared to the related technologies that may determine the running state of a Flink job solely based on its throughput, i.e., whether the message input and message output are balanced, and thus determine whether its stream computing performance needs optimization, while ignoring its message processing capabilities, this embodiment determines the running state of a Flink job by identifying the target job vertex, thereby improving the real-time performance and accuracy of stream computing performance optimization.

[0079] In addition, in this embodiment, the preset adjustment strategy rule can be a rule used to instruct the adjustment of the running parameters of the streaming computing job based on a first ratio between the message processing volume and the message input volume corresponding to the target job vertex. That is, after determining the running state of the Flink job, the step of adjusting the running parameters during the running of the streaming computing job according to the running state and using the preset adjustment strategy rule corresponding to the target object includes: obtaining the first ratio between the message processing volume and the message input volume; when the running state is a strong message processing capability state, reducing the concurrency based on the first ratio during the running of the streaming computing job; or, when the running state is a weak message processing capability state, increasing the concurrency based on the first ratio during the running of the streaming computing job.

[0080] In other words, considering that the message processing capability of a Flink job is directly proportional to its concurrency (i.e., the higher the concurrency, the stronger the message processing capability of the Flink job), when determining the target object's runtime parameters to be adjusted based on the Flink job's runtime status, a first ratio between the message processing volume and the message input volume can be obtained. If the runtime status indicates that the Flink job is in a strong message processing capability state, the concurrency of the Flink job is reduced according to this first ratio during the Flink job's execution to decrease its message processing capability and avoid unnecessary task resource consumption. Otherwise, if the runtime status indicates that the Flink job is in a weak message processing capability state, the concurrency of the Flink job is amplified according to this first ratio during the Flink job's execution to improve its message processing capability, so that it can match its message input volume and meet the real-time task processing requirements of message processing.

[0081] It should be noted that, in this implementation, when adjusting the concurrency of a Flink job based on the first ratio, the current concurrency can be directly divided or multiplied by the first ratio. For example, if the first ratio is 2 and the original concurrency is 6, the concurrency can be adjusted to 3 if the message processing capacity needs to be reduced; or, if the message processing capacity needs to be increased, the concurrency can be adjusted to 12.

[0082] Of course, to further improve accuracy, the concurrency level can be divided or multiplied by the ratio and then multiplied by a preset weighting coefficient. This weighting coefficient can be a value obtained from prior experiments to correct the adjustment result. For example, if the weighting coefficient is 0.8, then with a ratio of 2 and an original concurrency level of 6, if it is necessary to reduce the message processing capacity, the concurrency level can be adjusted to [6 / 2*0.8] = 2; if it is necessary to increase the message processing capacity, the concurrency level can be adjusted to [6*2*0.8] = 9. It is understood that in this implementation, if the reduced or increased concurrency level is not an integer, its value can be rounded down.

[0083] In addition, to avoid abnormal operation of the stream computing job due to excessively high or low concurrency after adjustment, in this embodiment of the disclosure, a minimum and a maximum value of concurrency can be preset. After calculating the adjusted value corresponding to the concurrency based on the first ratio, if the adjusted value is between the minimum and the maximum value, the concurrency of the target job vertex is directly adjusted according to the adjusted value; if the adjusted value is less than the minimum value, the concurrency is adjusted according to the minimum value; if the adjusted value is greater than the maximum value, the concurrency is adjusted according to the maximum value.

[0084] The above provides a detailed explanation of how to determine the running state of a Flink job based on its running parameters when the target object is the vertex of the target job, and how to adjust the running parameters based on the determined running state. The following describes how to determine the running state and how to adjust the running parameters based on the determined running state in the implementation where the target object is the memory resource allocated for the container where the Flink job is located.

[0085] Please refer to Figure 3 This is a schematic diagram of the second running state determination process provided in the embodiments of this disclosure. As can be seen from the above description, in the implementation of this disclosure where the target object is the memory resources allocated to the container where the Flink job is located, the running parameters may include the memory occupied by the old generation.

[0086] like Figure 3 As shown, in this embodiment, determining the running state of the stream computing job based on the running parameters may include: step S301, obtaining a recommended value for the memory resource based on the memory occupied by the old generation and a preset memory configuration algorithm; step S302, calculating the absolute value of the difference between the real-time value and the recommended value of the memory resource, and using the absolute value as the third difference; if the third difference is greater than or equal to a third preset threshold and the recommended value is less than the real-time value, step S303 is executed to determine the running state as a strong message processing capability state; if the third difference is greater than or equal to a third preset threshold and the recommended value is greater than the real-time value, step S304 is executed to determine the running state as a weak message processing capability state.

[0087] The third preset threshold can be used to determine whether there is a large difference between the recommended value and the real-time value. Its value can be set as needed, and there are no special restrictions here.

[0088] Additionally, if M represents the memory occupied by the old generation, then the preset memory configuration algorithm can be: the recommended value of memory resources is 3 to 4 times M, the size of memory resources occupied by the young generation is 1 to 1.5 times M, and the size of memory resources occupied by the old generation is 2 to 3 times M. The specific coefficient of each value can be set as needed, and no special limitation is made here.

[0089] As can be seen from the above description, in this implementation, when the target object is the memory resources configured in the container where the Flink job is located, it is only necessary to compare whether the difference between the recommended value of the memory resource and its current real-time value is too large to conveniently and accurately determine the running status of the Flink job, and thus determine whether it is necessary to optimize its stream computing performance.

[0090] In addition, in this embodiment, the preset adjustment strategy rule can be used to instruct the running parameters of the stream computing job to be adjusted based on the recommended value of memory resources. That is, after obtaining the running status of the Flink job, the step of adjusting the running parameters during the running of the stream computing job according to the running status and using the preset adjustment strategy rule corresponding to the target object includes: when the running status is a strong message processing capability state or a weak message processing capability state, updating the memory resources allocated to the container according to the recommended value during the running of the stream computing job.

[0091] In this implementation, if the obtained running status indicates that the Flink job is in a strong message processing capability state or a weak message processing capability state, it means that the memory resources allocated to the container where the Flink job resides need to be adjusted. Since the calculated recommended value is based on the current memory resource usage, directly updating the size of the memory resources allocated to the container to the recommended value can usually meet the resource requirements of stream computing. Of course, in actual implementation, when updating the memory resources according to the recommended value, the recommended value can also be multiplied by the memory weight coefficient obtained from pre-experimentation to improve the accuracy of parameter adjustment, thereby more accurately optimizing the stream computing performance of the Flink job and reducing unnecessary waste of memory resources.

[0092] Understandably, in practice, to improve the stability of Flink jobs, minimum and maximum values ​​of memory resources can be preset. Before updating the memory resources allocated to the container based on the recommended values, it can be determined whether the recommended values ​​are between the minimum and maximum values. If they are, the memory resources allocated to the container are updated based on the recommended values; if the recommended values ​​are less than the minimum values, the memory resources allocated to the container are updated based on the minimum values; and if the recommended values ​​are greater than the maximum values, the memory resources allocated to the container are updated based on the maximum values.

[0093] The above describes how to determine the running state and adjust the running parameters based on the determined running state in the implementation where the target object is the memory resources allocated to the container where the Flink job resides. The following describes how to determine the running state and adjust the running parameters based on the determined running state in the implementation where the target object is the CPU resources used by the container where the Flink job resides.

[0094] As can be seen from the above description, in the implementation method described in this disclosure where the target object is the CPU resources of the container where the Flink job is located, the running parameters can be the average CPU resource utilization of the container within a preset time period.

[0095] In this implementation, determining the running state of a Flink job based on running parameters includes: determining the running state as a strong message processing capability state when the average CPU resource utilization is greater than or equal to a fifth preset threshold; and determining the running state as a weak message processing capability state when the average CPU resource utilization is less than the fifth preset threshold.

[0096] The value of the fifth preset threshold can be set as needed, for example, it can be 10%, and there is no special limitation here.

[0097] In addition, in this embodiment, the preset adjustment strategy rule can be used to instruct the adjustment of the running parameters of the streaming computing job based on a second ratio between a preset maximum CPU resource utilization and a preset average CPU resource utilization. That is, after obtaining the running state of the Flink job, the step of adjusting the running parameters during the running of the streaming computing job according to the running state and using the preset adjustment strategy rule corresponding to the target object includes: obtaining the second ratio between the preset maximum CPU resource utilization and the average CPU resource utilization; when the running state is a strong message processing capability state, reducing the concurrency and / or memory resources corresponding to the streaming computing job according to the second ratio during the running of the streaming computing job; when the running state is a weak message processing capability state, increasing the concurrency and / or memory resources corresponding to the streaming computing job according to the second ratio during the running of the streaming computing job, so as to optimize the streaming computing performance of the streaming computing job.

[0098] This preset maximum CPU resource utilization rate represents the maximum CPU resource utilization rate. This value can be set as needed, for example, it can be 70. It is understood that this preset maximum CPU resource utilization rate needs to be set according to the aforementioned fifth preset threshold. For example, if the fifth preset threshold is 10%, then this preset maximum CPU resource utilization rate will generally not exceed 10%.

[0099] In other words, if it is determined that the current message processing capability of a Flink job is strong or weak, its message processing capability can be improved or reduced by increasing the concurrency and / or memory resources of the Flink job, or by reducing its concurrency and / or memory resources, so as to achieve accurate optimization of stream computing performance.

[0100] In this embodiment, the process of amplifying or reducing the concurrency and / or memory resources of the streaming computation job based on the second ratio is similar to the process of amplifying or reducing the concurrency of the target job vertex based on the first ratio described above.

[0101] That is, the concurrency and / or memory resources of the stream computing job can be directly divided or multiplied by the second ratio to obtain the adjusted concurrency value and the adjusted memory resource value.

[0102] Alternatively, weight coefficients corresponding to the concurrency and memory resources can be obtained in advance to correct the values ​​obtained after division or multiplication, so as to accurately optimize the performance of streaming computing jobs.

[0103] Alternatively, minimum and maximum values ​​for concurrency and memory resources can be preset. The system then checks if the calculated, adjusted concurrency value falls between these two values. If it does, the concurrency of the streaming job is adjusted based on the adjusted value. Otherwise, if the adjusted value is less than the minimum value, the concurrency is adjusted accordingly. Similarly, if the adjusted value is greater than the maximum value, the concurrency is adjusted based on the maximum value. Likewise, the system checks if the calculated, adjusted memory resource value falls between these two values. If it does, the memory resources for the streaming job are adjusted accordingly. Otherwise, if the adjusted value is less than the minimum value, the memory resources are adjusted accordingly. And if the adjusted value is greater than the maximum value, the memory resources are adjusted accordingly.

[0104] The above explanations illustrate how to determine the runtime parameters of a target object, how to determine the running state of a Flink job based on these parameters, and how to optimize the stream processing performance of the Flink job by adjusting these parameters. It should be noted that in actual implementation, the target object can include multiple objects simultaneously; this is not a specific limitation.

[0105] In the embodiments of this disclosure, when the target object includes multiple objects, the running parameters of different objects can be adjusted separately to optimize the stream computing performance of the Flink job; or, when the running parameters of different objects overlap, the average value of the overlapping running parameters can be calculated to update the overlapping running parameters.

[0106] For example, if the target object is both the target job vertex and the memory resources allocated to the container where the Flink job is located, the concurrency of the Flink job can be adjusted to 3 based on the target job vertex, and the memory resources allocated to the container can be adjusted to 4G based on the recommended value of memory resources.

[0107] For example, if the target object is both the memory resources allocated to the container where the Flink job resides and the average CPU resource utilization of the container within a preset time period, then based on the recommended value of memory resources, it is determined that the container's memory resources need to be adjusted to 4G. Based on the average CPU resource utilization, it is determined that the concurrency needs to be adjusted to 2. If the container's memory resources need to be adjusted to 6G, then ultimately the container's memory resources will be adjusted to 5G, and the concurrency of the Flink job will be adjusted to 2.

[0108] It should be noted that, in this embodiment of the disclosure, step S103 above can also be: obtaining the running parameters of the stream computing job in the same historical time period as reference running parameters; and adjusting the running parameters during the running of the stream computing job using a preset adjustment strategy rule corresponding to the target object based on the reference running parameters and the running status, so as to optimize the stream computing performance of the stream computing job.

[0109] In practice, the message input volume of a Flink job typically follows a pattern across different time periods. For example, in an office building turnstile system, with user authorization, facial image data can be collected from the user using an image acquisition device on the front-end turnstile. This facial image data is then transmitted to the back-end Flink job as a task message for streaming facial recognition processing. In this application scenario, the message input volume of the back-end Flink job is usually higher during peak commuting hours, such as 7:00-9:00 AM and 5:00-7:00 PM on weekdays, while the message processing volume is usually lower at other times. Therefore, based on the current time period of the Flink job, its historical running parameters for the same time period can be obtained as reference running parameters. By using these reference running parameters and the running status determined based on the current target object's running parameters, the running parameters of the target object can be adjusted more accurately to precisely optimize the streaming computing performance of the Flink job and meet its real-time task processing requirements.

[0110] In this embodiment, the step of adjusting the running parameters during the execution of the stream computing job based on the reference running parameters and the running state, using a preset adjustment strategy rule corresponding to the target object, to optimize the stream computing performance of the stream computing job includes: obtaining recommended running parameters based on the running state using the preset adjustment strategy rule; and adjusting the running parameters of the target object during the execution of the stream computing job based on the reference running parameters and the recommended running parameters, to optimize the stream computing performance of the stream computing job.

[0111] In this embodiment, adjusting the operating parameters of the target object according to the reference operating parameters and the recommended operating parameters includes: calculating any one of the maximum, minimum, or average values ​​of the same parameter in the reference operating parameters and the recommended operating parameters as the target adjustment value of the same parameter.

[0112] That is, the operating parameters determined directly based on the operating status in the above implementations where the target objects are different objects can be used as the recommended operating parameters in this implementation. In this way, the reference operating parameters in the same historical time period and the recommended operating parameters can be combined to obtain operating parameters that better meet business needs, so as to avoid frequent adjustment of operating parameters and instability of container clusters.

[0113] For example, when the target object is the target job vertex, if the recommended value for the concurrency of the Flink job is determined to be 3 according to the preset adjustment strategy rules corresponding to the target job vertex, and the reference running parameter in the same historical time period is 4, then the running parameter can be adjusted according to the actual business needs by taking any one of the maximum, minimum or average values ​​between the two.

[0114] Please refer to Figure 4 This is a schematic diagram illustrating an application scenario of the stream computing performance optimization method provided in this disclosure embodiment. For ease of understanding, the following is combined with... Figure 4 Taking the application scenario of using Flink jobs to process face recognition tasks as an example, this disclosure illustrates the stream computing performance optimization method provided in the embodiments. Figure 4 As shown, to perform real-time login security verification for users, with user authorization, terminal device 1000 can acquire the user's facial image data through its built-in or connected image acquisition device, and transmit the facial image data to the Flink job deployed in container 1 via communication network 2000, as follows. Figure 4As shown, container 1 can be any container node located in server 3000 within the container cluster. In this application scenario, a Flink job runs in container 1. This Flink job receives face image data to be recognized transmitted by terminal device 1000 in the form of task messages, consumes the face image data from its corresponding message queue, and performs feature extraction, feature matching, and other processing to complete real-time streaming recognition processing of the face image data. Simultaneously, to optimize the streaming computing performance of this Flink job, container 2 in the container cluster is also deployed in server 3000. Container 2 is used to execute steps S101-S103, namely, executing step S101 to obtain the running parameters of the target object during the Flink job's execution; step S102 to determine the running status of the Flink job based on these running parameters; and step S103 to adjust the running parameters during the Flink job's execution using a preset adjustment strategy rule corresponding to the target object, based on the running status. When it is necessary to adjust the running parameters, the adjustment can be sent to the node in the container cluster responsible for task resource scheduling, such as... Figure 4 The container 3 shown sends a runtime parameter adjustment request. After receiving the runtime parameter adjustment request, the container 3 can adaptively and accurately adjust its runtime parameters without restarting the Flink job. After successfully adjusting the runtime parameters, the Flink job can load the adjusted runtime parameters in real time for task processing to meet the real-time processing requirements of the face recognition task.

[0115] It should be noted that, in Figure 4 For ease of explanation, this example uses container nodes in a container cluster, specifically container 1, container 2, and container 3, all deployed and running on server 3000. In practice, however, container nodes in a container cluster can be deployed on different servers to facilitate the rational allocation of task resources; no special limitation is made here. Furthermore, it is understood that in actual implementation, there can be multiple terminal devices (1000) and multiple servers (3000). Figure 4 For ease of explanation, the detection node is represented as container 2 and the scheduling node as container 3. In practice, container 2 and container 3 can also be the same node. In this implementation, the container can directly adjust the operating parameters when it detects that the operating parameters need to be adjusted.

[0116] It should also be noted that the above explanation uses a Flink job for face image processing, i.e., for performing image processing tasks. In actual implementation, the Flink job can also be applied to perform one or more tasks such as speech processing, text processing, and video processing. No special limitation is made here.

[0117] In summary, the stream computing performance optimization method according to the embodiments of this disclosure obtains the running parameters of a target object that reflects the real-time running performance of a Flink job during its execution, and determines the running state of the Flink job based on these running parameters. This allows the electronic device implementing the embodiments of this disclosure to adaptively adjust the running parameters based on the running state during the execution of the Flink job without human intervention, using a preset adjustment strategy rule corresponding to the target object, without stopping the Flink job. This facilitates and efficiently optimizes the stream computing performance of the Flink job.

[0118] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0119] In addition, this disclosure also provides stream computing performance optimization apparatus, electronic equipment, and computer-readable storage medium, all of which can be used to implement any of the stream computing performance optimization methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0120] Figure 5 This is a block diagram of a stream computing performance optimization device provided in an embodiment of the present disclosure.

[0121] Reference Figure 5 This disclosure provides a stream computing performance optimization device that can be applied to electronic devices. The stream computing performance optimization device 500 includes: an acquisition unit 501, a determination unit 502, and an adjustment unit 503.

[0122] The acquisition unit 501 is used to acquire the running parameters of the target object during the running of the stream computing job, wherein the target object is an object that reflects the real-time running performance of the stream computing job.

[0123] The determining unit 502 is used to determine the running state of the stream computing job based on the running parameters, wherein the running state is used to represent the strength of the message processing capability of the stream computing job.

[0124] The adjustment unit 503 is used to adjust the running parameters during the running of the stream computing job according to the running state and using a preset adjustment strategy rule corresponding to the target object, so as to optimize the stream computing performance of the stream computing job.

[0125] In some embodiments, the target object includes a target job vertex, which is a logical job vertex in the stream computing job that includes a target computing task. The target computing task is the computing task with the lowest message processing volume per unit time among multiple computing tasks in the stream computing job. The running parameters include the concurrency of the target job vertex, the message output volume of the target job vertex per unit time, the message input volume of the target job vertex per unit time, and the maximum message processing time of the target computing task per unit time. When determining the running state of the stream computing job based on the running parameters, the determining unit 502 can be used to: calculate the message processing volume of the target job vertex per unit time based on the concurrency and the maximum message processing time; and determine the running state of the stream computing job based on the message input volume, message output volume, and message processing volume.

[0126] In this embodiment, when determining the running state of a stream computing job based on message input, message output, and message processing volume, the determining unit 502 can be used to: obtain the absolute value of the difference between message input and message output as a first difference; if the first difference is less than a first preset threshold, obtain the absolute value of the difference between message processing volume and message input as a second difference; if the message processing volume is greater than the message input and the second difference is greater than or equal to a second preset threshold, determine the running state as a strong message processing capability state; if the message output is less than the message input, the message processing volume is less than the message input, and the second difference is greater than or equal to a second preset threshold, determine the running state as a weak message processing capability state.

[0127] In this embodiment, the preset adjustment strategy rule is used to instruct the adjustment of the running parameters based on a first ratio between the message processing volume and the message input volume corresponding to the target job vertex. When the adjustment unit 503 adjusts the running parameters during the running of the stream computing job according to the running state and using the preset adjustment strategy rule corresponding to the target object, it can be used to: obtain the first ratio between the message processing volume and the message input volume; reduce the concurrency based on the first ratio during the running of the stream computing job when the running state is a strong message processing capability state; or increase the concurrency based on the first ratio during the running of the stream computing job when the running state is a weak message processing capability state.

[0128] In some implementations, the target object includes memory resources allocated to the container where the streaming job resides; the running parameters include the old generation memory usage value, which is the size of the remaining old generation memory resources after the virtual machine carrying the streaming job has performed multiple heap memory reclamation processes; when determining the running state of the streaming job based on the running parameters, the determining unit 502 can be used to: obtain a recommended value of memory resources based on the old generation memory usage value and a preset memory configuration algorithm; calculate the absolute value of the difference between the real-time value of the memory resources and the recommended value, and use the absolute value as a third difference; if the third difference is greater than or equal to a third preset threshold and the recommended value is less than the real-time value, then determine the running state as a strong message processing capability state; if the third difference is greater than or equal to the third preset threshold and the recommended value is greater than the real-time value, then determine the running state as a weak message processing capability state.

[0129] In this embodiment, the preset adjustment strategy rule is used to instruct the adjustment of the running parameters based on the recommended values ​​of memory resources. When the adjustment unit 503 adjusts the running parameters during the running of the stream computing job according to the running state and using the preset adjustment strategy rule corresponding to the target object, it can be used to: update the memory resources allocated to the container according to the recommended values ​​during the running of the stream computing job when the running state is a strong message processing capability state or a weak message processing capability state.

[0130] In some embodiments, the target object includes the central processing unit (CPU) resources used by the container where the stream computing job is located; the operating parameters include the average CPU resource utilization of the container within a preset time period; when determining the operating state of the stream computing job based on the operating parameters, the operating parameter determination module 502 can be used to: determine the operating state as a strong message processing capability state when the average CPU resource utilization is greater than or equal to a fifth preset threshold; and determine the operating state as a weak message processing capability state when the average CPU resource utilization is less than the fifth preset threshold.

[0131] In some embodiments, the preset adjustment strategy rule is used to instruct the adjustment of operating parameters based on a second ratio between a preset maximum CPU resource utilization and a preset average CPU resource utilization. When the adjustment unit 503 adjusts the operating parameters during the execution of a stream computing job according to the operating state and using the preset adjustment strategy rule corresponding to the target object, it can be used to: obtain the second ratio between the preset maximum CPU resource utilization and the average CPU resource utilization; when the operating state is a strong message processing capability state, reduce the concurrency and / or memory resources corresponding to the stream computing job according to the second ratio during the execution of the stream computing job; when the operating state is a weak message processing capability state, increase the concurrency and / or memory resources corresponding to the stream computing job according to the second ratio during the execution of the stream computing job.

[0132] In some embodiments, when the adjustment unit 503 adjusts the running parameters during the running of the stream computing job according to the running state and using a preset adjustment strategy rule corresponding to the target object, it can also be used to: obtain the running parameters of the stream computing job in the same historical time period as reference running parameters; and adjust the running parameters during the running of the stream computing job according to the reference running parameters and the running state and using a preset adjustment strategy rule corresponding to the target object.

[0133] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0134] Reference Figure 6 This disclosure provides an electronic device, which includes: at least one processor 601; at least one memory 602; and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 to enable the at least one processor 601 to perform the above-described stream computing performance optimization method.

[0135] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described stream computing performance optimization method. The computer-readable storage medium may be volatile or non-volatile.

[0136] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described stream computing performance optimization method.

[0137] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0138] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0139] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0140] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0141] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0142] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0143] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0144] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0146] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A method for stream computing performance optimization, the method comprising: include: The runtime parameters of a target object during the execution of a stream computing job are obtained. The target object is an object reflecting the real-time performance of the stream computing job. The target object includes a target job vertex, which is a logical job vertex within the stream computing job that includes a target computing task. The target computing task is the computing task with the lowest message processing volume per unit time among multiple computing tasks in the stream computing job. The runtime parameters include the concurrency of the target job vertex, the message output volume of the target job vertex per unit time, the message input volume of the target job vertex per unit time, and the maximum message processing time of the target computing task per unit time. Determining the running status of the stream computing job based on the running parameters includes: calculating the message processing volume of the target job vertex per unit time based on the concurrency and the maximum message processing time; when the message input and message output volumes of the target job vertex are approximately equal, determining the running status of the stream computing job by comparing the message processing volume and the message input volume; wherein, the running status is used to represent the strength of the message processing capability of the stream computing job; Based on the running status, the running parameters are adjusted during the running of the stream computing job using a preset adjustment strategy rule corresponding to the target object, so as to optimize the stream computing performance of the stream computing job.

2. The method of claim 1, wherein, The step of determining the running status of the stream computing job by comparing the amount of messages processed and the amount of messages input includes: The absolute value of the difference between the message input amount and the message output amount is taken as the first difference; If the first difference is less than the first preset threshold, the absolute value of the difference between the message processing volume and the message input volume is obtained as the second difference. If the message processing volume is greater than the message input volume and the second difference is greater than or equal to the second preset threshold, the operating state is determined to be a strong message processing capability state. If the message output is less than the message input, the message processing volume is less than the message input, and the second difference is greater than or equal to the second preset threshold, the operating state is determined to be a weak message processing capability state.

3. The method of claim 2, wherein, The preset adjustment strategy rule is used to instruct the operation parameters to be adjusted based on a first ratio between the message processing volume and the message input volume corresponding to the target job vertex; The step of adjusting the running parameters during the execution of the stream computing job based on the running state and using a preset adjustment strategy rule corresponding to the target object includes: Obtain a first ratio between the message processing volume and the message input volume; When the running state is the strong message processing capability state, during the running of the stream computing job, the concurrency degree is reduced according to the first ratio; When the running state is the weak message processing capability state, during the running of the stream computing job, the concurrency is amplified according to the first ratio.

4. The method of claim 1, wherein, The step of adjusting the running parameters during the execution of the stream computing job based on the running state and using a preset adjustment strategy rule corresponding to the target object includes: Obtain the running parameters of the stream computing job within the same historical time period as reference running parameters; Based on the reference operating parameters and the operating status, the operating parameters are adjusted during the operation of the stream computing job using a preset adjustment strategy rule corresponding to the target object.

5. The method according to any one of claims 1 to 4, characterized in that, The stream computing job is used to perform any one of the following tasks: image processing, speech processing, text processing, and video processing.

6. A stream computing performance optimization apparatus, characterized by, include: The acquisition unit is used to acquire the running parameters of a target object during the execution of a stream computing job. The target object is an object reflecting the real-time performance of the stream computing job. The target object includes a target job vertex, which is a logical job vertex in the stream computing job that includes a target computing task. The target computing task is the computing task with the lowest message processing volume per unit time among multiple computing tasks in the stream computing job. The running parameters include the concurrency of the target job vertex, the message output volume of the target job vertex per unit time, the message input volume of the target job vertex per unit time, and the maximum message processing time of the target computing task per unit time. A determining unit is configured to determine the running state of the stream computing job based on the running parameters, including: calculating the message processing volume of the target job vertex per unit time based on the concurrency and the maximum message processing time; when the message input and message output volumes of the target job vertex are approximately equal, determining the running state of the stream computing job by comparing the magnitude of the message processing volume and the message input volume; wherein, the running state is used to represent the strength of the message processing capability of the stream computing job; The adjustment unit is used to adjust the running parameters during the running of the stream computing job according to the running state and using a preset adjustment strategy rule corresponding to the target object, so as to optimize the stream computing performance of the stream computing job.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the stream computing performance optimization method as described in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the stream computing performance optimization method as described in any one of claims 1-5.

Images