Data flow write-in control method, system, device and equipment and readable storage medium

Abstract

The invention relates to a data flow write-in control method, system, device and equipment, a readable storage medium and a program product. Comprising the following steps: receiving write-in performance characteristics of a data stream processing engine in a first cycle period; according to the write-in performance characteristics, a first operator routing table of a first cycle period is adjusted, a second operator routing table of a second cycle period is obtained, the second operator routing table comprises a routing record in the second cycle period, and the routing record indicates a to-be-written data stream in the data stream processing engine and a corresponding data write-in operator; a second operator routing table is sent to the data flow processing engine, the second operator routing table is used for indicating the data flow processing engine to write the data flow to be written corresponding to the data write operator by using the data write operator indicated by each routing record in the second operator routing table in a second cycle period, and the data flow to be written corresponding to the data write operator is sent to the data flow processing engine. And writing the data writing operator and a file corresponding to the data stream to be written. By adopting the method, the data stream writing performance can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data stream writing control method, system, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] A data stream refers to a set of data sequences with the same data type and originating from the same data source. For example, the log data of each video application playing a video, the log data of each shared device being used, and the log data of each terminal device being started.

[0003] In traditional technologies, most of them first write the data stream to the intermediate file system through all the data writing operators in the data writing engine. Each data writing operator will generate a corresponding small file when writing the data stream to the intermediate file system, so that the same data stream is stored in the form of multiple small files in the intermediate file system. The small files of the same data stream in the intermediate file system are then integrated and processed, and the large files corresponding to the integrated data stream are stored in the final file system.

[0004] However, although this method solves the problem that the same data stream is stored in the form of multiple small files in the file system, resulting in poor data reading efficiency, setting up an intermediate file system and integrating the small files of the same data stream in the intermediate file system not only makes the data stream writing cost higher, but also makes the data stream writing efficiency lower, which in turn leads to poor data stream writing performance. Summary of the invention

[0005] Based on this, it is necessary to provide a data stream writing control method, system, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems, which can effectively improve the data stream writing performance while ensuring the data reading and writing efficiency.

[0006] In a first aspect, the present application provides a data stream writing control method, comprising:

[0007] receiving a write performance characteristic of a data stream processing engine in a first cycle, wherein the write performance characteristic represents a write performance of the data stream processing engine in writing each data stream in the first cycle;

[0008] According to the write performance characteristic, a first operator routing table of a first cycle is adjusted to obtain a second operator routing table of a second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator;

[0009] The second operator routing table is sent to the data stream processing engine, and the second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle period to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0010] In a second aspect, the present application also provides a data stream writing control method, comprising:

[0011] Obtaining a first operator routing table of a first cycle, the first operator routing table comprising at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator;

[0012] Using the data writing operator indicated by each routing record, the data is written into the data stream to be written corresponding to the operator, and into the file corresponding to the data writing operator and the data stream to be written;

[0013] For each data stream to be written, obtaining a write performance characteristic of writing by at least one data writing operator in the first cycle;

[0014] The write performance characteristic is sent, and the sent write performance characteristic is used to indicate adjusting the routing record in the first operator routing table according to the write performance characteristic to obtain the second operator routing table of the second cycle period, and the write performance characteristic represents the write performance of writing each data stream in the first cycle period.

[0015] In a third aspect, the present application also provides a data stream write control system, including a control node, a data stream processing engine, a file system and a cache, wherein the data stream processing engine includes a data write operator and a data source operator;

[0016] The data source operator is used to receive a first operator routing table of a first cycle from a control node, the first operator routing table comprising at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator; read data segments of each data stream from the cache, and distribute the data segments to each data writing operator according to the routing records in the operator routing table; wherein the write performance characteristics are obtained and sent by the data writing operator;

[0017] The data writing operator is used to write the data stream to be written corresponding to the data writing operator indicated by the routing record into the file system, the file corresponding to the data writing operator and the data stream to be written; obtain the writing performance characteristics of writing by the data writing operator in the first cycle; and send the writing performance characteristics to the control node;

[0018] The control node is used to receive the write performance characteristics of the data write operator in the first cycle, and the write performance characteristics represent the write performance of the data stream processing engine in writing each data stream in the first cycle; according to the write performance characteristics, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, the first operator routing table contains routing records in the first cycle, and the second operator routing table contains routing records in the second cycle, each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data write operator; send the second operator routing table to the data source operator, the operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle, write the data stream to be written corresponding to the data write operator, and write it into the file system, the data write operator and the file corresponding to the data stream to be written.

[0019] In a fourth aspect, the present application further provides a data stream writing control device, comprising:

[0020] A receiving module, used for receiving a write performance characteristic of a data stream processing engine in a first cycle, wherein the write performance characteristic represents a write performance of the data stream processing engine in writing each data stream in the first cycle;

[0021] A first execution module, configured to adjust a first operator routing table of a first cycle according to the write performance characteristic, and obtain a second operator routing table of a second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator;

[0022] A sending module is used to send the second operator routing table to the data stream processing engine. The second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle period, write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0023] In one embodiment, the first execution module is specifically used to determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is greater than a first threshold, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, route expansion is performed to generate an expanded routing record indicating the data stream; based on the existing routing record and the expanded routing record, a second operator routing table of the second cycle is generated.

[0024] In one embodiment, the first execution module is also used to obtain the number of operators of the data write operator on the data stream processing engine, and obtain the concurrent routing upper limit value of each data write operator on the data stream processing engine; determine the concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value; the first execution module is specifically used to generate an expanded routing record indicating the data stream for each data stream written in the first cycle period, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance indicator value is greater than the first threshold value, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle period.

[0025] In one embodiment, the first execution module is specifically used to, for each data stream written in the first cycle, if the write performance indicator value is greater than the first threshold, determine the number of expansion routing records for the data stream according to the preset expansion strategy; determine the candidate data write operator for which no routing record has been established with the data stream according to the existing routing record indicating the data stream in the first operator routing table of the first cycle; for each of the candidate data write operators, count the number of routing records corresponding to the candidate data write operator according to the routing records in the first operator routing table; from the candidate data write operators, select the newly added data write operators whose number is consistent with the number of the expansion routing records in ascending order of priority of the number of routing records; respectively establish routing records indicating each of the newly added data write operators and the data stream, and obtain the expansion routing record indicating the data stream.

[0026] In one embodiment, the first execution module is specifically used to determine the timing order of this routing expansion in the continuous routing expansion operation for each data stream written in the first cycle if the write performance indicator value is greater than the first threshold and an expansion routing record indicating the data stream has been generated for the first cycle; determine the number of expansion routing records for the data stream according to the timing order, wherein the number of expansion routing records is positively correlated with the timing order; based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and according to the number of expansion routing records, generate an expansion routing record indicating the data stream.

[0027] In one embodiment, the first execution module is specifically used to determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is less than or equal to a second threshold value, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, route shrinkage is performed to determine the shrunken routing record indicating the data stream; based on the existing routing record and the shrunken routing record, a second operator routing table of the second cycle is generated.

[0028] In one embodiment, the first execution module is specifically used to determine the number of reduced routing records for each data stream written in the first cycle according to a preset reduction strategy if the write performance indicator value is less than or equal to a second threshold; determine a candidate data write operator for which a routing record has been established with the data stream based on an existing routing record indicating the data stream in a first operator routing table of the first cycle; for each candidate data write operator, count the number of routing records corresponding to the candidate data write operator based on the routing records in the first operator routing table; from the candidate data write operators, select a reduced data write operator whose number is consistent with the number of reduced routing records in descending priority order of the number of routing records, and obtain a reduced routing record indicating the data stream and the reduced data write operator.

[0029] In one embodiment, the first execution module is specifically used to determine the multi-cycle period to be counted, and the multi-cycle period includes the first cycle period and at least one historical cycle period before the first cycle period; obtain the write performance characteristics of each of the at least one historical cycle period; perform statistical analysis of the multi-cycle period based on the write performance characteristics of each of the multi-cycle periods, and obtain the write performance index value of each data stream written in the first cycle period.

[0030] In one embodiment, each data stream includes multiple data segments, and the write performance characteristics include a release timestamp and a consumption timestamp of each data segment, the release timestamp represents the time when the data segment is written into the cache, and the consumption timestamp represents the time when the data segment read from the cache is written into the file; the write performance indicator value of each data stream written in the first cycle includes a data segment delay statistic value of a multi-cycle cycle; the first execution module is specifically used to determine the delay value of each data segment according to the release timestamp and the consumption timestamp of each data segment written in each cycle within the multi-cycle cycle; for each data stream, determine the total delay value of the data stream according to the delay values ​​of each data segment in the data stream written in the multi-cycle cycle; for each data stream, count the total number of data segments of the data segments in the data stream written in the multi-cycle cycle; for each data stream, determine the data segment delay statistic value of the data stream in the multi-cycle cycle according to the total delay value of the data stream and the total number of data segments of the data stream.

[0031] In one embodiment, the write performance characteristics include the amount of data written to each data stream in each cycle; the write performance indicator value of each data stream written in the first cycle, including the data stream load statistics of multiple cycles; the first execution module is specifically used to count the total amount of data written to the data stream in the multiple cycles according to the amount of data written to the data stream in each cycle for each data stream written in the first cycle; for each data stream written in the first cycle, count the number of routing records indicating the routing records of the data stream from the first operator routing table; for each data stream written in the first cycle, determine the data stream load statistics of the data stream in the multiple cycles according to the total amount of data, the number of routing records and the number of cycles in the multiple cycles.

[0032] In one embodiment, the first execution module is also used to scan the data stream newly added by the data stream processing engine in the first cycle; determine the number of default routing records of the newly added data stream according to the default operator allocation strategy; count the number of routing records of each data write operator on the data stream processing engine according to the routing records in the first operator routing table; select the target data write operator whose number is consistent with the number of default routing records from the data write operators on the data stream processing engine according to the ascending priority order of the number of routing records; and add the target routing record indicating the target data write operator and the data stream to the second operator routing table of the second cycle.

[0033] In one embodiment, the first execution module is specifically used to traverse the data stream indicated by the routing record in the first operator routing table; in response to the traversed data stream being in a locked state, continue to traverse the next data stream; in response to the traversed data stream being in an unlocked state, adjust the routing record for the traversed data stream according to the write performance characteristics; until the traversal is completed, a second operator routing table of the second cycle is obtained.

[0034] In a fifth aspect, the present application further provides a data stream writing control device, comprising:

[0035] A first acquisition module, used to acquire a first operator routing table of a first cycle, the first operator routing table comprising at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator;

[0036] A second execution module is used to use the data writing operator indicated by each routing record to write the data into the data stream to be written corresponding to the operator, and write the data into a file corresponding to the data writing operator and the data stream to be written;

[0037] A second acquisition module is used to acquire, for each data stream to be written, a write performance characteristic of writing by at least one data writing operator in the first cycle;

[0038] A sending module is used to send the write performance characteristic, and the sent write performance characteristic is used to indicate adjusting the routing record in the first operator routing table according to the write performance characteristic to obtain the second operator routing table of the second cycle period, and the write performance characteristic characterizes the write performance of writing each data stream in the first cycle period.

[0039] In one embodiment, the first acquisition module is specifically used to use the data source operator to receive the first operator routing table of the first cycle period from the control node; the first acquisition module is also used to use the data source operator to read the data segments of each data stream from the cache, and distribute the data segments to each data write operator according to the routing records in the first operator routing table; wherein the write performance characteristics are acquired and sent by the data write operator.

[0040] In a sixth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any embodiment in the first aspect when executing the computer program.

[0041] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any embodiment in the above-mentioned first aspect.

[0042] In an eighth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method of any embodiment in the above-mentioned first aspect.

[0043] The above-mentioned data stream writing control method, system, device, computer equipment, computer-readable storage medium and computer program product first receive the writing performance characteristics of the data stream processing engine in the first cycle, and the writing performance characteristics characterize the writing performance of the data stream processing engine in writing each data stream in the first cycle; then, according to the writing performance characteristics, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, the first operator routing table contains routing records in the first cycle, and the second operator routing table contains routing records in the second cycle, each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator; then, send the second operator routing table to the data stream processing engine, the second operator routing table is used to indicate that the data stream processing engine, in the second cycle, uses the data writing operator indicated by each routing record in the second operator routing table to write the data stream to be written corresponding to the data writing operator, and writes the data writing operator and the file corresponding to the data stream to be written. The data stream writing control method provided by the present application dynamically adjusts the data writing operator used to write the data stream in each cycle according to the writing performance characteristics of the data stream. Since not all data writing operators are used to write the same data stream, but an adapted number of data writing operators are used to write the data stream, the same data stream corresponds to as few small files as possible, and there is no need to set up an additional intermediate file system and perform small file integration processing. While ensuring data reading and writing efficiency, the cost of data stream writing is reduced, the writing efficiency of the data stream is improved, and the data stream writing performance is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 An application environment diagram of a data stream writing control method in one embodiment;

[0046] Figure 2 A schematic diagram of a flow chart of a data stream writing control method in one embodiment;

[0047] Figure 3is a schematic diagram of a first operator routing table in one embodiment;

[0048] Figure 4 is a schematic diagram of a second operator routing table in one embodiment;

[0049] Figure 5 A flowchart of routing capacity reduction and routing capacity expansion in one embodiment;

[0050] Figure 6 It is a flowchart of a data stream writing control method in another embodiment;

[0051] Figure 7 It is a flowchart of a data stream writing control method in another embodiment;

[0052] Figure 8 is a schematic diagram of a data stream processing engine in one embodiment;

[0053] Fig. 9 is a schematic diagram of a data stream processing engine in another embodiment;

[0054] Fig.10 is a schematic diagram of a data stream processing engine in another embodiment;

[0055] Fig.11 is a schematic diagram of an intermediate file system in one embodiment;

[0056] Fig.12 A schematic diagram of a data stream writing control system in one embodiment;

[0057] Fig.13 It is a structural block diagram of a data stream writing control device in one embodiment;

[0058] Fig.14 It is a structural block diagram of a data stream writing control device in another embodiment;

[0059] Fig.15 is an internal structure diagram of a computer device in one embodiment;

[0060] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In order to clearly describe the technical solution of the present application and facilitate understanding of the technical solution of the present application, the key concepts involved in the present application are explained below.

[0063] A data stream refers to a set of data sequences with the same data type and originating from the same data source.

[0064] A data stream processing engine refers to an engine that has a data stream processing function and can process data streams to write the data streams into a file system.

[0065] A cycle period refers to a fixed time period set when a data stream processing engine performs a write operation on a data stream.

[0066] The first cycle refers to any cycle in multiple cycles when the data stream processing engine performs a write operation on the data stream.

[0067] The second cycle period refers to a cycle period after the first cycle period.

[0068] The write performance characteristic characterizes the write performance of the data stream processing engine in writing each data stream in the first cycle.

[0069] The operator routing table contains routing records, each of which indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator.

[0070] The data writing operator refers to the operator set in the data stream processing engine for writing to the data stream.

[0071] The file corresponding to the data writing operator and the data stream to be written refers to the file created when the data stream to be written is first written into the file system by the data writing operator.

[0072] The write performance index value refers to a specific value used to quantify the performance of the data stream processing engine when writing each data stream in the first cycle.

[0073] Route expansion refers to the operation of adding new route records based on the original route records of the data flow.

[0074] The upper limit of concurrent routes refers to the maximum value of routes that the data writing operator can write to concurrently.

[0075] The concurrent routing constraint value refers to the maximum value of the concurrent routing that the data stream processing engine can perform simultaneously.

[0076] The timing order refers to the cumulative number of times this route expansion occurs in the continuous route expansion operation.

[0077] The data stream writing control method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the control node in the computer device can be connected to the data stream processing engine for communication. The control node and the data stream processing engine can be set in the same computer device or in different computer devices.

[0078] Exemplarily, a computer device may receive write performance characteristics of the write performance of each data stream written by a data stream processing engine in a first cycle, and adjust a first operator routing table of the first cycle based on the write performance characteristics to obtain a second operator routing table of a second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data write operator.

[0079] Furthermore, the computer device may send the second operator routing table to the data stream processing engine to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0080] The computer device may be a terminal or a server. The terminal may be, but is not limited to, various personal computers, laptops, and tablet computers. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0081] In an exemplary embodiment, Figure 2 As shown, a data stream writing control method is provided, the method comprising the following steps:

[0082] Step 201: Receive the write performance characteristics of the data stream processing engine in the first cycle.

[0083] A data stream refers to a set of data sequences with the same data type and originating from the same data source. Exemplarily, the data stream can be a log file type data stream, a user interaction type data stream, a sensor network type data stream, etc. Specifically, all types of systems and applications have log files. For example, a server log file can record user visits to a website, and a database log file can record the operation history of the database. These log files all exist in the form of data streams.

[0084] An engine is a key component or program module that has a specific function and can process input and generate output to support the operation of a specific task. A data stream processing engine is an engine that has a data stream processing function and can process data streams to write data streams into a file system.

[0085] Exemplarily, the data stream processing engine may be Apache Flink. Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. The core is a distributed stream data flow engine written in Java and Scala. Apache Flink executes any stream data program in a data parallel and pipeline manner. The pipeline runtime system of Apache Flink can execute batch and stream processing programs. The runtime of Apache Flink itself also supports the execution of iterative algorithms.

[0086] The cycle period refers to a fixed time period set when the data stream processing engine performs a write operation on the data stream. The duration of each cycle period can be pre-set by the technician according to actual needs. The duration of each cycle period can be 1 minute, 10 minutes, 30 minutes, 1 hour, 5 hours, 10 hours, 24 hours, etc.

[0087] Exemplarily, assuming that the duration of each cycle is 1 minute, the first cycle begins when the data stream processing engine starts to execute the write operation on the data stream. After 1 minute, the first cycle ends and the second cycle begins.

[0088] The first cycle refers to any cycle in multiple cycles when the data stream processing engine performs a write operation on the data stream.

[0089] The write performance characteristics of the first cycle period characterize the write performance of the data stream processing engine in writing each data stream in the first cycle period. The write performance characteristics may specifically include the data stream identifier (dataFlowId), the data write operator identifier (Sink taskIndex), the cycle duration, the consumption time, the release time, the total data volume, and the total number of data items.

[0090] The data in the data stream is first stored in the cache and then retrieved from the cache by the data stream processing engine for writing. The consumption time refers to the time when the data in the data stream is written, and the release time refers to the time when the data in the data stream is cached. The total data volume refers to the total data volume of each data stream in the first cycle, and the total number of data items refers to the total number of data items in each data stream in the first cycle.

[0091] Furthermore, if the data in a data stream is written by a data writing operator in a data stream processing engine, the total data volume of the data stream can be represented by the total size. If the data in a data stream is written by multiple data writing operators in a data stream processing engine, the total data volume of the data stream can be represented by the total number of fields.

[0092] In some exemplary embodiments, after the first cycle ends, the data stream processing engine may actively report the write performance characteristics of the first cycle to the computer device.

[0093] Furthermore, after the data stream processing engine actively reports the write performance characteristics of the first cycle to the computer device, the computer device can receive the write performance characteristics of the data processing engine in the first cycle.

[0094] In other exemplary embodiments, the computer device may also first send a write performance characteristic acquisition request to the data stream processing engine. After receiving the write performance characteristic acquisition request sent by the computer device, the data stream processing engine will send the write performance characteristics of the first cycle to the computer device after the first cycle ends.

[0095] Furthermore, after the data stream processing engine sends the write performance characteristics of the first cycle to the computer device, the computer device can receive the write performance characteristics of the data stream processing engine in the first cycle.

[0096] Step 202: According to the write performance characteristic, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle.

[0097] The second cycle refers to a cycle after the first cycle. For example, if the first cycle is the first cycle, then the second cycle is the second cycle. If the first cycle is the fifth cycle, then the second cycle is the sixth cycle.

[0098] The operator routing table contains routing records, each of which indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator. That is, the operator routing table is used to indicate the mapping relationship between the data stream and the data writing operator used to write the data stream. The operator routing table can specifically be a Sink routing table in the data stream processing engine.

[0099] The first operator routing table can be Figure 3 As shown, the first operator routing table includes routing records within a first cycle, and the routing records within the first cycle indicate a data stream to be written and a corresponding data writing operator of the data stream processing engine within the first cycle.

[0100] The second operator routing table can be Figure 4 As shown, the second operator routing table includes routing records within the second cycle, and the routing records within the second cycle indicate a data stream to be written and a corresponding data writing operator of the data stream processing engine within the second cycle.

[0101] The data writing operator refers to an operator set in the data stream processing engine for writing to the data stream. The data writing operator can be a Sink, which is a computing unit in the data stream processing engine for writing the data stream. Multiple Sinks can be set in the data stream processing engine, and each Sink has an identification number.

[0102] In some exemplary embodiments, after receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device can adjust the first operator routing table of the first cycle according to the write performance characteristics to obtain the second operator routing table of the second cycle.

[0103] Specifically, since the write performance characteristics of the first cycle can characterize the write performance of the data stream processing engine in writing each data stream in the first cycle, and the data stream processing engine uses the data write operator to write the data stream in the first cycle, therefore, when the write performance characteristics of the first cycle indicate that the write performance of a certain data stream is poor, the data write operator corresponding to the data stream can be adjusted to improve the write performance of the data stream.

[0104] For example, after receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device determines that the write performance of data stream a in the first cycle is poor based on the write performance characteristics, and then the data write operator corresponding to the data stream a can be adjusted. Specifically, assuming that the amount of data in the data stream a is large, and the data stream a is written by the data write operator x in the first cycle, the data write operator corresponding to the data stream a is adjusted, which can be to make the data write operator x and the data write operator y write to the data stream a together in the second cycle, that is, in the first operator routing table, add the routing records of the data write operator y and the data stream a to obtain the second operator routing table. The above process is only an adjustment process for a certain data stream. In actual applications, the above process can be performed on each data stream to obtain the second operator routing table.

[0105] Furthermore, in addition to the case of adding routing records, there is also the case of deleting routing records. For example, the amount of data written by data stream a in the N cycle periods before the first cycle period is large, so that in the N+1 operator routing table of the N+1 cycle period, there are multiple routing records corresponding to the data stream a, such as data stream a-data write operator x, data stream a-data write operator y, data stream a-data write operator z.

[0106] After receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device determines that the amount of data in the data stream a in the first cycle is greatly reduced based on the write performance characteristics, that is, there is no need to use more data write operators to write to the data stream a, and the data write operator corresponding to the data stream a can be adjusted.

[0107] Specifically, the routing record of data stream a can be deleted in the first operator routing table, for example, the routing record of "data stream a-data write operator x, data stream a-data write operator y" can be deleted to obtain the second operator routing table. The above process is only an adjustment process for a certain data stream. In actual applications, the above process can be performed on each data stream to obtain the second operator routing table.

[0108] Step 203: Send the second operator routing table to the data stream processing engine.

[0109] The second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0110] The file corresponding to the data writing operator and the data stream to be written refers to the file created when the data stream to be written is first written into the file system by the data writing operator.

[0111] Exemplarily, the file system may be a distributed file system (HDFS, Hadoop Distributed File System).

[0112] In some exemplary embodiments, after the computer device adjusts the first operator routing table of the first cycle according to the write performance characteristics and obtains the second operator routing table of the second cycle, it can send the second operator routing table to the data stream processing engine.

[0113] Furthermore, after the data stream processing engine receives the second cycle period table sent by the computer device, it can use the data writing operator indicated by each routing record in the second operator routing table to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0114] The above-mentioned data stream writing control method first receives the writing performance characteristics of the data stream processing engine in the first cycle, and the writing performance characteristics characterize the writing performance of the data stream processing engine in writing each data stream in the first cycle; then, according to the writing performance characteristics, adjusts the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, the first operator routing table contains routing records in the first cycle, and the second operator routing table contains routing records in the second cycle, each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator; then, sends the second operator routing table to the data stream processing engine, and the second operator routing table is used to instruct the data stream processing engine to use the data writing operator indicated by each routing record in the second operator routing table in the second cycle, write the data stream to be written corresponding to the data writing operator, and write the data writing operator and the file corresponding to the data stream to be written. The data stream writing control method provided by the present application dynamically adjusts the data writing operator used to write the data stream in each cycle according to the writing performance characteristics of the data stream. Since not all data writing operators are used to write the same data stream, but an adapted number of data writing operators are used to write the data stream, the same data stream corresponds to as few small files as possible, and there is no need to set up an additional intermediate file system and perform small file integration processing. While ensuring data reading and writing efficiency, the cost of data stream writing is reduced, the writing efficiency of the data stream is improved, and the data stream writing performance is effectively improved.

[0115] In an exemplary embodiment, according to the write performance characteristics, the first operator routing table of the first cycle is adjusted to obtain the second operator routing table of the second cycle, including: according to the write performance characteristics, determining the write performance indicator value of each data stream written in the first cycle; for each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, routing expansion is performed to generate an expanded routing record indicating the data stream; based on the existing routing record and the expanded routing record, a second operator routing table of the second cycle is generated.

[0116] The write performance index value refers to a specific value used to quantify the performance of the data stream processing engine when writing each data stream in the first cycle. The write performance index value can be a write speed index value, a throughput index value, a latency index value, or a resource utilization index value.

[0117] The first threshold may be a value pre-set by a technician according to actual needs and used for comparison with the write performance indicator value of the data stream to determine whether to perform a routing expansion operation on the routing of the data stream.

[0118] Route expansion refers to the operation of adding new route records based on the original route records of the data flow.

[0119] In some exemplary embodiments, after receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device can determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics. Specifically, the computer device can determine the write performance index value of each data stream written in the first cycle using a preset algorithm and the write performance characteristics.

[0120] Furthermore, after the computer device determines the write performance indicator value of each data stream written in the first cycle, for each data stream written in the first cycle, if the computer device determines that the write performance indicator value of the data stream is greater than the first threshold, the computer device performs route expansion based on the existing route record indicating the data stream in the first operator routing table of the first cycle, and generates an expanded route record indicating the data stream.

[0121] Specifically, for data stream a written in the first cycle, if the computer device determines that the write performance index value of the data stream a is greater than the first threshold, it first determines whether there is a record of the route of the data stream a in the first operator routing table. Assuming that the routing record indicating the data stream in the first operator routing table is "data stream a-data write operator x", then the routing of data stream a can be expanded based on other data write operators except data write operator x in the data stream processing engine. Assuming that the other selected data write operator is data write operator y, the generated expansion routing record can be "data stream a-data write operator y".

[0122] If the computer device determines that the write performance indicator value of the data flow is less than or equal to the first threshold, it does not perform routing expansion based on the existing routing record indicating the data flow in the first operator routing table of the first cycle.

[0123] Furthermore, for each data stream written in the first cycle, the computer device can perform the above operations, and after determining the expansion routing record of each data stream, the second operator routing table of the second cycle can be generated according to the existing routing record and the expansion routing record. Specifically, the expansion routing record can be added to the first operator routing table to generate the second operator routing table of the second cycle.

[0124] The above method determines the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is greater than the first threshold, the route is expanded based on the existing route record indicating the data stream in the first operator routing table of the first cycle, and an expanded routing record indicating the data stream is generated; according to the existing routing record and the expanded routing record, a method for generating a second operator routing table of the second cycle, when the write performance index value of the data stream is greater than the first threshold, the route is expanded for the data stream to improve the writing efficiency of the data stream, thereby effectively improving the writing performance of the data stream.

[0125] In an exemplary embodiment, the method further includes: obtaining the number of operators of the data write operator on the data stream processing engine, and obtaining the concurrent routing upper limit value of each data write operator on the data stream processing engine; determining the concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value; for each data stream written in the first cycle, if the write performance index value is greater than a first threshold value, then based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, perform routing expansion, and generate an expanded routing record indicating the data stream, including: for each data stream written in the first cycle, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance index value is greater than the first threshold value, then based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, generate an expanded routing record indicating the data stream.

[0126] The concurrent route upper limit refers to the maximum value of routes that can be concurrently processed by the data writing operator. For example, if a data writing operator can concurrently process 10 routes, then the concurrent route upper limit of the data writing operator can be determined to be 10.

[0127] The concurrent routing constraint value refers to the maximum value of the routing that the data stream processing engine can perform concurrently.

[0128] For example, if 10 data writing operators are set in the data stream processing engine, and the concurrent routing upper limit value of each data writing operator is 10, then the concurrent routing constraint value of the data stream processing engine may be 10×10.

[0129] In some exemplary embodiments, the computer device may obtain the number of operators of the data writing operator on the data stream processing engine, and obtain the concurrent routing upper limit value of each data writing operator on the data stream processing engine.

[0130] Furthermore, the computer device can determine the concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value. For each data stream written in the first cycle, the computer device, when determining that the write performance index value is greater than the first threshold, also needs to determine whether the number of routing records in the first operator routing table is less than the concurrent routing constraint value.

[0131] If the number of routing records in the first operator routing table is less than the concurrent routing constraint value, it can be determined that the data stream processing engine can still add new routing records, then based on the existing routing records indicating the data stream in the first operator routing table of the first cycle period, an expanded routing record indicating the data stream is generated.

[0132] If the number of routing records in the first operator routing table is greater than or equal to the concurrent routing constraint value, it can be determined that the data stream processing engine cannot add new routing records, and the first operator routing table of the first cycle is determined as the second operator routing table of the second cycle.

[0133] For each data stream written in the first cycle, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance index value is greater than the first threshold, then based on the existing routing records indicating the data stream in the first operator routing table of the first cycle, a method for generating an expanded routing record indicating the data stream is used. Before generating the expanded routing record, it is first determined whether the current number of routing records is less than the concurrent routing constraint value of the data stream processing engine, so as to avoid the situation where the number of routing records is greater than the concurrent routing constraint value, resulting in an abnormal problem caused by the full load of the data stream processing engine, thereby effectively improving the writing performance of the data stream.

[0134] In an exemplary embodiment, for each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, route expansion is performed based on the existing route record indicating the data stream in the first operator routing table of the first cycle, and an expanded routing record indicating the data stream is generated, including: for each data stream written in the first cycle, if the write performance indicator value is greater than the first threshold, the number of expanded routing records for the data stream is determined according to a preset expansion strategy; based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, a candidate data write operator for which no routing record has been established with the data stream is determined; for each of the candidate data write operators, based on the routing records in the first operator routing table, the number of routing records corresponding to the candidate data write operator is counted; from the candidate data write operators, in the order of priority in ascending order of the number of routing records, a newly added data write operator with a number consistent with the number of the expanded routing records is selected; and routing records indicating each of the newly added data write operators and the data stream are respectively established to obtain an expanded routing record indicating the data stream.

[0135] The preset expansion strategy refers to a strategy that is pre-set by the technicians based on actual needs and is used to determine the number of expanded route records when performing route expansion.

[0136] Exemplarily, the preset expansion strategy may indicate that the number of expansion routing records is determined based on the degree of performance index exceeding. Specifically, the number of expansion routing records may be determined according to the extent to which the write performance index value exceeds the first threshold. For example, if the write speed index value is 20% higher than the first threshold, one expansion routing record is added; if it is 50% higher, three expansion routing records are added.

[0137] The preset expansion strategy can also indicate expansion at a fixed ratio. Specifically, the number of expanded routing records can be determined according to a fixed ratio. For example, no matter how much the write performance index value exceeds the first threshold, the capacity is expanded according to a certain ratio of the number of existing routing records. Assuming that the number of existing routing records is 5, the preset expansion strategy stipulates that the capacity is expanded at a ratio of 20%, then 1 expansion routing record needs to be added.

[0138] In some exemplary embodiments, for each data stream written in the first cycle, if the computer device determines that the write performance indicator value is greater than a first threshold, the computer device determines the number of expansion routing records for the data stream according to a preset expansion strategy.

[0139] Furthermore, after determining the number of expansion routes, the computer device can determine the specific expansion object, that is, the data writing operator for the data flow expansion. Specifically, the computer device can determine the candidate data writing operator for which no routing record is established with the data flow based on the existing routing record indicating the data flow in the first operator routing table of the first cycle.

[0140] For example, the data stream processing engine is provided with five data writing operators, namely, data writing operator 0 to data writing operator 4. The first operator routing table indicates that the existing routing record of data stream a is "data stream a-data writing operator 0". Then, the candidate data writing operators determined for which no routing record has been established with the data stream are data writing operator 1, data writing operator 2, data writing operator 3 and data writing operator 4.

[0141] Furthermore, the computer device may count the number of routing records corresponding to each candidate data write operator according to the routing records in the first operator routing table.

[0142] Specifically, the computer device may index each candidate data write operator in the first operator routing table to obtain the number of routing records corresponding to each candidate data write operator. For example, if there are 3 routing records for data write operator 1 in the first operator routing table, it can be determined that the number of routing records corresponding to data write operator 1 is 3.

[0143] After the computer device determines the number of routing records corresponding to each candidate data write operator according to the routing records in the first operator routing table, the candidate data write operators can be sorted in ascending order of the number of routing records. For example, if the number of routing records corresponding to data write operator 1 is 3, the number of routing records corresponding to data write operator 2 is 4, the number of routing records corresponding to data write operator 3 is 5, and the number of routing records corresponding to data write operator 4 is 6, then the number of routing records corresponding to data write operator 4 is 6, and the numbers of routing records corresponding to data write operator 1, data write operator 2, data write operator 3, and data write operator 4 are sorted in ascending order of the number of routing records.

[0144] Furthermore, after the computer device sorts the candidate data write operators in ascending order of the number of routing records, it can select new data write operators whose number is consistent with the number of expanded routing records from the candidate data write operators in ascending order of priority of the number of routing records, and establish routing records indicating each of the new data write operators and the data flow, respectively, to obtain the expanded routing records indicating the data flow.

[0145] Specifically, assuming that the number of expansion routing records is 1, the data write operator that is ranked first after sorting is selected as the newly added data write operator, and the routing record of the newly added data write operator and the data stream is established to obtain the expansion routing record indicating the data stream. Assuming that the number of expansion routing records is 2, the data write operators that are ranked first and second after sorting are selected as the newly added data write operators, and the routing records of the newly added data write operator and the data stream are established respectively to obtain the expansion routing record indicating the data stream.

[0146] For each data stream written in the first cycle, if the write performance index value is greater than the first threshold, the number of expansion routing records for the data stream is determined according to the preset expansion strategy; according to the existing routing records indicating the data stream in the first operator routing table of the first cycle, the candidate data write operator for which no routing record is established with the data stream is determined; for each of the candidate data write operators, according to the routing records in the first operator routing table, the number of routing records corresponding to the candidate data write operator is counted; from the candidate data write operators, according to the priority order of the number of routing records in ascending order, the newly added data write operator with the same number as the number of the expansion routing records is selected Operator; respectively establish routing records indicating each newly added data write operator and the data stream, and obtain a method for obtaining an expanded routing record indicating the data stream. When performing routing expansion, the candidate data write operators for which no routing records have been established with the data stream can be first determined, and then the newly added data write operator can be determined from the candidate data write operators according to the number of routing records and the number of expanded routing records of the candidate data write operators. Since the number of routing records can indirectly reflect the idleness of the data write operator, the relatively idle data write operator can be selected as the newly added data write operator according to the number of routing records, so as to achieve load balancing of the data write operators and improve the stability of the data stream processing engine.

[0147] In an exemplary embodiment, for each data stream written in the first cycle, if the write performance indicator value is greater than the first threshold, then based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, an expanded routing record indicating the data stream is generated, including: for each data stream written in the first cycle, if the write performance indicator value is greater than the first threshold, and an expanded routing record indicating the data stream has been generated for the first cycle, then determining the timing order of this routing expansion in the continuous routing expansion operation; determining the number of expanded routing records for the data stream according to the timing order; based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and according to the number of expanded routing records, generating an expanded routing record indicating the data stream.

[0148] The timing order refers to the cumulative number of times this expansion routing occurs in continuous routing expansion operations. For example, if two expansion routings have been performed in succession in the two cycles before this expansion routing, the timing order of this routing expansion is 3. If one expansion routing has been performed in a cycle before this expansion routing, the timing order of this routing expansion is 2.

[0149] The number of expansion route records is positively correlated with the timing order. That is, as the timing order increases, the number of expansion route records will also increase. For example, when the timing order is 1, the number of expansion routes can be 1, when the timing order is 2, the number of expansion routes can be 2, when the timing order is 3, the number of expansion routes can be 4, and when the timing order is 4, the number of expansion routes can be 8.

[0150] It should be noted that after the continuous route expansion operation is interrupted, when the route expansion is performed again, the number of expanded route records will return to the initial number of expanded routes. For example, if the initial number of expanded routes is 1, after the route expansion operation is performed in three consecutive cycles, the number of expanded route records in the third cycle is 4. If the route expansion operation is not performed in the fourth cycle, then if the route expansion operation is performed in the fifth cycle, the number of expanded route records will be 1.

[0151] In some exemplary embodiments, for each data stream written in the first cycle, if the computer device determines that the write performance index value of the data stream is greater than the first threshold, and an expansion routing record indicating the data stream has been generated for the first cycle, then the timing order of this routing expansion in the continuous routing expansion operation can be determined.

[0152] Specifically, for each data stream written in the first cycle, if the computer device determines that the write performance index value of the data stream is greater than the first threshold, it can first determine whether the first cycle has generated an expansion routing record indicating the data stream. If generated, it can be determined as a continuous routing expansion operation, and the timing order of this routing expansion in the continuous routing expansion operation is determined. If not generated, it can be determined that the timing order of this routing expansion is 1.

[0153] Furthermore, after determining the timing sequence, the computer device can determine the number of expanded routing records for the data flow according to the timing sequence, and based on the existing routing records indicating the data flow in the first operator routing table of the first cycle, generate expanded routing records indicating the data flow according to the number of expanded routing records.

[0154] For each data stream written in the first cycle, if the write performance index value is greater than the first threshold value, and an expansion routing record indicating the data stream has been generated for the first cycle, the timing order of this routing expansion in the continuous routing expansion operation is determined; according to the timing order, the number of expansion routing records for the data stream is determined; based on the existing routing records indicating the data stream in the first operator routing table of the first cycle, and according to the number of expansion routing records, the method of generating the expansion routing record indicating the data stream, according to the timing order of this routing expansion in the continuous routing expansion operation, the number of expansion routing records for the data stream is determined, which can speed up the expansion rate during the continuous expansion operation and effectively improve the writing performance of the data stream.

[0155] In an exemplary embodiment, according to the write performance characteristics, the first operator routing table of the first cycle is adjusted to obtain the second operator routing table of the second cycle, including: according to the write performance characteristics, determining the write performance indicator value of each data stream written in the first cycle; for each data stream written in the first cycle, if the write performance indicator value is less than or equal to a second threshold, performing routing shrinkage based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and determining the shrunken routing record indicating the data stream; generating the second operator routing table of the second cycle according to the existing routing record and the shrunken routing record.

[0156] The second threshold may be a value preset by a technician according to actual needs and used for comparison with the write performance indicator value of the data stream to determine whether to perform a routing capacity reduction operation on the routing of the data stream.

[0157] Route shrinking refers to the operation of deleting the original route records of the data flow.

[0158] In some exemplary embodiments, after receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device can determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics. Specifically, the computer device can determine the write performance index value of each data stream written in the first cycle using a preset algorithm and the write performance characteristics.

[0159] Furthermore, after the computer device determines the write performance indicator value of each data stream written in the first cycle, for each data stream written in the first cycle, if the computer device determines that the write performance indicator value of the data stream is less than or equal to the second threshold, the computer device performs routing shrinkage based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and determines the shrunken routing record indicating the data stream.

[0160] Specifically, for data stream a written in the first cycle, if the computer device determines that the write performance index value of the data stream a is less than or equal to the second threshold, it first determines whether there is a record of the route of the data stream a in the first operator routing table. Assuming that the routing record indicating the data stream in the first operator routing table is "data stream a-data write operator x, data stream a-data write operator y", the route of data stream a can be reduced based on the original data write operator of the data stream. Assuming that the selected data write operator is data write operator y, the reduced routing record indicating the data stream is determined to be "data stream a-data write operator y".

[0161] If the computer device determines that the write performance indicator value of the data flow is greater than the second threshold, it does not perform routing shrinkage based on the existing routing record indicating the data flow in the first operator routing table of the first cycle.

[0162] Furthermore, for each data stream written in the first cycle, the computer device can perform the above operations. After determining the reduced capacity routing record of each data stream, the second operator routing table of the second cycle can be generated according to the existing routing record and the reduced capacity routing record. Specifically, the reduced capacity routing record can be deleted in the first operator routing table to generate the second operator routing table of the second cycle. For example, the routing record indicating the data stream in the first operator routing table is "data stream a-data write operator x, data stream a-data write operator y", and the reduced capacity routing record of the data stream is "data stream a-data write operator y", then the routing record of the data stream in the second operator routing table is "data stream a-data write operator x".

[0163] The above method determines the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is less than or equal to the second threshold, the route is reduced based on the existing route record indicating the data stream in the first operator routing table of the first cycle, and the reduced route record indicating the data stream is determined; based on the existing routing record and the reduced routing record, a method for generating a second operator routing table for the second cycle is used, and when the write performance index value of the data stream is less than or equal to the second threshold, the data stream is routed reduced to reduce the small files corresponding to the data stream, thereby improving the efficiency of the subsequent data acquisition process.

[0164] In an exemplary embodiment, for each data stream written in the first cycle, if the write performance indicator value is less than or equal to the second threshold, routing shrinkage is performed based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and the shrunk routing record indicating the data stream is determined, including: for each data stream written in the first cycle, if the write performance indicator value is less than or equal to the second threshold, the number of shrunk routing records for the data stream is determined according to a preset shrunk strategy; based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, a candidate data write operator with an established routing record for the data stream is determined; for each of the candidate data write operators, based on the routing records in the first operator routing table, the number of routing records corresponding to the candidate data write operator is counted; from the candidate data write operators, in descending priority order of the number of routing records, a shrunk data write operator with a number consistent with the number of shrunk routing records is selected to obtain a shrunk routing record indicating the data stream and the shrunk data write operator.

[0165] The preset shrinking strategy refers to a strategy that is preset by a technician according to actual needs and is used to determine the number of shrinking route records when executing route shrinking. Exemplarily, the preset shrinking strategy may indicate that the number of shrinking route records for each shrinking is 1.

[0166] In some exemplary embodiments, for each data stream written in the first cycle, if the computer device determines that the write performance indicator value is less than or equal to the second threshold, it determines the number of expansion routing records for the data stream according to a preset reduction strategy.

[0167] Furthermore, after determining the number of shrinking routes, the computer device can determine the specific shrinking object, that is, the data writing operator for shrinking the data flow. Specifically, the computer device can determine the candidate data writing operator with which the routing record has been established with the data flow based on the existing routing record indicating the data flow in the first operator routing table of the first cycle.

[0168] For example, the data stream processing engine is provided with five data writing operators, namely, data writing operator 0 to data writing operator 3. The first operator routing table indicates that the existing routing record of data stream a is "data stream a-data writing operator 0, data stream a-data writing operator 1, data stream a-data writing operator 2". Then, the candidate data writing operators determined to establish routing records with the data stream are data writing operator 0, data writing operator 1 and data writing operator 2.

[0169] Furthermore, the computer device may count the number of routing records corresponding to each candidate data write operator according to the routing records in the first operator routing table.

[0170] Specifically, the computer device may index each candidate data write operator in the first operator routing table to obtain the number of routing records corresponding to each candidate data write operator. For example, if there are 3 routing records for data write operator 0 in the first operator routing table, it can be determined that the number of routing records corresponding to data write operator 0 is 3.

[0171] After the computer device determines the number of routing records corresponding to each candidate data write operator according to the routing records in the first operator routing table, the candidate data write operators can be sorted in descending order of the number of routing records. For example, if the number of routing records corresponding to data write operator 0 is 5, the number of routing records corresponding to data write operator 1 is 4, and the number of routing records corresponding to data write operator 2 is 3, then the order of the number of routing records in descending order is "data write operator 0, data write operator 1, data write operator 2".

[0172] Furthermore, after the computer device sorts the candidate data write operators in descending order of the number of routing records, it can select reduction data write operators whose number is consistent with the number of reduction routing records from the candidate data write operators in descending order of priority of the number of routing records, and obtain the reduction routing records indicating the data flow and the reduction data write operator.

[0173] Specifically, assuming that the number of the shrinking routing records is 1, the data writing operator that is ranked first after sorting is selected as the shrinking data writing operator, and the shrinking routing record indicating the data flow and the shrinking data writing operator is obtained. Assuming that the number of the expanding routing records is 2, the data writing operators that are ranked first and second after sorting are selected as the shrinking data writing operators, and the shrinking routing records indicating the data flow and the shrinking data writing operator are obtained respectively.

[0174] For each data stream written in the first cycle, if the write performance index value is less than or equal to the second threshold, the number of shrinking routing records for the data stream is determined according to the preset shrinking strategy; based on the existing routing records indicating the data stream in the first operator routing table of the first cycle, the candidate data write operator with which the routing record has been established is determined; for each candidate data write operator, the number of routing records corresponding to the candidate data write operator is counted according to the routing records in the first operator routing table; from the candidate data write operators, the number of routing records corresponding to the shrinking routing record is selected in descending order of priority of the number of routing records. A method for obtaining a reduced data write operator indicating the data stream and the reduced data write operator with the same number of reduced data write operators. When performing route reduction, the candidate data write operator for establishing a route record with the data stream can be determined first, and then the reduced data write operator can be determined from the candidate data write operators according to the number of route records and the number of reduced route records of the candidate data write operators. Since the number of route records can indirectly reflect the idleness of the data write operator, the relatively busy data write operator can be selected as the reduced data write operator through the number of route records, so as to achieve load balancing of the data write operators and improve the stability of the data stream processing engine.

[0175] In an exemplary embodiment, based on the write performance characteristics, the write performance index value of each data stream written in the first cycle period is determined, including: determining the multi-cycle period to be counted; obtaining the write performance characteristics of each of the at least one historical cycle period; performing statistical analysis of the multi-cycle period based on the write performance characteristics of each of the multi-cycle periods to obtain the write performance index value of each of the data streams written in the first cycle period.

[0176] The multi-cycle period includes the first cycle period and at least one historical cycle period before the first cycle period.

[0177] In some exemplary embodiments, the computer device may first determine the multiple cycle periods to be counted, and then obtain the write performance characteristics of each of the at least one historical cycle period.

[0178] Furthermore, after obtaining the write performance characteristics of each of the first cycle period and at least one historical cycle period, the computer device can perform statistical analysis of multiple cycle periods based on the write performance characteristics of each of the multiple cycle periods to obtain the write performance index value of each data stream written in the first cycle period.

[0179] The above method of determining the multi-cycle periods to be counted; obtaining the write performance characteristics of each of the at least one historical cycle periods; performing statistical analysis of the multi-cycle periods based on the write performance characteristics of each of the multi-cycle periods, and obtaining the write performance index value of each of the data streams written in the first cycle period, and determining the write performance index value of each of the data streams written in the first cycle period through the write performance characteristics of each of the multi-cycle periods, can effectively improve the reliability and accuracy of the write performance index value.

[0180] In an exemplary embodiment, each data stream includes multiple data segments, and the write performance characteristics include a publishing timestamp and a consumption timestamp of each data segment, wherein the publishing timestamp indicates the time when the data segment is written into the cache, and the consumption timestamp indicates the time when the data segment read from the cache is written into the file; the write performance indicator values ​​of each data stream written in the first cycle include data segment delay statistics of multiple cycles; according to the write performance characteristics of each of the multiple cycles, a statistical analysis of the multiple cycles is performed to obtain the write performance characteristics of each of the data streams written in the first cycle. The performance indicator value includes: determining the delay value of each data segment according to the publishing timestamp and consumption timestamp of each data segment written in each cycle within the multi-cycle period; for each data stream, determining the total delay value of the data stream according to the delay values ​​of each data segment in the data stream written in the multi-cycle period; for each data stream, counting the total number of data segments of the data segments in the data stream written in the multi-cycle period; for each data stream, determining the data segment delay statistics of the data stream in the multi-cycle period according to the total delay value of the data stream and the total number of data segments of the data stream.

[0181] For example, a data stream can be understood as an orderly arrangement of a series of data packets in the time dimension. When data is transmitted in the form of a data stream, the data is divided into multiple data packets for transmission. Each data packet is a part of the data stream, which is transmitted in sequence and finally forms a complete data stream. The data segment is the data obtained by parsing the data packet. That is, the data stream consists of a series of data packets, and the data packet consists of a series of data segments.

[0182] The cache can be a storage device for temporarily storing data segments of a data stream. The cache can be Pulsar or TubeMQ. It should be noted that when the cache is TubeMQ, there is no publishing timestamp, so the above process is not involved. Only when the cache is Pulsar, there is a publishing timestamp, which involves the above process.

[0183] The delay value of each data segment refers to the length of time that the data segment is stored in the cache and waits to be written. The total delay value of a data stream refers to the total length of time that the data segments contained in the data stream are stored in the cache and wait to be written in a multi-cycle period. The data segment delay statistics of a multi-cycle period may be a maximum value, a median value, or an average value. In the embodiment of the present application, the data segment delay statistics of the multi-cycle period refers to the average length of time that the data segments contained in the data stream are stored in the cache and wait to be written.

[0184] In some exemplary embodiments, the computer device may first obtain the publishing timestamp and consumption timestamp of each data segment written in each cycle within the multi-cycle period, and then determine the delay value of each data segment according to the publishing timestamp and consumption timestamp of each data segment written in each cycle within the multi-cycle period. Specifically, the delay value is the absolute value of the difference between the publishing timestamp and the consumption timestamp, that is, the delay value = |consumption timestamp-publishing timestamp|.

[0185] Furthermore, for each data stream, the computer device determines the total delay value of the data stream according to the delay values ​​of each data segment in the data stream written in the multi-cycle period. Specifically, the total delay value is the sum of the delay values ​​of each data segment, that is, the total delay value = the delay value of data segment a + the delay value of data segment b + the delay value of data segment c + ...

[0186] For each data stream, the computer device counts the total number of data segments of the data segments in the data stream written in the multi-cycle period. For each data stream, the computer device determines the data segment delay statistics value of the data stream in the multi-cycle period according to the total delay value of the data stream and the total number of data segments of the data stream. Specifically, the data segment delay statistics value = total delay value / total number of data segments.

[0187] Further, when the write performance indicator value is a data segment delay statistic value of multiple cycle periods, the first threshold may be 10000 ms, and the second threshold may be 3000 ms.

[0188] The above method determines the delay value of each data segment according to the publishing timestamp and consumption timestamp of each data segment written in each cycle within the multi-cycle cycle; for each data stream, determines the total delay value of the data stream according to the delay values ​​of each data segment in the data stream written in the multi-cycle cycle; for each data stream, counts the total number of data segments of the data segments in the data stream written in the multi-cycle cycle; for each data stream, determines the data segment delay statistics of the data stream in the multi-cycle cycle according to the total delay value of the data stream and the total number of data segments of the data stream. The method can determine the data segment delay statistics of the data stream in the multi-cycle cycle according to the delay value of the data segment in each data stream and the total number of data segments, and use the value as the write performance indicator value, so that the routing expansion and routing reduction that can be performed based on the write performance indicator value are more accurate and reliable.

[0189] In an exemplary embodiment, the write performance characteristics include the amount of data written to each data stream in each cycle; the write performance index values ​​of each data stream written in the first cycle include data stream load statistics for multiple cycles; based on the write performance characteristics of each of the multiple cycles, statistical analysis of multiple cycles is performed to obtain write performance index values ​​for each of the data streams written in the first cycle, including: for each data stream written in the first cycle, based on the amount of data written to the data stream in each cycle, counting the total amount of data written to the data stream in the multiple cycles; for each data stream written in the first cycle, counting the number of routing records indicating the routing records of the data stream from the first operator routing table; for each data stream written in the first cycle, determining the data stream load statistics for the data stream in the multiple cycles based on the total amount of data, the number of routing records and the number of cycles in the multiple cycles.

[0190] In some exemplary embodiments, for each data stream written in the first cycle, the computer device may obtain the amount of data written to the data stream in each cycle, and count the total amount of data written to the data stream in the multiple cycles according to the amount of data written to the data stream in each cycle. The total amount of data may be the total size, i.e., the total number of bytes, or the total number of fields, i.e., the product of the total number of data items and the number of fields of each data item.

[0191] Further, for each data stream written in the first cycle, the computer device counts the number of routing records indicating the routing records of the data stream from the first operator routing table. For each data stream written in the first cycle, the computer device determines the data stream load statistics of the data stream in the multi-cycle period according to the total amount of data, the number of routing records and the number of cycles in the multi-cycle period. Specifically, the data stream load statistics of the data stream in the multi-cycle period = the total amount of data / the number of routing records / the number of cycles in the multi-cycle period. Optionally, the data stream load statistics of the multi-cycle period can also be a median, a maximum value, etc.

[0192] Further, when the write performance indicator value is a statistical value of the data flow load of the data flow in the multi-cycle period, the first threshold may be 500MB / 5000000, and the second threshold may be 50MB / 5000000.

[0193] In an optional embodiment, the following will be combined Figure 5 , the complete process of routing capacity reduction and routing capacity expansion is described. Specifically, after receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device can first determine the data stream load statistics and data segment delay statistics for each data stream according to the write performance characteristics of the data stream.

[0194] Furthermore, after determining the data flow load statistics and data segment delay statistics according to the write performance characteristics of the data flow, it is possible to first determine whether the data segment delay statistics are greater than the first delay threshold. If greater than, the number of expansion routing records for the data flow is determined according to the preset expansion strategy, and the newly added data write operator is determined from the data write operator in the data flow engine according to the routing record in the first operator routing table, and the expansion routing record is generated to obtain the second operator routing table.

[0195] If it is less than, determine whether the data flow load statistics value is greater than the first load threshold. If it is greater than, determine the number of expansion routing records for the data flow according to the preset expansion strategy, and determine the newly added data writing operator from the data writing operator in the data flow engine according to the routing records in the first operator routing table, and generate expansion routing records to obtain the second operator routing table.

[0196] If it is less than, determine whether the data segment delay statistic is less than or equal to the second delay threshold. If it is less than, determine the number of reduced capacity routing records for the data flow according to the preset reduction strategy, and generate reduced capacity routing records according to the routing records in the first operator routing table to obtain the second operator routing table.

[0197] If it is greater than, determine whether the data flow load statistics value is less than or equal to the second load threshold. If it is less than, determine the number of reduced capacity routing records for the data flow according to the preset reduction strategy, and generate reduced capacity routing records according to the routing records in the first operator routing table to obtain the second operator routing table. If it is greater than, use the first operator routing table as the second operator routing table.

[0198] The above method comprises the following steps: for each data stream written in the first cycle, based on the amount of data written to the data stream in each cycle, counting the total amount of data written to the data stream in the multi-cycle cycle; for each data stream written in the first cycle, counting the number of routing records indicating the routing records of the data stream from the first operator routing table; for each data stream written in the first cycle, determining the data flow load statistics value of the data stream in the multi-cycle cycle based on the total amount of data, the number of routing records and the number of cycles in the multi-cycle cycle, and using the data flow load statistics value of the data stream in the multi-cycle cycle as the write performance indicator value, so that the accuracy and reliability of routing expansion and routing reduction that can be performed based on the write performance indicator value are higher.

[0199] In an exemplary embodiment, the method further includes: scanning the data stream newly added by the data stream processing engine in the first cycle; determining the number of default routing records of the newly added data stream according to the default operator allocation strategy; counting the number of routing records of each data write operator on the data stream processing engine according to the routing records in the first operator routing table; selecting target data write operators whose number is consistent with the number of default routing records from the data write operators on the data stream processing engine in ascending order of priority of the number of routing records; and adding a target routing record indicating the target data write operator and the data stream to the second operator routing table of the second cycle.

[0200] The default operator allocation strategy refers to an operator allocation strategy that is pre-set by technical personnel based on actual needs.

[0201] Exemplarily, the default operator allocation strategy may indicate that when the newly added data stream is not a preset data stream, one data write operator is allocated to the newly added data stream, that is, the default number of route records for the newly added data stream is 1; when the newly added data stream is a preset data stream, a preset number of data write operators is allocated to the newly added data stream, that is, the default number of route records for the newly added data stream is a preset number. The preset data stream is a data stream that the technician determines based on prior knowledge that multiple data write operators need to be set.

[0202] The target data writing operator refers to a data writing operator used to write the newly added data stream.

[0203] In some exemplary embodiments, the computer device may also scan the data stream added by the data stream processing engine in the first cycle, and determine the number of default routing records of the newly added data stream according to the default operator allocation strategy.

[0204] Furthermore, after determining the number of default routing records for the newly added data stream, the computer device can count the number of routing records of each data writing operator on the data stream processing engine based on the routing records in the first operator routing table, and select target data writing operators whose number is consistent with the number of default routing records from the data writing operators on the data stream processing engine in ascending order of priority of the number of routing records.

[0205] Furthermore, after determining the target write operator, the computer device may add a target routing record indicating the target data write operator and the data stream to the second operator routing table of the second cycle.

[0206] The above-mentioned method scans the data stream newly added by the data stream processing engine in the first cycle; determines the number of default routing records of the newly added data stream according to the default operator allocation strategy; counts the number of routing records of each data writing operator on the data stream processing engine according to the routing records in the first operator routing table; selects the target data writing operator whose number is consistent with the number of default routing records from the data writing operators on the data stream processing engine according to the priority order of the number of routing records in ascending order; adds a method indicating the target data writing operator and the target routing record of the data stream to the second operator routing table of the second cycle, so that when it is necessary to write to the data stream, the default operator allocation strategy can be used to allocate as few data writing operators as possible to perform the write operation, so as to avoid generating a large number of small files and improve the efficiency of the subsequent data acquisition process.

[0207] In an exemplary embodiment, according to the write performance characteristics, the first operator routing table of the first cycle is adjusted to obtain the second operator routing table of the second cycle, including: traversing the data stream indicated by the routing record in the first operator routing table; in response to the traversed data stream being in a locked state, continuing to traverse the next data stream; in response to the traversed data stream being in an unlocked state, adjusting the routing record for the traversed data stream according to the write performance characteristics; until the traversal is completed, the second operator routing table of the second cycle is obtained.

[0208] The data flow in the locked state is a data flow whose routing record cannot be adjusted, that is, a data flow that cannot be routed for expansion or reduction.

[0209] The data flow in the unlocked state is a data flow whose routing record can be adjusted, that is, a data flow that can perform routing expansion or routing reduction.

[0210] In some exemplary embodiments, for some data streams with a small amount of data but time-consuming data writing, the state of these data streams can be set to a locked state. When the computer device traverses the data stream indicated by the routing record in the first operator routing table, if it is determined that the traversed data stream is in a locked state, it continues to traverse the next data stream. If it is determined that the traversed data stream is in an unlocked state, the routing record for the traversed data stream is adjusted according to the write performance characteristics until the traversal is completed, and the second operator routing table of the second cycle is obtained.

[0211] The above-mentioned method traverses the data stream indicated by the routing record in the first operator routing table; in response to the traversed data stream being in a locked state, continues to traverse the next data stream; in response to the traversed data stream being in an unlocked state, adjusts the routing record for the traversed data stream according to the write performance characteristics; until the traversal is completed, the second operator routing table of the second cycle is obtained. For a special data stream, its state can be set to a locked state, so that its routing capacity is not expanded or reduced, thereby avoiding low writing efficiency of the special data stream.

[0212] In an exemplary embodiment, Figure 6 As shown, a data stream writing control method is provided, the method comprising the following steps:

[0213] Step 601: Obtain a first operator routing table of a first cycle.

[0214] The cycle period refers to a fixed time period set when the data stream processing engine performs a write operation on the data stream. The duration of each cycle period can be pre-set by the technician according to actual needs. The duration of each cycle period can be 1 minute, 10 minutes, 30 minutes, 1 hour, 5 hours, 10 hours, 24 hours, etc.

[0215] Exemplarily, assuming that the duration of each cycle is 1 minute, the first cycle begins when the data stream processing engine starts to execute the write operation on the data stream. After 1 minute, the first cycle ends and the second cycle begins.

[0216] The first cycle refers to any cycle in multiple cycles when the data stream processing engine performs a write operation on the data stream.

[0217] The operator routing table contains routing records, each of which indicates a data stream to be written and a corresponding data writing operator. That is, the operator routing table is used to indicate the mapping relationship between the data stream and the data writing operator used to write the data stream. The operator routing table can specifically be a Sink routing table in the data stream processing engine.

[0218] The first operator routing table includes at least one routing record. The first operator routing table includes routing records within a first cycle. The routing records within the first cycle indicate a data stream to be written and a corresponding data writing operator within the first cycle.

[0219] In some exemplary embodiments, before each cycle starts, the data stream processing engine can obtain the operator routing table corresponding to the cycle. Specifically, before the first cycle starts, the data stream processing engine can obtain the first operator routing table corresponding to the first cycle.

[0220] Step 602: Use the data writing operator indicated by each routing record to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the data writing operator and the data stream to be written.

[0221] The data writing operator refers to an operator set in the data stream processing engine for writing to the data stream. The data writing operator can be a Sink, which is a computing unit in the data stream processing engine for writing the data stream. Multiple Sinks can be set in the data stream processing engine, and each Sink has an identification number.

[0222] A data stream refers to a set of data sequences with the same data type and originating from the same data source. Exemplarily, the data stream can be a log file type data stream, a user interaction type data stream, a sensor network type data stream, etc. Specifically, all types of systems and applications have log files. For example, a server log file can record user visits to a website, and a database log file can record the operation history of the database. These log files all exist in the form of data streams.

[0223] The file corresponding to the data writing operator and the data stream to be written refers to the file created when the data stream to be written is first written into the file system by the data writing operator. Exemplarily, the file system may be a distributed file system (HDFS, Hadoop Distributed File System).

[0224] In some exemplary embodiments, after obtaining the first operator routing table of the first cycle, the data stream processing engine can use the data writing operator indicated by each routing record in the first operator routing table to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0225] Step 603: For each data stream to be written, obtain a write performance characteristic of writing by at least one data writing operator in the first cycle.

[0226] The write performance characteristic table shows the write performance of each data stream in the first cycle. The write performance characteristic may specifically include the data flow identifier (dataFlowId), the data write operator identifier (Sink taskIndex), the cycle duration, the consumption time, the release time, the total data volume and the total number of data items.

[0227] The data in the data stream is first stored in the cache and then retrieved from the cache by the data stream processing engine for writing. The consumption time refers to the time when the data in the data stream is written, and the release time refers to the time when the data in the data stream is cached. The total data volume refers to the total data volume of each data stream in the first cycle, and the total number of data items refers to the total number of data items in each data stream in the first cycle.

[0228] Furthermore, if the data in a data stream is written by one data writing operator, the total data volume of the data stream can be represented by the total size. If the data in a data stream is written by multiple data writing operators, the total data volume of the data stream can be represented by the total number of fields.

[0229] In some exemplary embodiments, for each data stream to be written, the data stream processing engine may obtain, through the data writing operator, the writing performance characteristics of the data written by at least one data writing operator in the first cycle.

[0230] Step 604: Send the write performance characteristics.

[0231] The sent write performance characteristic is used to indicate adjusting the routing records in the first operator routing table according to the write performance characteristic to obtain the second operator routing table of the second cycle, and the write performance characteristic represents the write performance of writing each data stream in the first cycle.

[0232] In some exemplary embodiments, after determining the write performance characteristics, the data stream processing engine may send the write performance characteristics to a computer device, which may be a control node in the computer device.

[0233] Furthermore, after receiving the write performance characteristic, the control node in the computer device may adjust the routing record in the first operator routing table according to the write performance characteristic to obtain the second operator routing table of the second cycle.

[0234] The above-mentioned method obtains the first operator routing table of the first cycle, and the first operator routing table contains at least one routing record, each routing record indicates a data stream to be written and a corresponding data write operator; using the data write operator indicated by each routing record, the data stream to be written corresponding to the data write operator is written into the file corresponding to the data write operator and the data stream to be written; for each data stream to be written, the write performance characteristics of writing by at least one data write operator in the first cycle are obtained; the method of sending the write performance characteristics, by sending the write performance characteristics of each cycle, so that the computer device determines the operator routing table of the next cycle according to the write performance characteristics, so that the obtained operator routing table is more accurate, thereby effectively improving the data stream writing efficiency.

[0235] In an exemplary embodiment, the method of obtaining the first operator routing table of the first cycle period includes: using a data source operator to receive the first operator routing table of the first cycle period from a control node; the method also includes: using the data source operator to read data segments of each data stream from a cache, and according to the routing records in the first operator routing table, allocating the data segments to each data write operator.

[0236] The write performance characteristic is acquired and sent by the data write operator.

[0237] The data source operator refers to an operator set in the data stream processing engine, which is used to obtain data segments from the cache and assign the data segments to the data write operator so that the data write operator writes the data segments. The data source operator can be a Source, which is a computing unit in the data stream processing engine responsible for reading source data. Multiple Sources can be set in the data stream processing engine, and each Source has an identification number.

[0238] In some exemplary embodiments, the data stream processing engine may first use the data source operator to receive the first operator routing table of the first cycle from the control node. Specifically, the first operator routing table is stored in a routing table repository. The routing table repository may be Zookeeper.

[0239] Furthermore, after receiving the first operator routing table, the data stream processing engine can use the data source operator to read the data segments of each data stream from the cache, and distribute the data segments to each data write operator according to the routing records in the first operator routing table.

[0240] The above method uses the data source operator to read the data segments of each data stream from the cache, and allocates the data segments to each data write operator according to the routing records in the first operator routing table. The data segments can be allocated to each data write operator according to the routing records in the first operator routing table so that the data write operator writes the data segments, which can effectively improve the writing performance of the data stream.

[0241] In an optional embodiment of the present application, the method further includes: in the process of writing the data stream, using a checkpoint mechanism to periodically generate a state snapshot.

[0242] The checkpoint mechanism refers to the Checkpoint mechanism. Exemplarily, the checkpoint mechanism can be used to ensure fault tolerance and persistence during the data stream writing process, by periodically generating a state snapshot during the data stream processing process, so as to recover when the system fails.

[0243] In an optional embodiment of the present application, the data writing operator may be a data segment in a data stream written into a distributed file system. There are two types of nodes in a distributed file system, namely NameNode and DataNode. NameNode is responsible for storage-related scheduling in a distributed file system, and DataNode is responsible for specific storage tasks. Specifically, NameNode and DataNode will be described below in conjunction with the writing process of a distributed file system. The data writing operator may call the create() method through the distributed file system API to create a new file or append to an existing file. The data writing operator first communicates with NameNode, requests to create a file and obtains a set of DataNodes for storing data blocks. NameNode returns a list of DataNodes based on the number of copies of the data block and the load of the DataNode, and these DataNodes will store the data blocks in sequence. The data writing operator will establish a connection with the first DataNode and request it to prepare to receive data. After the first DataNode confirms, the data writing operator starts sending data to it. After receiving the data, the first DataNode forwards it to the next DataNode to form a data stream pipeline. During the data segment writing process, the data segment is written to the DataNode in the pipeline in the form of a data block. After receiving the data block, each DataNode will send a confirmation message to the data writing operator. When the data writing operator receives the confirmation messages from all DataNodes, it can be determined that the data segment has been written successfully. After the data segment is written, the data writing operator can call the close() method to close the file.

[0244] In an exemplary embodiment, Figure 7As shown, a data stream writing control method is provided, the method comprising the following steps:

[0245] Step 701: The data stream processing engine uses a data source operator to receive a first operator routing table of a first cycle from a control node, and uses the data source operator to read data segments of each data stream from a cache, and uses the data source operator to follow the routing records in the first operator routing table, and uses the data write operator indicated by each routing record to write the data to the data stream to be written corresponding to the operator, and writes the data to the file corresponding to the data write operator and the data stream to be written; for each data stream to be written, obtains the write performance characteristics of writing by at least one data write operator in the first cycle; and sends the write performance characteristics.

[0246] The first operator routing table includes at least one routing record, and each routing record indicates a data stream to be written and a corresponding data writing operator.

[0247] In some exemplary embodiments, the data stream processing engine may first utilize the data source operator to receive a first operator routing table of a first cycle from the control node.

[0248] Furthermore, after receiving the first operator routing table, the data stream processing engine can use the data source operator to read the data segments of each data stream from the cache, and according to the routing records in the first operator routing table, use the data write operator indicated by each routing record to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0249] Furthermore, for each data stream to be written, the data stream processing engine may obtain the write performance characteristics of the data written by at least one data write operator in the first cycle, and send the write performance characteristics to the control node in the computer device.

[0250] Step 702: The computer device receives the write performance characteristics of the data stream processing engine in the first cycle; traverses the data stream indicated by the routing record in the first operator routing table; and in response to the traversed data stream being in a locked state, continues to traverse the next data stream.

[0251] The write performance characteristic represents the write performance of the data stream processing engine in writing each data stream in the first cycle.

[0252] In some exemplary embodiments, for some data streams with a small amount of data but time-consuming data writing, the state of these data streams can be set to a locked state. When the computer device traverses the data stream indicated by the routing record in the first operator routing table, if it is determined that the traversed data stream is in a locked state, it continues to traverse the next data stream. If it is determined that the traversed data stream is in an unlocked state, the routing record for the traversed data stream is adjusted according to the write performance characteristics until the traversal is completed, and the second operator routing table of the second cycle is obtained.

[0253] Step 703: In response to the traversed data stream being in an unlocked state, the computer device determines a multi-cycle period to be counted, the multi-cycle period including the first cycle period and at least one historical cycle period before the first cycle period; obtains the write performance characteristics of each of the at least one historical cycle period; performs statistical analysis of the multi-cycle period based on the write performance characteristics of each of the multi-cycle periods, and obtains the write performance index value of each of the data streams written in the first cycle period; obtains the number of operators of the data write operator on the data stream processing engine, and obtains the concurrent routing upper limit value of each of the data write operators on the data stream processing engine; determines the concurrent routing constraint value of the data stream processing engine based on the number of operators and the concurrent routing upper limit value.

[0254] In some exemplary embodiments, the computer device may first determine the multiple cycle periods to be counted, and then obtain the write performance characteristics of each of the at least one historical cycle period.

[0255] Furthermore, after obtaining the write performance characteristics of each of the first cycle period and at least one historical cycle period, the computer device can perform statistical analysis of multiple cycle periods based on the write performance characteristics of each of the multiple cycle periods to obtain the write performance index value of each data stream written in the first cycle period.

[0256] The computer device may also obtain the number of operators of the data writing operator on the data stream processing engine, and obtain the concurrent routing upper limit value of each data writing operator on the data stream processing engine.

[0257] Furthermore, the computer device can determine the concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value. For each data stream written in the first cycle, the computer device, when determining that the write performance index value is greater than the first threshold, also needs to determine whether the number of routing records in the first operator routing table is less than the concurrent routing constraint value.

[0258] Step 704: for each data stream written in the first cycle, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance index value is greater than the first threshold, and an expansion routing record indicating the data stream has been generated for the first cycle, then determine the timing order of this routing expansion in the continuous routing expansion operation; determine the number of expansion routing records for the data stream based on the timing order; determine that no routing record has been established with the data stream based on the existing routing record indicating the data stream in the first operator routing table of the first cycle. candidate data write operators; for each of the candidate data write operators, according to the routing records in the first operator routing table, counting the number of routing records corresponding to the candidate data write operator; from the candidate data write operators, in the order of priority in ascending order of the number of routing records, selecting new data write operators whose number is consistent with the number of expanded routing records; respectively establishing routing records indicating each of the new data write operators and the data flow, and obtaining the expanded routing records indicating the data flow; generating a second operator routing table of a second cycle according to the existing routing records and the expanded routing records.

[0259] In some exemplary embodiments, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance indicator value is greater than the first threshold, and an expanded routing record indicating the data flow has been generated for the first cycle, then the timing order of this routing expansion in the continuous routing expansion operation can be determined.

[0260] Specifically, for each data stream written in the first cycle, if the computer device determines that the write performance index value of the data stream is greater than the first threshold, it can first determine whether the first cycle has generated an expansion routing record indicating the data stream. If generated, it can be determined as a continuous routing expansion operation, and the timing order of this routing expansion in the continuous routing expansion operation is determined. If not generated, it can be determined that the timing order of this routing expansion is 1.

[0261] Furthermore, after determining the timing sequence, the computer device can determine the number of expanded routing records for the data flow according to the timing sequence, and based on the existing routing records indicating the data flow in the first operator routing table of the first cycle, generate an expanded routing record indicating the data flow according to the number of expanded routing records, and generate a second operator routing table of the second cycle based on the existing routing record and the expanded routing record.

[0262] Step 705: The computer device determines a multi-cycle period to be counted, the multi-cycle period including the first cycle period and at least one historical cycle period before the first cycle period; obtains the write performance characteristics of each of the at least one historical cycle period; performs statistical analysis of the multi-cycle period according to the write performance characteristics of each of the multi-cycle periods, and obtains the write performance index value of each of the data streams written in the first cycle period; for each data stream written in the first cycle period, if the write performance index value is less than or equal to the second threshold value, determines the number of shrinkage routing records for the data stream according to the preset shrinkage strategy; and calculates the number of shrinkage routing records for the data stream according to the first cycle period. According to the present invention, the first operator routing table of the second cycle indicates the existing routing record of the data stream, and determines the candidate data write operator with the established routing record for the data stream; for each candidate data write operator, according to the routing record in the first operator routing table, counts the number of routing records corresponding to the candidate data write operator; from the candidate data write operators, according to the priority order of the number of routing records in descending order, selects the reduction data write operators whose number is consistent with the number of the reduction routing records, and obtains the reduction routing record indicating the data stream and the reduction data write operator; according to the existing routing record and the reduction routing record, generates the second operator routing table of the second cycle.

[0263] In some exemplary embodiments, the computer device may first determine the multiple cycle periods to be counted, and then obtain the write performance characteristics of each of the at least one historical cycle period.

[0264] Furthermore, after obtaining the write performance characteristics of each of the first cycle period and at least one historical cycle period, the computer device can perform statistical analysis of multiple cycle periods based on the write performance characteristics of each of the multiple cycle periods to obtain the write performance index value of each data stream written in the first cycle period.

[0265] After receiving the write performance characteristics of the data stream processing engine in the first cycle, the computer device can determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics. Specifically, the computer device can determine the write performance index value of each data stream written in the first cycle by using a preset algorithm and the write performance characteristics.

[0266] Furthermore, after the computer device determines the write performance indicator value of each data stream written in the first cycle, for each data stream written in the first cycle, if the computer device determines that the write performance indicator value of the data stream is less than or equal to the second threshold, the computer device performs routing shrinkage based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and determines the shrunken routing record indicating the data stream.

[0267] If the computer device determines that the write performance indicator value of the data flow is greater than the second threshold, it does not perform routing shrinkage based on the existing routing record indicating the data flow in the first operator routing table of the first cycle.

[0268] Furthermore, for each data stream written in the first cycle, the computer device can perform the above operations, and after determining the reduced capacity routing record of each data stream, a second operator routing table of the second cycle can be generated according to the existing routing record and the reduced capacity routing record.

[0269] Step 706, the computer device scans the data stream newly added by the data stream processing engine in the first cycle; determines the number of default routing records of the newly added data stream according to the default operator allocation strategy; counts the number of routing records of each data write operator on the data stream processing engine according to the routing records in the first operator routing table; selects the target data write operator whose number is consistent with the number of default routing records from the data write operators on the data stream processing engine according to the ascending priority order of the number of routing records; adds the target routing record indicating the target data write operator and the data stream to the second operator routing table of the second cycle.

[0270] In some exemplary embodiments, the computer device may also scan the data stream added by the data stream processing engine in the first cycle, and determine the number of default routing records of the added data stream according to the default operator allocation strategy.

[0271] Furthermore, after determining the number of default routing records for the newly added data stream, the computer device can count the number of routing records of each data writing operator on the data stream processing engine based on the routing records in the first operator routing table, and select target data writing operators whose number is consistent with the number of default routing records from the data writing operators on the data stream processing engine in ascending order of priority of the number of routing records.

[0272] Furthermore, after determining the target write operator, the computer device may add a target routing record indicating the target data write operator and the data flow to the second operator routing table of the second cycle.

[0273] In a specific application scenario, IoT devices are used to collect various data in real time. These devices temporarily store the data in the cache first, and the control node and the data stream processing engine write the data. Specifically, the data stream processing engine receives the first operator routing table of the first cycle sent by the control node through the data source operator. The routing table determines the writing path of different types of IoT device data (data streams), that is, which data writing operators write the corresponding storage files. These files can be used for long-term storage of device historical data or real-time analysis. The data source operator reads the data segment of the IoT device data from the cache, and assigns the data to the data writing operator according to the routing record indicated by the first operator routing table, so that the data writing operator writes the data segment. At the same time, for each device data stream, the data writing operator statistically obtains its performance characteristics for data writing in the first cycle and sends it to the control node.

[0274] After receiving the write performance characteristics, the control node traverses the device data stream indicated by the first operator routing table. Due to hardware performance issues, some old devices have small data volumes but unstable transmission and writing, and take a long time to process. The data streams of these devices are set to a locked state. During the traversal process, if the control node encounters a locked data stream, it will skip and continue to traverse the next one until it finds a device data stream in an unlocked state.

[0275] For the data stream of the device in the unlocked state, the control node determines the multi-cycle period to be counted, including the current first cycle period and the historical cycle periods of the previous time periods. The write performance characteristics of each historical cycle period are obtained, and statistical analysis is performed to obtain the write performance index values ​​of each device data stream in the first cycle period, such as the data segment delay statistics, data stream load statistics, etc. At the same time, the number of data write operators on the data stream processing engine and the concurrent routing upper limit value of each operator are obtained, and then the concurrent routing constraint value is determined.

[0276] For each device data stream, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance index value is greater than the first threshold, and the expansion routing record of the device data stream has been generated before, determine the timing order of this routing expansion in the continuous routing expansion operation. Then, from the candidate data writing operators that have not established routing records with the device data stream, select the newly added data writing operator in ascending order of the number of routing records, establish the newly added routing record, and generate the second operator routing table of the second cycle.

[0277] After completing the multi-cycle statistical analysis, for the device data stream whose write performance index value is less than or equal to the second threshold, the number of reduced routing records is determined according to the preset reduction strategy. From the candidate data writing operators that have established routing records with the device data stream, a corresponding number of reduced data writing operators are selected in descending order of the number of routing records to obtain the reduced routing records, and generate the second operator routing table of the second cycle.

[0278] With the update of IoT devices or new business needs, some devices may be added in the first cycle, such as new environmental monitoring devices. After the control node scans the data streams of these newly added devices, it determines the default number of routing records for each newly added device data stream according to the default operator allocation strategy, and selects the target data writing operator from the data writing operator in ascending order of the number of routing records, and adds the target routing records for these newly added device data streams to the second operator routing table in the second cycle.

[0279] In another specific application scenario, for the writing process of the application log data, it is assumed that the log data is three data streams, namely, data stream a, data stream b, and data stream c, and there are three data writing operators configured in the data stream processing engine, namely, Sink0, Sink1, and Sink2.

[0280] When the data stream processing engine starts writing log data, the default operator allocation strategy can be used, that is, each data stream is assigned a data writing operator by default. After the data source operator consumes the data, the data stream is assigned to the data writing operator according to the default operator allocation strategy. That is, data stream b-Sink0, data stream a-Sink1, data stream c-Sink2.

[0281] In a certain cycle, the traffic of data stream a increases, and the control node determines that its write performance index value is greater than the expansion threshold (first threshold) through the write performance characteristics, and then assigns another data write operator Sink-2 with the least routing records to data stream a. Then the operator routing table for the next cycle is data stream b-Sink0, data stream a-Sink1, data stream a-Sink2, and data stream c-Sink2. In the next cycle, the data processing engine will write data to the data stream of log data according to the operator routing table.

[0282] After at least one cycle, in another cycle, the control node determines through the write performance characteristics that the write performance index value of data stream a is again greater than the expansion threshold (first threshold), and then assigns another data write operator Sink-0 with the least routing records to data stream a. Then the operator routing table for the next cycle is data stream b-Sink0, data stream a-Sink1, data stream a-Sink2, data stream a-Sink0, data stream c-Sink2. In the next cycle, the data processing engine will write data to the data stream of log data according to the operator routing table.

[0283] After at least one cycle, in another cycle, the control node determines that the write performance index value of data stream c is also greater than the expansion threshold (first threshold) through the write performance characteristics, and a data write operator Sink-1 with the least routing records is also assigned to data stream c. Then the operator routing table of the next cycle is data stream b-Sink0, data stream a-Sink1, data stream a-Sink2, data stream a-Sink0, data stream c-Sink2, data stream c-Sink1. In the next cycle, the data processing engine will write data to the data stream of log data according to the operator routing table.

[0284] After at least one cycle, in another cycle, the control node determines through the write performance characteristics that the write performance index values ​​of data stream a and data stream c are less than the reduction threshold (second threshold), and then reduces the data write operator with more routing records of data stream c and data stream a. Then the operator routing table of the next cycle is data stream b-Sink0, data stream a-Sink2, and data stream c-Sink1. In the next cycle, the data processing engine will write data to the data stream of log data according to the operator routing table.

[0285] In order to clearly describe the beneficial effects of the technical solution of the present application, some prior arts are explained below.

[0286] In some prior arts, a data stream processing engine is separately set for each data stream, and the number of data writing operators in the corresponding data stream processing engine is configured according to the data volume of the data stream. Figure 8 , Fig. 9 and Fig.10 As shown, Figure 8 , Fig. 9 and Fig.10 Different data stream processing engines are set for three data streams with different data volumes. Figure 8The data stream in is data stream a, and its data volume is large. Therefore, the number of data write operators in the corresponding data stream processing engine is 3, namely, data write operator 0, data write operator 1 and data write operator 2. The data source operator first obtains data stream a from the cache, and then distributes data stream a to data write operator 0, data write operator 1 and data write operator 2. Then data write operator 0, data write operator 1 and data write operator 2 write data stream a to the file system, namely, file-data write operator 0-data stream a, file-data write operator 1-data stream a, file-data write operator 2-data stream a. Fig. 9 The data stream in is data stream b, and its data volume is small. Therefore, the number of data write operators in the corresponding data stream processing engine is 1, that is, data write operator 0. The data source operator first obtains data stream b from the cache, and then assigns data stream b to data write operator 0, and then data write operator 0 writes data stream b to the file system, that is, file-data write operator 0-data stream b. Fig.10 The data stream in is data stream c, and its data volume is smaller than that of data stream a, but larger than that of data stream b. Therefore, the number of data write operators in the corresponding data stream processing engine is 2, namely, data write operator 0 and data write operator 1. The data source operator first obtains data stream c from the cache, and then assigns data stream c to data write operator 0 and data write operator 1. Data write operator 0 and data write operator 1 then write data stream c to the file system, namely, file-data write operator 0-data stream c, file-data write operator 1-data stream c.

[0287] However, although this method realizes the independence of data stream resources, the cost of setting up a data stream processing engine for each data stream is high, especially in the scenario where the number of data streams is large.

[0288] Based on this, in some other existing technologies, the data stream is first written into the intermediate file system through all the data writing operators in the data writing engine. Each data writing operator generates a corresponding small file when writing the data stream into the intermediate file system, so that the same data stream is stored in the intermediate file system in the form of multiple small files, and then the small files of the same data stream in the intermediate file system are integrated and processed, and the large file corresponding to the integrated data stream is stored in the final file system. This method only needs to set up a data stream processing engine, specifically, Fig.11 As shown, in Fig.11In the data stream processing engine, the data source operator can obtain the data streams to be written from the cache, namely, data stream a, data stream b and data stream c, and assign the data streams to the data writing operators, namely, data writing operator 0, data writing operator 1 and data writing operator 2. Then, the data writing operator writes the data streams to be written into the intermediate file system to generate multiple small files, namely, file-data writing operator 0-data stream a, file-data writing operator 0-data stream b, ...

[0289] Furthermore, the Spark scheduled job can periodically obtain small files of the same data stream from the intermediate file system, aggregate them, and store the aggregated large files in the final file system.

[0290] However, although this method solves the problem that the same data stream is stored in the form of multiple small files in the file system, resulting in poor data reading efficiency, setting up an intermediate file system and integrating the small files of the same data stream in the intermediate file system not only makes the data stream writing cost higher, but also makes the data stream writing efficiency lower, which in turn leads to poor data stream writing performance.

[0291] In an exemplary embodiment, Fig.12 As shown, a data stream writing control system is provided, the system includes a control node, a data stream processing engine, a file system and a cache, the data stream processing engine includes a data writing operator and a data source operator;

[0292] The data source operator is used to receive a first operator routing table of a first cycle from a control node, the first operator routing table comprising at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator; read data segments of each data stream from the cache, and distribute the data segments to each data writing operator according to the routing records in the operator routing table; wherein the write performance characteristics are obtained and sent by the data writing operator;

[0293] The data writing operator is used to write the data stream to be written corresponding to the data writing operator indicated by the routing record into the file system, the file corresponding to the data writing operator and the data stream to be written; obtain the writing performance characteristics of writing by the data writing operator in the first cycle; and send the writing performance characteristics to the control node;

[0294] The control node is used to receive the write performance characteristics of the data write operator in the first cycle, and the write performance characteristics represent the write performance of the data stream processing engine in writing each data stream in the first cycle; according to the write performance characteristics, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, the first operator routing table contains routing records in the first cycle, and the second operator routing table contains routing records in the second cycle, each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data write operator; send the second operator routing table to the data source operator, the operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle, write the data stream to be written corresponding to the data write operator, and write it into the file system, the data write operator and the file corresponding to the data stream to be written.

[0295] In some exemplary embodiments, Fig.12 As shown, the data source operator can obtain the data source from the cache, namely, data source a, data source b and data source c, and allocate the data source to data write operator 0, data write operator 1 and data write operator 2 according to the operator routing table obtained from the control node. Then, data write operator 0, data write operator 1 and data write operator 2 write the data stream a, data source b and data source c to the file system respectively, namely, file-data write operator 0-data stream a, file-data write operator 0-data stream b, file-data write operator 1-data stream a, file-data write operator 1-data stream c, file-data write operator 2-data stream a, file-data write operator 2-data stream c. At the same time, the data write operator will also send the write performance characteristics to the control node.

[0296] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0297] Based on the same inventive concept, the embodiment of the present application also provides a data stream write control device for implementing the data stream write control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data stream write control device embodiments provided below can refer to the limitations of the data stream write control method above, and will not be repeated here.

[0298] In an exemplary embodiment, Fig.13 As shown, a data stream writing control device 1300 is provided, comprising: a receiving module 1301, a first execution module 1302 and a sending module 1303, wherein:

[0299] A receiving module 1301 is used to receive a write performance characteristic of a data stream processing engine in a first cycle, where the write performance characteristic represents a write performance of the data stream processing engine in writing each data stream in the first cycle;

[0300] A first execution module 1302 is used to adjust a first operator routing table of a first cycle according to the write performance characteristics, and obtain a second operator routing table of a second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator;

[0301] The sending module 1303 is used to send the second operator routing table to the data stream processing engine. The second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle period to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

[0302] In one embodiment, the first execution module 1302 is specifically used to determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is greater than a first threshold, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, route expansion is performed to generate an expanded routing record indicating the data stream; based on the existing routing record and the expanded routing record, a second operator routing table of the second cycle is generated.

[0303] In one embodiment, the first execution module 1302 is also used to obtain the number of operators of the data write operator on the data stream processing engine, and obtain the concurrent routing upper limit value of each data write operator on the data stream processing engine; determine the concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value; the first execution module 1302 is specifically used to generate an expanded routing record indicating the data stream for each data stream written in the first cycle period, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance indicator value is greater than the first threshold value, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle period.

[0304] In one embodiment, the first execution module 1302 is specifically used to, for each data stream written in the first cycle, if the write performance indicator value is greater than the first threshold, determine the number of expansion routing records for the data stream according to the preset expansion strategy; determine the candidate data write operator for which no routing record has been established with the data stream according to the existing routing record indicating the data stream in the first operator routing table of the first cycle; for each of the candidate data write operators, count the number of routing records corresponding to the candidate data write operator according to the routing records in the first operator routing table; from the candidate data write operators, select the newly added data write operators whose number is consistent with the number of the expansion routing records in ascending order of priority of the number of routing records; establish routing records indicating each of the newly added data write operators and the data stream respectively, and obtain the expansion routing record indicating the data stream.

[0305] In one embodiment, the first execution module 1302 is specifically used to determine the timing order of this routing expansion in the continuous routing expansion operation for each data stream written in the first cycle if the write performance indicator value is greater than the first threshold and an expansion routing record indicating the data stream has been generated for the first cycle; determine the number of expansion routing records for the data stream according to the timing order, wherein the number of expansion routing records is positively correlated with the timing order; based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, and according to the number of expansion routing records, generate an expansion routing record indicating the data stream.

[0306] In one embodiment, the first execution module 1302 is specifically used to determine the write performance index value of each data stream written in the first cycle according to the write performance characteristics; for each data stream written in the first cycle, if the write performance index value is less than or equal to the second threshold, based on the existing routing record indicating the data stream in the first operator routing table of the first cycle, route shrinkage is performed to determine the shrunken routing record indicating the data stream; based on the existing routing record and the shrunken routing record, a second operator routing table for the second cycle is generated.

[0307] In one embodiment, the first execution module 1302 is specifically used to determine the number of reduced routing records for each data stream written in the first cycle according to a preset reduction strategy if the write performance indicator value is less than or equal to a second threshold; determine a candidate data write operator for which a routing record has been established with the data stream based on an existing routing record indicating the data stream in a first operator routing table of the first cycle; for each candidate data write operator, count the number of routing records corresponding to the candidate data write operator based on the routing records in the first operator routing table; and select a reduced data write operator whose number is consistent with the number of reduced routing records from the candidate data write operators in descending order of priority of the number of routing records, to obtain a reduced routing record indicating the data stream and the reduced data write operator.

[0308] In one embodiment, the first execution module 1302 is specifically used to determine the multi-cycle period to be counted, which includes the first cycle period and at least one historical cycle period before the first cycle period; obtain the write performance characteristics of each of the at least one historical cycle period; perform statistical analysis of the multi-cycle period based on the write performance characteristics of each of the multi-cycle periods, and obtain the write performance index value of each data stream written in the first cycle period.

[0309] In one embodiment, each data stream includes multiple data segments, and the write performance characteristics include a release timestamp and a consumption timestamp of each data segment, the release timestamp represents the time when the data segment is written to the cache, and the consumption timestamp represents the time when the data segment read from the cache is written to the file; the write performance indicator value of each data stream written in the first cycle includes a data segment delay statistic value of a multi-cycle cycle; the first execution module 1302 is specifically used to determine the delay value of each data segment according to the release timestamp and the consumption timestamp of each data segment written in each cycle within the multi-cycle cycle; for each data stream, determine the total delay value of the data stream according to the delay values ​​of each data segment in the data stream written in the multi-cycle cycle; for each data stream, count the total number of data segments of the data segments in the data stream written in the multi-cycle cycle; for each data stream, determine the data segment delay statistic value of the data stream in the multi-cycle cycle according to the total delay value of the data stream and the total number of data segments of the data stream.

[0310] In one embodiment, the write performance characteristics include the amount of data written to each data stream in each cycle; the write performance indicator value of each data stream written in the first cycle, including the data stream load statistics of multiple cycles; the first execution module 1302 is specifically used to count the total amount of data written to the data stream in the multiple cycles according to the amount of data written to the data stream in each cycle for each data stream written in the first cycle; for each data stream written in the first cycle, count the number of routing records indicating the routing records of the data stream from the first operator routing table; for each data stream written in the first cycle, determine the data stream load statistics of the data stream in the multiple cycles according to the total amount of data, the number of routing records and the number of cycles in the multiple cycles.

[0311] In one embodiment, the first execution module 1302 is also used to scan the data stream newly added by the data stream processing engine in the first cycle; determine the number of default routing records of the newly added data stream according to the default operator allocation strategy; count the number of routing records of each data write operator on the data stream processing engine according to the routing records in the first operator routing table; select the target data write operator whose number is consistent with the number of default routing records from the data write operators on the data stream processing engine according to the ascending priority order of the number of routing records; and add a target routing record indicating the target data write operator and the data stream to the second operator routing table of the second cycle.

[0312] In one embodiment, the first execution module 1302 is specifically used to traverse the data stream indicated by the routing record in the first operator routing table; in response to the traversed data stream being in a locked state, continue to traverse the next data stream; in response to the traversed data stream being in an unlocked state, adjust the routing record for the traversed data stream according to the write performance characteristics; until the traversal is completed, the second operator routing table of the second cycle is obtained.

[0313] In an exemplary embodiment, Fig.14 As shown, a data stream writing control device 1400 is provided, comprising: a first acquisition module 1401, a second execution module 1402, a second acquisition module 1403 and a sending module 1404, wherein:

[0314] A first acquisition module 1401 is used to acquire a first operator routing table of a first cycle, where the first operator routing table includes at least one routing record, and each routing record indicates a data stream to be written and a corresponding data writing operator;

[0315] The second execution module 1402 is used to use the data writing operator indicated by each routing record to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the data writing operator and the data stream to be written;

[0316] The second acquisition module 1403 is used to acquire, for each data stream to be written, a write performance characteristic of writing by at least one data writing operator in the first cycle;

[0317] The sending module 1404 is used to send the write performance characteristics, and the sent write performance characteristics are used to indicate adjusting the routing records in the first operator routing table according to the write performance characteristics to obtain the second operator routing table of the second cycle period, and the write performance characteristics represent the write performance of writing each data stream in the first cycle period.

[0318] In one embodiment, the first acquisition module 1401 is specifically used to use the data source operator to receive the first operator routing table of the first cycle period from the control node; the first acquisition module is also used to use the data source operator to read the data segments of each data stream from the cache, and distribute the data segments to each data write operator according to the routing records in the first operator routing table; wherein the write performance characteristics are acquired and sent by the data write operator.

[0319] Each module in the above-mentioned data stream writing control device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0320] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data stream write control method is implemented.

[0321] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.16 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data stream write control method is implemented.

[0322] Those skilled in the art will understand that Fig.15 and Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0323] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned data stream writing control method embodiments are implemented.

[0324] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned data stream writing control method embodiments are implemented.

[0325] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above data stream writing control method embodiment when executed by a processor.

[0326] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0327] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0328] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data stream writing control method, characterized in that: The method comprises: receiving a write performance characteristic of a data stream processing engine in a first cycle, wherein the write performance characteristic represents a write performance of the data stream processing engine in writing each data stream in the first cycle; According to the write performance characteristics, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator; The second operator routing table is sent to the data stream processing engine, where the second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the data write operator and the data stream to be written.

2. The method according to claim 1, characterized in that According to the write performance characteristics, adjusting the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle includes: Determining, according to the write performance characteristics, respective write performance index values ​​of respective data streams written in the first cycle; For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, perform route expansion based on an existing route record indicating the data stream in the first operator routing table of the first cycle, and generate an expanded route record indicating the data stream; A second operator routing table of a second cycle is generated according to the existing routing record and the expanded routing record.

3. The method according to claim 2, characterized in that The method further comprises: Obtain the number of operators of the data writing operator on the data stream processing engine, and obtain the concurrent routing upper limit value of each data writing operator on the data stream processing engine; Determining a concurrent routing constraint value of the data stream processing engine according to the number of operators and the concurrent routing upper limit value; For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, performing route expansion based on an existing route record indicating the data stream in a first operator routing table of the first cycle, and generating an expanded route record indicating the data stream, including: For each data stream written in the first cycle, if the number of routing records in the first operator routing table is less than the concurrent routing constraint value, and the write performance indicator value is greater than the first threshold, then based on the existing routing records indicating the data stream in the first operator routing table of the first cycle, an expanded routing record indicating the data stream is generated.

4. The method according to claim 2, characterized in that: For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, performing route expansion based on an existing route record indicating the data stream in a first operator routing table of the first cycle, and generating an expanded route record indicating the data stream, including: For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, determining the number of expansion routing records for the data stream according to a preset expansion strategy; Determine, according to the existing routing record indicating the data flow in the first operator routing table of the first cycle, a candidate data writing operator for which no routing record is established with the data flow; For each of the candidate data writing operators, counting the number of routing records corresponding to the candidate data writing operator according to the routing records in the first operator routing table; From the candidate data writing operators, select new data writing operators whose number is the same as the number of the expanded routing records in ascending order of priority of the number of routing records; A routing record indicating each of the newly added data writing operators and the data flow is established respectively, and an expanded routing record indicating the data flow is obtained.

5. The method according to claim 2, characterized in that: For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold, based on an existing routing record indicating the data stream in a first operator routing table of the first cycle, generating an expanded routing record indicating the data stream, including: For each data stream written in the first cycle, if the write performance indicator value is greater than a first threshold value and an expansion route record indicating the data stream has been generated for the first cycle, determining a timing order of this route expansion in the continuous route expansion operation; Determining the number of expansion routing records for the data flow according to the timing order, wherein the number of expansion routing records is positively correlated with the timing order; Based on the existing routing records indicating the data flow in the first operator routing table of the first cycle period, and according to the number of the expanded routing records, an expanded routing record indicating the data flow is generated.

6. The method according to claim 1, characterized in that According to the write performance characteristics, adjusting the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle includes: Determining, according to the write performance characteristics, respective write performance index values ​​of respective data streams written in the first cycle; For each data stream written in the first cycle, if the write performance indicator value is less than or equal to the second threshold, perform routing shrinkage based on an existing routing record indicating the data stream in the first operator routing table of the first cycle, and determine a shrinkage routing record indicating the data stream; A second operator routing table of a second cycle is generated according to the existing routing record and the reduced capacity routing record.

7. The method according to claim 6, characterized in that: For each data stream written in the first cycle, if the write performance indicator value is less than or equal to the second threshold, scaling down the route based on the existing route record indicating the data stream in the first operator routing table of the first cycle, and determining the scaled-down route record indicating the data stream, including: For each data stream written in the first cycle, if the write performance indicator value is less than or equal to a second threshold, determining the number of shrinking routing records for the data stream according to a preset shrinking strategy; Determine, according to the existing routing record indicating the data flow in the first operator routing table of the first cycle, a candidate data writing operator for which a routing record has been established with the data flow; For each of the candidate data writing operators, counting the number of routing records corresponding to the candidate data writing operator according to the routing records in the first operator routing table; From the candidate data writing operators, in descending order of priority of the number of routing records, a reduction data writing operator having a number consistent with the number of the reduction routing records is selected to obtain a reduction routing record indicating the data flow and the reduction data writing operator.

8. The method according to any one of claims 2 to 7, characterized in that: Determining the write performance index value of each data stream written in the first cycle according to the write performance characteristic includes: Determine a multi-cycle period to be counted, wherein the multi-cycle period includes the first cycle period and at least one historical cycle period before the first cycle period; Obtaining a write performance characteristic of each of the at least one historical cycle; According to the write performance characteristics of each of the multiple cycle periods, a statistical analysis of the multiple cycle periods is performed to obtain the write performance index value of each data stream written in the first cycle period.

9. The method according to claim 8, characterized in that Each data stream includes a plurality of data segments, and the write performance characteristics include a publishing timestamp and a consumption timestamp of each data segment, wherein the publishing timestamp indicates a time when the data segment is written into a cache, and the consumption timestamp indicates a time when the data segment read from the cache is written into the file; The write performance index value of each data stream written in the first cycle period, including the data segment delay statistics of multiple cycle periods; According to the write performance characteristics of each of the multiple cycles, statistical analysis of the multiple cycles is performed to obtain the write performance index value of each of the data streams written in the first cycle, including: Determine a delay value of each data segment according to a publishing timestamp and a consumption timestamp of each data segment written in each cycle within the multi-cycle period; For each data stream, determining a total delay value of the data stream according to delay values ​​of each data segment in the data stream written in the multi-cycle period; For each data stream, counting the total number of data segments in the data stream written in the multi-cycle period; For each data flow, a data segment delay statistic value of the data flow in the multi-cycle period is determined according to a total delay value of the data flow and a total number of data segments of the data flow.

10. The method according to claim 8, characterized in that The write performance characteristics include the amount of data written to each data stream in each cycle; the write performance index values ​​of each data stream written in the first cycle, including the data stream load statistics of multiple cycles; According to the write performance characteristics of each of the multiple cycles, statistical analysis of the multiple cycles is performed to obtain the write performance index value of each of the data streams written in the first cycle, including: For each data stream written in the first cycle, according to the amount of data written to the data stream in each cycle, counting the total amount of data written to the data stream in the multiple cycles; For each data flow written in the first cycle, counting the number of routing records indicating the routing records of the data flow from the first operator routing table; For each data stream written in the first cycle, a data stream load statistic value of the data stream in the multi-cycle period is determined according to the total amount of data, the number of routing records and the number of periods in the multi-cycle period.

11. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Scan the data stream added by the data stream processing engine in the first cycle; Determine the number of default route records for the newly added data flow according to the default operator allocation strategy; According to the routing records in the first operator routing table, counting the number of routing records of each data writing operator on the data stream processing engine; From the data writing operators on the data stream processing engine, in order of priority in ascending order of the number of routing records, select a target data writing operator whose number is consistent with the number of the default routing records; In the second operator routing table of the second cycle, a target routing record indicating the target data write operator and the data flow is added.

12. The method according to any one of claims 1 to 7, characterized in that: According to the write performance characteristics, adjusting the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle includes: Traversing the data stream indicated by the routing record in the first operator routing table; In response to the traversed data stream being in a locked state, continuing to traverse the next data stream; In response to the traversed data stream being in an unlocked state, adjusting a routing record for the traversed data stream according to the write performance characteristic; After the traversal is completed, the second operator routing table of the second cycle is obtained.

13. A data stream writing control method, characterized in that: The method comprises: Obtaining a first operator routing table of a first cycle, wherein the first operator routing table includes at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator; Using the data writing operator indicated by each of the routing records, the data writing operator writes the data stream to be written corresponding to the data writing operator into a file corresponding to the data writing operator and the data stream to be written; For each data stream to be written, obtaining a write performance characteristic of writing by at least one data writing operator in the first cycle; The write performance characteristic is sent, and the sent write performance characteristic is used to indicate adjusting the routing records in the first operator routing table according to the write performance characteristic to obtain the second operator routing table of the second cycle period, and the write performance characteristic represents the write performance of writing each data stream in the first cycle period.

14. The method according to claim 13, characterized in that The obtaining of the first operator routing table of the first cycle includes: Using the data source operator, receiving a first operator routing table of a first cycle from the control node; The method further comprises: Using the data source operator, read the data segments of each data stream from the cache, and distribute the data segments to each data write operator according to the routing records in the first operator routing table; The write performance characteristics are acquired and sent by the data write operator.

15. A data stream writing control system, characterized in that: The system includes a control node, a data stream processing engine, a file system and a cache, and the data stream processing engine includes a data writing operator and a data source operator; The data source operator is used to receive a first operator routing table of a first cycle from a control node, wherein the first operator routing table includes at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator; read data segments of each data stream from the cache, and distribute the data segments to each data writing operator according to the routing records in the operator routing table; wherein the write performance characteristics are obtained and sent by the data writing operator; The data writing operator is used to write the data stream to be written corresponding to the data writing operator indicated by the routing record into the file system, the file corresponding to the data writing operator and the data stream to be written; obtain the writing performance characteristics of writing by the data writing operator in the first cycle; and send the writing performance characteristics to the control node; The control node is used to receive the write performance characteristics of the data write operator in the first cycle, the write performance characteristics characterizing the write performance of the data stream processing engine in writing each data stream in the first cycle; according to the write performance characteristics, adjust the first operator routing table of the first cycle to obtain the second operator routing table of the second cycle, the first operator routing table contains routing records in the first cycle, the second operator routing table contains routing records in the second cycle, each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data write operator; send the second operator routing table to the data source operator, the operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle to write the data stream to be written corresponding to the data write operator into the file system, the data write operator and the file corresponding to the data stream to be written.

16. A data stream writing control device, characterized in that: The device comprises: A receiving module, configured to receive a write performance characteristic of a data stream processing engine in a first cycle, wherein the write performance characteristic represents a write performance of the data stream processing engine in writing each data stream in the first cycle; A first execution module, configured to adjust a first operator routing table of a first cycle according to the write performance characteristics, and obtain a second operator routing table of a second cycle, wherein the first operator routing table includes routing records within the first cycle, and the second operator routing table includes routing records within the second cycle, and each routing record indicates a data stream to be written in the data stream processing engine and a corresponding data writing operator; A sending module is used to send the second operator routing table to the data stream processing engine, where the second operator routing table is used to instruct the data stream processing engine to use the data write operator indicated by each routing record in the second operator routing table in the second cycle period to write the data into the data stream to be written corresponding to the operator, and write the data into the file corresponding to the operator and the data stream to be written.

17. A data stream writing control device, characterized in that: The device comprises: A first acquisition module, used to acquire a first operator routing table of a first cycle, wherein the first operator routing table includes at least one routing record, each routing record indicating a data stream to be written and a corresponding data writing operator; A second execution module is used to use the data writing operator indicated by each routing record to write the data into the data stream to be written corresponding to the data writing operator, and write the data into the file corresponding to the data writing operator and the data stream to be written; A second acquisition module, configured to acquire, for each data stream to be written, a write performance characteristic of writing by at least one data writing operator in the first cycle; A sending module is used to send the write performance characteristics, and the sent write performance characteristics are used to indicate adjusting the routing records in the first operator routing table according to the write performance characteristics to obtain the second operator routing table of the second cycle period, and the write performance characteristics represent the write performance of writing each data stream in the first cycle period.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images