Data processing method, system, equipment and medium

By aggregating real-time stream data according to feature values ​​and adjusting the processing strategy according to the status of the processing node, the asynchronous errors and high cost problems of data processing under high concurrency are solved, and efficient data processing and cost reduction are achieved.

CN119088814APending Publication Date: 2024-12-06BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411213060.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the highly concurrent data processing mode, the read and write competition of multiple processing nodes on the database leads to asynchronous errors and increases computing costs.

Method used

By obtaining the eigenvalues ​​of the real-time stream data, the flow data of the same eigenvalue is allocated to the same processing node, and the processing strategy is determined based on the business status of the processing node, including adjusting the window time and update frequency to avoid data conflicts and reduce computing costs.

Benefits of technology

It effectively avoids database operation conflicts between different processing nodes on the same stream data, realizes efficient data processing, and reduces the increase in computing costs brought by traditional high concurrency methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088814A_ABST
    Figure CN119088814A_ABST
Patent Text Reader

Abstract

The invention relates to a data processing method, system and device and a medium, and the method comprises the steps: obtaining real-time streaming data, determining a feature value corresponding to the streaming data, and distributing the streaming data with the same feature value to the same processing node; for the streaming data in any processing node, determining a processing strategy for processing the streaming data according to the service state of the processing node; and processing the streaming data in the processing node according to the processing strategy to obtain a processing result of the streaming data, and updating the processing result to a database. And the conflict phenomenon of database operation of the same stream data by different processing nodes is avoided. And meanwhile, the processing strategy of the streaming data is determined according to the service state of the processing node, so that efficient processing of the streaming data can be realized, and the traditional increase of calculation cost for realizing high concurrency is further effectively reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method, system, device and medium. Background Art

[0002] With the development of the Internet, network data has grown explosively. For a network object, there may be tens of thousands of users using it at the same time, which will generate a large amount of streaming data for the network object at the same time. In order to process these operations with low latency, a high-concurrency method is usually used to update the corresponding operations in these streaming data to the database. However, the high-concurrency method will distribute these streaming data to different computing nodes to process the corresponding operations, which will inevitably cause asynchronous errors due to competition between multiple processing nodes in reading and writing network objects in the database. That is, the processing results of multiple processing nodes are directly updated to the database. If the types of processing results are the same, it is impossible to effectively merge multiple processing nodes, resulting in data loss; in addition, the use of a high-concurrency method will increase the burden on processing nodes and databases, exacerbating the horizontal expansion of servers and increasing costs.

[0003] Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0004] The present application provides a data processing method to solve the problems of data processing errors and excessively high data processing costs in traditional data processing methods.

[0005] The first aspect of the present application provides a data processing method, comprising:

[0006] Acquire real-time stream data, determine characteristic values ​​corresponding to the stream data, and assign stream data with the same characteristic values ​​to the same processing node;

[0007] For the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node;

[0008] The stream data in the processing node is processed according to the processing strategy to obtain a processing result of the stream data, and the processing result is updated to a database.

[0009] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0010] Determining a preset window time for periodically processing the stream data;

[0011] Calculating a growth rate corresponding to any of the stream data received by the processing node within an adjacent window time, and determining whether the growth rate exceeds a first predetermined value;

[0012] If the growth rate exceeds a first predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time;

[0013] If the growth rate does not exceed the first predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0014] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0015] Determining a preset window time for periodically processing the stream data;

[0016] Calculate the comprehensive growth rate corresponding to all flow data received by the processing node within the adjacent window time, and determine whether the comprehensive growth rate exceeds a second predetermined value;

[0017] If the growth rate exceeds a second predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time;

[0018] If the growth rate does not exceed the second predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0019] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0020] Determining a load rate of the processing node, and determining whether the load rate exceeds a third predetermined value;

[0021] If the load rate exceeds the third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0022] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0023] Determine the load rate of the node where the database corresponding to the stream data is located, and determine whether the load rate exceeds a fourth predetermined value;

[0024] If the load rate exceeds a fourth predetermined value, the processing strategy is determined to be a third processing strategy, and the third processing strategy includes increasing the window time to a second predetermined time.

[0025] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0026] If the load rate of the node where the database is located exceeds a fourth predetermined value, obtaining the load rate of the processing node, and determining whether the load rate of the processing node exceeds a third predetermined value;

[0027] If the load rate of the processing node exceeds the third predetermined value, the processing strategy is determined to be a fourth processing strategy, and the fourth processing strategy includes reducing the window time to the first predetermined time and setting the interval time for updating the processing result to the database to a predetermined multiple of the window time.

[0028] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0029] If the load rate of the node where the database is located exceeds a fourth predetermined value, the load rates of all processing nodes are obtained, and the processing nodes whose load rates exceed the third predetermined value are identified;

[0030] For the processing node whose load rate exceeds a third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0031] The second aspect of the present application also provides a data processing system, comprising:

[0032] A data distribution module is used to obtain real-time stream data, determine the characteristic values ​​corresponding to the stream data, and distribute the stream data with the same characteristic values ​​to the same processing node;

[0033] A strategy formulation module, used to formulate a processing strategy for the flow data assigned to the same processing node according to the business status of the processing node and the corresponding database;

[0034] The data processing module is used to execute the processing strategy to update the processing results of the flow data assigned to the same processing node to the database.

[0035] The third aspect of the present application also proposes a computer device: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to execute the data processing method described in any one of the above embodiments.

[0036] The fourth aspect of the present application further proposes a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the data processing method described in any one of the above embodiments.

[0037] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application aggregates the corresponding stream data according to the characteristic value and then sends it to the same processing node, so that the stream data operating on the same network object will not appear in different processing nodes, thereby avoiding the conflict of database operations on the same stream data by different processing nodes. At the same time, the processing strategy of the stream data is determined according to the business status of the processing node, which can realize efficient processing of the stream data, thereby effectively reducing the increase in traditional computing costs for achieving high concurrency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0041] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a process for determining a processing strategy based on individual stream data provided in an embodiment of the present application;

[0043] Figure 3A schematic diagram of a process for determining a processing strategy based on all stream data provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a process for determining a processing strategy according to a load status of a processing node provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a process for determining a processing strategy according to a database load state provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a process for determining a processing strategy according to a database load state and a processing node load state provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a flow chart for determining a processing strategy according to a database load state and a load state of some processing nodes provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0052] In order to solve the problems of data update errors and high costs in traditional implementation methods, the present application provides a data processing method that can aggregate data of the same object and then distribute it to processing nodes for concurrent processing for batch processing, effectively solving data errors caused by competition in traditional distributed solutions, and corresponding solutions formulated according to the business status of the processing nodes.

[0053] like Figure 1 As shown, the first aspect of the present application proposes a data processing method, comprising:

[0054] Step S1, acquiring real-time stream data, determining characteristic values ​​corresponding to the stream data, and assigning stream data with the same characteristic value to the same processing node;

[0055] Step S2: for the stream data in any of the processing nodes, determine a processing strategy for processing the stream data according to the service status of the processing node;

[0056] Step S3: Process the stream data in the processing node according to the processing strategy to obtain a processing result of the stream data, and update the processing result to a database.

[0057] In the embodiments of the present application, stream data refers to data that is operated on a network object. For example, the network object can be a video or an article. When a user likes a video or an article (including text content such as comments and bullet screen), corresponding data will be generated. For the server, when the number of likes by users is large, a corresponding data stream will be generated, and each data in the data stream is the corresponding stream data;

[0058] The characteristic value of the flow data refers to the unique identifier of the corresponding network object in a system, which is the data assigned by the corresponding system to the network object for uniquely identifying the network object;

[0059] A processing node is a computing unit that processes the flow data of a network object. It can be a computer or a process within a computer, or some virtualized computing unit, such as a pod or a virtual machine.

[0060] A processing strategy refers to a solution used on a processing node to control the processing behavior of the processing node on stream data. The processing method for stream data is a processing logic pre-implemented according to the type of stream data. The processing logic is different for stream data corresponding to different businesses.

[0061] In some embodiments of the present application, the business status of the processing node includes the load status of the processing node, the cache status of the processing node cached flow data, the number of all flow data assigned to the processing node, the number of each flow data assigned to the processing node, and other relevant data belonging to the processing node.

[0062] In step S1, stream data is received and the characteristic value of the network object corresponding to the stream data is determined. Then, according to the mapping relationship between the pre-assigned characteristic value and the corresponding processing node, the stream data with the same characteristic value is sent to the corresponding processing node, so that the stream data of the same network object is assigned to the same processing node.

[0063] In some embodiments of the present application, the mapping relationship between the characteristic value of the flow data and the processing node can be dynamically determined, that is, without affecting the update of the processing result of the previously allocated flow data, the characteristic value of the flow data allocated to the corresponding processing node can be dynamically determined according to the state of the corresponding processing node. After the corresponding mapping relationship is re-determined, the corresponding flow data will no longer be allocated to the corresponding processing node, but will be allocated to the corresponding other processing nodes. For example, assuming that there are 100 network objects and 10 processing nodes, each node is allocated with flow data of 10 networks. Then establish a mapping relationship table corresponding to the characteristic values ​​of 100 network objects and 10 processing nodes. If the data corresponding to the 10 network objects allocated by a certain node is obviously too high or above the preset reference value, resulting in a high load on the corresponding processing node, while the number of flow data of other processing nodes is very low or below the preset reference value, then 9 of the 10 network objects can be evenly redistributed to the other 9 processing nodes, and one network object is selected from the other 9 processing nodes to be allocated to the processing node.

[0064] In step S2, after the corresponding stream data is assigned to the processing node, the processing node does not immediately process the stream data, but periodically processes the corresponding stream data according to the preset processing strategy and the preset processing logic of the stream data. The processing strategy includes the window time for processing the stream data. For example, when the business state of the processing node is always under high load, the window time corresponding to the processing strategy is shortened; when the processing node is assigned a large amount of stream data, the corresponding processing strategy can shorten the corresponding window time to quickly process the received stream data.

[0065] In some embodiments of the present application, the processing strategy also includes a control strategy for updating the processing results to the database. When the load of the database corresponding to the stream data processed in the processing node is too high, the operation behavior of updating the processing results to the database can be reduced according to preset conditions.

[0066] In step S3, according to the processing strategy determined in step S2, the data that meets the window time is batch processed at any processing node for the stream data assigned to the node, and the processing results are updated to the corresponding database in a timely manner. If there is a preset control strategy for the behavior of updating the processing results to the database, the processing results are updated to the database according to the corresponding control strategy.

[0067] In a typical embodiment of the present application, the present application implements the diversion and processing of stream data through the Flink tool. Specifically, the stream data from Kafka is obtained through the Flink interface. Then, the received stream data is cleaned and converted to obtain stream data in a predetermined data format. Further, through the KeyBy component of Flink, the cleaned stream data is grouped according to the characteristic value of the network object corresponding to the data, that is, the corresponding Key value, and the grouped data is automatically distributed to the corresponding processing node.

[0068] Furthermore, at the processing node, the window time corresponding to the processing strategy is combined with the window component in the Flink framework to allocate a corresponding window time, for example, 10 seconds, to each processing node. That is, every 10 seconds, the processing node processes the stream data allocated in these 10 seconds according to the processing program pre-set for processing the stream data.

[0069] Furthermore, after processing the stream data allocated to the 10 seconds, the current value of the corresponding stream data in the database is obtained, and after merging the processing result with the current value, the merged result is updated to the corresponding database.

[0070] Furthermore, in some cases, when the processing node processes the received stream data according to the window time, the received stream data outside the window time is temporarily cached. If the amount of cached data exceeds the preset reference value, the window time corresponding to the cached stream data will be appropriately reduced. For example, if the processing node is currently processing the stream data assigned to the last window time, and the stream data assigned to the next processing node exceeds the preset reference value, then in this case, the corresponding window time is reduced by 50%. That is, if the window time is 10 seconds in the previous example, the window time is changed to 5 seconds at this time. The cached data can be processed in time to prevent the cache of the stream data from being penetrated by the rapidly increasing stream data when hot data appears.

[0071] like Figure 2 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0072] Step S21, determining a preset window time for periodically processing the stream data;

[0073] Step S22, calculating the growth rate corresponding to any of the flow data received by the processing node within the adjacent window time, and determining whether the growth rate exceeds a first predetermined value;

[0074] Step S23: if the growth rate exceeds a first predetermined value, determining that the processing strategy is a first processing strategy, wherein the first processing strategy includes reducing the window time to a first predetermined time;

[0075] Step S24: If the growth rate does not exceed the first predetermined value, determine that the processing strategy is a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0076] In some embodiments of the present application, when allocating stream data, when allocating stream data, the number of corresponding stream data allocated to each processing node within the window time is counted according to the preset window time, and the number of stream data corresponding to the corresponding characteristic value allocated to each processing node in each window time is calculated, and the growth rate of the stream data corresponding to each characteristic value of adjacent window times is further calculated. When the growth rate corresponding to a certain characteristic value exceeds a first predetermined value, the processing strategy corresponding to the processing node is changed, specifically, the predetermined window time is modified to the first predetermined time.

[0077] When the growth rate is lower than the first predetermined value, the corresponding window time is restored to the preset window time or the default window time.

[0078] In some embodiments of the present application, the first predetermined time can be set based on experience, or it can be set based on the processing capacity of the corresponding processing node when processing the corresponding stream data, that is, the time to cache a certain amount of stream data is equal to the time to process a certain amount of stream data. This is to achieve the ultimate balance between the cached stream data and the stream data being processed. This can effectively deal with the phenomenon of some hot data emerging. Timely data feedback can be given to users at the user level, allowing users to promptly feel the changes in the content they are concerned about, thereby enhancing the user's viewing experience.

[0079] like Figure 3 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0080] Step S31, determining a preset window time for periodically processing the stream data;

[0081] Step S32: Calculate the comprehensive growth rate corresponding to all the flow data received by the processing node within the adjacent window time, and determine whether the comprehensive growth rate exceeds a second predetermined value;

[0082] Step S33: if the growth rate exceeds a second predetermined value, determining that the processing strategy is a first processing strategy, wherein the first processing strategy includes reducing the window time to a first predetermined time;

[0083] Step S34: If the growth rate does not exceed the second predetermined value, determine that the processing strategy is the second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0084] In this embodiment, the number of all flow data assigned to each processing node is counted with the preset window time as the time interval, and the number of all flow data counted in the adjacent time window is combined. Further, the growth rate of all flow data assigned to the processing node within the time range of this adjacent time window is calculated. If the growth rate exceeds the preset second predetermined value, the corresponding window time is reduced to the first predetermined time in the processing strategy for the processing node, for example, the original window time is changed from 10 seconds to 5 seconds. If the calculated growth rate is lower than the second predetermined value, the window time is set to the default preset time, that is, if the preset window time is 10 seconds and the current window time is 5 seconds, the next window time is set to 5 seconds; if the current window time is 10 seconds, the window time is kept unchanged.

[0085] like Figure 4 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0086] Step S41, determining the load rate of the processing node, and judging whether the load rate exceeds a third predetermined value;

[0087] Step S42: If the load rate exceeds the third predetermined value, determine that the processing strategy is the first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0088] In this embodiment, for each processing node, the load rate of the corresponding processing node is detected in real time. When the load rate of the corresponding processing node exceeds the preset third predetermined value, it is necessary to adjust the processing strategy of the processing node. The adjustment of the processing strategy mainly adjusts the window time for processing the received stream data, that is, reduces the window time for processing the stream data. For example, the window time is reduced from the original 10 seconds to 5 seconds.

[0089] In some embodiments of the present application, the load rate includes status data of different dimensions related to CPU usage, memory usage, disk usage, and network bandwidth usage, which are used to characterize the usage status of the processing node. For status data of different dimensions, the same third predetermined value, that is, the same ratio, can be set. Of course, specific values ​​corresponding to the dimensions of the corresponding dimensions can also be set separately for status data of different dimensions. For example, the third predetermined value of the CPU usage rate is 80%, the third predetermined value of the memory usage rate is 90%, the third predetermined value corresponding to the disk usage rate is 70%, and the network bandwidth usage rate is 85%. The above load rate is only an example reference, and the third predetermined value of the effective load rate needs to be determined after debugging according to the specific scenario. In this way, the processing strategy for processing flow data of the processing node can be accurately set according to the usage status of multiple dimensions, to prevent the emergence of the "barrel short board effect" from affecting the normal operation of the processing node, and to improve the business stability of the processing node.

[0090] like Figure 5 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0091] Step S51, determining the load rate of the node where the database corresponding to the stream data is located, and judging whether the load rate exceeds a fourth predetermined value;

[0092] Step S52: If the load rate exceeds a fourth predetermined value, determine that the processing strategy is a third processing strategy, and the third processing strategy includes increasing the window time to a second predetermined time.

[0093] In this embodiment, when determining the processing strategy for the processing node, the business status of the database corresponding to the stream data also needs to be considered. Because the processing results of the stream data need to be updated to the database for storage, if the update operation on the database is frequent, it is bound to increase the business burden of the database. Moreover, it is not only the processing node that accesses and operates the database, other businesses or services that provide data display services to users will access the business database more frequently.

[0094] Therefore, it is necessary to first determine the database corresponding to the stream data in the processing node and obtain the load rate of the database. If the load rate exceeds the fourth predetermined value set in advance, the processing strategy of the corresponding processing node needs to extend the window time to the second predetermined time. That is, as in the above example, assuming that the default window time is 10 seconds, the window time can be correspondingly increased to a longer window time such as 15 seconds or 20 seconds.

[0095] Furthermore, in some embodiments of the present application, the load rate of the database corresponding to the stream data can be obtained by communicating with the node of the corresponding database, that is, the load rate of the node deployed by the database is obtained, and the load rate of the node is regarded as the load rate of the database.

[0096] Furthermore, in some embodiments of the present application, the load rate of the database corresponding to the streaming data can also be determined based on the amount of streaming data, that is, the load rate of the corresponding database can be represented by the amount of streaming data of a certain characteristic value. This is because the generation of streaming data is actively or passively generated by the user in response to the content seen, and the corresponding content description seen by the user is obtained by the corresponding content platform, which displays the content in the database to the user. When the corresponding streaming data generated actively or passively by the user's viewing of the content increases, it reflects from the side that the corresponding platform will also have more access behaviors or operations corresponding to the network objects corresponding to the streaming data obtained from the database at this time. Therefore, the load rate of the corresponding database can be reflected to a certain extent based on the amount of streaming data.

[0097] Further, in some embodiments of the present application, the load rate of the database is determined according to the amount of stream data, and the correlation between the amount of a large amount of stream data and the real-time load rate of the corresponding database can be collected within a certain range to determine the corresponding proportional relationship. In subsequent applications, the load rate of the corresponding database is determined according to the amount of stream data and the corresponding proportional relationship. In this way, there is no need to communicate with the corresponding database or detect the load rate of the database in real time, and the corresponding processing strategy can be determined, thereby reducing the operation behavior of the processing node on the database.

[0098] like Figure 6 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0099] Step S61: if the load rate of the node where the database is located exceeds a fourth predetermined value, the load rate of the processing node is obtained, and it is determined whether the load rate of the processing node exceeds a third predetermined value;

[0100] Step S62: If the load rate of the processing node exceeds the third predetermined value, determine that the processing strategy is the fourth processing strategy, and the fourth processing strategy includes reducing the window time to the first predetermined time, and setting the interval time for updating the processing result to the database to a predetermined multiple of the window time.

[0101] In this embodiment, if the load rate of the node where the database corresponding to a certain stream data is located exceeds the fourth predetermined value, the load rate of the processing node where the stream data is located is obtained. Then it is determined whether the load rate of the processing node exceeds the third predetermined value. If the load rate of the processing node also exceeds the third predetermined value. When determining the processing strategy of the processing node, on the one hand, the window time for the processing node to process the stream data is reduced to the first predetermined time, and on the other hand, according to the value of the first predetermined time, the access to the database of the processing node to update the processing result to the database is extended, and the extended time is a predetermined multiple of the first predetermined time. Specifically, if the load rate of the database exceeds 90%, and the load rate of the processing node also exceeds 80%, the processing node and the database are both in a busy state. In this case, for the processing node, the corresponding window time can be reduced to 5 seconds, and the time for updating the processing result to the database is increased from 5 seconds (by default, it will be updated to the corresponding database immediately after the stream data is processed according to the window time) to 25 seconds.

[0102] It should be noted that, in the case of the above processing strategy, when the processing results are not updated to the database, the processing results corresponding to multiple window times can be merged, and when the time reaches the time to update the database, the merged results of the processing results in multiple window times are updated to the database. In this process, the processing results of the previous window time need to be saved, and when the corresponding processing results of the next window time are generated, they are directly merged and then saved until the time to update the database is met, and then the merged results are updated to the corresponding database.

[0103] like Figure 7 As shown, in some embodiments of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0104] Step S71: if the load rate of the node where the database is located exceeds the fourth predetermined value, the load rates of all processing nodes are obtained, and the processing nodes whose load rates exceed the third predetermined value are identified;

[0105] Step S72: for the processing node whose load rate exceeds the third predetermined value, determine that the processing strategy is the first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0106] In this embodiment, when the load rate of the node where the database is located exceeds the fourth predetermined value, the load rate of all processing nodes is further determined, and the processing nodes whose load rate exceeds the third predetermined value are screened out. The corresponding processing strategy is further determined for the stage where the load rate exceeds the third predetermined value, that is, the window time of the processing node whose load rate exceeds the third predetermined value is reduced to the first predetermined time. For example, if only 3 of the 10 processing nodes have a load rate exceeding 90%, and the load rate of the node where the corresponding database is located exceeds 80% at this time. Then only the window time of these 3 processing nodes can be reduced from the default 10 seconds to 5 seconds, and the window time of the other 7 processing nodes is still 10 seconds. In this way, only the frequency of accessing the database by 3 processing nodes increases, and the access frequency of the other 7 processing nodes remains unchanged. It is possible to balance the business status of the database and the business status of the processing nodes.

[0107] It should be noted that the database in this embodiment is a database associated with the flow data in the corresponding processing nodes, that is, the above 10 processing nodes have corresponding flow data to be stored in the database.

[0108] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application aggregates the corresponding stream data according to the characteristic value and then sends it to the same processing node, so that the stream data operating on the same network object will not appear in different processing nodes, thereby avoiding the conflict of database operations on the same stream data by different processing nodes. At the same time, the processing strategy of the stream data is determined according to the business status of the processing node, which can realize efficient processing of the stream data, thereby effectively reducing the increase in traditional computing costs for achieving high concurrency.

[0109] Furthermore, according to the relationship between the processing nodes and the database, balancing the access frequency of one or more processing nodes and the database can effectively reduce the number of communication behaviors between the processing nodes and the database, and fully release the processing capacity of the database while ensuring data reliability.

[0110] like Figure 8 As shown, the second aspect of the present application also proposes a data processing system, comprising:

[0111] Data distribution module 1, used to obtain real-time stream data, determine the characteristic values ​​corresponding to the stream data, and distribute the stream data with the same characteristic value to the same processing node;

[0112] A strategy formulation module 2, for formulating a processing strategy for the flow data assigned to the same processing node according to the business status of the processing node and the corresponding database;

[0113] The data processing module 3 is used to execute the processing strategy to update the processing results of the flow data assigned to the same processing node to the database.

[0114] like Fig. 9 As shown, the embodiment of the present application provides a computer device, including a processor 911, a communication interface 912, a memory 913 and a communication bus 914, wherein the processor 911, the communication interface 912, and the memory 913 communicate with each other through the communication bus 914.

[0115] Memory 913, used for storing computer programs;

[0116] In one embodiment of the present application, the processor 911 is used to execute the program stored in the memory 913 to implement any one of the above method embodiments to provide a data processing method, including:

[0117] Acquire real-time stream data, determine characteristic values ​​corresponding to the stream data, and assign stream data with the same characteristic values ​​to the same processing node;

[0118] For the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node;

[0119] The stream data in the processing node is processed according to the processing strategy to obtain a processing result of the stream data, and the processing result is updated to a database.

[0120] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0121] Determining a preset window time for periodically processing the stream data;

[0122] Calculating a growth rate corresponding to any of the stream data received by the processing node within an adjacent window time, and determining whether the growth rate exceeds a first predetermined value;

[0123] If the growth rate exceeds a first predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time;

[0124] If the growth rate does not exceed the first predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0125] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0126] Determining a preset window time for periodically processing the stream data;

[0127] Calculate the comprehensive growth rate corresponding to all flow data received by the processing node within the adjacent window time, and determine whether the comprehensive growth rate exceeds a second predetermined value;

[0128] If the growth rate exceeds a second predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time;

[0129] If the growth rate does not exceed the second predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

[0130] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0131] Determining a load rate of the processing node, and determining whether the load rate exceeds a third predetermined value;

[0132] If the load rate exceeds the third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0133] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0134] Determine the load rate of the node where the database corresponding to the stream data is located, and determine whether the load rate exceeds a fourth predetermined value;

[0135] If the load rate exceeds a fourth predetermined value, the processing strategy is determined to be a third processing strategy, and the third processing strategy includes increasing the window time to a second predetermined time.

[0136] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0137] If the load rate of the node where the database is located exceeds a fourth predetermined value, obtaining the load rate of the processing node, and determining whether the load rate of the processing node exceeds a third predetermined value;

[0138] If the load rate of the processing node exceeds the third predetermined value, the processing strategy is determined to be a fourth processing strategy, and the fourth processing strategy includes reducing the window time to the first predetermined time and setting the interval time for updating the processing result to the database to a predetermined multiple of the window time.

[0139] In some implementations of the present application, for the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node includes:

[0140] If the load rate of the node where the database is located exceeds a fourth predetermined value, the load rates of all processing nodes are obtained, and the processing nodes whose load rates exceed the third predetermined value are identified;

[0141] For the processing node whose load rate exceeds the third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

[0142] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data processing method provided by any one of the aforementioned method embodiments is implemented.

[0143] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0146] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that: include: Acquire real-time stream data, determine characteristic values ​​corresponding to the stream data, and assign stream data with the same characteristic values ​​to the same processing node; For the stream data in any of the processing nodes, determining a processing strategy for processing the stream data according to the service status of the processing node; The stream data in the processing node is processed according to the processing strategy to obtain a processing result of the stream data, and the processing result is updated to a database.

2. The method according to claim 1, characterized in that: The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: Determining a preset window time for periodically processing the stream data; Calculating a growth rate corresponding to any of the stream data received by the processing node within an adjacent window time, and determining whether the growth rate exceeds a first predetermined value; If the growth rate exceeds a first predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time; If the growth rate does not exceed the first predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

3. The method according to claim 1, characterized in that The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: Determining a preset window time for periodically processing the stream data; Calculate the comprehensive growth rate corresponding to all flow data received by the processing node within the adjacent window time, and determine whether the comprehensive growth rate exceeds a second predetermined value; If the growth rate exceeds a second predetermined value, determining that the processing strategy is a first processing strategy, the first processing strategy includes reducing the window time to a first predetermined time; If the growth rate does not exceed the second predetermined value, the processing strategy is determined to be a second processing strategy, and the second processing strategy includes setting the window time to a preset value.

4. The method according to claim 1, characterized in that: The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: Determining a load rate of the processing node, and determining whether the load rate exceeds a third predetermined value; If the load rate exceeds the third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

5. The method according to claim 1, characterized in that The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: Determine the load rate of the node where the database corresponding to the stream data is located, and determine whether the load rate exceeds a fourth predetermined value; If the load rate exceeds a fourth predetermined value, the processing strategy is determined to be a third processing strategy, and the third processing strategy includes increasing the window time to a second predetermined time.

6. The method according to claim 5, characterized in that The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: If the load rate of the node where the database is located exceeds a fourth predetermined value, obtaining the load rate of the processing node, and determining whether the load rate of the processing node exceeds a third predetermined value; If the load rate of the processing node exceeds the third predetermined value, the processing strategy is determined to be a fourth processing strategy, and the fourth processing strategy includes reducing the window time to the first predetermined time and setting the interval time for updating the processing result to the database to a predetermined multiple of the window time.

7. The method according to claim 5, characterized in that The step of determining a processing strategy for processing the stream data in any of the processing nodes according to a service state of the processing node includes: If the load rate of the node where the database is located exceeds a fourth predetermined value, the load rates of all processing nodes are obtained, and the processing nodes whose load rates exceed the third predetermined value are identified; For the processing node whose load rate exceeds a third predetermined value, the processing strategy is determined to be a first processing strategy, and the first processing strategy includes reducing the window time to a first predetermined time.

8. A data processing system, characterized in that: include: A data distribution module is used to obtain real-time stream data, determine the characteristic values ​​corresponding to the stream data, and distribute the stream data with the same characteristic values ​​to the same processing node; A strategy formulation module, used to formulate a processing strategy for the flow data assigned to the same processing node according to the business status of the processing node and the corresponding database; The data processing module is used to execute the processing strategy to update the processing results of the flow data assigned to the same processing node to the database.

9. A computer device: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein: The processor is configured to execute the data processing method according to any one of claims 1 to 7.

10. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the data processing method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Data storage processing method and device, computer equipment and storage medium

    CN112000703A

  • Node resource management method and device

    CN113467956A

  • Data processing method and apparatus, storage medium, and program product

    WO2023273727A1