Data processing method, apparatus, medium, and electronic device

Abstract

The present disclosure relates to the technical field of data processing, and provides a data processing method, device, medium and electronic equipment. The method comprises: in response to a policy configuration instruction, obtaining a policy configuration parameter, and performing configuration legality verification on the policy configuration parameter; performing parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification to obtain a target format parameter; receiving to-be-processed data, and performing stream computing integration processing on the to-be-processed data and the target format parameter to obtain integrated data; performing stream aggregation calculation on the integrated data to obtain a stream aggregation result, and outputting the stream aggregation result. The present disclosure enables business users to not need to pay attention to the writing logic of data processing, greatly reduces the use difficulty and complexity of business users, fully reduces the occurrence of stream aggregation calculation delay, and guarantees the real-time performance and effectiveness of data processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a data processing method, a data processing apparatus, a computer readable medium and an electronic device. BACKGROUND

[0002] In the data processing scene of the Internet, the real-time requirement of data processing is gradually increasing. Generally, the Flink real-time stream computing framework is used to meet the requirement of real-time processing data.

[0003] However, the use of the Flink real-time stream computing framework requires business personnel to master programming languages such as Java or SQL, which cannot meet the use requirement of business users who lack programming ability.

[0004] Therefore, there is an urgent need in the art to develop a new data processing method and apparatus.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the technical background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a data processing method, a data processing apparatus, a computer readable medium and an electronic device, thereby at least partially overcoming the technical problems of poor applicability and low real-time performance.

[0007] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0008] According to an aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0009] In response to a policy configuration instruction, a policy configuration parameter is obtained, and a configuration legality verification is performed on the policy configuration parameter;

[0010] A parameter format conversion process is performed on the policy configuration parameter that passes the configuration legality verification to obtain a target format parameter;

[0011] The data to be processed is received, and a stream computing integration process is performed on the data to be processed and the target format parameter to obtain integrated data;

[0012] A stream aggregation calculation is performed on the integrated data to obtain a stream aggregation result, and the stream aggregation result is output.

[0013] According to an aspect of an embodiment of the present disclosure, a data processing apparatus is provided, the apparatus comprising:

[0014] The configuration verification module is configured to acquire a policy configuration parameter in response to a policy configuration instruction, and perform configuration legality verification on the policy configuration parameter.

[0015] The format conversion module is configured to perform parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification to obtain a target format parameter.

[0016] The data integration module is configured to receive to-be-processed data, and perform stream computing integration processing on the to-be-processed data and the target format parameter to obtain integrated data.

[0017] The aggregation computing module is configured to perform stream aggregation computing on the integrated data to obtain a stream aggregation result, and output the stream aggregation result.

[0018] In some embodiments of the present disclosure, based on the above technical solutions, the data integration module comprises a request receiving sub-module configured to receive a data processing request sent by a business data source, wherein the data processing request comprises a data attribute of the to-be-processed data.

[0019] The attribute matching sub-module is configured to perform receiving matching processing on the data attribute to obtain a receiving matching result, so as to receive the to-be-processed data according to the receiving matching result.

[0020] In some embodiments of the present disclosure, based on the above technical solutions, the data processing apparatus further comprises a result storage module configured to perform distributed storage processing on the stream aggregation result, so as to store the stream aggregation result in a distributed storage database.

[0021] In some embodiments of the present disclosure, based on the above technical solutions, the aggregation computing module comprises a result query sub-module configured to perform result query processing on the distributed storage database according to the data attribute, so as to determine the stream aggregation result from the distributed storage database.

[0022] The path query sub-module is configured to perform path query processing on the target format parameter to determine an output path of the stream aggregation result.

[0023] The result output sub-module is configured to output the stream aggregation result according to the output path.

[0024] In some embodiments of the present disclosure, based on the above technical solutions, the result output sub-module comprises an original path unit configured to output the stream aggregation result to the business data source according to an original path if the output path is the original path.

[0025] A designated path unit is configured to output the stream aggregation result according to the designated path if the output path is the designated path.

[0026] In some embodiments of the present disclosure, based on the above technical solution, the data processing apparatus further comprises: a duration statistical module configured to statistically calculate the duration of the stream aggregation calculation process to obtain the calculation duration of the stream aggregation calculation process.

[0027] A default output module is configured to output a default result corresponding to the target format parameter if the calculation duration is greater than the configured delay duration.

[0028] In some embodiments of the present disclosure, based on the above technical solution, the configuration verification module comprises: a rule identification submodule configured to perform rule identification processing on the policy configuration parameter to obtain a configuration parameter rule corresponding to the policy configuration parameter.

[0029] A rule verification submodule is configured to perform configuration legality verification on the policy configuration parameter by using the configuration parameter rule.

[0030] In some embodiments of the present disclosure, based on the above technical solution, the format conversion module comprises: a configuration storage submodule configured to store the policy configuration parameter that passes the configuration legality verification to save the policy configuration parameter that passes the configuration legality verification in a configuration management center.

[0031] A parameter conversion submodule is configured to perform parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification and is saved in the configuration management center to obtain a target format parameter.

[0032] In some embodiments of the present disclosure, based on the above technical solution, the data integration module comprises: a parameter identification submodule configured to perform configuration parameter identification processing on the target format parameter to determine an integration bit parameter in the target format parameter.

[0033] A calculation integration submodule is configured to perform stream calculation integration processing on the to-be-processed data and the target format parameter based on the integration bit parameter to obtain integrated data.

[0034] In some embodiments of the present disclosure, based on the above technical solution, the aggregation calculation module comprises: a cache processing submodule configured to perform data cache processing on the integrated data to store the integrated data in an integrated message queue corresponding to the integrated data.

[0035] The queue reading submodule is configured to read the integrated data from the integrated message queue and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0036] In some embodiments of the present disclosure, based on the above technical solution, the aggregation calculation module comprises a port configuration submodule configured to perform port configuration processing on the integrated data to store the integrated data in a port corresponding to the integrated data.

[0037] The port reading submodule is configured to read the integrated data from the port and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0038] In some embodiments of the present disclosure, based on the above technical solution, the aggregation calculation module comprises a file storage submodule configured to perform file storage processing on the integrated data to store the integrated data in a target file corresponding to the integrated data.

[0039] The file reading submodule is configured to read the integrated data from the target file and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0040] According to an aspect of an embodiment of the present disclosure, a computer readable medium having a computer program stored thereon is provided, and the computer program is executed by a processor to implement the data processing method in the above technical solution.

[0041] According to an aspect of an embodiment of the present disclosure, an electronic device is provided, which comprises a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the executable instructions to perform the data processing method in the above technical solution.

[0042] In the technical scheme provided in the embodiments of the present disclosure, the parameter configuration of the data processing mode can be realized through the policy configuration instruction, so that the business user does not need to pay attention to the writing logic of data processing, greatly reducing the use difficulty and complexity of the business user, enriching the use scenarios and universality of the data processing mode, and further, the configured policy configuration parameters are subjected to configuration legality verification, providing an automatic verification mode for the policy configuration parameters, avoiding the occurrence of human verification errors of the business user, and improving the accuracy and intelligent degree of the policy configuration parameters. On the other hand, the integrated data is obtained through the stream computing integration processing, which facilitates the unified and universal use of the integrated data, and improves the reusability of the integrated data. In addition, the stream aggregation calculation is performed on the integrated data, which sufficiently reduces the delay of the stream aggregation calculation, guarantees the real-time and effectiveness of the data processing, and provides strong support for the application scenarios of the security aggregation policy.

[0043] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0045] Figure 1 The architecture schematic diagram of an exemplary system to which the technical scheme of the present disclosure is applied is schematically shown;

[0046] Figure 2 The step flowchart of a data processing method in some embodiments of the present disclosure is schematically shown;

[0047] Figure 3 The step flowchart of a configuration legality verification method in some embodiments of the present disclosure is schematically shown;

[0048] Figure 4 The step flowchart of a parameter format conversion processing method in some embodiments of the present disclosure is schematically shown;

[0049] Figure 5 The step flowchart of a method for receiving data to be processed in some embodiments of the present disclosure is schematically shown;

[0050] Figure 6 The step flowchart of a stream computing integration processing method in some embodiments of the present disclosure is schematically shown;

[0051] Figure 7 An interface schematic diagram of some stream computing integration processes in the present disclosure is schematically shown;

[0052] Figure 8 A step flowchart of a first stream aggregation computing method in some embodiments of the present disclosure is schematically shown;

[0053] Figure 9 A step flowchart of a second stream aggregation computing method in some embodiments of the present disclosure is schematically shown;

[0054] Figure 10 A step flowchart of a third stream aggregation computing method in some embodiments of the present disclosure is schematically shown;

[0055] Figure 11 A step flowchart of a method of outputting stream aggregation results in some embodiments of the present disclosure is schematically shown;

[0056] Figure 12 A step flowchart of a method of outputting stream aggregation results according to an output path in some embodiments of the present disclosure is schematically shown;

[0057] Figure 13 A step flowchart of a method of outputting default results in some embodiments of the present disclosure is schematically shown;

[0058] Figure 14 A whole step flowchart of a data processing method in some embodiments of the present disclosure is schematically shown;

[0059] Figure 15 A system framework diagram of a data processing method in some embodiments of the present disclosure in an application scenario is schematically shown;

[0060] Figure 16 An interface schematic diagram of setting policy configuration parameters in some embodiments of the present disclosure in an application scenario is schematically shown;

[0061] Figure 17 A step flowchart of a stream aggregation computing method in some embodiments of the present disclosure in an application scenario is schematically shown;

[0062] Figure 18 An interface schematic diagram of a configuration situation of policy configuration parameters in some embodiments of the present disclosure in an application scenario is schematically shown;

[0063] Figure 19 A system architecture diagram of another data processing method in some embodiments of the present disclosure in an application scenario is schematically shown;

[0064] Figure 20 a structural block diagram of a data processing apparatus in some embodiments of the present disclosure is shown schematically;

[0065] Figure 21 a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0066] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0067] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the implementations of the disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, and operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

[0068] The block diagrams in the drawings show only the functional entities and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0069] The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily have to include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further broken down, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0070] In the related art in the field, in the application system of the Internet, the real-time requirement of the business scene for data processing is gradually increasing. Among them, the Flink real-time streaming computing framework has an absolute advantage in the streaming processing of data.

[0071] Among them, Apache Flink is a real-time stream computing framework and distributed processing engine, which is used for stateful computing on unbounded and bounded data streams, and Flink can be deployed in various cluster environments to quickly compute various sizes of data. Therefore, Flink is widely used in large-scale real-time data computing scenarios, and can process large-scale data with low latency and perform aggregation operation on data.

[0072] The unbounded data is a growing data set without boundaries. Such data cannot be determined when to terminate, and is also called stream data; and the bounded data is static data, which is commonly found in already saved data.

[0073] Generally, real-time processing of data is mainly realized through Flink SQL (Structured Query Language) or Java API (Application Programming Interface), which is not friendly to business users who lack programming ability and cannot meet the use requirements of business users. On the other hand, it requires business users to learn SQL, Java or Python language, which has a large learning cost and poor implementability.

[0074] Based on the problems existing in the above scheme, the present disclosure provides a new cloud technology-based data processing method, data processing device, computer readable medium and electronic equipment.

[0075] Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or local area network to realize data calculation, storage, processing and sharing.

[0076] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology and other technologies based on cloud computing business model application, which can form a resource pool and be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item may have its own identification mark in the future, and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system support, which can only be realized through cloud computing.

[0077] The cloud security refers to the security software, hardware, user, institution, and security cloud platform based on the cloud computing business model application. The cloud security combines emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior judgment, and through the abnormal monitoring of software behavior in the network by a large number of clients in a network, the latest information of the Trojan and malicious programs in the Internet is obtained and sent to the server for automatic analysis and processing, and the solutions of the viruses and Trojans are distributed to each client.

[0078] The main research directions of the cloud security include: 1. Cloud computing security, mainly studying how to protect the security of the cloud itself and various applications on the cloud, including cloud computer system security, secure storage and isolation of user data, user access authentication, information transmission security, network attack protection, compliance audit, etc.; 2. Cloudification of security infrastructure, mainly studying how to build and integrate security infrastructure resources by using cloud computing, optimize the security protection mechanism, including building a large-scale security event, information collection and processing platform through cloud computing technology, realizing the collection and correlation analysis of massive information, and improving the network security event control ability and risk control ability; 3. Cloud security service, mainly studying various security services provided by the cloud computing platform for users, such as antivirus service, etc.

[0079] The data processing method using the cloud security technology in the cloud technology makes the business users not need to pay attention to the writing logic of the data processing, greatly reduces the use difficulty and complexity of the business users, enriches the use scene and universality of the data processing method, provides an automatic verification method for the policy configuration parameters, avoids the error situation caused by the manual verification of the business users, and also improves the accuracy and intelligent degree of the policy configuration parameters. On the other hand, the delay of the stream aggregation calculation is greatly reduced, the real-time and effectiveness of the data processing are guaranteed, and strong support is provided for the application scene of the security aggregation policy.

[0080] Figure 1 An exemplary system architecture schematic diagram applying the technical solutions of the present disclosure is shown.

[0081] As shown in Figure 1 , the system architecture 100 can include a terminal 110, a network 120, and a server end 130. The terminal 110 and the server end 130 are connected through the network 120.

[0082] The terminal 110 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The network 120 can be a communication medium of various connection types capable of providing a communication link between the terminal 110 and the server end 130, such as a wired communication link, a wireless communication link, or an optical fiber cable, etc., which are not limited herein. The server 130 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0083] Specifically, the terminal 110 first acquires the policy configuration parameter in response to the policy configuration instruction, and performs configuration legality verification on the policy configuration parameter.

[0084] Then, the policy configuration parameter that passes the configuration legality verification is subjected to parameter format conversion processing to obtain a target format parameter.

[0085] Further, the to-be-processed data is received, and the to-be-processed data and the target format parameter are subjected to stream computing integration processing to obtain integrated data.

[0086] Finally, the integrated data is subjected to stream aggregation calculation to obtain a stream aggregation result, and the stream aggregation result is output.

[0087] In addition, the data processing method in the embodiments of the present disclosure can be applied to a terminal or a server end, and the present disclosure does not specially limit this. The embodiments of the present disclosure mainly take the data processing method applied to the terminal 110 as an example for illustration.

[0088] The data processing method, the data processing apparatus, the computer readable medium, and the electronic device provided by the present disclosure will be described in detail in combination with specific implementation manners.

[0089] Figure 2 The step flowchart of the data processing method in some embodiments of the present disclosure is schematically shown as Figure 2 As shown in the figure, the data processing method can mainly include the following steps:

[0090] Step S210. In response to a policy configuration instruction, a policy configuration parameter is acquired, and configuration legality verification is performed on the policy configuration parameter.

[0091] Step S220. The policy configuration parameter that passes the configuration legality verification is subjected to parameter format conversion processing to obtain a target format parameter.

[0092] Step S230. Receiving the data to be processed, and performing stream computing integration processing on the data to be processed and the target format parameter to obtain integrated data.

[0093] Step S240. Performing stream aggregation calculation on the integrated data to obtain stream aggregation results, and outputting the stream aggregation results.

[0094] In the example embodiments of the present disclosure, the parameter configuration of the data processing mode can be realized through the policy configuration instruction, so that the business user does not need to pay attention to the writing logic of the data processing, greatly reducing the use difficulty and complexity of the business user, enriching the use scenarios and universality of the data processing mode, and further, the configured policy configuration parameters are subjected to configuration legality verification, providing an automatic verification mode for the policy configuration parameters, avoiding the occurrence of human verification errors of the business user, and improving the accuracy and intelligent degree of the policy configuration parameters. On the other hand, the integrated data is obtained through stream computing integration processing, which facilitates the subsequent unified and universal use of the integrated data, and improves the reusability of the integrated data. In addition, the stream aggregation calculation is performed on the integrated data, which greatly reduces the occurrence of delay of the stream aggregation calculation, guarantees the real-time and effectiveness of the data processing, and provides strong support for the application scenarios of the security aggregation policy.

[0095] The steps of the data processing method will be described in detail below.

[0096] In step S210, in response to the policy configuration instruction, the policy configuration parameters are obtained, and the policy configuration parameters are subjected to configuration legality verification.

[0097] In the example embodiments of the present disclosure, the policy configuration instruction can be an instruction triggered by the business user through the front-end page of the terminal device, which is used to indicate the policy of subsequent stream aggregation calculation on the data to be processed.

[0098] The policy configuration instruction can include a configuration instruction of an input policy and a configuration instruction of an output policy.

[0099] For example, the input policy configuration instruction can be a configuration instruction of a time window, a configuration instruction of a time window duration setting, a configuration instruction of setting a grouping field, or a configuration instruction of a delay duration, etc., which is not specially limited in the example embodiments.

[0100] The output policy configuration instruction can be a configuration instruction of an output field, a configuration instruction of selecting different calculation functions, or a configuration instruction of setting an expression of a grouping field for stream aggregation calculation, etc.

[0101] Further, when the input policy configuration instruction is a configuration instruction of a time window, the window type configured by the business user can be acquired; when the policy configuration instruction is a configuration instruction of a time length setting of the time window, the window time length configured by the business user can be acquired; when the policy configuration instruction is a configuration instruction of setting a grouping field, the grouping field configured by the business user can be acquired; when the policy configuration instruction is a configuration instruction of a delay time length, the delay time length configured by the business user can be acquired, and the like.

[0102] When the output policy configuration instruction is a configuration instruction of an output field, the output field configured by the business user can be acquired; when the policy configuration instruction is a configuration instruction of selecting different calculation functions, the calculation function configured by the business user can be acquired; when the policy configuration instruction is a configuration instruction of setting an expression of a grouping field for performing a stream aggregation calculation, the expression configured by the business user can be acquired.

[0103] Therefore, the policy configuration parameters can include a window type, a window time length, a grouping field, and a delay time length, and can also include an output field, a calculation function, and an expression.

[0104] The window type can include two types of a rolling window and a sliding window.

[0105] Specifically, the rolling window can slice data according to a fixed window time length, has the characteristics of time alignment, fixed window length, and no overlap. That is, the rolling window allocates each data to a window of a specified window time length, the window has a fixed size, and the data of one window does not overlap / occur in other windows.

[0106] The sliding window is a more general form of the fixed window, which is composed of a fixed window time length and a sliding interval, has the characteristics of time alignment, fixed window length, and can have overlap. That is, the sliding window allocates data to a window of a fixed length, the size of the window is configured by the window time length, and another window sliding parameter controls the frequency at which the sliding window starts, so that the window can overlap if the sliding parameter is less than the window time length, that is, the data can be allocated to multiple windows. Obviously, the sliding window does not jump from one group of data to the next group of data, but slides through the input data stream, so that more smooth aggregation of data can be performed.

[0107] The window time length is a parameter for setting the window size of the rolling window or the sliding window.

[0108] The grouping field can be the field name set when grouping is needed according to a certain field of the data. For example, the grouping field can be field a or field b, and the like.

[0109] The delay duration can be set as the maximum allowed time for the streaming aggregation computation process to take, for example, it can be 80ms. This delay setting can ensure the real-time processing of data to be processed by different business processes.

[0110] The output field can be a representation field of the streaming aggregation result, such as AGG_ACOUNT.

[0111] The calculation function can be a function used to perform streaming aggregation calculations on the subsequently integrated data. For example, the calculation function can be a count function, a discount function, a max function, a min function, or a sum function.

[0112] Among them, the count function is a cumulative function used to count the number of records; the discount function is a deduplication function used to remove duplicate records; the max function is a maximum value function used to find the maximum value in the data; the min function is a minimum value function used to find the minimum value in the data; and the sun function is a summation function used to find the sum of all data.

[0113] An expression is a mathematical expression used to filter data retrieved based on a grouping field. When the grouping field is 'a', all data under the grouping field 'a' can be retrieved. However, if the expression is set to 'a>5', the data under the grouping field 'a' must also satisfy 'a>5' for subsequent streaming aggregation calculations to be performed.

[0114] After obtaining the policy configuration parameters, you can further verify the validity of the policy configuration parameters.

[0115] In an optional embodiment, Figure 3 A flowchart illustrating the steps of configuring a validity check method is shown, such as... Figure 3 As shown, the method includes at least the following steps: In step S310, the policy configuration parameters are processed by rule recognition to obtain the configuration parameter rules corresponding to the policy configuration parameters.

[0116] For different policy configuration parameters, corresponding configuration parameter rules can be preset. These configuration parameter rules are used to validate the format, type, or setting range of the corresponding policy configuration parameters.

[0117] For example, when the output field is AGG_ACOUNT, the corresponding field type must be uint8. Therefore, using uint8 as the field type is one of the configuration parameter rules for the output field. In addition, when the calculation function is a max or min function, the corresponding statistical field cannot be empty; this is another configuration parameter rule.

[0118] Therefore, the rule recognition processing can be performed on the policy configuration parameter and the preset configuration parameter rule in the case that the policy configuration parameter and the strategy configuration parameter rule are known, and the configuration parameter rule corresponding to the output field or the calculation function known at this time can be obtained.

[0119] In step S320, the policy configuration parameter is configured for the configuration legality verification by using the configuration parameter rule.

[0120] In the case that the configuration parameter rule is determined, the configuration legality verification can be performed on whether the policy configuration parameter meets the configuration parameter rule.

[0121] For example, when the output field is AGG_ACOUNT, the field type is required to be uint8, and when the subsequent obtained to-be-processed data is of the type uint8, it is determined that the policy configuration parameter meets the corresponding configuration parameter rule, that is, the configuration legality verification is passed. When the calculation function is the max function or the min function, the corresponding statistical field position is required to be not empty, but the business use unit is set to be empty, or the statistical field is empty due to the error setting or the non-setting, which indicates that the policy configuration parameter does not meet the corresponding configuration parameter rule, that is, the configuration legality verification is failed.

[0122] When the policy configuration parameter does not meet the corresponding configuration parameter rule, that is, the configuration legality verification is failed, a corresponding prompt information can be sent, so that the business user can perceive the policy configuration parameter that fails the configuration legality verification, and modify and update the policy configuration parameter, so as to ensure the smooth performance of the subsequent stream aggregation calculation.

[0123] In the example embodiment, the configuration legality verification is performed on the policy configuration parameter by using the configuration parameter rule, which ensures the setting accuracy of the policy configuration parameter by the business user, and the business user does not need to check and query the related rule by himself / herself, thereby reducing the labor cost and the verification time, and improving the accuracy and efficiency of the verification.

[0124] In step S220, the parameter format conversion processing is performed on the policy configuration parameter that passes the configuration legality verification, to obtain the target format parameter.

[0125] In the example embodiment of the present disclosure, after the configuration legality verification is ended, the configuration legality verification result that the policy configuration parameter passes the configuration legality verification or the policy configuration parameter fails the configuration legality verification can be obtained.

[0126] When the configuration legality verification result is that the policy configuration parameter passes the configuration legality verification, the parameter format conversion processing can be further performed on the policy configuration parameter that passes the configuration legality verification.

[0127] In the optional embodiment,Figure 4 A step flow chart of a method of parameter format conversion processing is shown as Figure 4 The method includes at least the following steps: in step S410, storing the policy configuration parameter that passes the configuration legality check to save the policy configuration parameter that passes the configuration legality check in the configuration management center.

[0128] Specifically, the policy configuration parameter that passes the configuration legality check can be stored in the configuration management center such as Zookeeper.

[0129] Zookeeper is a distributed and open-source coordination service designed for distributed applications. Zookeeper provides a basic service, namely a distributed lock service, and also provides a maintenance and management mechanism for distributed application data, including a unified naming service, a state synchronization service, cluster management, a distributed message queue, and management of distributed application configuration items.

[0130] In step S420, the policy configuration parameter that passes the configuration legality check and is saved in the configuration management center is subjected to parameter format conversion processing to obtain a target format parameter.

[0131] Specifically, the policy configuration parameter that passes the configuration legality check can be converted into a target format parameter in JSON (JavaScript Object Notation) format.

[0132] The JSON format refers to a string format obtained by converting a set of data in a JavaScript object, namely the policy configuration parameter. A file in JSON format can be transmitted between networks or programs, and can also be restored to a data format supported by various programming languages.

[0133] In this step, the policy configuration parameter is converted into a target format parameter in JSON format, and specifically, the parameter configuration parameter can be saved in the form of a field name and a value. When a max function is calculated for a grouping field a, it can be converted into the form of a-max.

[0134] Therefore, after the parameter format conversion processing of the policy configuration parameter, the target format parameter obtained can satisfy the data storage and query of the back end, and facilitates the reuse of the same configuration in the future.

[0135] In step S230, the to-be-processed data is received, and the to-be-processed data and the target format parameter are subjected to stream computing integration processing to obtain integrated data.

[0136] In the example embodiment of the present disclosure, after the configuration legality verification and parameter format conversion processing of the policy configuration parameter, the processed data capable of being processed according to the configured policy configuration parameter can be received.

[0137] In an optional embodiment, Figure 5 A step flowchart of a method for receiving processed data is shown in FIG. 5, and the method comprises at least the following steps: Figure 5 As shown in FIG. 5, the method comprises at least the following steps: in step S510, a data processing request sent by a service data source is received, and the data processing request comprises data attributes of processed data.

[0138] Specifically, when the processed data exists in the service data source, a data processing request can be sent to actively initiate a request and processing.

[0139] The data processing request carries information representing the data attributes of the processed data. The data attributes comprise data types and data names, etc. In addition, other data attributes can also be included, which are not specially limited in the example embodiment.

[0140] It is worth noting that the data processing request is actively sent by the service data source, actively requests to process the processed data, is more active, and improves the defects of complicated process and poor timeliness caused by the problem of passive acquisition of processed data by multiple subscriptions.

[0141] In step S520, the data attributes are received and matched to obtain a received matching result, so as to receive the processed data according to the received matching result.

[0142] When the data processing request of the processed data is received, different data receiving modules, such as access layers or access modules, are matched with different processed data of different services, so that the data attributes can be received and matched to determine the corresponding data receiving module according to the data types and data names, etc. so as to receive the processed data by using the data receiving module.

[0143] In the example embodiment, through the received matching processing, different data receiving modules can receive corresponding processed data, expand the service capability of data receiving, facilitate parallel expansion in the case of excessive request of the service data source, and mutual disaster recovery between the data receiving modules, so as to provide more secure high-availability services.

[0144] After the processed data is received, the processed data and the target format parameter can be processed by stream computing and integration.

[0145] In an optional embodiment, Figure 6 A step flowchart of a method for processing stream computing and integration is shown in FIG. 6, and the method comprises at least the following steps:Figure 6 As shown, the method at least includes the following steps: in step S610, performing configuration parameter identification processing on the target format parameter to determine the integration bit parameter in the target format parameter.

[0146] For example, the target format parameter can be

[0147]

[0148] The configuration parameter identification processing on the target format parameter can determine that the integration bit parameter in the target format parameter is agg_conf, and thus it can be determined that the position for performing stream computing integration processing on the to-be-processed data and the target format parameter is the row above agg_conf.

[0149] In addition, the integration bit parameter can also be set as window_type according to actual conditions, and in this case it can be determined that the position for performing stream computing integration processing on the to-be-processed data and the target format parameter is the row below window_type.

[0150] Obviously, the integration bit parameter is a flag bit indicating that the to-be-processed data and the target format parameter are subjected to stream computing integration processing.

[0151] In step S620, based on the integration bit parameter, the to-be-processed data and the target format parameter are subjected to stream computing integration processing to obtain integrated data.

[0152] After determining the integration bit parameter in the target format parameter, the to-be-processed data and the target format parameter are subjected to stream computing integration processing.

[0153] Specifically, the to-be-processed data is inserted into the target format parameter of the integration bit parameter as a whole, and a flag row of agg_data is added to obtain integrated data.

[0154] For example, Figure 7 An interface diagram of stream computing integration processing is shown, as Figure 7 As shown, the target format parameter is

[0155]

[0156]

[0157] The to-be-processed data is

[0158] [{

[0159] “name”:“A”,

[0160] “type”:“uint8”,

[0161] " value ": " 10 ",

[0162] },

[0163] {

[0164] " name ": " B ",

[0165] " type ": " uint8 ",

[0166] " value ": " 20 ",

[0167] } ]

[0169] }.

[0170] In the case that the integration bit parameter is agg_conf, it can be determined to insert the to-be-processed data into the previous line of the target format parameter; in the case that the integration bit parameter is window_type, it can be determined to insert the to-be-processed data into the next line of the target format parameter.

[0171] Specifically, the integrated data can be obtained as

[0172]

[0173]

[0174] In the present exemplary embodiment, the integrated data can be obtained by performing stream computing and integration processing on the to-be-processed data and the target format parameter, so as to facilitate the generalization and unification processing of the to-be-processed data of the same service subsequently, and improve the configuration reusability.

[0175] In step S240, stream aggregation calculation is performed on the integrated data to obtain a stream aggregation result, and the stream aggregation result is output.

[0176] In the exemplary embodiments of the present disclosure, after the integrated data is obtained, stream aggregation calculation can be performed on the integrated data stored in different positions.

[0177] Figure 8 、 Figure 9 and Figure 10 respectively show the step flowcharts of three methods of obtaining the integrated data stored in different positions and performing stream aggregation calculation. Among them, Figure 8 shows the step flowchart of a method of obtaining the integrated data stored in the integrated message queue and performing stream aggregation calculation; Figure 9 shows the step flowchart of a method of obtaining the integrated data stored in the specified port and performing stream aggregation calculation; Figure 10A step flow chart of a method of acquiring integrated data stored in a target file and performing stream aggregation calculation is shown.

[0178] In an optional embodiment, Figure 8 A step flow chart of a first method of stream aggregation calculation is shown, which comprises at least the following steps: in step S810, performing data caching processing on the integrated data to store the integrated data in an integrated message queue corresponding to the integrated data. Figure 8

[0179] To avoid data processing abnormality caused by excessive data processing request of the business data source, data caching processing can be performed on the integrated data.

[0180] Specifically, the integrated data can be stored in an integrated message queue. The integrated message queue is a queue for storing integrated data for processing, which is equivalent to a storage space for caching integrated data.

[0181] In step S820, integrated data is read from the integrated message queue, and stream aggregation calculation is performed on the integrated data to obtain stream aggregation results.

[0182] After the integrated data is stored in the integrated message queue, the integrated data can be read from the integrated message queue to perform stream aggregation calculation on the integrated data.

[0183] Specifically, the keyby operator is used to perform stream aggregation calculation. In most cases, the keyby operator can be used when data is grouped according to a certain field and processed.

[0184] The keyby operator converts the data stream of the integrated data into a KeyedStream, which is logical. KeyedStream is a special data stream. In fact, KeyedStream inherits the data stream, and the elements of the data stream are randomly distributed in the task slots. The elements of KeyedStream are grouped according to the grouping field and distributed to the task slots. Therefore, the keyby operator needs to pass a parameter, i.e. the grouping field, to determine the field as the Key for grouping.

[0185] For example, when the grouping field is a, the parameter passed to the keyby operator is a, i.e. the integrated data under the field a can be subjected to stream aggregation calculation.

[0186] ​The stream aggregation computing needs to be implemented according to the computing function. Meanwhile, the stream computing technology refers to real-time processing of a continuously generated data stream, and the stream computing has higher timeliness than the batch processing computing. Since the integrated data is processed by the stream computing in sequence, the stream aggregation computing is performed on the integrated data.

[0187] For example, when the grouping field is a and the computing function is a count function, the stream aggregation computing performed on the integrated data under the field a can be counting the number of records of the integrated data under the field a as the stream aggregation result.

[0188] In the example embodiment, the stream aggregation computing is performed by reading the integrated data in the integrated message queue, which smoothes the abnormal situation caused by too large request quantity of the integrated data burst, and improves the safety and stability of data processing.

[0189] Figure 9 A step flowchart of a second method of stream aggregation computing is shown in FIG. 9, which includes at least the following steps: in step S910, performing port configuration processing on the integrated data to store the integrated data in a port corresponding to the integrated data. Figure 9

[0190] The port is an outlet for the device to communicate with the outside world, and specifically refers to a register in an interface circuit, which can be used to store the integrated data to facilitate the stream aggregation computing on the integrated data.

[0191] To determine the port corresponding to the integrated data, the IP (Internet Protocol Address) address and the port number can be used. Specifically, the server location can be determined by the IP address, and the server can start multiple services, and then the location for storing the integrated data can be determined by the port number to store the integrated data.

[0192] In step S920, the integrated data is read from the port, and the stream aggregation computing is performed on the integrated data to obtain a stream aggregation result.

[0193] After the integrated data is stored in the port, the integrated data can be read from the port to perform the stream aggregation computing.

[0194] For example, when the grouping field is b and the computing function is a max function, the stream aggregation computing performed on the integrated data under the field b can be finding the maximum value in the integrated data under the field b as the stream aggregation result.

[0195] ​In this exemplary embodiment, by reading the integrated data in the port for streaming aggregation calculation, the data reading speed is faster, which meets the timeliness requirements of streaming aggregation calculation of integrated data.

[0196] Figure 10 A flowchart illustrating the steps of the third method for streaming aggregation computation is shown, as follows: Figure 10 As shown, the method includes at least the following steps: In step S1010, the integrated data is processed for file storage so that the integrated data is stored in the target file corresponding to the integrated data.

[0197] The target file can be a file specifically used to store integrated data, and the integrated data can be stored in the target file by writing the integrated data to a location determined by the directory of the target file.

[0198] In step S1020, integrated data is read from the target file, and streaming aggregation calculation is performed on the integrated data to obtain streaming aggregation results.

[0199] After the integrated data is stored in the target file, it can be read from the target file for streaming aggregation computation.

[0200] For example, when the grouping field is c and the calculation function is the min function, the streaming aggregation calculation performed on the integrated data under field c can be to find the minimum value in the integrated data under field c as the streaming aggregation result.

[0201] In this exemplary embodiment, streaming aggregation computation is performed by reading the integrated data stored in the target file. The storage method is simpler and more convenient, and the timeliness of streaming aggregation computation is improved by the simplicity of the storage path.

[0202] After obtaining the streaming aggregation result through streaming aggregation calculation, the streaming aggregation result can be stored.

[0203] In an optional embodiment, the streaming aggregation results are processed for distributed storage to store the streaming aggregation results in a distributed storage database.

[0204] This distributed storage database can be a key-value (KV) database. A KV database is a database that stores data in key-value pairs, where each key corresponds to a unique value.

[0205] For example, when the stream aggregation calculation on the integrated data under field a is performed, the number of records of the integrated data under field a is 5, the stream aggregation result stored in the KV database can be a=5; when the stream aggregation calculation on the integrated data under field b is performed, the maximum value of the integrated data under field b is 2, the stream aggregation result stored in the KV database can be b=2; when the stream aggregation calculation on the integrated data under field c is performed, the minimum value of the integrated data under field c is 1, the stream aggregation result stored in the KV database can be c=1.

[0206] In the present exemplary embodiment, the stream aggregation result is stored in the distributed storage database, which has the advantages of large data storage capacity and high concurrency support, and the stream aggregation result can be queried subsequently with extremely fast query speed.

[0207] After the stream aggregation result is stored in the distributed storage database, the stream aggregation result can be output from the distributed storage database.

[0208] In an optional embodiment, Figure 11 A step flowchart of a method for outputting the stream aggregation result is shown, as Figure 11 The method at least includes the following steps: in step S1110, the distributed storage database is subjected to result query processing according to the data attribute, so as to determine the stream aggregation result from the distributed storage database.

[0209] Since the stream aggregation results of different services are stored in the distributed storage database, the data attribute can be used for result query processing to determine the stream aggregation result to be output.

[0210] Specifically, the field name in the data attribute can be used for result query processing of the distributed database to determine the stream aggregation result to be output.

[0211] For example, when the field name is a, the stream aggregation result can be determined as a=5; when the field name is b, the stream aggregation result can be determined as b=2; when the field name is c, the stream aggregation result can be determined as c=1.

[0212] In step S1120, the target format parameter is subjected to path query processing to determine the output path of the stream aggregation result.

[0213] The target format parameter is obtained by performing parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification, and therefore, when the policy configuration parameter includes the output path selected by the service user through the policy configuration instruction, the target format parameter can be subjected to path query processing to determine the output path configured by the service user.

[0214] In step S1130, the stream aggregation result is output according to the output path.

[0215] In an optional embodiment, Figure 12 A step flowchart of a method of outputting the stream aggregation result according to the output path is shown in FIG. 12, which comprises at least the following steps: in step S1210, if the output path is an original path, the stream aggregation result is output to the service data source according to the original path. Figure 12 The output path includes the original path and a specified path. The original path is the path of the service data source. When the service user needs to return the stream aggregation result to the service data source, the original path can be specified by a policy configuration instruction.

[0216] In step S1220, if the output path is the specified path, the stream aggregation result is output according to the specified path.

[0217] The output path includes the original path and a specified path. The specified path is a storage path specified by a policy configuration instruction when the service user needs to output the stream aggregation result to a target position.

[0218] For example, the specified path can include a storage path of a relational database or a data cluster to meet the requirement of storing the stream aggregation result in the corresponding relational database or data cluster. In addition, it can also be other storage paths, which are not specially limited in the present exemplary embodiment.

[0219] In the present exemplary embodiment, through the result query processing and the path query processing, the stream aggregation result queried can be output according to the output path queried, meeting the requirement of the service user in different output situations and optimizing the use experience of the service user.

[0220] In addition, since the stream aggregation calculation requires real-time of the calculation process, a default result can also be output as the stream aggregation result when the time consumption of the stream aggregation calculation is too long.

[0221] In an optional embodiment, the target format parameter includes a configuration delay time length,

[0222] A step flowchart of a method of outputting the default result is shown in FIG. 13, which comprises at least the following steps: in step S1310, the calculation time length of the stream aggregation calculation process is counted to obtain the calculation time length of the stream aggregation calculation process. Figure 13 Figure 13

[0223] ​​Specifically, the process execution time of the stream aggregation calculation in the step S820, the step S920 and the step S1020 is counted to obtain the calculation time.

[0224] In the step S1320, if the calculation time is greater than the configuration delay time, a default result corresponding to the target format parameter is outputted.

[0225] The configuration delay time is a strategy configuration parameter of the delay time of the parameter format conversion processing after the configuration legality verification.

[0226] After the calculation time is obtained, the calculation time can be compared with the configuration delay time. When the comparison result is that the calculation time is greater than the configuration delay time, a default result corresponding to the target format parameter is outputted.

[0227] Specifically, the default result can correspond to the field type of the group field in the target format parameter. For example, when the field type of the group field is uint8, the corresponding default result is 0, so the default result 0 can be outputted.

[0228] In the present exemplary embodiment, when the process of the stream aggregation calculation takes a long time, a default result can be determined to be outputted, which meets the real-time requirement of the stream aggregation calculation and sufficiently reduces the delay of the stream aggregation calculation, which is of great significance in the application scenarios such as the actual aggregation strategy.

[0229] To further illustrate the relationship between the figures, Figure 14 The overall step flowchart of the data processing method is shown in FIG. 1. Figure 14As shown, in step S1401, in response to the policy configuration instruction, the policy configuration parameter is acquired; in step S1402, the policy configuration parameter is subjected to rule identification processing, the configuration parameter rule corresponding to the policy configuration parameter is obtained, and the policy configuration parameter is subjected to configuration legality verification by using the configuration parameter rule; in step S4103, the policy configuration parameter that passes the configuration legality verification is subjected to configuration parameter storage, so as to save the policy configuration parameter that passes the configuration legality verification in the configuration management center; in step S1404, the policy configuration parameter that passes the configuration legality verification and is saved in the configuration management center is subjected to parameter format conversion processing, and the target format parameter is obtained; in step S1405, the data processing request sent by the business data source is received, and the data attribute of the to-be-processed data is included in the data processing request; in step S1406, the data attribute is subjected to receiving matching processing, and the receiving matching result is obtained, so as to receive the to-be-processed data according to the receiving matching result; in step S1407, the target format parameter is subjected to configuration parameter identification processing to determine the integration bit parameter in the target format parameter; in step S1408, based on the integration bit parameter, the to-be-processed data and the target format parameter are subjected to stream computing integration processing to obtain the integrated data; in step S1409, the integrated data is subjected to data caching processing, so as to store the integrated data in the integrated message queue corresponding to the integrated data; in step S1410, the integrated data is read from the integrated message queue, and the stream aggregation result is obtained by performing stream aggregation calculation on the integrated data; in step S1411, the integrated data is subjected to port configuration processing, so as to store the integrated data in the port corresponding to the integrated data; in step S1412, the integrated data is read from the port, and the stream aggregation result is obtained by performing stream aggregation calculation on the integrated data; in step S1413, the integrated data is subjected to file storage processing, so as to store the integrated data in the target file corresponding to the integrated data; in step S1414, the integrated data is read from the target file, and the stream aggregation result is obtained by performing stream aggregation calculation on the integrated data; in step S1415, the stream aggregation result is subjected to distributed storage processing, so as to store the stream aggregation result in the distributed storage database; in step S1416, the distributed storage database is subjected to result query processing according to the data attribute, so as to determine the stream aggregation result from the distributed storage database; in step S1417, the target format parameter is subjected to path query processing, so as to determine the output path of the stream aggregation result; in step S1418, if the output path is the original path, the stream aggregation result is output to the business data source according to the original path; in step S1419, if the output path is the specified path, the stream aggregation result is output according to the specified path.

[0230] The data processing method provided in the embodiments of the present disclosure will be described in detail in combination with a specific application scenario.

[0231] Figure 15 A system framework diagram of a data processing method in an application scenario is shown in FIG. 1. Figure 15 As shown in FIG. 1, the data processing system includes a configuration layer, a data logic layer, and a stream computing layer.

[0232] The configuration layer includes a front-end page, a configuration management interface, and a configuration management center; the data logic layer includes an access layer, a background computing layer, and an output layer; and the stream computing layer includes a message queue layer, a Flink layer, and a distributed storage database, i.e., a KV storage layer.

[0233] Specifically, the front-end page is an operation page for business users, through which the business users can send a policy configuration instruction to configure the policy of stream aggregation computing; the configuration management interface is used to receive a storage request of the policy configuration parameter sent by the front-end page after the business user configures the policy configuration parameter, so as to send the policy configuration parameter to the configuration management center for storage through the configuration management result; the access layer mainly responds to the data processing request sent by the service calling party such as a business data source; the background computing layer can perform stream computing integration processing on the received to-be-processed data and target format parameter to obtain integrated data; the message queue layer is used for data caching processing of the integrated data, which smoothes the abnormal situation caused by too large request volume of the integrated data burst; the Flink layer performs stream aggregation computing on the integrated data to obtain a stream aggregation result; the KV storage layer is used for distributed storage processing of the stream aggregation result; and the output layer outputs the stream aggregation result according to an output path.

[0234] Among them, multiple access layers are set for different to-be-processed data of different businesses sent by different business data sources, which can make different access layers receive corresponding to-be-processed data, expand the service capability of data reception, facilitate parallel expansion in the case of too large request volume of the business data source, and provide more secure high-availability services through mutual disaster recovery between the access layers.

[0235] In addition, the message queue layer can store the integrated output in the integrated message queue, which smoothes the abnormal situation caused by too large request volume of the integrated data burst, and improves the security and stability of data processing.

[0236] In addition, a business data source is provided in the system framework. When there is to-be-processed data in the business data source, a data processing request can be sent to actively initiate the request and processing. This active request processing mode is more proactive, and also improves the defects of process complexity and poor timeliness caused by the multi-party subscription problem of passively obtaining to-be-processed data.

[0237] Figure 16 An interface diagram for setting a policy configuration parameter in an application scenario is shown in FIG. 2.Figure 16 As shown, a policy configuration instruction can be sent through the front-end page to configure corresponding policy configuration parameters.

[0238] The policy configuration instruction can be an instruction triggered by a business user through a front-end page of a terminal device, and the instruction is used to indicate a stream aggregation calculation policy of to-be-processed data for subsequent stream aggregation calculation.

[0239] The policy configuration instruction can include a configuration instruction of an input policy and a configuration instruction of an output policy.

[0240] For example, the input policy configuration instruction can be a configuration instruction of a time window, a configuration instruction of a time window duration, a configuration instruction of setting a grouping field, or a configuration instruction of a delay duration, and the present example embodiment does not specially limit this.

[0241] The output policy configuration instruction can be a configuration instruction of an output field, a configuration instruction of selecting different calculation functions, or a configuration instruction of setting an expression of a grouping field for stream aggregation calculation.

[0242] Further, when the input policy configuration instruction is a configuration instruction of a time window, a window type configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction of a time window duration, a window duration configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction of setting a grouping field, a grouping field configured by the business user can be obtained; and when the policy configuration instruction is a configuration instruction of a delay duration, a delay duration configured by the business user can be obtained.

[0243] When the output policy configuration instruction is a configuration instruction of an output field, an output field configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction of selecting different calculation functions, a calculation function configured by the business user can be obtained; and when the policy configuration instruction is a configuration instruction of setting an expression of a grouping field for stream aggregation calculation, an expression configured by the business user can be obtained.

[0244] Therefore, the policy configuration parameters can include a window type, a window duration, a grouping field, and a delay duration, and can also include an output field, a calculation function, and an expression.

[0245] The window type can include a rolling window and a sliding window.

[0246] Specifically, the rolling window can slice the data according to a fixed window length, and has the characteristics of time alignment, fixed window length, and no overlap. That is, the rolling window assigns each data to a window of a specified window length, the window has a fixed size, and the data of a window does not overlap / occur in other windows.

[0247] The sliding window is a more general form of the fixed window, which is composed of a fixed window length and a sliding interval, and has the characteristics of time alignment, fixed window length, and can have overlap. That is, the sliding window assigns data to a window of a fixed length, the size of the window is configured by the window length, and another window sliding parameter controls the frequency of the start of the sliding window, so the window can overlap if the sliding parameter is less than the window length, that is, the data will be assigned to multiple windows. Obviously, the sliding window does not jump from one set of data to the next set of data, but slides through the input data stream, so it can perform more smooth aggregation on the data.

[0248] The window length is a parameter for setting the window size of the rolling window or the sliding window.

[0249] The grouping field can be the name of the field set when grouping is required according to a certain field of the data. For example, the grouping field can be a or b.

[0250] The delay length can be the maximum length set for the flow aggregation calculation execution process to take, for example, it can be 80ms. The setting of the delay length can ensure the real-time processing of the to-be-processed data of different services.

[0251] The output field can be a representation field of the flow aggregation result of the flow aggregation calculation, for example, it can be AGG_ACOUNT.

[0252] The calculation function can be a function used for flow aggregation calculation on subsequent integrated data. For example, the calculation function can be a count function, a discount function, a max function, a min function, or a sum function.

[0253] The count function is a cumulative function for counting the number of records; the discount function is a deduplication function for removing duplicate records in the records; the max function is a maximum value function for finding the maximum value in the data; the min function is a minimum value function for finding the minimum value in the data; and the sun function is a sum function for finding the sum of all data.

[0254] The expression is a mathematical expression used to filter data obtained according to a grouping field. When the grouping field is a, all data under the grouping field a can be obtained, but when the expression is set to a>5, the data under the grouping field a must also satisfy a>5 before subsequent stream aggregation calculation can be performed.

[0255] Figure 17 A step flowchart of the method of stream aggregation calculation in an application scenario is shown as Figure 17 As shown in step S1710, the input and output fields and addresses of the Web configuration request data are obtained.

[0256] On the front-end Web (World Wide Web, global wide area network) graphical page, the policy configuration parameters are obtained in response to a policy configuration instruction triggered by a business user.

[0257] The policy configuration instruction can include an input policy configuration instruction and an output policy configuration instruction.

[0258] For example, the input policy configuration instruction can be a configuration instruction for a time window, a configuration instruction for setting the duration of a time window, a configuration instruction for setting a grouping field, or a configuration instruction for setting a delay duration, and the like, which is not particularly limited in the present example embodiment.

[0259] The output policy configuration instruction can be a configuration instruction for an output field, a configuration instruction for selecting different calculation functions, or a configuration instruction for setting an expression for a grouping field for stream aggregation calculation, and the like.

[0260] In step S1720, the Web configuration data aggregates related parameters, such as a key field, an aggregation function, and the like.

[0261] When the input policy configuration instruction is a configuration instruction for a time window, the window type configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction for setting the duration of a time window, the window duration configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction for setting a grouping field, the grouping field configured by the business user can be obtained; and when the policy configuration instruction is a configuration instruction for setting a delay duration, the delay duration configured by the business user can be obtained, and the like.

[0262] When the output policy configuration instruction is a configuration instruction for an output field, the output field configured by the business user can be obtained; when the policy configuration instruction is a configuration instruction for selecting different calculation functions, the calculation function configured by the business user can be obtained; and when the policy configuration instruction is a configuration instruction for setting an expression for a grouping field for stream aggregation calculation, the expression configured by the business user can be obtained.

[0263] Therefore, the policy configuration parameter can include a window type, a window duration, a delay duration, a grouping field, i.e., a key field, and the like, and can further include an output field, an expression, a calculation function, i.e., an aggregation function, and the like.

[0264] The window type can include a tumble window and a sliding window.

[0265] Specifically, the tumble window can slice data according to a fixed window duration, has the characteristics of time alignment, fixed window length, and no overlap. That is, the tumble window allocates each data to a window of a specified window duration, the window has a fixed size, and the data of one window does not overlap / occur in other windows.

[0266] The sliding window is a more general form of the fixed window, which is composed of a fixed window duration and a sliding interval, has the characteristics of time alignment, fixed window length, and can have overlap. That is, the sliding window allocates data to a window of a fixed length, the size of the window is configured by the window duration, and another window sliding parameter controls the frequency of the start of the sliding window. Therefore, if the sliding parameter is less than the window duration, the window can overlap, i.e., the data can be allocated to multiple windows. Obviously, the sliding window does not jump from one group of data to the next group of data, but slides through the input data stream, so that more smooth aggregation of data can be performed.

[0267] The window duration is a parameter for setting the window size of the tumble window or the sliding window.

[0268] The grouping field can be a field name set when grouping is required according to a certain field of the data. For example, the grouping field can be a or b.

[0269] The delay duration can be a maximum duration set for allowing the stream aggregation calculation execution process to take, for example, 80 ms. The setting of the delay duration can ensure the processing real-time performance of the to-be-processed data of different services.

[0270] The output field can be a representation field of the stream aggregation result of the stream aggregation calculation, for example, AGG_ACOUNT.

[0271] The calculation function can be a function used for stream aggregation calculation of subsequent integrated data. For example, the calculation function can be a count function, a discount function, a max function, a min function, or a sum function.

[0272] Wherein, the count function is a cumulative function, used for counting the number of records; the discount function is a deduplication function, used for removing duplicate records in the records; the max function is a maximum value function, used for obtaining the maximum value in the data; the min function is a minimum value function, used for obtaining the minimum value in the data; and the sun function is a summation function, used for obtaining the sum of all data.

[0273] The expression is a mathematical expression used for filtering data obtained according to the grouping field. When the grouping field is a, all data under the grouping field a can be obtained, but when the expression is set to a>5, the data under the grouping field a must also satisfy a>5, and then subsequent stream aggregation calculation can be performed.

[0274] Figure 18 An interface schematic diagram of the configuration of the policy configuration parameter in the application scenario is shown, as shown in Figure 18 The grouping field (groupby) is removed from the field a, the window type is selected as a sliding window, the window size is set to a window duration of 120 seconds, the output field is set to AGG_ACOUNT, the calculation function is set to the count function, the expression is filled in as b>10, and the configured delay duration is 80 milliseconds.

[0275] In step S1730, the aggregation configuration parameter is saved to the configuration center.

[0276] Before being saved in the configuration management center, the policy configuration parameter can be subjected to configuration legality verification and parameter format conversion processing.

[0277] The configuration legality verification can be first subjected to rule recognition processing of the policy configuration parameter to obtain a configuration parameter rule corresponding to the policy configuration parameter, and then the configuration legality verification of the policy configuration parameter is performed by using the configuration parameter rule.

[0278] Specifically, for different policy configuration parameters, corresponding configuration parameter rules can be pre-set, which are used to verify the format, type or setting range of the corresponding policy configuration parameter.

[0279] For example, when the output field is AGG_ACOUNT, the corresponding field type must be uint8, so that the field type uint8 is one of the configuration parameter rules of the output field. In addition, when the calculation function is the max function or the min function, the corresponding statistical field position cannot be empty, which is the corresponding configuration parameter rule.

[0280] Therefore, the rule recognition processing can be performed on the policy configuration parameter and the preset configuration parameter rule in the case that the policy configuration parameter and the strategy configuration parameter rule are known, and the configuration parameter rule corresponding to the one or more policy configuration parameters known at this time, such as the output field or the calculation function, can be obtained.

[0281] In the case that the configuration parameter rule has been determined, the configuration legality verification can be performed on whether the policy configuration parameter meets the configuration parameter rule.

[0282] For example, when the output field is AGG_ACOUNT, the field type is required to be uint8, and when the subsequently obtained to-be-processed data is of the type uint8, it is determined that the policy configuration parameter meets the corresponding configuration parameter rule, i.e., passes the configuration legality verification; when the calculation function is the max function or the min function, the corresponding statistical field position is required to be non-empty, but the business use unit is set to be empty, or the statistical field is empty due to error setting or no setting, which indicates that the policy configuration parameter does not meet the corresponding configuration parameter rule, i.e., does not pass the configuration legality verification.

[0283] When the policy configuration parameter does not meet the corresponding configuration parameter rule, i.e., does not pass the legality verification, a corresponding prompt information can be sent, so that the business user can perceive the policy configuration parameter that does not pass the legality verification and modify and update it, so as to ensure the smooth progress of the subsequent stream aggregation calculation.

[0284] The parameter format conversion processing can be first to store the policy configuration parameter that passes the configuration legality verification, so as to save the policy configuration parameter that passes the configuration legality verification in the configuration management center, and then to perform the parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification and is saved in the configuration management center to obtain the target format parameter.

[0285] Specifically, the save button of the web page can be clicked to store the policy configuration parameter that passes the configuration legality verification in the configuration management center, such as Zookeeper.

[0286] Further, the policy configuration parameter that passes the configuration legality verification is converted into the target format parameter in the JSON format.

[0287] The JSON format refers to the string format obtained by converting a group of data, i.e., the policy configuration parameter, in the JavaScript object. The file in the JSON format can be transmitted between the network or the programs, and can also be restored to the data format supported by each programming language.

[0288] In this step, the strategy configuration parameters are converted into target format parameters in JSON format. Specifically, the parameter configuration parameters can be saved as field names and values. When performing a max function calculation on the grouping field 'a', it can be saved in the form of 'a-max'.

[0289] Therefore, the target format parameters obtained after converting the strategy configuration parameters can meet the data storage and query requirements of the backend, facilitating the reuse of the same configuration in the future.

[0290] In step S1740, the service caller sends request data to the input access node.

[0291] First, it receives data processing requests sent by service callers such as business data sources. The data processing requests include data attributes of the data to be processed. Then, it performs receive matching processing on the data attributes to obtain the receive matching results, and receives the data to be processed according to the receive matching results.

[0292] Data processing requests are proactively sent by the business data source. By actively requesting the processing of the data to be processed, the initiative is stronger, and it also improves the shortcomings of cumbersome processes and poor timeliness caused by the passive acquisition of data to be processed through multiple subscriptions.

[0293] Specifically, when there is data to be processed in the business data source, a data processing request can be sent to proactively initiate the request and processing.

[0294] The data processing request carries information representing the data attributes of the data to be processed. These data attributes include data type and data name, among other things. Other data attributes may also be included, but this exemplary embodiment does not impose any specific limitations on them.

[0295] When a data processing request for data to be processed is received, since different services have different data receiving modules (such as the access layer or access module) to match the data to be processed, the data attributes can be matched to determine the corresponding data receiving module based on the data type and data name, so that the data receiving module can be used to receive the data to be processed.

[0296] In step S1750, the background logic module combines the request data and the corresponding aggregation parameters into new aggregation request data.

[0297] Specifically, configuration parameter identification processing is performed on the target format parameters to determine the integration bit parameter. This configuration parameter identification processing identifies the integration bit parameter as `agg_conf`. Therefore, the position for stream computing integration processing of the data to be processed and the target format parameters can be determined as the line preceding `agg_conf`.

[0298] In addition, the integration bit parameter can be set as window_type according to actual conditions, and in this case, the position of the stream computing integration processing of the to-be-processed data and the target format parameter can be determined as the next row of the window_type.

[0299] Obviously, the integration bit parameter is a flag bit representing the stream computing integration processing of the to-be-processed data and the target format parameter.

[0300] Then, based on the integration bit parameter, the stream computing integration processing of the to-be-processed data and the target format parameter is performed to obtain integrated data.

[0301] After determining the integration bit parameter in the target format parameter, the stream computing integration processing of the to-be-processed data and the target format parameter can be performed.

[0302] Specifically, the to-be-processed data is inserted into the target format parameter of the integration bit parameter as a whole, and a flag row of agg_data is added to obtain integrated data, that is, aggregated request data.

[0303] In step S1760, the aggregated request data is issued to the message queue.

[0304] The integrated data is subjected to data caching processing to store the integrated data in an integrated message queue corresponding to the integrated data.

[0305] To avoid data processing exceptions caused by excessive data processing requests of the business data source, the integrated data can be subjected to data caching processing.

[0306] Specifically, the integrated data can be stored in an integrated message queue. The integrated message queue is a queue for storing integrated data to be processed, which is equivalent to a storage space for caching integrated data.

[0307] In step S1770, the Flink consumes the request data in the message queue and performs aggregation, and writes the aggregated result data to the distributed storage database.

[0308] After storing the integrated data in the integrated message queue, the integrated data can be read from the integrated message queue to perform stream aggregation computing on the integrated data.

[0309] Specifically, the stream aggregation computing is performed by using a keyby operator. In most cases, the data is grouped according to a certain field, and the keyby operator can be used when processing the data in a group.

[0310] For example, when the grouping field is a, the parameter passed to the keyby operator is a, that is, the integrated data under the field a can be subjected to the stream aggregation calculation.

[0311] The stream aggregation calculation needs to be implemented according to the calculation function. Meanwhile, the stream computing technology refers to the real-time processing of the continuously generated data stream, and the timeliness of the stream computing is higher than that of the batch processing calculation. Therefore, the integrated data is subjected to the stream calculation in sequence, and thus the integrated data is subjected to the stream aggregation calculation.

[0312] For example, when the grouping field is a and the calculation function is the count function, the stream aggregation calculation performed on the integrated data under the field a can be to count the number of records of the integrated data under the field a as the stream aggregation result.

[0313] Further, the stream aggregation result is subjected to the distributed storage processing, so as to store the stream aggregation result in the distributed storage database.

[0314] The distributed storage database can be a KV database. The KV database is a database that stores data in the form of key-value pairs, and each key corresponds to a unique value.

[0315] For example, when the stream aggregation calculation performed on the integrated data under the field a counts the number of records of the integrated data under the field a as 5, the stream aggregation result stored in the KV database can be a=5; when the stream aggregation calculation performed on the integrated data under the field b obtains the maximum value of the integrated data under the field b as 2, the stream aggregation result stored in the KV database can be b=2; and when the stream aggregation calculation performed on the integrated data under the field c obtains the minimum value of the integrated data under the field c as 1, the stream aggregation result stored in the KV database can be c=1.

[0316] In step S1780, the background logic module queries the distributed storage database according to the key.

[0317] Since the stream aggregation results of different businesses are stored in the distributed storage database, to determine the stream aggregation result to be output, the result query processing can be performed by using the data attribute.

[0318] Specifically, the result query processing of the distributed database can be performed by using the field name in the data attribute to determine the stream aggregation result to be output.

[0319] For example, when the field name is a, it can be determined that the stream aggregation result is a=5; when the field name is b, it can be determined that the stream aggregation result is b=2; and when the field name is c, it can be determined that the stream aggregation result is c=1.

[0320] In step S1790, the aggregation result data is output.

[0321] If the output path is the original path, the stream aggregation result is output to the service data source according to the original path.

[0322] The original path is the path of the service data source. When the service user needs to return the stream aggregation result to the service data source, the original path can be specified by the policy configuration instruction.

[0323] If the output path is the specified path, the stream aggregation result is output according to the specified path.

[0324] The specified path is a storage path specified by the policy configuration instruction when the service user needs to output the stream aggregation result to a target location.

[0325] For example, the specified path can include a storage path of a relational database or a data cluster to meet the requirement of storing the stream aggregation result in the corresponding relational database or data cluster. In addition, it can also be other storage paths, which are not specially limited in the example embodiment.

[0326] Figure 19 A system architecture diagram of another data processing method in an application scenario is shown, as shown in Figure 19 The data processing system includes a configuration layer, a data logic layer and a stream computing layer.

[0327] The configuration layer includes a front-end page, a configuration management interface and a configuration management center; the data logic layer includes an access layer, a background computing layer and an output layer; the stream computing layer includes a Flink layer and a distributed storage database, i.e., a KV storage layer.

[0328] Specifically, the front-end page is an operation page for the service user, and the service user can send a policy configuration instruction through the operation page to configure the policy of the stream aggregation calculation; the configuration management interface is used to receive a storage request of the policy configuration parameter sent by the front-end page after the service user configures the policy configuration parameter, so as to send the policy configuration parameter to the configuration management center for storage through the configuration management result; the access layer mainly responds to the data processing request sent by the service calling party such as the service data source; the background computing layer can perform stream computing and integration processing on the received to-be-processed data and target format parameter to obtain integrated data; the Flink layer performs stream aggregation calculation on the integrated data to obtain a stream aggregation result; the KV storage layer is used for distributed storage processing of the stream aggregation result; and the output layer is used for outputting the stream aggregation result according to the output path.

[0329] Compared with Figure 15The system architecture diagram in the figure is removed from the message queue layer in the stream computing layer, so that the integrated data can be directly stored in the corresponding port or target file for direct use by Flink.

[0330] The port is an outlet for the device to communicate with the outside world, specifically a register in the interface circuit, which can be used to store integrated data for direct stream aggregation calculation of the integrated data.

[0331] The target file can be a file specially used to store integrated data, and the integrated data can be stored in the target file by writing the integrated data to the directory of the target file.

[0332] The data processing system in the application scenario has been put into application and promotion, and so far, the stream aggregation calculation tasks processed by the data processing system have reached 30, covering various aggregation strategy attack scenarios such as verification code verification, financial data processing, and security business management, and the application scenarios are very rich and effective.

[0333] Based on the above application scenarios, the data processing method provided by the embodiments of the present disclosure can realize parameter configuration of the data processing mode through the policy configuration instruction, so that the business user does not need to pay attention to the writing logic of data processing, greatly reducing the use difficulty and complexity of the business user, enriching the use scenarios and universality of the data processing mode, and further, the configured policy configuration parameters are configured for legality verification, providing an automatic verification method for the policy configuration parameters, avoiding the occurrence of human verification errors, and improving the accuracy and intelligent degree of the policy configuration parameters. On the other hand, the integrated data obtained by stream computing integration processing facilitates the subsequent unified and universal use of the integrated data, and improves the reusability of the integrated data. In addition, the stream aggregation calculation of the integrated data greatly reduces the occurrence of stream aggregation calculation delay, ensures the real-time and effectiveness of data processing, and provides strong support for the application scenarios of security aggregation strategy.

[0334] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0335] The device embodiments of the present disclosure are introduced below, which can be used to execute the data processing method in the above-mentioned embodiments of the present disclosure. For details not disclosed in the device embodiments of the present disclosure, please refer to the above-mentioned embodiments of the data processing method of the present disclosure.

[0336] Figure 20 A structural block diagram of a data processing apparatus in some embodiments of the present disclosure is schematically shown as Figure 20 As shown, the data processing apparatus 2000 can mainly include a configuration verification module 2010, a format conversion module 2020, a data integration module 2030 and an aggregated calculation module 2040.

[0337] The configuration verification module 2010 is configured to acquire a policy configuration parameter in response to a policy configuration instruction, and perform configuration legality verification on the policy configuration parameter; the format conversion module 2020 is configured to perform parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification to obtain a target format parameter; the data integration module 2030 is configured to receive to-be-processed data, and perform stream computing integration processing on the to-be-processed data and the target format parameter to obtain integrated data; and the aggregated calculation module 2040 is configured to perform stream aggregation calculation on the integrated data to obtain a stream aggregation result, and output the stream aggregation result.

[0338] In some embodiments of the present disclosure, the data integration module includes a request receiving sub-module configured to receive a data processing request sent by a business data source, the data processing request including a data attribute of to-be-processed data;

[0339] The attribute matching sub-module is configured to perform receiving matching processing on the data attribute to obtain a receiving matching result, so as to receive the to-be-processed data according to the receiving matching result.

[0340] In some embodiments of the present disclosure, the data processing apparatus further includes a result storage module configured to perform distributed storage processing on the stream aggregation result, so as to store the stream aggregation result in a distributed storage database.

[0341] In some embodiments of the present disclosure, the aggregated calculation module includes a result query sub-module configured to perform result query processing on the distributed storage database according to the data attribute, so as to determine the stream aggregation result from the distributed storage database;

[0342] The path query sub-module is configured to perform path query processing on the target format parameter to determine an output path of the stream aggregation result;

[0343] The result output sub-module is configured to output the stream aggregation result according to the output path.

[0344] In some embodiments of the present disclosure, the result output sub-module includes an original path unit configured to, if the output path is an original path, output the stream aggregation result to the business data source according to the original path;

[0345] A designated path unit is configured to output the stream aggregation result according to the designated path if the output path is the designated path.

[0346] In some embodiments of the present disclosure, the data processing apparatus further comprises a duration statistical module configured to perform duration statistics on the process of the stream aggregation calculation to obtain a calculation duration of the process of the stream aggregation calculation.

[0347] A default output module is configured to output a default result corresponding to the target format parameter if the calculation duration is greater than the configured delay duration.

[0348] In some embodiments of the present disclosure, the configuration verification module comprises a rule identification submodule configured to perform rule identification processing on the policy configuration parameter to obtain a configuration parameter rule corresponding to the policy configuration parameter.

[0349] A rule verification submodule is configured to perform configuration legality verification on the policy configuration parameter by using the configuration parameter rule.

[0350] In some embodiments of the present disclosure, the format conversion module comprises a configuration storage submodule configured to perform configuration parameter storage on the policy configuration parameter that passes the configuration legality verification, so as to save the policy configuration parameter that passes the configuration legality verification in a configuration management center.

[0351] A parameter conversion submodule is configured to perform parameter format conversion processing on the policy configuration parameter that passes the configuration legality verification and is saved in the configuration management center to obtain the target format parameter.

[0352] In some embodiments of the present disclosure, the data integration module comprises a parameter identification submodule configured to perform configuration parameter identification processing on the target format parameter to determine an integration bit parameter in the target format parameter.

[0353] A calculation integration submodule is configured to perform stream calculation integration processing on the to-be-processed data and the target format parameter based on the integration bit parameter to obtain integrated data.

[0354] In some embodiments of the present disclosure, the aggregation calculation module comprises a cache processing submodule configured to perform data cache processing on the integrated data to store the integrated data in an integrated message queue corresponding to the integrated data.

[0355] A queue reading submodule is configured to read the integrated data from the integrated message queue and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0356] In some embodiments of the present disclosure, the aggregation calculation module comprises a port configuration submodule configured to perform port configuration processing on the integrated data to store the integrated data in a port corresponding to the integrated data.

[0357] The port reading submodule is configured to read the integrated data from the port and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0358] In some embodiments of the present disclosure, the aggregation calculation module comprises a file storage submodule configured to perform file storage processing on the integrated data to store the integrated data in a target file corresponding to the integrated data.

[0359] The file reading submodule is configured to read the integrated data from the target file and perform stream aggregation calculation on the integrated data to obtain a stream aggregation result.

[0360] The specific details of the data processing apparatus provided in the embodiments of the present disclosure have been described in detail in the corresponding method embodiments, and thus will not be described here again.

[0361] Figure 21 A structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present disclosure is shown.

[0362] It should be noted that, Figure 21 The computer system 2100 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0363] As Figure 21 shown, the computer system 2100 includes a central processing unit (CPU) 2101, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 2102 or loaded from a storage portion 2108 into a random access memory (RAM) 2103. In the RAM 2103, various programs and data required for system operation are also stored. The CPU 2101, the ROM 2102, and the RAM 2103 are connected to each other through a bus 2104. An input / output (I / O) interface 2105 is also connected to the bus 2104.

[0364] The following components are connected to the I / O interface 2105: an input part 2106 including a keyboard, a mouse, etc.; an output part 2107 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 2108 including a hard disk, etc.; and a communication part 2109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 2109 performs communication processing via a network such as the Internet. A drive 2110 is also connected to the I / O interface 2105 as necessary. A removable medium 2111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 2110 as necessary, so that a computer program read therefrom is installed in the storage part 2108 as necessary.

[0365] In particular, according to embodiments of the present disclosure, the processes described in the various method flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 2109, and / or installed from the removable medium 2111. When the computer program is executed by the central processing unit (CPU) 2101, various functions defined in the system of the present application are performed.

[0366] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carrying a computer-readable program code in a baseband or as a part of a carrier wave. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0367] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0368] It should be noted that, although several modules or units for a device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the disclosure, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into embodied by a plurality of modules or units.

[0369] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the disclosure.

[0370] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure that are deemed to fall within the general principles of the disclosure and include examples of the disclosure that are not disclosed herein.

[0371] It should be understood that the present disclosure is not limited to the precise structures described and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A data processing method, characterized in that, The method is applied to an electronic device, including: In response to a policy configuration command, policy configuration parameters are obtained, and the policy configuration parameters are validated for legality. The policy configuration parameters that have passed the configuration validity verification are converted to obtain the target format parameters. The system receives data to be processed and performs configuration parameter identification processing on the target format parameters to determine the integration bit parameter in the target format parameters. The integration bit parameter is a flag bit that represents the integration processing of the data to be processed and the target format parameters in streaming computation. Based on the integrated bit parameters, the data to be processed is inserted into the target format parameters to obtain integrated data; The integrated data is subjected to streaming aggregation calculation to obtain streaming aggregation results, and the streaming aggregation results are output.

2. The data processing method according to claim 1, characterized in that, The receiving of data to be processed includes: Receive a data processing request sent by a business data source, wherein the data processing request includes the data attributes of the data to be processed; The data attributes are subjected to a receiving matching process to obtain a receiving matching result, and the data to be processed is received according to the receiving matching result.

3. The data processing method according to claim 2, characterized in that, After performing streaming aggregation calculations on the integrated data to obtain streaming aggregation results, the method further includes: The streaming aggregation results are processed for distributed storage so that they are stored in a distributed storage database.

4. The data processing method according to claim 3, characterized in that, The output of the streaming aggregation result includes: The distributed storage database is queried according to the data attributes to determine the streaming aggregation result from the distributed storage database; Perform path lookup processing on the target format parameters to determine the output path of the streaming aggregation result; The streaming aggregation results are output according to the output path.

5. The data processing method according to claim 4, characterized in that, The step of outputting the streaming aggregation result according to the output path includes: If the output path is the original path, the streaming aggregation result is output to the business data source according to the original path; If the output path is a specified path, the streaming aggregation result is output according to the specified path.

6. The data processing method according to claim 1, characterized in that, The target format parameters include the configuration delay duration. The method further includes: The computation time of the streaming aggregation computation process is statistically analyzed to obtain the computation time of the streaming aggregation computation process; If the calculation time is greater than the configured delay time, the default result corresponding to the target format parameter is output.

7. The data processing method according to claim 1, characterized in that, The step of validating the configuration validity of the strategy configuration parameters includes: The policy configuration parameters are subjected to rule recognition processing to obtain configuration parameter rules corresponding to the policy configuration parameters; The configuration parameter rules are used to verify the validity of the policy configuration parameters.

8. The data processing method according to claim 1, characterized in that, The step of converting the parameter format of the policy configuration parameters that have passed the configuration validity verification to obtain the target format parameters includes: The policy configuration parameters that pass the configuration validity verification are stored in the configuration management center. The policy configuration parameters that have passed the configuration validity verification and are stored in the configuration management center are converted into target format parameters.

9. The data processing method according to claim 1, characterized in that, The process of performing streaming aggregation calculations on the integrated data to obtain streaming aggregation results includes: The integrated data is cached to store the integrated data in an integrated message queue corresponding to the integrated data. The integrated data is read from the integrated message queue, and streaming aggregation calculation is performed on the integrated data to obtain the streaming aggregation result.

10. The data processing method according to claim 1, characterized in that, The process of performing streaming aggregation calculations on the integrated data to obtain streaming aggregation results includes: The integrated data is configured to store the integrated data in a port corresponding to the integrated data. The integrated data is read from the port, and streaming aggregation calculation is performed on the integrated data to obtain the streaming aggregation result.

11. The data processing method according to claim 1, characterized in that, The process of performing streaming aggregation calculations on the integrated data to obtain streaming aggregation results includes: The integrated data is processed for file storage so that the integrated data is stored in a target file corresponding to the integrated data; The integrated data is read from the target file, and streaming aggregation is performed on the integrated data to obtain the streaming aggregation result.

12. A data processing apparatus, characterized in that, The device is deployed in an electronic device, including: The configuration verification module is configured to respond to a policy configuration command, obtain policy configuration parameters, and perform configuration validity verification on the policy configuration parameters. The format conversion module is configured to perform parameter format conversion processing on the policy configuration parameters that have passed the configuration validity verification to obtain target format parameters; The data integration module is configured to receive data to be processed and perform streaming computation integration processing on the data to be processed and the target format parameters to obtain integrated data; The aggregation calculation module is configured to perform streaming aggregation calculations on the integrated data to obtain streaming aggregation results and output the streaming aggregation results; The data integration module includes: a parameter identification submodule, configured to perform configuration parameter identification processing on the target format parameters to determine the integration bit parameter in the target format parameters, wherein the integration bit parameter is a flag bit representing the stream computing integration processing of the data to be processed and the target format parameters; The calculation integration submodule is configured to insert the data to be processed into the target format parameter based on the integration bit parameter to obtain integrated data.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 11.

14. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method of any one of claims 1 to 11 by executing the executable instructions.

15. A computer program product, characterized in that, It includes a computer program carried on a computer-readable medium, the computer program containing program code for performing the data processing method according to any one of claims 1 to 11.

Images