Stream computing data processing method and device and electronic equipment

Abstract

The invention provides a stream computing data processing method and device and electronic equipment, and the method comprises the steps: obtaining production parameter data in a blast furnace ironmaking process, carrying out the preprocessing of the production parameter data, obtaining the preprocessing data, converting the preprocessing data into stream data, and carrying out the processing of the stream data according to the processing demands of the stream data. Target processing operators and the execution sequence of the target processing operators are determined, the target processing operators calculate and process the streaming data according to the execution sequence, and a streaming data processing result is obtained; according to the method, the preprocessed data is converted into the stream data, the target processing operators and the execution sequence of the target processing operators are determined according to the processing requirements of the stream data, the target processing operators calculate and process the stream data according to the execution sequence, and the real-time processing requirements and the batch processing requirements of the blast furnace production parameter data are met.

Description

Technical Field

[0001] The present application relates to the technical field of industrial Internet of Things data processing, and in particular to a stream computing data processing method, device and electronic device. Background Art

[0002] The steelmaking process involves numerous equipment and systems, such as blast furnaces, converters, and rolling mills. Each device potentially generates a large amount of data of varying types, including real-time monitoring data, equipment status data, and process parameters. This results in complex data sources and diverse formats. Furthermore, steelmaking is a continuous production process, generating enormous amounts of data that requires real-time or near-real-time processing. However, existing data processing methods lack sufficient processing capabilities to meet the demands of real-time or near-real-time data processing. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present application provides a stream computing data processing method, device and electronic device to solve the above-mentioned technical problems.

[0004] According to one aspect of an embodiment of the present application, a stream computing data processing method is provided, the method comprising: obtaining production parameter data in a blast furnace ironmaking process; preprocessing the production parameter data to obtain preprocessed data; the preprocessing method comprising: filtering and cleaning, data integration, and data encryption; converting the preprocessed data into stream data, and determining a target processing operator and an execution order of the target processing operator based on the processing requirements of the stream data; the target processing operator comprising: a data aggregation operator, a compound operation operator, and a script calculation operator; the target processing operator performs calculation processing on the stream data in the execution order to obtain a stream data processing result.

[0005] In one embodiment of the present application, the process of preprocessing the production parameter data includes: classifying and marking the production parameter data to obtain classified marked data; filtering and cleaning the classified marked data to obtain classified cleaned data; and integrating the classified cleaned data according to preset dimensions to obtain classified integrated data; encrypting the classified integrated data to obtain the preprocessed data.

[0006] In one embodiment of the present application, the process of classifying and labeling the production parameter data includes: dividing the production parameter data into measurement parameter data, configuration parameter data and derived parameter data; the derived parameter data is determined by the measurement parameter data and / or the configuration parameter data; using the measurement parameter data, the configuration parameter data and the derived parameter data as input parameters of a classification labeling operator, and classifying and labeling the measurement parameter data, the configuration parameter data and the derived parameter data through the classification labeling operator to obtain the classified labeling data.

[0007] In one embodiment of the present application, the process of converting the preprocessed data into stream data includes: decrypting the preprocessed data to obtain decrypted data; and converting the decrypted data according to a stream data format to obtain the stream data.

[0008] In one embodiment of the present application, the process of determining the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data includes: determining the operation type of the stream data and the execution order of the operation type according to the processing requirements of the stream data; the operation type includes data aggregation, compound operation, and script calculation; using the processing operator that executes the operation type as the target processing operator; and using the execution order of the operation type as the execution order of the target processing operator.

[0009] In one embodiment of the present application, if the target processing operator processes the stream data in groups and there are multiple target processing operators, then after obtaining the stream data processing result, the method further includes: extracting process data of each target processing operator in the stream data processing result; the process data includes: input time, output time and stream data volume of each group of stream data; calculating the processing time of each group of stream data in each target processing operator based on the input time and output time of each group of stream data in each target processing operator; calculating the real-time stream data volume processed by each target processing operator per unit time based on the stream data volume of each group of stream data and the processing time of each group of stream data in each target processing operator; and determining the maximum real-time stream data volume processed by each target processing operator from the real-time stream data volume processed by each target processing operator within the unit time.

[0010] In one embodiment of the present application, after obtaining the processing time of each group of stream data in each target processing operator, the method further includes: calculating the processing time of each group of stream data in the calculation and processing process based on the input time and output time of each group of stream data in each target processing operator; calculating the time proportion of each target processing operator performing calculation and processing on each group of stream data based on the processing time of each group of stream data in each target processing operator and the processing time of each group of stream data in the calculation and processing process; calculating the average time proportion of each target processing operator performing calculation and processing on each group of stream data based on the time proportion of each target processing operator performing calculation and processing on each group of stream data.

[0011] In one embodiment of the present application, after obtaining the average time share, the method further includes: storing and outputting the stream data processing results, the average time share and the maximum real-time stream data volume processed by each target processing operator; and / or, displaying the stream data processing results, the average time share and the maximum real-time stream data volume processed by each target processing operator.

[0012] According to one aspect of an embodiment of the present application, a stream computing data processing device is provided, including: a data acquisition module for acquiring production parameter data in a blast furnace ironmaking process; a data processing module for preprocessing the production parameter data to obtain preprocessed data; the preprocessing method includes: filtering and cleaning, data integration, and data encryption; an operator determination module for converting the preprocessed data into stream data, and determining a target processing operator and an execution order of the target processing operator according to the processing requirements of the stream data; the target processing operator includes: a data aggregation operator, a compound operation operator, and a script calculation operator; a data calculation module for the target processing operator to perform calculation processing on the stream data in the execution order to obtain a stream data processing result.

[0013] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the stream computing data processing method as described above.

[0014] The beneficial effects of the present application are as follows: The present application obtains production parameter data in the blast furnace ironmaking process, pre-processes the production parameter data, obtains pre-processed data, converts the pre-processed data into stream data, and determines the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data. The target processing operator calculates and processes the stream data according to the execution order to obtain the stream data processing result. The above process, by converting the pre-processed data into stream data, and determining the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data, the target processing operator calculates and processes the stream data according to the execution order, meets the real-time processing requirements and batch processing requirements of the blast furnace production parameter data.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0017] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;

[0018] Figure 2 This is a flow chart of a stream computing data processing method shown in an exemplary embodiment of the present application;

[0019] Figure 3 is a flowchart of a stream computing data processing method shown in another exemplary embodiment of the present application;

[0020] Figure 4 is a block diagram of a stream computing data processing device shown in an exemplary embodiment of the present application;

[0021] Figure 5 is a block diagram of a stream computing data processing device shown in another exemplary embodiment of the present application;

[0022] Figure 6 It is a structural diagram of a computer system of an electronic device shown in another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand other advantages and functions of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0025] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0026] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.

[0027] Reference Figure 1 As shown, the system architecture may include an acquisition device 101 and a computer device 102. The computer device 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, and the like. Relevant technicians may use the computer device 102 to obtain production parameter data from a blast furnace ironmaking process, preprocess the production parameter data to obtain preprocessed data, convert the preprocessed data into stream data, and determine target processing operators and the execution order of the target processing operators based on the processing requirements of the stream data. The target processing operators perform computational processing on the stream data in the execution order to obtain stream data processing results. The acquisition device 101 is used to acquire production parameter data from a blast furnace ironmaking process. In this embodiment, the acquisition device 101 uses sensors installed on the blast furnace to acquire the production parameter data and provides the data to the computer device 102 for processing.

[0028] Schematically, after obtaining the production parameter data of the acquisition device 101, the computer device 102 preprocesses the production parameter data to obtain preprocessed data, converts the preprocessed data into stream data, and determines the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data. The target processing operator calculates and processes the stream data in the execution order to obtain the stream data processing result. The above process converts the preprocessed data into stream data, and determines the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data. The target processing operator calculates and processes the stream data in the execution order, thereby meeting the real-time processing requirements and batch processing requirements of the blast furnace production parameter data.

[0029] It should be noted that the stream computing data processing method provided in the embodiment of the present application is generally executed by the computer device 102 , and accordingly, the stream computing data processing apparatus is generally provided in the computer device 102 .

[0030] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0031] Figure 2 This is a flow chart of a stream computing data processing method according to an exemplary embodiment of the present invention. The stream computing data processing method can be executed by a computing processing device. The computing processing device can be Figure 1 The computer device 102 shown in FIG. Figure 2 As shown, the stream computing data processing method includes at least steps S210 to S240, which are described in detail as follows:

[0032] In step S210, production parameter data of the blast furnace ironmaking process is obtained. In one embodiment of the present application, the production parameter data includes temperature data inside the blast furnace, pressure data on the top of the blast furnace, current data of the blast furnace equipment, rolling temperature data, weight data and size data of finished steel castings, etc.

[0033] In step S220, the production parameter data is preprocessed to obtain preprocessed data. In one embodiment of the present application, the preprocessing methods include filtering and cleaning, data integration, and data encryption. After obtaining the preprocessed data, it is sent to the data processing platform via a data transmission channel, which is an HTTPS (HyperText Transfer Protocol Secure) transmission channel.

[0034] In step S230, the preprocessed data is converted into stream data, and the target processing operator and the execution order of the target processing operators are determined based on the processing requirements of the stream data. In one embodiment of the present application, the processing requirements of the stream data are determined by the configuration information of the stream data in the data processing platform. The configuration information of the stream data includes the binding relationship between the stream data and the processing operators, as well as the execution order of the processing operators. The target processing operators include: data aggregation operators, composite operation operators, script calculation operators, etc.

[0035] In step S240, the target processing operator performs computational processing on the stream data in the execution order to obtain a stream data processing result. In one embodiment of the present application, by converting the preprocessed data into stream data and determining the target processing operator and the execution order of the target processing operators based on the processing requirements of the stream data, the target processing operator performs computational processing on the stream data in the execution order, thereby meeting the real-time processing requirements and batch processing requirements of blast furnace production parameter data.

[0036] In one embodiment of the present application, the process of preprocessing the production parameter data includes:

[0037] The production parameter data is classified and labeled to obtain classified and labeled data. In one embodiment of the present application, the production parameter data is divided into three categories according to its source, including: acquisition category, configuration category, and derived category. The acquisition category is measurement parameter data obtained through sensors, etc., the configuration category is configuration parameter data obtained through pre-configuration or real-time configuration on a data processing platform, and the derived category is obtained by processing the measurement parameter data and / or the configuration parameter data.

[0038] The classified labeled data is filtered and cleaned to obtain classified cleaned data; and the classified cleaned data is integrated according to preset dimensions to obtain classified integrated data. In one embodiment of the present application, the process of filtering and cleaning the classified labeled data includes: deleting abnormal data from the classified labeled data, deleting duplicate data from the classified labeled data, etc., to obtain the deleted classified labeled data, and filling the missing values ​​in the deleted classified labeled data to obtain the classified cleaned data, and filling the missing values ​​in the deleted classified labeled data includes: mean filling, median filling or mode filling, etc. The preset dimensions include: time dimension or production process dimension, etc. The process of integrating the classified cleaned data according to the preset dimensions includes: integrating the classified cleaned data according to different data types (for example, temperature, pressure, current, weight, size) in different time sequences or integrating the cleaned production parameter data according to different data types (for example, temperature, pressure, current, weight, size) in different production batches. For example, for all the data that have been cleaned and integrated in a steel production process, the blast furnace internal temperature data, blast furnace top pressure data, equipment current data, and the corresponding weight and size data of the finished steel castings within the same time period are integrated into a data unit. In the data unit, the data corresponding to each time node or each process node is regarded as a group of data. When the data in the data unit is subsequently converted into stream data processing, each group of data can be conveniently processed according to the time series or production process, so as to better explore the correlation between the data.

[0039] The classified integrated data is encrypted to obtain preprocessed data. In one embodiment of the present application, the classified integrated data is encrypted using a symmetric encryption algorithm and a key, and the encrypted classified integrated data is stored in a database or other storage medium. During storage, the encrypted classified integrated data, the collection timestamp of the classified integrated data, and the data source of the classified integrated data are stored together, and an encrypted data transmission channel is established using the SSL (Secure Sockets Layer) / TLS (Transport Layer Security) protocol to transmit the preprocessed data.

[0040] In one embodiment of the present application, the process of classifying and labeling production parameter data includes:

[0041] Production parameter data is divided into measured parameter data, configured parameter data, and derived parameter data. In one embodiment of the present application, the derived parameter data is determined by the measured parameter data and / or the configured parameter data; the measured parameter data is obtained by measuring parameters using sensors installed on the blast furnace, the configured parameter data is obtained by pre-configuration or real-time configuration via a data processing platform, and the derived parameter data is obtained by processing the measured parameter data and / or the configured parameter data.

[0042] The measurement parameter data, configuration parameter data and derived parameter data are used as input parameters of the classification marker operator, and the measurement parameter data, configuration parameter data and derived parameter data are classified and marked by the classification marker operator to obtain classification marker data. In one embodiment of the present application, in the data processing platform, the classification marker operator generally refers to a mathematical or logical operator used to mark or classify data in data classification, pattern recognition or machine learning. After the measurement parameter data, configuration parameter data and derived parameter data are input into the classification marker operator, data carrying the classification marker (i.e., classification marker data) is obtained.

[0043] In one embodiment of the present application, the process of converting preprocessed data into streaming data includes:

[0044] Decrypting the pre-processed data to obtain decrypted data. In one embodiment of the present application, the data processing platform receives the pre-processed data and decrypts the pre-processed data using the encryption algorithm and key used in the encryption process.

[0045] The decrypted data is converted according to a streaming data format to obtain streaming data. In one embodiment of the present application, a streaming data format is a structured data representation method designed for real-time or continuous data transmission. The streaming data format includes a binary stream format, a text stream format, etc., which are not specifically limited here.

[0046] In one embodiment of the present application, the process of determining the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data includes:

[0047] According to the processing requirements of the stream data, the operation type of the stream data and the execution order of the operation type are determined. In one embodiment of the present application, the user or designer can determine the processing requirements of the stream data according to the production requirements of the blast furnace, and according to the processing requirements of the stream data, determine the binding relationship between the stream data and the processing operator, as well as the execution order of the processing operator, and configure the binding relationship between the stream data and the processing operator, as well as the execution order of the processing operator in the stream data processing flow design interface of the data processing platform. In the stream data processing flow design interface of the data processing platform, the operator types that can be provided include: data input operator, data output operator, application programming interface input operator, data extraction operator, data reorganization operator, data filtering operator, data aggregation operator, composite operation operator, script calculation operator, etc. Users or technicians can drag and drop the required operators one by one into the stream data processing flow design interface, connect them end to end according to the execution order of the processing operators, and then pop up the parameter setting panel to set the required operator parameters. After the setting is completed, manually enter data and view the output data to check whether the required operators can accurately perform their intended functions. Operation types include data aggregation, compound operations, data input, data output, application programming interface input, data extraction, data reorganization, data filtering, and script calculations.

[0048] The processing operator that performs the operation type is used as the target processing operator; and the execution order of the operation type is used as the execution order of the target processing operator. In one embodiment of the present application, if the operation types of the stream data are data reorganization and data aggregation, the data reorganization operator and the data aggregation operator are used as the target processing operators. If the execution order of data reorganization is the first execution order and the execution order of data aggregation is the second execution order, the execution order of the data reorganization operator is the first execution order and the execution order of the data aggregation operator is the second execution order.

[0049] In another embodiment of the present application, if the target processing operator processes stream data in groups and there are multiple target processing operators, the operation type for each group of stream data and the execution order of the operation type for each group of stream data are determined according to the processing requirements of each group of stream data; the processing operator that executes the operation type for each group of stream data is used as the target processing operator, and the execution order of the target processing operator is determined according to the successive operation order of all groups of stream data and the execution order of the operation type of each group of stream data.

[0050] In one embodiment of the present application, if the target processing operator processes stream data in groups and there are multiple target processing operators, after obtaining the stream data processing result, the stream computing data processing method further includes:

[0051] Extract process data for each target processing operator from the stream data processing results. In one embodiment of the present application, the process data includes the input time, output time, and volume of each set of stream data. Each set of stream data is distinguished by a different tag. The volume of each set of stream data is calculated based on the number of bytes in each set of stream data.

[0052] The processing time of each set of stream data at each target processing operator is calculated based on the input time and output time of each set of stream data at each target processing operator. In one embodiment of the present application, the processing time of each set of stream data at each target processing operator is the time difference between the input time and the output time of each set of stream data at each target processing operator.

[0053] Based on the amount of stream data for each group of stream data and the processing time of each group of stream data in each target processing operator, the amount of real-time stream data processed by each target processing operator per unit time is calculated; and from the amount of real-time stream data processed by each target processing operator per unit time, the maximum amount of real-time stream data processed by each target processing operator is determined. In one embodiment of the present application, the formula for calculating the amount of real-time stream data processed by each target processing operator per unit time is as follows:

[0054]

[0055] Among them, W i represents the amount of real-time stream data processed by the i-th target processing operator per unit time, S j represents the amount of flow data of the jth group of flow data, ΔT i,j Indicates the processing time of the j-th group of stream data at the i-th target processing operator.

[0056] In one embodiment of the present application, the data processing platform also provides a stream data processing process query interface, through which analysis data of the stream data processing process can be queried. The analysis data includes: the processing time of each group of stream data in each target processing operator, the real-time stream data volume processed by each target processing operator per unit time, the maximum real-time stream data volume processed by each target processing operator, etc., so that users can have an in-depth understanding of the various performance indicators and data processing status of the stream data processing process during operation, providing a strong basis for further optimizing the stream data processing process and improving process processing efficiency.

[0057] In one embodiment of the present application, after obtaining the processing time of each set of stream data in each target processing operator, the stream computing data processing method further includes:

[0058] The processing time of each set of stream data in the computational processing process is calculated based on the input time and output time of each target processing operator for each set of stream data. In one embodiment of the present application, the processing time of each set of stream data in each target processing operator is determined by the input time of each set of stream data into the first target processing operator and the output time of each set of stream data from the last target processing operator. For example, if a set of stream data needs to be processed and computed by three target processing operators, the processing time of this set of stream data in the computational processing process is the difference between the output time of this set of stream data from the third target processing operator and the input time of this set of stream data into the first target processing operator.

[0059] Based on the processing time of each set of stream data in each target processing operator and the processing time of each set of stream data in the calculation and processing process, the time proportion of each target processing operator performing calculation and processing on each set of stream data is calculated. In one embodiment of the present application, the calculation formula for the time proportion of each target processing operator performing calculation and processing on each set of stream data is as follows:

[0060]

[0061] Among them, p i,j It represents the time proportion of the i-th target processing operator to calculate and process the j-th group of stream data, ΔT i,j Indicates the processing time of the jth group of stream data at the i-th target processing operator, ΔT j Indicates the processing time of the jth group of stream data during the calculation process.

[0062] Based on the time proportion of each target processing operator performing calculation processing on each set of stream data, the average time proportion of each target processing operator performing calculation processing on each set of stream data is calculated. In one embodiment of the present application, the calculation formula for the average time proportion of each target processing operator performing calculation processing on each set of stream data is as follows:

[0063]

[0064] Among them, B i represents the average time proportion of the i-th target processing operator to calculate and process each set of stream data, p i,j represents the time proportion of the i-th target processing operator to calculate and process the j-th group of stream data, n represents the stream data group number, ΔT i,j Indicates the processing time of the jth group of stream data at the i-th target processing operator, ΔT j Indicates the processing time of the jth group of stream data during the calculation process.

[0065] In another embodiment of the present application, the data processing platform also provides a stream data processing process query interface, through which analysis data of the stream data processing process can be queried. The analysis data includes: the processing time of each group of stream data in each target processing operator, the amount of real-time stream data processed by each target processing operator per unit time, the maximum amount of real-time stream data processed by each target processing operator, the average time proportion of each target processing operator for calculating and processing each group of stream data, etc., so that users can have an in-depth understanding of the various performance indicators and data processing conditions of the stream data processing process during operation, providing a strong basis for further optimizing the stream data processing process and improving process processing efficiency.

[0066] In one embodiment of the present application, after obtaining the average time proportion, the stream computing data processing method further includes:

[0067] The stream data processing results, average time proportion, and maximum real-time stream data volume processed by each target processing operator are stored and output; and / or, the stream data processing results, average time proportion, and maximum real-time stream data volume processed by each target processing operator are displayed.

[0068] In one embodiment of the present application, the stream data processing results, average time proportion and the maximum real-time stream data volume processed by each target processing operator are stored, output, displayed, etc., so that users or technicians can view, query or obtain analysis data and stream data processing results.

[0069] Figure 3 is a flow chart of a stream computing data processing method shown in another exemplary embodiment of the present application. Figure 3 As shown, the process of the stream computing data processing method includes: (310) accessing the production parameter data in the blast furnace ironmaking process; (320) processing the production parameter data and outputting the processed production parameter data; the processing methods include: classification marking, filtering and cleaning, and data integration; (330) converting the processed production parameter data into stream data, and matching the stream data with the target processing operator, and the target processing operator performs calculation processing on the stream data in the execution order to obtain the stream data processing result; (340) analyzing the stream data processing result to obtain analysis data, and storing and outputting the stream data processing result and analysis data; the analysis data includes: the processing time of each group of stream data in each target processing operator, the real-time stream data volume processed by each target processing operator per unit time, the maximum real-time stream data volume processed by each target processing operator, the average time proportion of each target processing operator for calculating and processing each group of stream data, etc.; (350) visualizing the stream data processing results and analysis data, and providing them to the design interface of the stream data processing process.

[0070] Figure 4This is a block diagram of a stream computing data processing device shown in an exemplary embodiment of the present application. Figure 4 In the embodiment, the stream computing data processing device includes:

[0071] The data access module 401 is used to access production parameter data during the blast furnace ironmaking process.

[0072] The intelligent gateway 402 is used to process the production parameter data and output the processed production parameter data. The intelligent gateway has built-in data preprocessing program, encryption storage program and encryption transmission program.

[0073] The data processing platform 403 is used to convert the processed production parameter data into stream data and match the stream data with target processing operators. The target processing operators calculate and process the stream data in the execution order to obtain the stream data processing results.

[0074] The storage module 404 is used to analyze the stream data processing results, obtain analysis data, and store and output the stream data processing results and analysis data. The storage module has a built-in data quality assessment program and an information export interface.

[0075] The interactive module 405 is used to visualize the stream data processing results and analysis data, and provide them to the stream data processing flow design interface.

[0076] The following describes an embodiment of the device of the present application, which can be used to execute the stream computing data processing method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the stream computing data processing method in the above embodiment of the present application.

[0077] Figure 5 This is a block diagram of a stream computing data processing device shown in another exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is specifically configured in the computer device 102. The apparatus may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the apparatus is applicable.

[0078] like Figure 5 As shown, the exemplary stream computing data processing device 500 includes:

[0079] The data acquisition module 501 is used to obtain production parameter data during the blast furnace ironmaking process.

[0080] The data processing module 502 is used to pre-process the production parameter data to obtain pre-processed data;

[0081] The operator determination module 503 is used to convert the pre-processed data into stream data, and determine the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data.

[0082] The data calculation module 504 is used for the target processing operator to perform calculations on the stream data in the execution order to obtain the stream data processing results.

[0083] In one embodiment of the present application, the production parameter data includes temperature data inside the blast furnace, pressure data on the top of the blast furnace, current data of the blast furnace equipment, rolling temperature data, weight data and size data of finished steel castings, etc.

[0084] In one embodiment of the present application, the pre-processing method includes: filtering and cleaning, data integration, and data encryption. After the pre-processed data is obtained, it is sent to the data processing platform through a data transmission channel, and the data transmission channel is an HTTPS transmission channel.

[0085] In one embodiment of the present application, the process of converting preprocessed data into stream data and determining the target processing operators and the execution order of the target processing operators based on the processing requirements of the stream data is performed within the data processing platform. The processing requirements of the stream data are determined by the configuration information of the stream data in the data processing platform. The configuration information of the stream data includes the binding relationship between the stream data and the processing operators, as well as the execution order of the processing operators. The target processing operators include: data aggregation operators, composite operation operators, script calculation operators, etc.

[0086] In one embodiment of the present application, by converting the preprocessed data into stream data, and determining the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data, the target processing operator calculates and processes the stream data according to the execution order to meet the real-time processing requirements and batch processing requirements of blast furnace production parameter data.

[0087] It should be noted that the stream computing data processing apparatus provided in the above embodiments and the stream computing data processing method provided in the above embodiments are based on the same concept. The specific manner in which the various modules and units perform their operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the stream computing data processing apparatus provided in the above embodiments can, as needed, allocate the aforementioned functions to different functional modules, i.e., divide the internal structure of the apparatus into different functional modules to perform all or part of the functions described above. This is not a limitation herein.

[0088] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the stream computing data processing method provided in the above-mentioned embodiments.

[0089] Figure 6 This is a structural diagram of a computer system of an electronic device according to another exemplary embodiment of the present application. Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0090] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 into the random access memory (RAM) 603, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0092] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the various functions defined in the system of the present application are executed.

[0093] Another aspect of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a computer processor, the computer executes the stream computing data processing methods provided in the above embodiments. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0094] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0095] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A stream computing data processing method, characterized in that: The method comprises: Obtain production parameter data during blast furnace ironmaking; Preprocessing the production parameter data to obtain preprocessed data; the preprocessing method includes: filtering and cleaning, data integration, and data encryption; Convert the pre-processed data into stream data, and determine the target processing operators and the execution order of the target processing operators according to the processing requirements of the stream data; the target processing operators include: data aggregation operators, composite operation operators, and script calculation operators; The target processing operator performs calculation processing on the stream data in the execution order to obtain a stream data processing result.

2. The stream computing data processing method according to claim 1, characterized in that: The process of preprocessing the production parameter data includes: Classifying and marking the production parameter data to obtain classified marked data; Filtering and cleaning the classified marked data to obtain classified cleaned data; and integrating the classified cleaned data according to preset dimensions to obtain classified integrated data; The classified integrated data is encrypted to obtain the pre-processed data.

3. The stream computing data processing method according to claim 2, characterized in that: The process of classifying and labeling the production parameter data includes: Dividing the production parameter data into measurement parameter data, configuration parameter data, and derived parameter data; the derived parameter data is determined by the measurement parameter data and / or the configuration parameter data; The measurement parameter data, the configuration parameter data and the derived parameter data are used as input parameters of a classification labeling operator, and the measurement parameter data, the configuration parameter data and the derived parameter data are classified and labeled by the classification labeling operator to obtain the classification labeling data.

4. The stream computing data processing method according to any one of claims 1 to 3, characterized in that: The process of converting the pre-processed data into streaming data includes: decrypting the preprocessed data to obtain decrypted data; The decrypted data is converted according to a stream data format to obtain the stream data.

5. The stream computing data processing method according to any one of claims 1 to 3, characterized in that: The process of determining target processing operators and the execution order of the target processing operators according to the processing requirements of the stream data includes: Determine the type of operation on the stream data and the execution order of the operation types according to the processing requirements of the stream data; the operation types include data aggregation, compound operation, and script calculation; The processing operator that executes the operation type is used as the target processing operator; and the execution order of the operation type is used as the execution order of the target processing operator.

6. The stream computing data processing method according to any one of claims 1 to 3, characterized in that: If the target processing operator processes the stream data in groups and there are multiple target processing operators, after obtaining the stream data processing result, the method further includes: Extracting process data of each target processing operator in the stream data processing result; the process data includes: input time, output time and stream data volume of each set of stream data; Based on the input time and output time of each set of stream data in each target processing operator, the processing time of each set of stream data in each target processing operator is calculated; The real-time stream data volume processed by each target processing operator per unit time is calculated based on the stream data volume of each group of stream data and the processing time of each group of stream data in each target processing operator; and the maximum real-time stream data volume processed by each target processing operator is determined from the real-time stream data volume processed by each target processing operator per unit time.

7. The stream computing data processing method according to claim 6, characterized in that: After obtaining the processing time of each set of stream data in each target processing operator, the method further includes: Based on the input time and output time of each set of stream data in each target processing operator, the processing time of each set of stream data in the computing process is calculated; Based on the processing time of each set of stream data in each target processing operator and the processing time of each set of stream data in the calculation process, calculate the proportion of time each target processing operator spends on calculating and processing each set of stream data; According to the proportion of time that each target processing operator performs calculation and processing on each group of stream data, the average proportion of time that each target processing operator performs calculation and processing on each group of stream data is calculated.

8. The stream computing data processing method according to claim 7, characterized in that: After obtaining the average time proportion, the method further includes: Storing and outputting the stream data processing results, the average time proportion, and the maximum real-time stream data volume processed by each target processing operator; And / or, the stream data processing result, the average time proportion and the maximum real-time stream data volume processed by each target processing operator are displayed.

9. A stream computing data processing device, characterized in that: include: Data acquisition module, used to obtain production parameter data during the blast furnace ironmaking process; A data processing module, configured to preprocess the production parameter data to obtain preprocessed data; Preprocessing methods include: filtering and cleaning, data integration, and data encryption; An operator determination module is used to convert the pre-processed data into stream data and determine the target processing operator and the execution order of the target processing operator according to the processing requirements of the stream data; the target processing operator includes: data aggregation operator, composite operation operator, script calculation operator; The data calculation module is used for the target processing operator to perform calculation processing on the stream data according to the execution order to obtain a stream data processing result.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the stream computing data processing method according to any one of claims 1 to 8.

Images