Industrial manufacturing real-time stream processing method, equipment and medium

Abstract

The invention discloses an industrial manufacturing real-time stream processing method and device and a medium, relates to the field of data processing, and solves the problems of data time disorder and message accumulation. The method comprises the following steps: splitting service data into corresponding sub-queues according to a service theme identification field; and dividing the sub-queues into batches with preset specifications according to a preset time window. For each piece of data of each batch, extracting a dimension key, a unique key and a time identifier, and packaging the dimension key, the unique key and the time identifier into a first tuple; and generating a data group according to the dimension key grouping, splicing the group identifier of each group of unique keys according to a preset rule, and packaging the group identifier and the data group into a second tuple. And performing hash partitioning on the second tuple according to the group identifier, splitting the second tuple of each partition into single data, and calculating an index according to a time sequence. According to the method, the message queues are split according to the identifier combination, the three-dimensional identifier groups such as the dimension keys are extracted, and the unique key hash partitions are spliced, so that the real-time streaming data are processed according to the time sequence, meanwhile, real-time index calculation is efficiently completed, and the processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to an industrial manufacturing real-time stream processing method, device and medium. BACKGROUND

[0002] In the field of industrial intelligent manufacturing, production processes continuously generate business data with real-time stream attributes, such as Glass substrate processing records generated by continuous operation of equipment in a semiconductor chip processing process. Such data not only has high dimensions, but also needs to be strictly processed in time sequence.

[0003] Current real-time stream data processing methods mostly use a single message queue plus single-thread processing. However, in a complex production process, the amount of data generated is huge, which can easily cause message accumulation and slow down the real-time stream data processing efficiency. When using a single message queue plus multi-thread partition calculation, the time sequence of data writing into the database is different when multiple partitions are concurrently updated, which can cause data with later time to be written into the database first compared with data with earlier time, making it difficult to guarantee the overall time sequence of data and easily causing index calculation errors. If a distributed lock is used, the real-time stream data processing efficiency will be greatly reduced. SUMMARY

[0004] Embodiments of the present application provide an industrial manufacturing real-time stream processing method, device and medium, which are used to solve the problem of data time disorder caused by multi-thread concurrent calculation and the problem of message accumulation caused by single-thread calculation.

[0005] Embodiments of the present application use the following technical solutions: On the one hand, the embodiments of the present application provide an industrial manufacturing real-time stream processing method, which comprises: according to a business topic identification field, splitting an original message queue of business data into sub-message queues of different business topics; according to a preset time window, dividing each sub-message queue into batches of a preset specification; extracting a dimension key, a unique key and a time identifier for each data in each batch, and encapsulating the dimension key, the unique key, the time identifier and the corresponding data as a first tuple; grouping all first tuples according to the dimension key to generate data groups; according to a preset splicing rule, splicing the unique keys of all first tuples in each data group respectively to generate a unique group identifier for each data group, and encapsulating the group identifier and all first tuples of the corresponding data group to generate a second tuple corresponding to each data group; according to the group identifier, performing hash partitioning on all second tuples so that second tuples with the same group identifier are allocated to the same partition; splitting the second tuples of each partition into single message data, and performing index calculation on the single message data according to time sequence to obtain index calculation results.

[0006] In one example, the dimension key, unique key and time identifier are extracted for each piece of data in each batch, specifically including: determining a set of dimension attributes participating in the calculation in each piece of data according to the data dependency relationship of real-time index calculation; extracting the corresponding attribute values from the data fields according to the set of dimension attributes, and generating the dimension key according to the preset mapping rule; extracting the key value that can globally uniquely identify the data record from the fields contained in each piece of data, and determining the key value as the unique key; extracting the data generation time information of each piece of data, converting the data generation time information into a standardized timestamp format, and determining the time identifier.

[0007] In one example, all first tuples are grouped according to the dimension key to generate data groups, specifically including: extracting the attribute values corresponding to the dimension attributes on which the real-time index calculation depends in the dimension key; arranging the attribute values in a preset order and splicing them using a preset separator to generate a composite string; comparing the composite strings of all first tuples to divide the first tuples with the same composite string into the same data group.

[0008] In one example, the unique keys of all first tuples in each data group are spliced according to a preset splicing rule to generate a unique group identifier for each data group, specifically including: sorting the first tuples in the same data group according to the time identifier, and sequentially extracting the unique keys in each first tuple after sorting; splicing all the unique keys extracted by a preset connector at the beginning and end according to the sorting order of the first tuples to generate a unique string identifier, and determining the string identifier as the group identifier.

[0009] In one example, all second tuples are hashed partitioned according to the group identifier, so that second tuples with the same group identifier are allocated to the same partition, specifically including: determining the number of partitions according to the number of computing resources and the total amount of data; calculating the absolute value of the hash value of the group identifier according to a preset hash partitioner; obtaining the partition number by taking the modulus of the absolute value with respect to the number of partitions; and allocating second tuples with the same group identifier to the same partition according to the partition number.

[0010] In one example, the second tuples in each partition are split into single message data, and the index calculation is performed on the single message data according to the time sequence to obtain the index calculation result, specifically including: traversing the second tuples in each partition to split the ordered message list of the second tuples into single message data; performing data cleaning and data conversion operations on each message data according to the time sequence of each message to generate standardized data corresponding to each message data; and performing index calculation on the standardized data to obtain the index calculation result.

[0011] In one example, before splitting the original message queue of the business data into sub-message queues of different business topics according to the business topic identification field, the method further comprises: determining whether the original message queue is a single-topic message queue; and when the original message queue is a single-topic message queue, dividing the original message queue into batches of a preset specification according to a preset time window.

[0012] In one example, after splitting the second tuple of each partition into single message data and performing index calculation on the single message data according to time sequence, the method further comprises: writing the index results calculated by each partition and the index corresponding dimension data into a parallel processing database concurrently.

[0013] In another aspect, an embodiment of the present application provides an industrial manufacturing real-time stream processing device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the industrial manufacturing real-time stream processing methods.

[0014] In another aspect, an embodiment of the present application provides an industrial manufacturing real-time stream processing nonvolatile computer storage medium, which stores computer executable instructions, and the computer executable instructions can perform any one of the industrial manufacturing real-time stream processing methods.

[0015] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects: The present application solves the problem of message accumulation caused by excessive single data quantity by splitting the original message queue according to the business topic identification and taking into account the data quantity balance of each queue, and lays a balanced data foundation for subsequent data processing. Secondly, by dividing each sub-message queue data into batches according to a dynamically adjustable preset time interval, extracting the dimension key, unique key and time identification from each data and encapsulating them into a tuple, a method is obtained to adapt to different data quantity scenarios, provide clear data identification for subsequent grouping and sorting, and avoid the confusion of index calculation due to the high dimension characteristics of industrial data. Furthermore, by grouping, sorting and splicing the unique keys in the group to generate a group identification and encapsulating it into a new tuple according to the dimension key, the out-of-order problem caused by the same dimension data belonging to different partitions during subsequent concurrent calculation is avoided, and distributed locks are not needed, without reducing processing efficiency. By hashing partitioning with the group identification as the partition key, the calculation delay caused by data skew is avoided, and multi-thread concurrent processing is supported to improve real-time calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application, some embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A schematic flowchart of a real-time stream processing method for industrial manufacturing provided in an embodiment of this application; Figure 2 A diagram illustrating the overall execution steps of a real-time stream processing method for industrial manufacturing, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a real-time stream processing device for industrial manufacturing, provided as an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] Figure 1 This is a schematic flowchart illustrating a real-time stream processing method for industrial manufacturing, provided as an embodiment of this application. This method can be applied to various business domains. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.

[0020] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0021] Based on this Figure 1 The process may include the following steps: S101: Based on the business topic identifier field, split the original message queue of business data into sub-message queues for different business topics.

[0022] In some embodiments of this application, for processing real-time industrial manufacturing streams, it is first necessary to obtain the original message queue corresponding to the business data generated in real time during the industrial manufacturing process (such as the operation log during equipment operation). Then, it is determined whether the original message queue contains data of multiple different business topics. If it contains multiple different business topics, a field that can identify the business topic (the business topic identified by this field must meet the conditions of independent business logic and a quantity of 2-100) is selected as the basis for splitting. The original message queue is split into multiple message queue topics (denoted as topic_1 to topic_n) according to this field. When splitting, it is necessary to ensure that the data volume of each topic in topic_1 to topic_n is balanced. If the original message queue has been divided by business theme (i.e., it contains data for a single business theme) or contains only one business theme data, then there is no need to split it. The original message queue can be directly used as a sub-message queue for subsequent processing.

[0023] This application addresses the problem of message backlog caused by excessively large single data volumes by splitting the original message queues according to business theme identifiers while ensuring a balanced data volume across each queue, thus laying a balanced data foundation for subsequent data processing.

[0024] S102: Divide each sub-message queue into batches of preset specifications according to the preset time window.

[0025] In some embodiments of this application, before batching, the data flow of the sub-message queues (including the split-generated topic_1 to topic_n or the unsplit original message queue topic) is monitored in real time. Then, based on the monitoring results, the interval of the time window is dynamically adjusted. The initial preset time window is set to 15 seconds, and the interval ranges from 1 second to 60 seconds. When the data flow increases, the interval of the time window is shortened accordingly; when the data flow decreases, the interval of the time window is increased accordingly, so that the data volume of each batch adapts to the processing capacity of the stream computing engine, avoiding message backlog or idle processing resources. After adjustment, the real-time streaming data of the sub-message queues is divided into multiple batches of preset specifications.

[0026] S103: Extract the dimension key, unique key, and time identifier for each data item in each batch, and encapsulate the dimension key, unique key, time identifier, and corresponding data into a first tuple; the dimension key includes the real-time indicator calculation dependency; the unique key includes the unique identifier of each data item.

[0027] In some embodiments of this application, for each piece of real-time data in each preset batch, three types of identifiers are extracted: First, based on the data dependency relationship of the real-time indicator calculation, the set of dimensional attributes involved in the calculation of each piece of data is determined. Then, based on the set of dimensional attributes, the corresponding attribute values ​​are extracted from the data fields, and dimensional keys are generated according to preset mapping rules. At the same time, key values ​​that can globally and uniquely identify data records are extracted from the fields contained in each piece of data, and these key values ​​are determined as unique keys. Simultaneously, the data generation time information of each piece of data is extracted, and the data generation time information is converted into a standardized timestamp format and determined as a time identifier.

[0028] Furthermore, the dimension key, unique key, and time identifier are associated and encapsulated with the corresponding original data record to form a structured first tuple, in which the first tuple maintains the correspondence between the dimension key, unique key, time identifier, and the original data record.

[0029] By dividing the data of each sub-message queue into batches according to a dynamically adjustable preset time interval, and extracting the dimension key, unique key, and time identifier from each data item and encapsulating them into tuples, a method is obtained that adapts to different data volume scenarios, provides clear data identifiers for subsequent grouping and sorting, and avoids the confusion of dimensional dependence in indicator calculation caused by the high-dimensional characteristics of industrial data.

[0030] S104: Group all first tuples according to the dimension key to generate data groups; wherein all first tuples in the same data group have the same dimension key.

[0031] In some embodiments of this application, after obtaining the first tuple, the attribute values ​​corresponding to the dimension attributes on which the real-time indicator calculation depends are extracted from the dimension key of the first tuple; then the attribute values ​​are arranged according to a preset order and concatenated using preset delimiters (such as hash symbols and underscores) to generate a composite string. If the first tuple contains only one tuple, the concatenation result is directly equal to the unique key of that tuple. Finally, the composite strings of all first tuples are compared to divide the first tuples with the same composite string into the same data group.

[0032] S105: According to the preset splicing rules, the unique keys of all first tuples in each data group are spliced ​​together to generate a unique group identifier for each data group, and the group identifier is encapsulated with all first tuples of the corresponding data group to generate a second tuple corresponding to each data group.

[0033] In some embodiments of this application, after dividing the data into groups, the first tuples within the same data group are sorted according to time identifiers, and the unique keys in each sorted first tuple are extracted sequentially.

[0034] Furthermore, based on the sorting order of the first tuple, all extracted unique keys are concatenated end to end using a preset connector (as in underscore) to generate a unique string identifier, which is then used as the group identifier.

[0035] Furthermore, the group identifier is encapsulated with all the first tuples of the corresponding data group to generate the second tuple corresponding to each data group.

[0036] By grouping, sorting, and concatenating unique keys within the groups to generate group identifiers and encapsulate them into new tuples, the method avoids the out-of-order problem caused by data of the same dimension belonging to different partitions during subsequent concurrent computations, and does not rely on distributed locks or reduce processing efficiency.

[0037] S106: Based on the group identifier, hash partition all second tuples so that second tuples with the same group identifier are assigned to the same partition.

[0038] In some embodiments of this application, the number of partitions is determined based on the amount of computing resources and the total amount of data; For example, it can be determined based on the data volume per second (TPS) of the real-time streaming data, with one additional partition corresponding to every 1000 increase in TPS.

[0039] Furthermore, the hash partitioner provided by the Spark or Flink computing engine is invoked, using the group identifier in the second tuple as the partition key, to calculate the absolute hash value of the group identifier; then, based on the absolute hash value, the modulo is taken with the number of partitions to obtain the partition number; finally, based on the partition number, second tuples with the same group identifier are assigned to the same partition. The difference in data volume between partitions does not exceed a preset load balancing threshold.

[0040] By using the group identifier as the partition key for hash partitioning, computational delays caused by data skew are avoided, while multi-threaded concurrent processing is supported to improve real-time computation efficiency.

[0041] S107: Split the second tuple of each partition into individual message data, and calculate the metrics for each individual message data according to the time order to obtain the metric calculation results.

[0042] In some embodiments of this application, after partitioning the second tuple, the second tuple of each partition is traversed, and the ordered message list of the second tuple is split into individual message data using Spark or Flink operators. Then, according to the time order of each message, data cleaning (such as removing null data and correcting data with format errors) and data transformation operations are performed on each message data to generate standardized data corresponding to each message data. Finally, the standardized data is used to calculate the index to obtain the index calculation result.

[0043] Furthermore, the metric results calculated for each partition and the corresponding dimension data are concurrently written to the parallel processing database.

[0044] By splitting the data list into individual data entries for each partition, performing cleaning and processing, and calculating indicators, and writing them concurrently to the MPP database, each partition can be processed independently without interfering with each other. This ensures the accuracy of indicator calculations in chronological order and further improves the overall data processing and storage efficiency through concurrent writing, thus adapting to the needs of high-dimensional real-time streaming data processing in industry.

[0045] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S107 will be described sequentially, but this does not mean that steps S101 to S107 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S107 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S107 can be appropriately adjusted according to actual needs.

[0046] pass Figure 1 This application addresses the issue of message backlog caused by excessively large single data volumes by splitting the original message queues according to business theme identifiers while ensuring balanced data volume across queues. This lays a balanced data foundation for subsequent data processing. Secondly, by dividing the data in each sub-message queue into batches at dynamically adjustable preset time intervals, and extracting the dimension key, unique key, and time identifier from each data entry and encapsulating them into tuples, a method is developed that adapts to different data volume scenarios, provides clear data identifiers for subsequent grouping and sorting, and avoids the dimensional chaos caused by the high-dimensionality of industrial data in indicator calculations. Furthermore, by grouping and sorting tuples according to dimension keys, concatenating unique keys within groups to generate group identifiers, and encapsulating them into new tuples, the method avoids the out-of-order problem caused by data of the same dimension belonging to different partitions during subsequent concurrent calculations, without relying on distributed locks or reducing processing efficiency. Finally, by using the group identifier as the partition key for hash partitioning, the method avoids computational delays caused by data skew and supports multi-threaded concurrent processing to improve real-time computation efficiency.

[0047] Figure 2 This is a diagram illustrating the overall execution steps of a real-time stream processing method for industrial manufacturing, provided in an embodiment of this application.

[0048] exist Figure 2 The document illustrates the overall execution steps of this application, including the data acquisition step, the identification extraction and grouping after the data acquisition is completed, and the subsequent data partitioning and multi-threaded data processing.

[0049] Figure 3 A schematic diagram of a real-time stream processing device for industrial manufacturing, provided as an embodiment of this application, includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, An industrial manufacturing real-time stream processing method is described, in which a memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform any of the above-mentioned tasks.

[0050] Some embodiments of this application provide a non-volatile computer storage medium for real-time stream processing in industrial manufacturing, which stores computer-executable instructions capable of executing any of the above-described real-time stream processing methods for industrial manufacturing.

[0051] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0052] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0053] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0057] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0058] Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0059] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0060] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0061] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A real-time stream processing method for industrial manufacturing, characterized in that, The method includes: Based on the business topic identifier field, the original message queue of business data is split into sub-message queues for different business topics; Based on a preset time window, each sub-message queue is divided into batches of a preset size; For each data entry in each batch, extract the dimension key, unique key, and time identifier, and encapsulate the dimension key, unique key, time identifier, and corresponding data into a first tuple; the dimension key includes the real-time metric calculation dependency; the unique key includes the unique identifier of each data entry. Group all first tuples according to the dimension key to generate data groups; wherein all first tuples within the same data group have the same dimension key; According to the preset splicing rules, the unique keys of all first tuples in each data group are spliced ​​together to generate a unique group identifier for each data group. The group identifier is then encapsulated with all first tuples of the corresponding data group to generate a second tuple for each data group. Based on the group identifier, hash partition all second tuples so that second tuples with the same group identifier are assigned to the same partition; The second tuple of each partition is split into individual message data, and the metrics of the individual message data are calculated according to the time order to obtain the metric calculation results.

2. The method according to claim 1, characterized in that, The extraction of dimension key, unique key, and time identifier for each data entry in each batch specifically includes: Based on the data dependencies calculated by real-time metrics, determine the set of dimensional attributes involved in the calculation for each data entry; extract the corresponding attribute values ​​from the data fields according to the set of dimensional attributes, and generate dimensional keys according to preset mapping rules; extract key values ​​that can globally and uniquely identify data records from the fields contained in each data entry, and determine the key values ​​as unique keys; extract the data generation time information for each data entry, convert the data generation time information into a standardized timestamp format, and determine it as a time identifier.

3. The method according to claim 1, characterized in that, The step of grouping all first tuples according to the dimension key to generate data groups specifically includes: Extract the attribute values ​​corresponding to the dimension attributes that the real-time metric calculation depends on from the dimension key; The attribute values ​​are arranged in a preset order and concatenated using a preset separator to generate a compound string; Compare all the first tuples of the compound strings to group the first tuples of the same compound strings into the same data group.

4. The method according to claim 1, characterized in that, The step involves concatenating the unique keys of all first tuples in each data group according to preset concatenation rules to generate a unique group identifier for each data group. Specifically, this includes: For the first tuple within the same data set, sort them according to the time identifier, and extract the unique key from each sorted first tuple in turn; Based on the sorting order of the first tuple, all extracted unique keys are concatenated end to end using a preset connector to generate a unique string identifier, and the string identifier is determined as the group identifier.

5. The method according to claim 1, characterized in that, The step of hash partitioning all second tuples according to the group identifier, so that second tuples with the same group identifier are assigned to the same partition, specifically includes: The number of partitions is determined based on the amount of computing resources and the total amount of data. Calculate the absolute value of the hash value of the group identifier according to the preset hash partitioner; Based on the absolute value, take the modulo of the number of partitions to obtain the partition number; Based on the partition number, second tuples with the same group identifier are assigned to the same partition.

6. The method according to claim 1, characterized in that, The process of splitting the second tuple of each partition into individual message data and calculating metrics for each individual message data according to time order to obtain the metric calculation results specifically includes: Iterate through the second tuple of each partition and split the ordered list of messages in the second tuple into individual message data; Based on the chronological order of the messages, each message is cleaned and transformed sequentially to generate standardized data corresponding to each message. The standardized data is used to calculate the indicators, and the results are obtained.

7. The method according to claim 1, characterized in that, Before splitting the original message queue of business data into sub-message queues for different business topics based on the business topic identifier field, the method further includes: Determine whether the original message queue is a single-topic message queue; When the original message queue is a single-topic message queue, the original message queue is divided into batches of a preset specification according to a preset time window.

8. The method according to claim 1, characterized in that, After splitting the second tuple of each partition into individual message data and calculating the metrics for each individual message data according to time order, the method further includes: The metric results and corresponding dimension data calculated for each partition are concurrently written to the parallel processing database.

9. A real-time stream processing device for industrial manufacturing, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform an industrial manufacturing real-time stream processing method as described in any one of claims 1-8.

10. An industrial manufacturing real-time stream processing storage medium, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the industrial manufacturing real-time stream processing method described in any one of claims 1-8.

Images