A method and system for organizing and storing discrete industrial time series data based on production processes

Abstract

The application discloses a kind of organization storage method and system of discrete industrial time series data based on production process, the method first uses production process description language to express the production process of single product, forms product production process description file.Secondly, product data template is generated according to production process description file.Then, according to product data template, each sensor data is associated with production process, and single product data block is generated.Finally, multiple product data blocks are integrated to form a data page composed of multiple product data blocks and the data page is organized and managed.In digital twin, production process decision analysis, time series prediction and anomaly detection, etc.Application scenarios, the convenience and efficiency of data processing are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of industrial data storage technology, and specifically relates to a method for organizing and storing discrete industrial time series data based on production processes, and in particular to a method and system for storing discrete industrial data based on the correlation between production processes and time series. Background Art

[0002] The amount of industrial data generated during factory production is rapidly increasing. This data is typically generated by sensors, equipment, and control systems, and is characterized by time series and diversity. Data collection covers all stages of the production process, providing critical information for production monitoring, quality analysis, and decision support. However, existing data storage methods are primarily non-relational, based on sensors. This lacks overall time series relevance, resulting in data being separated in independent storage and unable to reflect the inherent connections between different nodes in the production process.

[0003] Existing storage methods also suffer from inefficiencies in data query and utilization. Comprehensive analysis of production data within a specific time period often requires repeated reading and processing from multiple data sources, resulting in inefficient queries. Furthermore, due to the underutilization of data associations, redundant information exists in the storage structure, further increasing storage costs and impacting data management and utilization.

[0004] Therefore, there is an urgent need for a storage and organization method for discrete industrial data that establishes the relationship between data with the production process as the core. Organizing and storing data in a structured manner can not only improve storage efficiency, but also greatly increase the utilization value of data during query and analysis. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a method and system for organizing and storing discrete industrial time series data based on production processes. This method effectively leverages the correlation between production time series and data to achieve efficient storage and management of industrial data. This method meets the demand for data relevance and analyzability in industrial scenarios, providing strong support for industrial production optimization and intelligent decision-making.

[0006] In order to solve the above problems, the technical solutions of the present invention are as follows:

[0007] A method for organizing and storing discrete industrial time series data based on a production process includes the following steps:

[0008] S10: Use a production process description language to represent the production process of a single product, and form a product production process description file. The product production process description file describes the sensors and data types related to the product production action according to the production sequence.

[0009] S20: Generate a product data template according to the production process description file; the product data template specifies the type, name and location of each sensor data in a single product data block.

[0010] S30: Correlate each sensor data with the production process according to the product data template to generate a single product data block.

[0011] S40: Integrate multiple product data blocks to form a data page consisting of multiple product data blocks and organize and manage the data page.

[0012] In step S10, the production process description file F is defined by the production process description language L. The production process description file is a file ending with pdesc. The production process description file includes a process action name A = {a1, a2, ..., a m}(A is a set of process action names, a m is the process action name), sensor number S={s1,s2,…,s m}(S is the sensor number set, s m is the sensor number), data attribute D={d1,d2,…,d m}(D is the data attribute set, d m Data attributes for corresponding process actions) and key action tags (K is the key action tag set, It is a key action mark, that is, the name is a i whether the process action is a key action).

[0013] The production process description file is written in JSON. The process action name, sensor ID, and critical action flag are represented by the ActionName, SensorID, and isCriticalAction fields, respectively. Data attributes are represented by the key DataAttribute, and their corresponding values ​​include the data type DataType and the data length DataBytes. Finally, the production process is described using an integer sequence number as the key and the set of process action name, sensor ID, data attributes, and critical action flag as the value.

[0014] In step S20, the product data template M includes: the size of a single product data block BSize, which satisfies where b i It is action a i The data size of a i English code n for each process action represented i and data type d i , and its position in the product data block p iThe product data template is a binary file that follows the aforementioned format and has the suffix pdtem. The file begins with a header that stores metadata, including the product data template identifier, file version, number of actions, and data block size. Next is the product data template's data area, which first specifies the type and length of the product ID and then arranges the process action tuples in the order of the production process. Each tuple represents an action data item and includes the process action name, sensor ID, data type, data value, and its position offset within the data block.

[0015] The product data template M in step S20 is generated by parsing the product production process description file F.

[0016] The parsing process includes the following sub-steps:

[0017] S201: Read the process action name set A, sensor set S and its data attribute D in the production process description file F.

[0018] S202: Create a mapping f:A→N, and change the process action name a i Mapped to the action data name n in the data template i N is the set of English codes of the process action names after mapping.

[0019] S203: Determine the storage type T of each action data according to the sensor ID and data attributes d .

[0020] S204: According to the sequence of the process actions represented by the process action name set A in the production process, the offset positions of the data in the product data block are allocated to generate an offset position set P = {p1, p2, ..., p m}, p i The process action is named a i The offset position of the process action represented in the data block satisfies

[0021] S205: Generate standardized product data template M = (N, P, T d ).

[0022] The mapping f:A→N in step S202 converts semantically distinct Chinese process action names into their corresponding English codes. This mapping can be specified by the program by default or manually. In step S203, the computer-recognizable data type is determined based on the data type DataType and data length DataBytes in the data attributes. In step S204, the production process sequence is determined based on the key sequence in the production process description file, and the offset position set P is then generated based on the production process sequence.

[0023] The single product data block in step S30 is a fixed-length structure and contains the following information:

[0024] Product unique identification ID i .

[0025] Production time range T = [t start , t end ]; where t start is the start time of product production, and t end is the end time of product production.

[0026] Action data area V = {v1, v2, …, v m}. Where v i is the action data generated by a single flow action. The storage area of each action data v i is determined by P defined by M.

[0027] Where the length and type of product unique identification ID i are defined by the product data template M, and the production time range T is represented by the Unix timestamp. According to the definition of the product data template M, the production time range T can be a Unix timestamp at the level of seconds, milliseconds, or nanoseconds. The storage area of each action data v i in the action data area is determined by the offset P defined by the product data template M.

[0028] As a further improvement, the data page containing multiple product data blocks in step S40 is composed of a page header H, a data area C, and an index area I. The page header is the meta-information area of the data page, recording the offset position of the data area and the index area in the page, containing the page number, the page size, the number of data blocks, and the index position offset. The index area is used to store the index for locating the data, composed of the data block time order index, the data block ID index, and the single variable time order index.

[0029] The data block time order index, the data block ID index, and the single variable time order index contained in the index area are defined as follows:

[0030] Data block ID index: maps the product unique identification ID i to the data block position set P B , i.e., g: ID i → P B .

[0031] Data block time order index: maps the time range T to the data block B position set P B , i.e., f: T → P B .

[0032] Monovariable time sequence index: record the sequence of variable v at different time points, form a sequence {(t i ,v i )}, meet t1<t2<...<t n . Where t i is time.

[0033] The data page containing multiple product data blocks in step S40 is organized in chronological order when generated, and the positioning of the data page is guided by the secondary index. The secondary index also adopts the three index forms described above: data block time sequence index, data block ID index, and monovariable time sequence index.

[0034] Another aspect of the present application provides an organization and storage system for discrete industrial time sequence data based on production process, comprising the following modules:

[0035] Product production process description module, for representing the production process of a single product through a production process description language, forming a product production process description file describing the sensors and data types related to the production action of the product according to the production time sequence.

[0036] Product data module, for generating a product data template specifying the type, name and location of each sensor data in a single product data block according to the production process description file.

[0037] Single product data module, for associating each sensor data with the production process according to the product data template, generating a single product data block.

[0038] Combined product data module, for integrating multiple product data blocks, forming a data page composed of multiple product data blocks and organizing and managing the data page.

[0039] Advantages of the present application:

[0040] The present application uses a production process description language to represent the production process of a product and generates a corresponding product data template, establishing a time sequence relationship between data and production process, making up for the lack of consideration of the time sequence relationship of production process in traditional non-relational data storage. Thanks to the introduction of data block time sequence index and monovariable time sequence index, the time complexity of time range query for monovariable data and single product data block can reach O(log n ). At the same time, unlike traditional non-relational data storage which needs to query and reorganize data when used, the production process-based product data block directly stores the production data of a single product according to the process, providing a more efficient data access method. This storage method greatly improves the convenience and efficiency of data processing in digital twin, production process decision analysis, time sequence prediction and anomaly detection, etc. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. The drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0042] Figure 1 The method flowchart of the embodiments of the present application;

[0043] Figure 2 The production flow description file template provided by the embodiments of the present application;

[0044] Figure 3 The product data template of the embodiments of the present application;

[0045] Figure 4 The production flowchart from the production flow description file to the product data template of the embodiments of the present application;

[0046] Figure 5 The data block format and the description example of the embodiments of the present application;

[0047] Figure 6 The data page format of the embodiments of the present application;

[0048] Figure 7 The form of time sequence index of the present application;

[0049] Figure 8 The system schematic diagram of the present application. DETAILED DESCRIPTION

[0050] The technical solutions provided by the present application will be further described in combination with the accompanying drawings.

[0051] A production flow-based organization and storage method of discrete industrial time sequence data, comprising the following steps:

[0052] S10: using production flow description language to represent the production flow of a single product, forming a product production flow description file. The product production flow description file describes the sensors and data types related to the production action of the product according to the production time sequence.

[0053] S20: generating a product data template according to the production flow description file; the product data template specifies the type, name and position of each sensor data in a single product data block.

[0054] S30: associating each sensor data with the production flow according to the product data template, and generating a single product data block.

[0055] S40: integrate the plurality of product data blocks, form a data page composed of the plurality of product data blocks and organize and manage the data page.

[0056] Figure 1 The flow chart of the present application is shown.

[0057] The present embodiment is described in detail by using a specific product production data example and combining the implementation steps.

[0058] Step S10: use the production flow description language to represent the production flow of a single product, and form a product production flow description file. The product production flow description file describes the sensors and data types related to the product production actions according to the production sequence.

[0059] The present application can be applied to a flow-based product production line. For a product produced by a flow-based production line, the production process will pass through a plurality of physical devices, and each physical device has its corresponding action meaning. The process of describing the action meaning of the physical device is the process of using the production flow description language to describe the production flow of a single product.

[0060] The production flow description file includes a flow action name A={a1, a2, …, a m}(A is a flow action name set, a i is a flow action name), a sensor number S={s1, s2, …, s m}(S is a sensor number set, s i is a sensor number), a data attribute D={d1, d2, …, d m}(D is a data attribute set, d i is a data attribute corresponding to a flow action), and a key action marker (K is a key action marker set, is a key action marker, i.e., whether the flow action with the name a i is a key action).

[0061] The production process description file is written in JSON. The process action name, sensor ID, and critical action flag are represented by the ActionName, SensorID, and isCriticalAction fields, respectively. Data attributes are represented by the key DataAttribute, and their corresponding values ​​include the data type DataType and the data length DataBytes. For example, the incoming photoelectric sensing action in a production process might be described as {"ActionName":"Incoming photoelectric sensing","SensorID":"I3.4","isCriticalAction":"true","DataAttribute":{"DataType":"timestamp","DataBytes":8}}. This indicates that the sensor ID corresponding to the incoming photoelectric sensing action is I3.4, that the action is a critical action, and that its data attribute is an 8-byte timestamp. Next, using the process action description as the value and the process action sequence number as the key, a representation of the action within the process is formed. For example, {1:{"ActionName":"Feed Photoelectric Sensor","SensorID":"I3.4","isCriticalAction":"true","DataAttribute":{"DataType":"timestamp","DataBytes":8}}}} indicates that the feed photoelectric sensor action is the first action in the production process action sequence. A collection of key-value pairs consisting of the process action description as the value and the process action sequence number as the key, plus the data attribute description of the product ID, defines a complete production process description file.

[0062] For example:

[0063]

[0064]

[0065] An example of a complete production process description file is shown in Figure 2 ; Figure 2 A production process description file consisting of 6 process actions is shown, where the data type of the product ID is unsigned int and the data length is 8 bytes.

[0066] Step S20: Generate a product data template according to the production process description file; the product data template specifies the type, name and location of each sensor data in a single product data block.

[0067] The product data template M contains: the size of a single product data block BSize, which satisfies where b i It is action ai The data size of each action data a i Name i , data type d i , and its position in the product data block p i A product data template is a binary file that follows the format described above. Its suffix is ​​pdtem. The file begins with a header that stores metadata, including the product data template identifier, file version, number of actions, and data block size. Next is the template data area, which first specifies the type and length of the product ID and then arranges the process action tuples in the order of the production process. Each tuple represents an action data item and includes the action name, data type, data value, and its position offset within the data block. Figure 3 The format of the product data template is Figure 3 The order from left to right and from top to bottom is the organization order of each element in the product data template in the binary file. The byte size of each element is Figure 3 Already marked.

[0068] The product data template M in step S20 is generated by parsing the product production process description file F.

[0069] The parsing process includes the following sub-steps:

[0070] S201: Read the process action name set A, sensor set S and its data attribute D in the production process description file F.

[0071] S202: Create a mapping f:A→N, and change the process action name a i Mapped to the action data name n in the data template i N is the set of English codes of the process action names after mapping.

[0072] S203: Determine the storage type T of each action data according to the sensor ID and data attributes d .

[0073] S204: According to the sequence of the process actions represented by the process action name set A in the production process, the offset positions of the data in the product data block are allocated to generate an offset position set P = {p1, p2, ..., p m}, p i The process action is named a i The offset position of the process action represented in the data block satisfies

[0074] S205: Generate standardized product data template M = (N, P, T d ).

[0075] Figure 4 The mapping f:A→N in step S202 is to convert the Chinese action name with clear semantics into the corresponding English code. The mapping can be specified by the program by default or manually. Figure 4 The "in" field is manually assigned, giving each action a more readable English code. If the default assignment is used, the generated code will be in the form of P01, P02, etc., starting with the letter P and followed by a number.

[0076] In step S203, the data type that can be finally recognized by the computer is determined based on the data type DataType and data length DataBytes in the data attributes. Figure 4 The timestamp type plus the 8-byte length ultimately determines that the action data type is UINT64, which is an 8-byte unsigned integer.

[0077] In step S204, the production process sequence is determined according to the key serial number in the production process description file, and then the offset position set P is generated according to the production process sequence. The position offset is related to the production sequence and data type. For example, the action data types in 4 are all 8-byte unsigned integers, so their position offset is the action serial number multiplied by the data length plus the starting position.

[0078] Finally, the product data template is generated. The product data template is in binary byte format. Figure 4 The product data template (byte form) shown in FIG. 1 is shown in FIG. 2 . For ease of presentation, hexadecimal notation is used to represent binary data.

[0079] Step S30: Correlating each sensor data with the production process according to the product data template to generate a single product data block;

[0080] A single product data block contains the product's unique ID i , production time range T=[t start ,t end ] and action data area V={v1,v2,…,v m}Three parts.

[0081] Among them, the product unique identification ID i The length and type of is defined by the product data template M, and the production time range T is represented by a Unix timestamp. According to the definition of the product data template M, the production time range T can be a Unix timestamp in seconds, milliseconds, or nanoseconds. Each action data v in the action data area i The storage area is determined by the offset P defined by the product data template M.

[0082] Figure 5It shows the form of a product data block. The first 16 bytes are the product ID, which is the ID set by the factory for the product during the actual production process. If the factory does not set an ID for the product, a default ID is set for the product in the form of an integer self-increment; the following 16 bytes are the production time range of the product. The first 8 bytes of the production time range are the production start time, and the last 8 bytes are the production end time. Next is the action data area, which stores the actual action data. Figure 5 In the data block example shown, the product ID is 100001410402668; the production start time is UNIX timestamp 1732979729860, which is 23:15:29 on November 30, 2024; and the production end time is UNIX timestamp 1732980131960, which is 23:22:11 on November 30, 2024. In the corresponding product data template, the first action in this data block example is the InSense action, which is the photoelectric sensing action for feeding. In this data block example, the InSense action is triggered at 1732979739860, which is 23:15:39 on November 30, 2024.

[0083] Step S40: Integrate multiple product data blocks to form a data page consisting of multiple product data blocks and organize and manage the data page.

[0084] A data page consists of a page header H, a data area C, and an index area I. The page header is the metadata area of ​​the data page, recording the offset positions of the data and index areas within the page. It includes the page number, page size, number of data blocks, and index offset. The index area is used to store indexes used to locate data and consists of a data block time-sequential index, a data block ID index, and a univariate time-sequential index.

[0085] Figure 6 The following table shows the data page format, where the page number is represented by an integer and increments as the number of data pages increases. The page size is the size of the data page itself, which includes the page header, data area, and index area. The number of data blocks is the number of data blocks contained in the page. Combined with the data block size recorded in the product data template, all data blocks can be located. The index position offset is the offset of the index area start position relative to the data page start position. Since the index area is at the end of the data page, the index position offset is used to locate the index area.

[0086] The data block time sequence index, data block ID index, and univariate time sequence index contained in the index area are defined as follows:

[0087] Data block ID index: uniquely identifies the product ID i Mapped to the data block location set P B , i.e. g:IDi →P B .

[0088] Data block time order index: maps the time range T to the location set P of data block B B , that is, f:T→P B .

[0089] Univariate time sequence index: records the order of variables v at different time points, forming a sequence {(t i ,v i )}, satisfying t1 <t2<…<t n .

[0090] The data block ID index takes the form of a key-value pair. Using the data block ID index hash function, a hash calculation is performed on a specific data block ID. Using the hash calculation as the key and the corresponding position offset as the value, the data block ID index can be quickly located. If a hash collision occurs, where different data block IDs have the same hash value, a chain address method is used to avoid it.

[0091] The data block time-sequential index and the univariate time-sequential index use the same index structure: both are time-sequential indexes. The data block time-sequential index establishes a time-sequential index for the production time range field in a data block, while the univariate time-sequential index establishes a time-sequential index for the same action data in different data blocks.

[0092] The time sequence index is a tree structure in which nodes are connected sequentially. Figure 7 This is an example of a single variable time sequence index. First, the data points of the same variable are indexed in time order to form a linear order index at the bottom of the index tree. Then, a tree-shaped time index is established based on the linear order index. Its characteristic is that it uses a binary partitioning method to establish the index layer by layer. The node of the tree-shaped time index is a time range, and the tree-shaped time index can be used to quickly locate a single variable node. For example, locating Figure 7 The data point 1732979749860 in the figure can be located through the time sequence structure via the nodes [1732979729860, 1732979749860] and [1732979739860, 1732979749860]. Data can also be located sequentially from the data point 1732979749860.

[0093] As production progresses, product data gradually increases, and the number of data pages also increases. Multiple data pages are organized in chronological order during generation, and secondary indexes guide data page location. Secondary indexes also use the three aforementioned indexing methods: data block chronological index, data block ID index, and univariate chronological index. The principles behind these indexes are the same as those described above.

[0094] like Figure 8 As shown, another aspect of the present invention provides a system for organizing and storing discrete industrial time series data based on a production process, comprising the following modules:

[0095] The product production process description module is used to represent the production process of a single product through a production process description language, and form a product production process description file that describes the sensors and data types related to the product production actions according to the production sequence.

[0096] The product data module is used to generate a product data template that specifies the type, name and location of each sensor data in a single product data block based on the production process description file.

[0097] The single product data module is used to associate each sensor data with the production process according to the product data template to generate a single product data block.

[0098] The combined product data module is used to integrate multiple product data blocks to form a data page consisting of multiple product data blocks and organize and manage the data page.

Claims

1. A method for organizing and storing discrete industrial time series data based on production processes, characterized in that: The following steps are involved: S10: Use the production process description language to express the production process of a single product and form a product production process description file; S20: Generate a product data template according to the production process description file; S30: Correlating each sensor data with the production process according to the product data template to generate a single product data block; S40: Integrate multiple product data blocks to form a data page consisting of multiple product data blocks and organize and manage the data page. The specific implementation process is as follows: The data page containing multiple product data blocks consists of a page header H, a data area C, and an index area I. The page header is the metadata area of ​​the data page, recording the offset positions of the data area and index area in the page, including the page number, page size, number of data blocks, and index position offset. The index area is used to store indexes for locating data, and consists of a data block time sequence index, a data block ID index, and a single variable time sequence index. The data block time sequence index, data block ID index, and univariate time sequence index contained in the index area are defined as follows: Data block ID index: uniquely identifies the product ID i Mapped to the data block location set P B ; Data block time order index: maps the time range T to the location set P of data block B B ; Univariate time sequence index: records the order of variables v at different time points, forming a sequence {(t i , v i )}, satisfying t1 <t2<…<t n , where t i For time; The data page containing multiple product data blocks is organized in chronological order when generated, and the location of the data page is guided by the secondary index; the secondary index also uses the three aforementioned index forms: data block chronological index, data block ID index and univariate time sequence index.

2. The method for organizing and storing discrete industrial time series data based on production processes according to claim 1 is characterized in that: The specific implementation process of step S10 is as follows: The production process description file F is defined by the production process description language L. The production process description file includes a process action name set A = {a1, a2, ..., a m }, a m is the name of the process action, the sensor number set S = {s1, s2, ..., s m }, s m is the sensor number, the data attribute set D={d1,d2,…,d m }, d m Data attributes corresponding to process actions, key action set It is a key action mark, that is, the name is a i Whether the process action is a key action; The production process description file is expressed in JSON language. The process action name, sensor ID and critical action tag are represented by the ActionName, SensorID and isCriticalAction fields respectively. The data attribute is represented by the key name DataAttribute, and its corresponding value includes the data type DataType and the data length DataBytes; finally, the production process is described by an integer serial number as the key and the set of process action name, sensor ID, data attribute and key action tag as the value.

3. The method for organizing and storing discrete industrial time series data based on production processes according to claim 2, characterized in that: The generated product data template is generated by parsing the product production process description file F; The product data template M contains: the size of a single product data block BSize, which satisfies where b i It is action a i The data size of a i English code n for each process action represented i and data type d i , and its position in the product data block p i The product data template is a binary file that follows the above format. The beginning of the file contains a file header that stores the file metadata, including the product data template identifier, file version, number of actions, and data block size. Data area of ​​the product data template: This area first specifies the type and length of the product ID, and then arranges the process action tuples in the order of the production process. Each tuple represents an action data, including the process action name, sensor ID, data type, data value and its position offset in the data block.

4. The method for organizing and storing discrete industrial time series data based on production processes according to claim 3 is characterized in that: The specific process of parsing the product production process description file F is as follows S201: Read the process action name set A, sensor set S and its data attribute D in the production process description file F; S202: Create a mapping f:A→N, and change the process action name a i Mapped to the action data name n in the data template i , N is the set of English codes of process action names after mapping; S203: Determine the storage type T of each action data according to the sensor ID and data attributes d ; S204: According to the sequence of the process actions represented by the process action name set A in the production process, the offset positions of the data in the product data block are allocated to generate an offset position set P = {p1, p2, ..., p m }, p i The process action is named a i The offset position of the process action represented in the data block satisfies S205: Generate standardized product data template M = (V, P, T d ).

5. The method for organizing and storing discrete industrial time series data based on production processes according to claim 4 is characterized in that: The single product data block is a fixed-length structure and contains the following information: Unique product ID i ; Production time range T=[t start , t end ]; where t start is the start time of product production, t end The end time of product production; Action data area V = {v1, v2, ..., v m }; where v i Action data generated for a single process action; each action data v i The storage area is determined by P defined by M; Among them, the product unique identification ID i The length and type of the product data template M are defined, and the production time range T is represented by a Unix timestamp.

6. A system for organizing and storing discrete industrial time series data based on a production process, used to implement the method for organizing and storing data according to any one of claims 1 to 5, characterized in that: Includes the following modules: The product production process description module is used to express the production process of a single product using a production process description language, and to generate a product production process description file that describes the sensors and data types related to the product production actions according to the production sequence; The product data module is used to generate a product data template that specifies the type, name and location of each sensor data in a single product data block based on the production process description file; Single product data module, used to associate each sensor data with the production process according to the product data template to generate a single product data block; The combined product data module is used to integrate multiple product data blocks to form a data page consisting of multiple product data blocks and organize and manage the data page.

Images