Association method of discrete multi-modal data signal and instance

Abstract

The invention belongs to the technical field of data processing, and relates to a discrete multi-modal data signal and instance association method, which comprises the following steps of: setting a weak constraint process template in a discrete multi-modal industrial system based on industrial prior logic, acquiring a signal and guiding the signal to be associated with an instance; the weak constraint process template defines a buffer number, a time range, a stage to which the signal belongs and a logical relationship between the signals, and the logical relationship comprises a clear constraint relationship and a mappable relationship between the signals; according to the template, for the multi-modal data signals arranged according to the time sequence, based on the continuity of the process and the connection relation of the stages, inference association is conducted in all the stages, the association sequence serves as a candidate sequence to be put into system buffering, and finally cross-stage integration is conducted on the candidate sequence formed in all the stages to obtain a complete instance association sequence. The method can meet the requirements for data positioning and action traceability in an industrial scene, and provides powerful support for optimization and intelligent decision making of industrial production.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology and relates to a method for associating discrete multimodal data signals with instances. Background Technology

[0002] With the continuous improvement of industrial automation and intelligence, a large number of sensors, controllers, vision devices, and information systems are deployed on production sites to collect multi-source data such as equipment status, process parameters, operating instructions, and quality inspection results. This data encompasses various types, including continuous time-series signals, event logs, alarm records, barcode or identification information, and image / video detection results, exhibiting strong multimodal, heterogeneous, and real-time characteristics. To conduct advanced applications such as data mining, production cycle analysis, anomaly diagnosis, and quality traceability, it is necessary to first integrate this discrete multimodal data by product instance to construct an instance-level event sequence with a unified instance identifier and temporal order.

[0003] In typical discrete manufacturing scenarios, multiple products are often processed in parallel on the same production line. The timestamp accuracy varies across different data sources, the granularity of events and their semantic representations are inconsistent, and various types of data lack unified instance identifiers. For example, control logs only record the occurrence of actions, sensor signals only reflect physical changes, vision systems only output partial recognition results, and information systems use work orders or barcodes as primary keys. There is no natural one-to-one correspondence between these different types of data. This makes it difficult to directly match cross-modal events to the same product instance.

[0004] In existing technologies, achieving instance association for multimodal events typically relies on deterministic matching rules, such as fixed timing windows, explicit behavioral templates, strict equipment state transition logic, or predefined process path structures. While these methods are effective under specific operating conditions, they are highly sensitive to the accuracy and completeness of the rules. Once data is missing, modalities are heterogeneous, states are abnormal, or processes are disturbed, deterministic rules struggle to adapt, leading to unstable instance associations, insufficient robustness, and difficulty in generalizing to complex scenarios.

[0005] Therefore, it is necessary to propose an instance association method that does not rely on strict deterministic rules, but rather utilizes flexible, weakly constrained prior knowledge for constraint guidance. Such methods only require weak relationships, weak constraint patterns, or incomplete prior knowledge between events, rather than a complete, precise, and fixed rule system. Through prior association mechanisms, stable instance-level structure construction can be achieved even under conditions of inconsistent multimodal data, incomplete events, or even process perturbations, thereby significantly improving adaptability and robustness in complex industrial environments. Summary of the Invention

[0006] To address the aforementioned technical problems in existing technologies, this invention proposes a method for associating discrete multimodal data signals with instances. This method utilizes weakly constrained prior knowledge for constraint-guided instance association, and its specific technical solution is as follows: A method for associating discrete multimodal data signals with instances is proposed. Based on industrial prior logic, a weakly constrained process template is set up in a discrete multimodal industrial system to acquire signals and guide their association with instances. The weakly constrained process template defines the number of buffers, time range, the process and stage to which the signals belong, and the logical relationships between signals. The logical relationships include explicit constraint relationships and mappable relationships between signals. According to the template, for multimodal data signals arranged in time sequence, association is inferred within each stage based on the continuity of the process and the connection relationship between stages. The associated sequences are placed into the system buffer as candidate sequences. Finally, the candidate sequences formed in each stage are integrated across stages to obtain a complete instance association sequence.

[0007] Furthermore, each execution step of the instance target process is encoded, and the process is divided into several stages.

[0008] Furthermore, the explicit constraints between the signals include: the logical connection between the sequential arrangement of the signals, and the branching dependencies established by determining the preconditions of each signal.

[0009] Furthermore, the branch dependency relationship specifically means that if the current signal depends on a certain control signal, then the current signal is logically bound as a branch of that control signal.

[0010] Furthermore, the mappable relationship specifically refers to mapping signals and unifying them into an abstract action with an alias, which is used to consistently represent the connection relationship of actions in the process template.

[0011] Furthermore, alias mapping is performed on sequentially interchangeable signals, with at least two interchangeable signals.

[0012] Furthermore, alias mapping is performed on signals with branch dependencies.

[0013] Furthermore, for each action in the original input stream, each action signal is identified and parsed according to the weakly constrained process template, specifically including the following sub-steps: S001: Determine if it is a start signal. If it is and the system buffer is not full, then directly allocate a new buffer. Otherwise, according to the replacement policy, replace the old buffer and allocate a new buffer. S002: Based on the predefined rules in the weak constraint process template, determine the current signal and place it into the corresponding buffer based on the specific description of the signal; S003: If the current signal is the end signal and the signals that have completed a stage are arranged in order, then the current buffer should be popped through the elimination strategy.

[0014] Furthermore, S002 specifically refers to: When multimodal data and control signals for general steps are received, they are directly placed into the corresponding buffers in chronological order. The principle at this time is to place the data into the buffer that is earliest in the chronological order. When a branch signal is received, find all the buffers containing the control signal to which it belongs, and put the buffer that is first in time into it; When a signal with an alias is received, the alias is used instead of the real name and placed in the buffer.

[0015] Furthermore, the elimination strategy adopts the earliest expired first elimination strategy (FIFO).

[0016] The beneficial effects of this invention are as follows: It enables reusability across multiple production lines without requiring any coding. By providing only weak constraint patterns between events and employing a priori association mechanism, stable instance associations can be achieved even in cases of inconsistent multimodal data or process disturbances, significantly improving adaptability and robustness in complex industrial environments. It effectively leverages the universality of characteristics exhibited in industrial production environments to achieve generalized and accurate analysis of industrial data, thereby meeting the needs for data location and action tracing in industrial scenarios and providing strong support for industrial production optimization and intelligent decision-making. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for associating discrete multimodal data signals with instances according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the weak constraint process template in this embodiment; Figure 3 This is a schematic diagram of the original signal flow stage division in this embodiment; Figure 4 This is a schematic diagram of signal partitioning and buffering in this embodiment; Figure 5 This is a schematic diagram of the signal cross-stage integration result in this embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0019] This embodiment discloses a method for associating discrete multimodal data with instances, providing a weakly constrained process template as a description file for instance production processes. This template can be used to locate the current processing stage through process information, thereby providing a precise basis for decision-making in discrete manufacturing quality traceability. The process rules defined by the weakly constrained process template include predefined time ranges, target unit correspondences, and temporal dependencies between stages. Specifically, such as... Figure 1 and Figure 2 As shown, the above association method specifically includes the following: Step 1: In a discrete multimodal industrial production scenario, each execution step of the physical devices on the target process line is encoded, and the process is divided into several stages, thus forming a process structure framework with clear boundaries. For each stage, the maximum number of instances in the production line is determined and specified in the template. Each physical device has its corresponding action meaning, so the process of describing the action meaning of the physical devices is the process of describing the production process of a single product.

[0020] Each stage may contain the following signals: Key signals: These are signals that inevitably occur during the operation of the production line, including signals detected by input devices such as sensors and signals emitted by output devices such as actuators. Non-critical signals: These are signals whose triggering is uncertain, such as auxiliary detection signals of certain sensors or prompt signals that are triggered only under specific conditions. Control signals: These are signals that can change the direction of a process or determine the selection of subsequent steps. Branch signal: refers to a signal that depends on a certain control signal.

[0021] Any execution action in different stages should be able to occur in parallel, while any execution action in the same stage can only occur sequentially. The process is described as dividing the system into m stages based on the parallelism of signals. Each stage C i and action signals A i，l The relationship satisfies: A i，1， A i，2···· ∈ C i , i、l This represents the sequence number value. For any stage C... i and C j The condition is that there are no identical actions: .

[0022] Step 2: Establish Action Dependencies and Branch Control Relationships: Based on the actual process, determine the preconditions for each action signal and the logical connections between signals to form a hierarchical constraint system. Specifically, for each control signal, clearly define the branch signals it controls, that is, logically bind the branch signals to the control signals to construct branch dependencies: branch signals are generated by the output of control signals, but the presence of a control signal does not necessarily mean the presence of a branch signal, while the presence of a branch signal always implies the presence of a control signal.

[0023] Step 3: Map action signals with branch dependencies, commutative or unpredictable order to a single abstract action with an alias, used to consistently represent the connection between actions in the process template.

[0024] Specifically, for cases where action signals are interchangeable or their order is variable, these action signals are uniformly abstracted into aliases. This allows the process template to consistently express these action signals without relying on a fixed order, thereby maintaining the stability and describability of the process structure. A mapping relationship between real names and aliases is established: ,For example Figure 2 The real names 100051 and 100052 are represented by the alias 10005. For example, during product assembly, due to the different arrival order of different components, the timing of the signals received by the sensor from the arrival of the components may differ. The signal order may be A->B or B->A. Therefore, A and B are uniformly described as alias C, and the two situations mentioned above are uniformly summarized by C->C.

[0025] For situations where control signals cause differences in process flow, the control signals and their branch signals are mapped to a unified alias identifier to accurately represent the process branch structure triggered by the control signals in the process template. For example, if control signal A1 -> branch signal B1 and control signal A2 -> branch signal B2, then control signals A1 and A2 are assigned the alias A, and branch signals B1 and B2 are assigned the alias B. Thus, the unified template for both cases can be completed using control signal A -> branch signal B.

[0026] Step 4: Based on the process structure framework obtained in Step 1, combined with the branch action dependencies obtained in Step 2 and the alias mapping results obtained in Step 3, the signal sequence within the stage, the logical relationship between signals, and the execution flow of the stage are clarified. At the same time, the correspondence of instance products, the time range, and the number of buffers for signal parsing by the system are predefined, thereby constructing a weakly constrained process template describing discrete scenarios.

[0027] In one embodiment, the constrained template can be used to divide the multimodal data scenario into stages, including an infeed stage, a testing stage, and an outfeed stage. For example, after entering the production line, the electricity meters are blocked by a blocking cylinder and queue up to be picked up by the grippers. In this stage, each later-entering electricity meter must wait for the previous electricity meter to complete a certain action before it can perform the same action, thus exhibiting seriality. Specifically, assuming that in this stage the electricity meters need to pass through the following stages in sequence: S1. Receive the product; S2, the blocking position has a meter blocking the meter; S3, the product flows to the next workstation.

[0028] Therefore, for meter B, which enters the production line later, each step can only be executed after meter A, which entered the production line earlier, has completed its execution. For example, step S2 of meter B must be after step S2 of meter A in time, thus exhibiting a sequential nature, and this stage can be divided into an independent stage.

[0029] The grippers then clamp the meters, grasping a maximum of three meters at a time, hereinafter referred to as one group of meters. For ease of description, the grippers will cycle through the following six steps: S01, Waiting for control signal from host computer; S02, Turning feed inlet; S03, clamp a set of electricity meters; S04. Rotate to the designated slot according to the signal from the host computer; S05. Put down the electricity meter; S06. Lift the gripper.

[0030] In this process, the movements of the grippers themselves are sequential, while any movement of the grippers has no temporal relationship with any movement in the feeding stage. Therefore, these two stages are independent and parallel. It's important to note that the temporal relationship here refers to the fact that any movement in the gripper stage and the feeding stage has no sequential order, but the stages still exhibit a sequential relationship. For example, meter A must first go through the feeding stage and then the gripper stage.

[0031] The same principle applies to the discharge stage, so it will not be elaborated further.

[0032] Based on the weak constraint process template definition, for any stage C i Each action signal in the sequence satisfies the following condition: the actions are arranged in a forward-to-back order. If the order is uncertain, aliases can be used for unification. For any stage... C i The starting signals should be arranged in the following order: .

[0033] Step 5: Following the template, for multimodal data signals arranged in chronological order, inference correlation is performed within each stage based on the continuity of the process and the connection between stages. The correlation sequence is then placed into the system buffer as a candidate sequence. Finally, the candidate sequences formed in each stage are integrated across stages to construct a complete instance correlation sequence.

[0034] Specifically, in the cross-stage integration phase, based on the predefined time range, product correspondence, and temporal dependencies between stages in the weakly constrained process template, cross-stage matching and splicing are performed to generate the final complete product full-process signal sequence. The core of this process lies in identifying multiple stage buffers belonging to the same product, determining their logical continuity and physical reachability through timestamps, location indexes, or control signal identifiers. During the matching process, a comprehensive judgment is made in conjunction with the time range, and finally, all stage buffers that meet the conditions are aggregated according to the product ID dimension.

[0035] During the splicing process between the material feeding and testing phases, if the template is set to a valid time range of [-3000, 0], meaning the completion timestamp of the material feeding phase sequence (i.e., the timestamp of the last step) should be within 3000 units before the start signal timestamp of the testing phase, calculations will show that if buffers 2 and 3 meet this time interval, while buffer 1 does not, then even if buffer 1 has already output, it cannot match this testing phase. Ultimately, the buffers that successfully match in the testing phase are buffers 2 and 3, which are then spliced ​​together and restored to the smallest unit. Here, the smallest unit is the product; therefore, output is performed at the product level. The final output result is as follows: Figure 5 As shown.

[0036] like Figure 3 and Figure 4 As shown, for each action in the original input stream, each action signal can be identified and parsed according to the weakly constrained process template, specifically including the following sub-steps: S001: Determine if it is a start signal. If it is and the system buffer is not full, then open a new buffer directly. Otherwise, according to the replacement policy, replace the old buffer and open a new buffer.

[0037] Since 10001 is the start signal in the template, a new buffer is allocated each time a 10001 signal flows out. Figure 3 In the example signal sequence, it can be observed that there are a total of 4 10001s, so 4 buffers need to be allocated: buffer number 1 to buffer number 4.

[0038] S002: Based on the predefined rules in the weak constraint process template, determine the current signal and place it into the appropriate buffer based on the signal's specific description. Specifically: When multimodal data and control signals for the general steps are received, they are directly placed into the corresponding buffers in chronological order, based on the principle of placing the data into the buffer with the earliest available time sequence. If the current control signal has a time range within the template, it is necessary to further determine whether the last action of the selected buffer occurred within the template's specified time; if not, it is replaced. For example... Figure 2 The template shown has two buffers: 1: [(20001,1), (20002,1001), (20003, 2001)]; 2: [(20001,500), (20002,1501), (20003, 2501)]; If we enter (20004, 5500) at this point, then according to... Figure 2 As shown in the 20004 time range, it can be seen that the last action of buffer 1 has exceeded the range, and the incoming actions must be in ascending order of time sequence. Therefore, it is determined that buffer 1 has expired and is directly discarded.

[0039] When a branch signal is received, the system searches all buffers containing the control signal to which it belongs and places the buffer with the earliest time sequence into it. For example, if the original input stream contains four actions (100052, 36715359), (100051, 36715712), (100061, 36715717), and (100062, 36715799), according to... Figure 2 The sequence marked in the weak constraint process template shows that the sub-signal 100062 belongs to the control signal 100052, and the branch signal 100061 belongs to the control signal 100051. Therefore, when the branch signal (100061, 36715717) flows in, the 100051 signal is searched in each stage and assigned to the corresponding buffer of the stage.

[0040] When a signal with an alias is received, the alias is used instead of the real name and placed in the buffer.

[0041] S003: If the current signal is the end signal and the signals that have completed a stage are arranged in order, then the elimination strategy determines whether to pop the current buffer and provide it for the inter-stage matching work in the next stage.

[0042] The system needs to dynamically maintain multiple signal sequence buffers to record the signal sequence of a product unit within a given stage. Due to the limited buffer capacity, especially on high-concurrency production lines with multiple products being processed in parallel, a buffer eviction strategy must be introduced to determine when to release historical data and allocate new buffers. This eviction strategy employs both First-to-First-Out (FIFO) and a custom eviction mechanism to adapt to the data characteristics of different production lines.

[0043] When a buffer accumulates in the buffer pool without output, and a start signal arrives, a replacement policy needs to be executed. For example, if for some reason none of the three buffers receive an end signal, and therefore all accumulate in the buffer pool, then an input start signal of 10001 should replace one buffer, placing it directly into the output queue. If a FIFO policy is being used, then the first buffer to enter the queue will be replaced, and the other buffers will be moved forward to create new buffers.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for associating discrete multimodal data signals with instances, based on industrial prior logic, setting a weakly constrained process template in a discrete multimodal industrial system, acquiring signals, and guiding the association between signals and instances, characterized in that, The weak constraint process template defines the number of buffers, time range, stage to which the signal belongs, and logical relationships between signals. The logical relationships include explicit constraint relationships and mappable relationships between signals. According to the template, for multimodal data signals arranged in time sequence, based on the continuity of the process and the connection between stages, inference association is performed in each stage, and the association sequence is put into the system buffer as a candidate sequence. Finally, the candidate sequences formed in each stage are integrated across stages to obtain a complete instance association sequence.

2. The association method as described in claim 1, characterized in that, Each execution step of the instance target process is coded, and the process is divided into several stages.

3. The association method as described in claim 1, characterized in that, The explicit constraints between the signals include: the logical connection between the sequential arrangement of the signals, and the branching dependencies established by determining the preconditions of each signal.

4. The association method as described in claim 3, characterized in that, The branch dependency relationship is as follows: if the current signal depends on a certain control signal, then the current signal is logically bound as a branch of that control signal.

5. The association method as described in claim 3, characterized in that, Specifically, the mappable relationship refers to mapping signals and unifying them into an abstract action with an alias, which is used to consistently represent the connection relationship of actions in the process template.

6. The association method as described in claim 5, characterized in that, Alias ​​mapping is performed on sequentially commutable signals, where the number of commutable signals is at least two.

7. The association method as described in claim 5, characterized in that, Alias ​​mapping is performed on signals with branch dependencies.

8. The association method as described in claim 1, characterized in that, For each action in the original input stream, the action signal is identified and parsed according to the weakly constrained process template, which includes the following sub-steps: S001: Determine if it is a start signal. If it is and the system buffer is not full, then directly allocate a new buffer. Otherwise, according to the replacement policy, replace the old buffer and allocate a new buffer. S002: Based on the predefined rules in the weak constraint process template, determine the current signal and place it into the corresponding buffer based on the specific description of the signal; S003: If the current signal is the end signal and the signals that have completed a stage are arranged in order, then the current buffer should be popped through the elimination strategy.

9. The association method as described in claim 8, characterized in that, Specifically, S002 is: When multimodal data and control signals for general steps are received, they are directly placed into the corresponding buffers in chronological order. The principle at this time is to place the data into the buffer that is earliest in the chronological order. When a branch signal is received, find all the buffers containing the control signal to which it belongs, and put the buffer with the earliest time sequence into it; When a signal with an alias is received, the alias is used instead of the real name and placed in the buffer.

10. The association method as described in claim 8, characterized in that, The elimination strategy adopts the earliest-to-expires (FIFO) strategy.

Images