An industrial visual monitoring intelligent management and control method based on a multi-modal large model

Abstract

The application relates to the technical field of industrial vision, and discloses an industrial vision monitoring intelligent management and control method based on a multimodal large model, which comprises the following steps: collecting key frame images and obtaining a bottom equipment continuous time sequence; respectively extracting a vision and a time sequence word element sequence through an encoder; inputting a cross attention network to fuse and output a multimodal representation sequence and extract an attention tensor; inputting a gate network to calculate a dangerous state probability; if the dangerous state probability is greater than a threshold value, a blocking instruction is issued; otherwise, the attention tensor is converted into a penalty vector, state features are calibrated by combining the representation sequence to obtain calibration parameters, a digital twin model is inputted to solve a safety control subspace; a mask vector is constructed according to the above and is superimposed on a decoder probability output layer for constraint; a management and control instruction sequence is generated by sampling and is issued for execution. The industrial vision monitoring intelligent management and control method based on the multimodal large model blocks the output of out-of-bound parameters through cross-modal alignment and mask constraint, and guarantees the safety of bottom equipment control.

Description

Technical Field

[0001] This invention relates to the field of industrial vision technology, specifically to an intelligent control method for industrial vision monitoring based on a multimodal large model. Background Technology

[0002] Industrial visual monitoring relies on image sensing equipment and computer vision algorithms to continuously capture equipment movements and environmental changes in industrial settings. This technology transforms optical changes in the physical environment into digital features, providing non-contact environmental perception for front-end control equipment and forming the fundamental prerequisite for automated operation in industrial settings.

[0003] Traditional intelligent control relies heavily on issuing fixed action commands based on established rules. With the evolution of algorithms, multimodal large models are gradually being applied to industrial control. These models can simultaneously receive visual features and text sensor data, utilize the feature extraction capabilities of neural networks to analyze multi-source heterogeneous information, and attempt to directly output serialized command content through end-to-end autonomous decision-making.

[0004] Existing control schemes have inherent shortcomings in practical applications. Conventional models fail to achieve deep physical alignment between image spatial features and the continuous temporal states of underlying devices, causing control decisions to deviate from the actual dynamic feedback from the underlying system. More critically, multimodal large models are essentially generated based on statistical distributions and lack physical boundary constraints on the underlying hardware. Purely data-driven models cannot perceive the kinematic extremes and stress limits of the devices, easily exceeding safety boundaries and generating illegal control parameters, thus directly causing operational failures in the underlying devices when control commands are issued.

[0005] Therefore, this invention proposes an intelligent control method for industrial visual monitoring based on a multimodal large model to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent control method for industrial visual monitoring based on a multimodal large model, which solves the problems of lack of physical boundary constraints when the large model outputs control commands and the failure of cross-modal data to effectively align, which leads to malfunctions in the underlying equipment.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an intelligent control method for industrial visual monitoring based on a multimodal large model, comprising the following steps: Keyframe images are acquired using image sensing devices, and continuous time sequences of underlying devices are obtained through monitoring and control nodes. The keyframe image is processed by a visual encoder to extract a visual word sequence, and the continuous temporal sequence is mapped into a temporal word sequence by a vector quantization encoder. The visual lexical sequence and the temporal lexical sequence are input into a cross-attention network, which outputs a multimodal representation sequence and extracts the attention tensor. The multimodal representation sequence is input into the attention tensor gating network to calculate the crisis probability. When the crisis probability is greater than a threshold, a blocking command is output to the monitoring and control node; otherwise, the parameter calibration step is entered. In the parameter calibration step, the attention tensor is converted into a penalty vector, and the original state features are calibrated using the penalty vector and the multimodal representation sequence to obtain calibration parameters. The calibration parameters are then input into the digital twin model to solve for the security control subspace. Based on the security control subspace, a mask vector is constructed in the decoder vocabulary of the multimodal large model. The mask vector is superimposed on the decoder probability output layer for constraint, and a control instruction sequence is sampled and sent to the monitoring and control node for execution.

[0008] Preferably, the step of acquiring keyframe images using an image sensing device and obtaining the continuous time sequence of the underlying device through a monitoring and control node includes: The image sensing device is used to acquire the monitoring video stream of the target workstation, and the key frame image is output by performing frame parsing operation on the monitoring video stream according to the preset detection interval. The monitoring and control node receives the digital conversion values ​​from the physical sensing elements and stores the digital conversion values ​​sequentially into a circular data queue. When the keyframe image is generated at the reference alignment point, the time observation window is traced back, and historical digital conversion values ​​whose time range falls within the time observation window interval are retrieved from the circular data queue. The retrieved historical digital values ​​are serialized and concatenated according to the chronological order of sampling time to obtain the continuous time sequence.

[0009] Preferably, the step of extracting visual word sequences from the keyframe image using a visual encoder and mapping the continuous temporal sequence to a temporal word sequence using a vector quantization encoder includes: The keyframe image is divided into two-dimensional image blocks without physical overlap. Each two-dimensional image block is flattened and mapped to a one-dimensional feature using the linear projection network layer configured inside the visual encoder. The visual word sequence is then serialized and output according to the spatial position order. The vector quantization encoder utilizes a one-dimensional convolutional network configured within it to perform sliding window convolution operations along the time step dimension of the continuous temporal sequence, and performs feature aggregation and downsampling in the channel dimension to output the temporal hidden layer feature sequence. Calculate the Euclidean distance between the temporal hidden layer feature sequence and each discrete feature in the preset discrete codebook space, and select the discrete feature with the smallest Euclidean distance to replace and generate the temporal lexical sequence.

[0010] Preferably, the step of inputting the visual lexical sequence and the temporal lexical sequence into a cross-attention network, outputting a multimodal representation sequence, and extracting the attention tensor includes: The visual word sequence is multiplied with the query weight matrix to output the query matrix, and the temporal word sequence is multiplied with the key weight matrix and the value weight matrix to output the key matrix and the value matrix respectively. Calculate the dot product of the query matrix and the transpose of the key matrix, divide the dot product by a scaling factor, and then apply a normalized exponential activation function to obtain the attention tensor. The attention tensor is multiplied by the value matrix to obtain a cross-modal weighted output. After performing dimensional back mapping on the cross-modal weighted output using the output projection matrix, a residual connection is performed with the visual lexical sequence, and a layer normalization operation is performed to obtain the multimodal representation sequence.

[0011] Preferably, the step of calculating the crisis probability by inputting the multimodal representation sequence and the attention tensor into the gating network, and outputting a blocking command to the monitoring and control node when the crisis probability is greater than a threshold, and otherwise proceeding to the parameter calibration step, includes: The mean of elements along the sequence length dimension of the multimodal representation sequence is calculated to output a one-dimensional visual temporal feature vector, and the attention tensor is flattened in two dimensions to generate a one-dimensional correlation distribution vector. The fused feature vector is obtained by concatenating the one-dimensional visual temporal feature vector and the one-dimensional correlation distribution vector along the feature dimension. The fused feature vector is input into the gated network for nonlinear mapping calculation to obtain the probability of the emergency. When the comparison determines that the probability of the emergency is greater than the threshold, the blocking command is output to the monitoring and control node via the bypass sending interface to cut off the servo driver power circuit of the underlying device. When the comparison determines that the probability of the emergency is not greater than the threshold, the parameter calibration step is entered.

[0012] Preferably, the step of converting the attention tensor into a penalty vector and using the penalty vector and the multimodal representation sequence to calibrate the original state features to obtain calibration parameters includes: Max pooling is performed on the attention tensor along the visual space dimension, and the output of the pooling operation is converted into the penalty vector by limiting it to a preset interval through an inverse mapping function; The penalty vector is extended along the state component dimension of the original state feature to obtain an extended penalty vector. The extended penalty vector is then multiplied element-wise with the original state feature to calculate the first feature term. The multimodal representation sequence is dimensionality reduced and projected to output a multimodal mapping tensor. A scalar 1 subtraction and inversion operation is performed on the extended penalty vector to generate a compensation coefficient. The compensation coefficient is then multiplied element-wise with the multimodal mapping tensor to calculate the second feature term. The calibration parameters are output by performing a vector addition operation on the first feature term and the second feature term.

[0013] Preferably, the original state features are obtained from the continuous time sequence through channel mapping, time downsampling, and normalization. The state components in the original state features are arranged in a fixed order; Each state component in the original state feature is aligned with the number of time steps in the temporal lexical sequence in the time dimension.

[0014] Preferably, the step of inputting the calibration parameters into the digital twin model to solve for the safety control subspace includes: The calibration parameters are decoupled and extracted according to the physical meaning of the state components in the calibration parameters to obtain the initial state vector, and the initial state vector is assigned as the initial state of the state space differential equation. The extreme values ​​of the predicted trajectory of the underlying device in the next control cycle are calculated iteratively along the time axis using a numerical integration algorithm. Sampling is performed within the preset control parameter search range, and candidate control parameters are substituted into the digital twin model for prediction and deduction to extract a set of compliant parameters that do not trigger the restricted area and meet the physical boundary constraints. The multidimensional geometric region encompassed by the set of compliance parameters in the mathematical space is taken as the security control subspace.

[0015] Preferably, the step of constructing a mask vector based on the decoder vocabulary of the multimodal large model according to the security control subspace includes: Initialize a one-dimensional tensor with all zeros as the mask vector, and traverse the decoder vocabulary of the multimodal large model to identify the attribute category of the word; If the term belongs to a semantic text term, the corresponding element of the mask vector remains zero. If the term belongs to a control term, the pre-maintained control term mapping table is called to parse the physical control parameter set corresponding to the control term, and the boundary relationship between the physical control parameter set and the security control subspace is verified. When the set of physical control parameters falls within the geometric boundary of the security control subspace, the corresponding element of the mask vector is assigned a value of zero; When the set of physical control parameters exceeds the boundary range of the security control subspace, the corresponding element of the mask vector is assigned a value of negative infinity.

[0016] Preferably, the step of superimposing the mask vector onto the decoder probability output layer for constraint, and sampling to generate a control instruction sequence and sending it to the monitoring and control node for execution includes: Extract the original log probability distribution tensor of the multimodal large model, and perform tensor addition operation of the corresponding dimension on the mask vector and the original log probability distribution tensor in the probability output layer of the decoder to obtain the constrained log probability tensor; The constrained log probability tensor is input into the activation function and subjected to exponential normalization to obtain the generated probability distribution, which constitutes the legal probability space. Within the legal probability space, autoregressive cyclic sampling is performed according to the set language model sampling strategy to output word units, and the output word units are concatenated to generate the control instruction sequence; The sequence of control instructions is encapsulated into an industrial control message and sent to the monitoring and control node for execution.

[0017] This invention provides an intelligent control method for industrial visual monitoring based on a multimodal large model. It has the following beneficial effects: 1. This invention solves the problem of effectively aligning image features with equipment operating status in traditional industrial visual monitoring by mapping keyframe images acquired by image sensing devices to continuous temporal sequences of underlying devices into word sequences and then inputting them into a cross-attention network for fusion. This method extracts multimodal representation sequences and attention tensors, constructing cross-modal connections between visual and physical data, thus establishing the underlying data foundation for subsequent logical judgments.

[0018] 2. This invention solves the problem of equipment failure caused by the lack of physical boundary constraints in conventional intelligent control methods by converting the attention tensor into a penalty vector, calibrating the state features in combination with a multimodal representation sequence, and inputting the calibration parameters into a digital twin model to solve for the safety control subspace. This approach utilizes model forward inference to predict extreme values ​​and preemptively eliminates parameters that trigger restriction regions, ensuring the physical execution safety of the underlying equipment during the control cycle.

[0019] 3. This invention solves the problem of illegal action sequences easily generated when directly generating control commands based on a multimodal large model by constructing a mask vector based on the security control subspace and superimposing it on the decoder probability output layer for constraint. This method intervenes in the probability distribution before lexical sampling, directly blocking parameter outputs that exceed the security boundary, ensuring that the final generated commands are strictly limited to the legal probability space and guaranteeing the compliance of on-site control commands. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of the industrial visual monitoring intelligent control system based on a multimodal large model according to the present invention; Figure 2 This is a flowchart of the intelligent control method for industrial visual monitoring based on a multimodal large model according to the present invention. Figure 3 This is a graph showing the boundary derivation of the safety control subspace of the present invention.

[0021] Among them, 10 is the data synchronization acquisition module; 20 is the feature discrete coding module; 30 is the cross-modal attention module; 40 is the gating blocking and diversion module; 50 is the parameter degradation calibration module; and 60 is the mask constraint control module. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See attached document Figure 1 The present invention provides an intelligent control system for industrial visual monitoring based on a multimodal large model, which includes a data synchronization acquisition module 10, a feature discrete encoding module 20, a cross-modal attention module 30, a gating blocking and diversion module 40, a parameter degradation calibration module 50, and a mask constraint control module 60.

[0024] This intelligent control system for industrial visual monitoring based on a multimodal large model runs on a computing platform with physical computing capabilities. The computing platform is equipped with an image processor dedicated to tensor matrix operations and parallel acceleration. The computing platform establishes high-bandwidth hardware communication connections with image sensing devices and monitoring and control nodes in the industrial field via an industrial Ethernet network. The monitoring and control nodes oversee several underlying devices connected to the production physical environment. This hardware topology provides the physical link foundation for the synchronous input of heterogeneous data from the multimodal large model and the low-latency transmission of underlying hardware real-time commands.

[0025] The data synchronization acquisition module 10 is located at the system input end. It acquires keyframe images output by the image sensing device in real time via a communication connection, and synchronously acquires the continuous time sequence of the underlying device transmitted back by the monitoring and control node according to a unified clock stamp. The data synchronization acquisition module 10 transmits the heterogeneous visual signals and time sequence signals to the feature discrete encoding module 20.

[0026] The feature discrete encoding module 20 internally maintains two independent neural network forward propagation computation graphs, used for visual lexical extraction of keyframe images and discrete quantization mapping operations on continuous temporal sequences, respectively. The output of the feature discrete encoding module 20 is connected to the cross-modal attention module 30, injecting the feature dimension-aligned visual lexical sequence and temporal lexical sequence into the tensor computation layer of the cross-modal attention module 30. The cross-modal attention module 30 performs high-dimensional space alignment operations on the two sequences and extracts an attention tensor representing cross-modal correlation.

[0027] The gated blocking and diversion module 40 is connected in series in the output path of the cross-modal attention module 30. It receives the multimodal representation sequence and the attention tensor, performs dimensionality reduction operations, and outputs the crisis probability. The gated blocking and diversion module 40 is equipped with a bypass sending interface that bypasses the multimodal large model decoding layer and directly connects to the monitoring and control node. When the crisis probability is determined to exceed the safety boundary, the gated blocking and diversion module 40 directly outputs a blocking command to the monitoring and control node through this bypass sending interface to cut off the servo drive power circuit of the underlying device.

[0028] The parameter degradation calibration module 50 is logically connected to the conventional data output port of the gated blocking and diversion module 40, the output of the cross-modal attention module 30, and the data buffer port of the data synchronization acquisition module 10. Within the operation cycle where the probability of a crisis does not exceed the limit, the parameter degradation calibration module 50 receives the attention tensor, the multimodal representation sequence, and the original state features obtained by channel mapping, time downsampling, and normalization of the continuous time sequence. It then performs corresponding mathematical transformations and feature compensation operations to obtain calibration parameters, and subsequently calls the digital twin model to solve the mechanism equations to obtain the safety control subspace.

[0029] The mask constraint control module 60 is deployed in the multimodal large model decoder layer of the system. Based on this security control subspace, it constructs a mask vector in the decoder probability output layer. Finally, the mask constraint control module 60 samples and generates a control instruction sequence within the legal probability space and sends it to the monitoring and control node via Ethernet link to execute the control closed loop.

[0030] See attached document Figure 2 This invention provides an intelligent control method for industrial visual monitoring based on a multimodal large model, comprising the following steps: S100: The data synchronization acquisition module 10 acquires key frame images using an image sensing device and obtains the continuous time sequence of the underlying device through the monitoring and control node. S200, the keyframe image is extracted into a visual word sequence by the visual encoder through the feature discrete coding module 20, and the continuous temporal sequence is mapped into a temporal word sequence by the vector quantization encoder; S300 inputs visual lexical sequences and temporal lexical sequences into the cross-attention network through the cross-modal attention module 30, outputs a multimodal representation sequence and extracts the attention tensor; S400 inputs the multimodal representation sequence and attention tensor into the gating network through the gating blocking and diversion module 40 to calculate the crisis probability. When the crisis probability is greater than the threshold, it outputs a blocking command to the monitoring and control node; otherwise, it enters the parameter calibration step. In the parameter calibration step, the attention tensor is converted into a penalty vector through the parameter downgrade calibration module 50. The original state features are calibrated using the penalty vector and the multimodal representation sequence to obtain calibration parameters. The calibration parameters are then input into the digital twin model to solve for the safety control subspace. S600 constructs a mask vector in the multimodal large model decoder vocabulary based on the security control subspace through the mask constraint control module 60. The mask vector is superimposed on the decoder probability output layer for constraint, and the sampled control instruction sequence is sent to the monitoring and control node for execution.

[0031] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the functional modules and method processing flow involved above.

[0032] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S100 includes the following sub-steps: S110, the data synchronization acquisition module 10 uses an image sensing device to collect the monitoring video stream of the target workstation and extract key frame images.

[0033] During the data acquisition process, image sensing devices (such as industrial area scan cameras or high frame rate vision sensors) deployed in the physical environment of the target workstation continuously send the captured monitoring video stream to the memory buffer of the computing platform via a gigabit Ethernet vision interface. The data synchronization acquisition module 10 performs frame parsing operations on the monitoring video stream according to a preset detection interval. During frame parsing, the data synchronization acquisition module 10 strips the encoded header of the video transmission protocol and truncates the continuous video stream into single static frames, outputting keyframe images.

[0034] The keyframe image is used to represent the three-dimensional pixel matrix tensor of the transient visual state of the target workstation, denoted as . ,in Image pixel height, The width of the image in pixels. This refers to the number of color channels. For the aperture closure control, exposure adjustment of the image sensor, and basic analog-to-digital signal conversion logic of image sensing devices, those skilled in the art can implement them using conventional machine vision configuration architectures. The underlying hardware operating mechanisms are well-known technologies in this field and will not be elaborated upon here.

[0035] S120, the data synchronization acquisition module 10 acquires the continuous time sequence of the underlying equipment associated with the target workstation through the monitoring and control node.

[0036] The underlying equipment (such as the drive joints of a multi-axis robotic arm or the spindle unit of a CNC machine tool) is equipped with physical sensors responsible for monitoring motor torque, voltage, and vibration amplitude. Monitoring and control nodes (such as programmable logic controllers or field industrial data gateways) act as data access nodes in the industrial field, reading the digital conversion values ​​from the physical sensors according to a fixed high-frequency polling cycle. The data synchronization acquisition module 10 receives the digital conversion values ​​and sequentially stores them in a circular data queue on the computing platform by establishing a transmission control protocol connection. Subsequently, the data synchronization acquisition module 10 truncates the digital conversion values ​​in the circular data queue according to a clock stamping mechanism, generating a continuous time sequence.

[0037] To achieve clock stamp alignment for heterogeneous data, a precise time protocol network service runs between the computing platform and the monitoring and control node to ensure the underlying clock stamp reference synchronization. When the data synchronization acquisition module 10 is in the clock stamp... Generate a keyframe image At that time, the data synchronization acquisition module 10 uses a clock stamp. Using the reference alignment point, backtrack by a length of... The time observation window. The data synchronization acquisition module 10 retrieves data from the circular data queue within a time range that falls within... All historical digital values ​​within the interval are converted and the retrieved data is serialized and spliced ​​according to the sampling time sequence to obtain a continuous time series.

[0038] From the perspective of data correspondence, keyframe images represent instantaneous snapshots of the device's operating state, while continuous time-series sequences represent the physical change trends over a period of time preceding those snapshots. Through this time window matching method, subsequent calculation modules can obtain prior physical change information related to specific visual phenomena.

[0039] As a preferred method, time observation window The value is determined based on the response hysteresis period of the underlying equipment's mechanical actions. Taking a specific industrial manufacturing scenario as an example, when the underlying equipment at the target workstation is an AC servo motor and its action hysteresis time is approximately 150 milliseconds, the time observation window... It can be configured between 150 milliseconds and 300 milliseconds to ensure that the continuous time series can fully cover the physical parameter fluctuations that cause changes in visual state.

[0040] A continuous time series represents a two-dimensional tensor containing the fluctuation trends of multiple physical quantities, denoted as ,in This represents the total number of independent physical sensor channels corresponding to the underlying devices. In the time observation window The total number of discrete data points collected internally based on the sampling frequency of the monitoring and control nodes. Utilizing the aforementioned window capture operation based on visual clock stamps, the data synchronization acquisition module 10 reduces the impact of communication latency differences between heterogeneous data sources on subsequent feature processing.

[0041] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S200 includes the following sub-steps: S210, the feature discrete coding module 20 extracts the visual word sequence from the keyframe image through the visual encoder.

[0042] In the specific calculation process, the feature discrete encoding module 20 will represent it as a three-dimensional pixel matrix tensor. The keyframe image is divided into multiple fixed-size, non-overlapping two-dimensional image patches. Subsequently, a linear projection network layer configured within the visual encoder flattens each two-dimensional image patch and maps it to a feature dimension of... One-dimensional features.

[0043] The feature discrete encoding module 20 serializes these one-dimensional vectors according to the two-dimensional spatial position order of the original image blocks. After spatial segmentation and linear dimensionality reduction mapping, the output is a visual word sequence. The visual word sequence represents a set of high-dimensional vectors that, after discretizing the global visual image, can be recognized by the attention mechanism of a large model and characterize the spatial morphological features of local regions. The visual word sequence is denoted as a tensor. ,in The total number of two-dimensional image blocks generated by segmentation.

[0044] As a preferred approach, to balance computational efficiency and spatial resolution, the segmentation size of the aforementioned two-dimensional image patch is set to 16×16 pixels, with a feature dimension of... The hidden layer configuration is set to 768 or 1024 depending on the subsequent large model. For the additional positional encoding embedding logic and the specific stacking structure of the multi-head self-attention extraction network within the visual encoder, those skilled in the art can implement it using a conventional visual transformer architecture. The configuration of its internal attention activation layer is well-known in the art and will not be elaborated upon here.

[0045] S220, the feature discrete coding module 20 inputs the continuous time sequence into the vector quantization encoder and uses a one-dimensional convolutional network to extract the temporal hidden layer feature sequence of the continuous time sequence.

[0046] The vector quantization encoder front end is configured with a multi-layered cascaded one-dimensional convolutional network. This one-dimensional convolutional network includes alternating stacked dilated convolutional layers, batch normalization layers, and modified linear unit activation layers. The feature discrete encoding module 20 utilizes the convolutional kernels of the one-dimensional convolutional network along the time step dimension of the continuous temporal sequence. Perform sliding window convolution operations and in the channel dimension Feature aggregation and downsampling are performed on the network. By setting the convolution stride and number of output channels of the one-dimensional convolutional network, the network outputs a temporal hidden layer feature sequence.

[0047] The temporal hidden layer feature sequence represents a continuous set of hidden layer features that encapsulates the local variation waveforms and fluctuation trends of the original high-frequency physical sensing signal. The temporal hidden layer feature sequence is denoted as... ,in This represents the total number of time-step features retained after convolutional downsampling, with its feature channel dimension forcibly mapped to the same feature dimension as the visual word sequence. .

[0048] S230, the feature discrete coding module 20 calculates the Euclidean distance between the temporal hidden layer feature sequence and each discrete feature in the preset discrete codebook space inside the vector quantization encoder, and selects the discrete feature with the smallest Euclidean distance to replace it, generating a temporal lexical sequence with the same feature dimension as the visual lexical sequence.

[0049] In the network structure of a vector quantization encoder, a preset discrete codebook space is initialized and frozen for maintenance. The preset discrete codebook space is denoted as... ,in Represents the first in the codebook A discrete feature, This represents the total number of discrete features contained in the pre-defined discrete codebook space. For the model construction here, the pre-defined discrete codebook space is generated iteratively through a training process that uses a joint optimization of reconstruction loss and commitment loss, based on pre-collected historical time-series datasets of industrial equipment's normal operation. This ensures that the discrete features can serve as standard templates to characterize various typical physical health states of the equipment.

[0050] For temporal hidden layer feature sequences Each time step component ( The feature discrete coding module 20 calculates the components respectively. With each discrete feature in the preset discrete codebook space The Euclidean distance. The specific formula for calculating the distance is: ; After traversing all After identifying each discrete feature, the feature discrete encoding module 20 executes nearest neighbor quantization logic, selecting the discrete feature with the smallest Euclidean distance to the original continuous components. Perform a hard replacement. The specific feature replacement formula is as follows: ,in ; In industrial data processing scenarios, the aforementioned Euclidean distance comparison and replacement operation is used to find the best-matching waveform template in a standard signal feature library. This process effectively filters out mechanical vibration noise and electromagnetic interference from high-frequency sensor signals in industrial environments. The feature discrete encoding module 20 then processes all the discrete components obtained after the replacement. Reassemble and combine them according to the original time step sequence.

[0051] After the aforementioned distance comparison and quantization replacement calculations, the feature discrete encoding module 20 outputs a temporal lexical sequence. This temporal lexical sequence represents the removal of underlying physical sensing noise interference and the discretization of feature distribution, thus enabling cross-modal interactive computation with the visual lexical sequence within the same high-dimensional tensor space. The temporal lexical sequence is denoted as... Its feature dimensions Strictly aligned with the feature dimensions of the visual word sequence.

[0052] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S300 includes the following sub-steps: S310, the cross-modal attention module 30 performs a linear transformation on the visual lexical sequence to generate a query matrix, and performs a linear transformation on the temporal lexical sequence to generate a key matrix and a value matrix respectively.

[0053] Since the visual lexical sequence and the temporal lexical sequence have already been mapped to the same feature dimension in the previous stage. The cross-modal attention module 30 utilizes its internal cross-attention network configuration of independent fully connected mapping network layers to perform linear projection on the two types of sequences, so as to calculate the correlation between different modal information in the same feature subspace.

[0054] In the specific operation, the cross-modal attention module 30 will process the visual word sequence. The query weight matrix is ​​multiplied by the query matrix to obtain the query matrix. Similarly, the cross-modal attention module 30 performs a matrix multiplication on the temporal lexical sequence. Matrix multiplication is performed with the key weight matrix and value weight matrix respectively to output the key matrix and value matrix. The above linear transformation network computation layer does not contain a non-linear activation function, and its calculation formula is as follows: ; ; ; In the formula, Represents the query matrix; Represents the bond matrix; Representative value matrix; Represents the query weight matrix; Represents the key weight matrix; Representative value weight matrix; and satisfying , , . is the hidden layer dimension of the mapping vector inside a single attention head in a cross-attention network.

[0055] All the weight matrices mentioned above are learnable parameters, and their values ​​are iteratively optimized during the overall backpropagation training phase of the multimodal large model using a gradient descent algorithm on the loss function. Through the above linear transformation, the visual word sequence and the temporal word sequence are projected onto the feature subspace that the cross-attention network can compute.

[0056] S320, the cross-modal attention module 30 performs a scaled dot product operation on the transpose of the query matrix and the key matrix and obtains the attention tensor through normalized exponential activation.

[0057] To support subsequent parameter degradation calibration steps, the cross-modal attention module 30 extracts the attention mapping results from the cross-attention network. The cross-modal attention module 30 then calculates the query matrix. AND key matrix transpose matrix The dot product is used to measure the similarity between each visual local region and the physical state at each historical time step. To prevent the gradient vanishing or exploding problem during backpropagation caused by the variance of the dot product result increasing with the dimension of the hidden layer, the cross-modal attention module 30 divides the dot product result by a scaling factor. Based on this, the cross-modal attention module 30 applies a normalized exponential activation function along the last tensor dimension for probability normalization, obtaining and explicitly extracting the attention tensor. The mathematical expression for this operation is: ; In the formula, Represents the extracted attention tensor. This represents the normalized exponential activation function. The dimension of the attention tensor is denoted as . The internal element values ​​are strictly limited to the interval [0, 1], and the sum of the elements in each row of the matrix is ​​forced to be normalized to 1.

[0058] In this embodiment, the attention tensor represents a two-dimensional weight distribution matrix characterizing cross-modal correlation. This matrix records the probability distribution of the correlation between the specific visual anomalies of the current target workstation and the fluctuations of underlying physical parameters over different time periods. This attention tensor not only supports subsequent feature fusion but also serves as a key penalty basis extracted by the parameter degradation calibration module 50 in subsequent parameter calibration steps.

[0059] S330, the cross-modal attention module 30 multiplies the attention tensor with the value matrix to output a multimodal representation sequence.

[0060] After establishing the correlation weight distribution between visual and temporal aspects, the cross-modal attention module 30 uses the attention tensor... To guide this process, the value matrix containing underlying temporal feature information is analyzed. Perform a weighted summation operation. This is done by performing matrix multiplication. The network strengthens temporal physical features that are highly correlated with the current visual state and suppresses historical redundant features with low correlation.

[0061] To enable the cross-modal weighted output to form a residual connection with the visual word sequence, the cross-modal attention module 30 further utilizes the output projection matrix to perform a dimension back mapping on the cross-modal weighted output, making its feature dimension match the feature dimension of the visual word sequence. To maintain consistency, the cross-modal attention module 30 then performs a residual connection between the cross-modal weighted output after dimension backmapping and the visual word sequence, and performs layer normalization to obtain the multimodal representation sequence. The corresponding calculation formula is expressed as: ; ; ; In the formula, The intermediate latent variables represent the cross-modal weighted output; Represents the output projection matrix; This represents the cross-modal weighted output after backmapping through the output projection matrix, and its dimension is the same as the visual word sequence. Consistent; Represents the normalized function after layers The final output after processing is a multimodal representation sequence.

[0062] The multimodal representation sequence achieves feature dimensional fusion between visual spatial morphological features and underlying high-frequency dynamic physical features. This multimodal representation sequence simultaneously includes visual representation information of the workstation and underlying physical change information related to this visual representation information, which is used to provide state input for subsequent gating judgment and control instruction sequence generation.

[0063] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S400 includes the following sub-steps: S410, the gated blocking and diversion module 40 performs pooling dimensionality reduction operations on the multimodal representation sequence and the attention tensor, and concatenates the processed features along a specific tensor dimension to generate a fused feature vector.

[0064] To adapt to the input dimension of the gated network and reduce the computational overhead caused by high-dimensional tensors, the gated blocking and decompression module 40 uses a global average pooling layer to compress the input high-dimensional tensors. Specifically, during the computation, the gated blocking and decompression module 40 processes the multimodal representation sequence output from the previous network computation layer. (Its tensor dimension is) ) along its sequence length dimension Calculate the element mean and output a one-dimensional visual temporal feature vector, denoted as . .

[0065] At the same time, the gating blocking and diversion module 40 extracts the attention tensor. Perform two-dimensional flattening to generate a one-dimensional correlation distribution vector, denoted as... ,in The total number of elements after flattening (i.e. Based on this, the gating blocking and diversion module 40 will convert the one-dimensional visual temporal feature vector... With one-dimensional correlation distribution vector The features are concatenated along the feature dimension, and the network operation outputs a fused feature vector.

[0066] The fused feature vector representation simultaneously encompasses the distribution patterns of underlying multimodal comprehensive features and the one-dimensional hidden layer features of cross-modal correlation weight relationships. It can provide feature-based evidence for determining whether an industrial site is in a fault state without requiring decoder inference. The fused feature vector is denoted as... .

[0067] S420, the gated blocking and diversion module 40 inputs the fused feature vector into the gated network for nonlinear mapping, calculates the output critical probability and compares it with the threshold.

[0068] The gated network employs a multilayer perceptron structure, which internally comprises an input layer, two cascaded hidden layers, and a single-neuron output layer. The hidden layers are connected by fully connected weight matrices and are coupled with batch normalization functions and modified linear unit activation functions to form a nonlinear mapping between the fused feature vector and the crisis probability.

[0069] The gated blocking and diversion module 40 will fuse feature vectors. The bottom layer of this gating network is input. When the network propagates forward to the single-neuron output layer, the gating network uses an exponential activation function to map the generated scalar value to the interval [0, 1]. The gating blocking and diversion module 40 calculates the critical situation probability through this single-neuron output layer. The mathematical expression for this operation is: ; In the formula, Represents the probability of a critical situation; Represents an exponential activation function; and These represent the weight matrix and bias vector of a single neuron's output layer, respectively. This represents the nonlinear transformation integration function executed by the hidden layer. The crisis probability represents the numerical value of the likelihood that a physical fault will occur at the current target workstation, requiring the mechanical action to be stopped.

[0070] The specific implementation of the model construction and training parameters for the aforementioned gated network is as follows: After the basic pre-training of the entire multimodal large model is completed, the gated network is trained separately using a thaw fine-tuning strategy. The sample set used for supervised fine-tuning comes from a collection of historical industrial fault recordings and corresponding monitoring video footage. The sample labels are marked as 0 or 1 based on whether a fault event triggering an emergency shutdown occurred in the corresponding underlying equipment at the time of recording, where 0 represents safety and 1 represents danger. The loss function used in the fine-tuning network is configured as a binary cross-entropy loss, and the network weights are iteratively updated using a stochastic gradient descent algorithm, enabling the gated network to output the probability of a critical situation corresponding to the critical operating condition.

[0071] After completing the probability calculation, the gated blocking and diversion module 40 will determine the probability of a critical situation. With threshold Perform numerical comparison. Threshold. The value is set between 0.80 and 0.95. As a preferred approach, when the underlying equipment of the target workstation is a high-risk actuator (such as a heavy-duty hydraulic press or a high-speed spindle), the threshold value is... It is set to 0.85 to improve the sensitivity of collision avoidance and overload protection.

[0072] S430, when the probability of a critical situation is determined to be greater than the threshold, the gated blocking and diversion module 40 outputs a blocking command to the monitoring and control node; otherwise, it enters the parameter calibration step.

[0073] When the comparison result is as follows At this time, the hardwired interrupt logic inside the gating blocking and diversion module 40 is triggered. To avoid software process queuing delays, the gating blocking and diversion module 40 skips the multimodal large model decoder layer in its physical architecture and directly generates a blocking command. The gating blocking and diversion module 40 sends the above blocking command to the monitoring and control node via the bypass sending interface, using the highest priority industrial Ethernet communication protocol. After receiving the blocking command, the monitoring and control node disconnects the control relay at the underlying firmware level, cutting off the power circuit of the servo drive of the underlying device. The blocking command is a forced shutdown control command directly driven by the underlying neural network feature layer in emergency situations.

[0074] As a complete algorithm for traffic splitting, when the comparison result is expressed as When this occurs, it indicates that the current operating condition has not triggered the safety threshold, and the target workstation may only have a steady-state machining deviation requiring minor adjustments. At this time, the gating blocking diversion module 40 maintains the original data flow channel without being blocked, allowing the attention tensor, multimodal representation sequence, and original state features to flow into the parameter degradation calibration module 50, thus directing the execution of the step to the subsequent parameter calibration steps. Through this diversion method, the system directly executes the blocking command when the blocking conditions are met, and continues to execute the parameter calibration steps when the blocking conditions are not met, thereby reducing erroneous shutdowns when the blocking conditions are not met.

[0075] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S500 includes the following sub-steps: S510, in the parameter calibration step, the parameter downgrade calibration module 50 performs pooling and inverse mapping of the attention tensor into a penalty vector, and multiplies the penalty vector element-wise with the original state features to obtain the first feature term.

[0076] The attention tensor extracted by the cross-modal attention module 30 in the preceding steps The parameter degradation calibration module 50 is along the visual space dimension. Max pooling is performed to extract the maximum cross-modal correlation weight at each time step. The parameter degradation and calibration module 50 limits the output of the pooling operation to the interval [0, 1] using an inverse mapping function to obtain the penalty vector. The specific calculation formula is as follows: ; In the formula, The penalty vector represents the degraded weight sequence of the contribution of the underlying physical state to the visual anomaly in each time step. The lower the value, the higher the correlation between the underlying physical state at that time step and the current visual anomaly. Therefore, it is necessary to reduce the direct proportion of its original measurement value and increase the proportion of the multimodal mapping compensation term in the subsequent parameter calibration process.

[0077] After obtaining the penalty vector, the parameter degradation calibration module 50 reads the raw state features. Raw state features refer to the set of state features reflecting the measured values ​​of each sensor in the underlying device before encoder processing, denoted as... ,in The number of state components. The state components in the original state features are arranged in a fixed order and include at least position feedback components, velocity feedback components, torque feedback components, and load estimation components; the number of time steps of each state component relative to the temporal lexical sequence. Alignment.

[0078] The original state characteristics are composed of continuous time series The data is obtained through channel mapping, time downsampling, and normalization. Channel mapping maps motor torque, voltage, vibration amplitude, position feedback, and speed feedback values ​​from different sensing channels to a unified state dimension. Time downsampling aligns the sampling length of the continuous time sequence to the number of time steps in the time sequence lexical sequence. Normalization is used to eliminate the dimensional differences between different physical quantities.

[0079] The parameter degradation calibration module 50 will penalize the vector. Features along the original state The state component dimensions are expanded to obtain the extended penalty vector. The extended penalty vector is then multiplied element-wise with the original state features to calculate the first feature term. The corresponding calculation formula is: ; In the formula, Represents the first characteristic term; This represents the element-wise multiplication operator. The first feature term represents the original state feature after adjustment based on the extended penalty vector.

[0080] S520, the parameter degradation calibration module 50 extracts the multimodal mapping tensor using the multimodal characterization sequence, multiplies the multimodal mapping tensor with the inverted result of the penalty vector element by element to obtain the second feature term, and adds the first feature term and the second feature term to obtain the calibration parameter.

[0081] To compensate for the lack of information in the first feature term caused by the reduced proportion of original measurements in the time steps of anomalous contributions, the parameter downgrade calibration module 50 introduces a multimodal representation sequence containing global context features. The parameter degradation calibration module 50 uses an internally configured fully connected regression network layer to process the multimodal representation sequence. (Tensor dimension is) The network is dimensionality reduced and projected to the same time step and state component dimensions as the original state features. The forward operation output of the network yields a multimodal mapping tensor, denoted as... The weight parameters of this fully connected regression network layer are simultaneously optimized through the overall backpropagation process of the multimodal large model.

[0082] Next, the parameter degradation calibration module 50 applies the extended penalty vector. Perform a scalar-1 subtraction inversion operation to generate compensation coefficients for the degradation time step. The parameter degradation calibration module 50 then maps the compensation coefficients to the multimodal mapping tensor. Perform element-wise multiplication to calculate the second characteristic term.

[0083] ; In the formula, This represents the second feature term. Through the above calculations, the parameter degradation calibration module 50 will adjust the first feature term... With the second characteristic term Perform vector addition and summarize the output calibration parameters.

[0084] ; In the formula, The calibration parameters represent the state components in the calibration parameters, which correspond one-to-one with the state components in the original state features, and are aligned with the temporal lexical sequence in the time dimension.

[0085] In the above calibration process, the penalty vector When the value approaches 0, it indicates that the correlation between the underlying physical state of the corresponding time step and the current visual anomaly is relatively high. The parameter downgrade calibration module 50 reduces the direct proportion of the original state features of this time step in the calibration parameters, and instead increases the proportion of the compensation term corresponding to the multimodal mapping tensor, thereby obtaining calibration parameters that take into account both the original sensing information and the multimodal context compensation information.

[0086] S530, the parameter degradation calibration module 50 inputs the calibration parameters into the digital twin model mechanism equation for iterative solution, generating a safety control subspace for the next control cycle.

[0087] The digital twin model incorporates state-space differential equations characterizing the kinematics and dynamics of the underlying target equipment. For modeling and constructing these dynamic equations, those skilled in the art can employ Lagrangian mechanics or the Newton-Euler method. For example, when the underlying equipment of the target workstation is a multi-axis industrial robotic arm, its dynamic differential equations are specifically constructed as follows: ; In the formula, , and These represent the position vector, velocity vector, and acceleration vector of each moving joint in the underlying device, respectively. The inertia matrix represents the distribution of equipment mass. This represents the coupling matrix reflecting the influence of the Coriolis force and the centrifugal force; Represents the gravity term vector; This represents the control torque vector output by the underlying servo driver.

[0088] The parameter degradation calibration module 50 will process the calibration parameters after feature compensation. Decoupling and extraction are performed based on the physical meaning of each state component in the calibration parameters to obtain the initial state vector required for the digital twin model. The initial state vector includes at least the initial position of the underlying device. Initial velocity The current torque estimate and the current load estimate, and the initial position. With initial velocity The assignment is the state-space differential equation at time t. The initial state at that time.

[0089] After establishing the initial state, the parameter degradation calibration module 50 uses the Runge-Kutta numerical integration algorithm to iteratively calculate the predicted trajectory extreme values ​​of physical parameters such as torque, temperature and position of the underlying equipment in the next control cycle.

[0090] The parameter degradation calibration module 50 uses a control parameter vector as the solution object, which includes the target motion velocity, target joint torque, target displacement, and acceleration limit values. The module performs discrete sampling or constraint optimization within a preset control parameter search range and substitutes each set of candidate control parameters into a digital twin model for prediction and deduction. The module extracts a set of compliant parameters from the untriggered equations, specifying the material yield limit and mechanical interference constraint regions, and simultaneously satisfying torque thresholds, temperature thresholds, position boundaries, and velocity boundary constraints. The multidimensional geometric region enclosed by this set of compliant parameters in the mathematical space constitutes the calculated safety control subspace.

[0091] As a concrete manifestation of physical boundaries, the security control subspace represents the permissible range of command parameters that ensure the physical safety of the underlying equipment under objective conditions such as wear, interference, or performance degradation. After obtaining the security control subspace through the above deduction process, the subsequent multimodal large model decoder layer will be strictly constrained by the boundaries of the security control subspace to ensure that the generated control command sequence will not cause damage to the underlying equipment.

[0092] See attached document Figure 1 and Figure 2 In this embodiment, the specific implementation of step S600 includes the following sub-steps: S610 traverses the multimodal large model decoder vocabulary through the mask constraint control module 60, and assigns conditional values ​​to the corresponding dimensions of the mask vector based on whether the extracted set of physical control parameters falls into the security control subspace.

[0093] Let the total capacity of the vocabulary of the multimodal large model decoder be... The mask constraint control module 60 initializes a one-dimensional tensor filled with zeros in memory as a mask vector, denoted as . During the traversal of this vocabulary, the mask constraint control module 60 identifies the first... The attribute categories of each word element, among which .

[0094] If the term belongs to a semantic text term, the mask constraint control module 60 maintains the mask vector. The Each element remains unchanged if it is 0. If the term belongs to a control term representing a device action, the mask constraint control module 60 calls the pre-maintained control term mapping table to parse the set of physical control parameters corresponding to the control term.

[0095] The control lexicon mapping table records the lexicon index, action category, device number, and corresponding physical control parameter field of each control lexicon in the multimodal large model decoder vocabulary. The physical control parameter field includes at least one of the following: target motion speed, target joint torque, target displacement, acceleration limit value, and start / stop flag. The mask constraint control module 60 parses the set of physical control parameters corresponding to the control lexicon based on the control lexicon mapping table, such as target motion speed or target joint torque.

[0096] For such control terms, the mask constraint control module 60 verifies the boundary relationship between the aforementioned set of physical control parameters and the security control subspace obtained in the preceding steps. When a control instruction is composed of multiple terms, the mask constraint control module 60 performs incremental parsing based on the currently generated partial instruction sequence to obtain partial or complete control parameters corresponding to the candidate next term, and updates the mask vector according to the relationship between the parsing result and the security control subspace.

[0097] When the set of physical control parameters represented by the control term falls within the geometric boundary of the safety control subspace, it indicates that the set of physical control parameters is within the physically permissible range. The mask constraint control module 60 will then apply the mask vector... The Each element is assigned a value of 0; conversely, if the set of physical control parameters exceeds the boundary range of the safety control subspace, the mask constraint control module 60 will assign a value of 0 to the mask vector. The Each element is assigned the value of negative infinity (-∞). The mathematical expression for the conditional assignment in this process is: ; In the formula, The index in the mask vector is The element value; The first in the representative vocabulary list The set of physical control parameters parsed from each control term; This represents the security control subspace.

[0098] By using the conditional assignment operation with lexical classification attributes, the semantic text lexical units are prevented from being erroneously intercepted, and a mapping relationship between the discrete vocabulary space and the continuous physical security boundary is established.

[0099] S620, the mask constraint control module 60 extracts the original logarithmic probability distribution tensor of the multimodal large model and superimposes the mask vector onto the probability output layer of the decoder for constraint calculation.

[0100] In each time step of the autoregressive generative loop of a multimodal large model, its internal feedforward network outputs the raw log probability distribution tensor without activation function processing, denoted as . This tensor reflects the initial tendency of the model to predict each word in the vocabulary as the next output based on the current context sequence. To introduce physical boundary constraints into the model generation process, the mask constraint control module 60 applies the constructed mask vector to the decoder probability output layer of the multimodal large model (i.e., before the feedforward computation node of the Softmax activation function). With the original logarithmic probability distribution tensor Perform tensor addition operations on the corresponding dimension. The corresponding calculation formula is: ; In the formula, This represents the constrained log-probability tensor after the superposition mask.

[0101] From the perspective of operational logic, for control words and semantic text words that satisfy the security control subspace constraints, since their corresponding values ​​in the mask vector are 0, the original generation tendency of these words remains unchanged during the addition operation; for control words that do not satisfy the security control subspace constraints, their corresponding mask values ​​are negative infinity, and the corresponding initial tendency values ​​are lowered to negative infinity after the addition, thereby suppressing the possibility of their generation at the numerical level.

[0102] S630, the mask constraint control module 60 performs normalization activation on the constrained log probability tensor after superposition operation, samples and generates a control instruction sequence in the legal probability space and sends it to the monitoring and control node for execution.

[0103] After completing the tensor superposition, the mask constraint control module 60 will control the constrained log probability tensor. The input is fed into the activation function for exponential normalization, and the final generation probability distribution of each word in the vocabulary is calculated, denoted as... The calculation formula is: ; In the formula, This represents the normalized exponential activation function.

[0104] After the above activation operation, control terms that do not satisfy the security control subspace constraints have a logarithmic probability value of negative infinity, and their corresponding natural exponential function output approaches zero, causing these control terms to have a generation probability distribution The middle part is effectively masked. The resulting generation probability distribution is as follows: This constitutes a controlled legal probability space.

[0105] Within this legal probability space, the mask constraint control module 60 performs autoregressive cyclic sampling according to the set language model sampling strategy. Regarding the execution principles and implementation methods of specific sampling algorithms (such as kernel sampling or beam search algorithms) in large language model decoding, those skilled in the art can select and configure them according to the resource requirements of the actual computing device; these are well-known technologies in the field of natural language processing and will not be elaborated upon here. The mask constraint control module 60 outputs lexical units that conform to grammatical rules and satisfy physical boundary constraints through cyclic sampling, and finally concatenates them to generate a complete control instruction sequence.

[0106] After a single generation process is completed, the mask constraint control module 60 encapsulates the control command sequence into corresponding industrial control messages and sends them to the monitoring and control nodes for execution using a high-priority communication protocol. This mask constraint mechanism ensures that the control command sequence output by the system is constrained by the safety control subspace, preventing mechanical damage to underlying devices caused by executing out-of-bounds commands.

[0107] To further illustrate the implementation process and technical effects of this invention, the following description is provided in conjunction with specific application scenarios and experimental data. The specific numerical values, scenario parameters, and comparative experiments described below are only used to explain the implementation principle of this invention and do not limit the scope of protection of this invention.

[0108] The underlying equipment at the target workstation is set to a multi-axis industrial robotic arm, and the computing platform acquires heterogeneous data through data synchronization module 10. (The last part, "clock stamp," appears to be an unrelated fragment and is omitted from the translation.) In milliseconds, the data synchronization acquisition module 10 uses an image sensing device to acquire keyframe images. The keyframe image The pixel matrix has dimensions of 224×224×3. The data synchronization acquisition module 10 uses this clock stamp as the reference alignment point and traces back a length of... The time observation window is in milliseconds. The sampling frequency of the monitoring and control node is set to 1000Hz. The underlying device includes four independent physical sensing channels: position feedback value, velocity feedback value, torque feedback value, and vibration amplitude value. The data synchronization acquisition module 10 captured 150 discrete data points (i.e., Generates a continuous time series with a dimension of 4×150. At this point, the data alignment input for the intelligent control method of industrial visual monitoring based on a multimodal large model was completed.

[0109] The feature discrete encoding module 20 receives the above data. Keyframe image. The visual encoder divides the image into 16×16 pixel two-dimensional blocks. The total number of two-dimensional image blocks generated by the segmentation is... After passing through a linear projection network layer, the feature dimensions are extracted. Visual word sequence Continuous time series The input vector quantization encoder performs feature aggregation and downsampling using a one-dimensional convolutional network, retaining the total number of features at each time step. Temporal hidden layer feature sequences were extracted. The total number of discrete features contained in the preset discrete codebook space is set to [value]. The feature discrete encoding module 20 calculates the Euclidean distance between each component in the temporal hidden layer feature sequence and these 512 discrete features, selects the discrete feature with the smallest distance for hard replacement, and generates a temporal lexical sequence with the same feature dimension as the visual lexical sequence. .

[0110] The cross-modal attention module 30 will visualize word sequences. Generate the query matrix by performing matrix multiplication with the query weight matrix. , time sequence Generate the key matrix by performing matrix multiplication with the key weight matrix and value weight matrix respectively. AND-value matrix Cross-modal attention module with 30 query matrices AND key matrix The transpose of the expression is subjected to a scaled dot product operation and then passed through a normalized exponential activation function to extract an attention tensor with a dimension of 196×30. The cross-modal attention module 30 will use the attention tensor AND-value matrix Multiplication, followed by residual connection and layer normalization, outputs a multimodal representation sequence with a dimension of 196×768. .

[0111] The gated blocking and diversion module 40 performs pooling dimensionality reduction operations on the above features. Multimodal representation sequence Average pooling yields a one-dimensional visual temporal feature vector with a feature dimension of 768. Attention tensor. After two-dimensional flattening, a one-dimensional correlation distribution vector is generated, with a total of 196 × 30 = 5880 elements. The gating and blocking diversion module 40 concatenates the one-dimensional visual temporal feature vector and the one-dimensional correlation distribution vector along the feature dimension to generate a fused feature vector with a dimension of 6648. The gated blocking and diversion module 40 will fuse feature vectors. Input a gating network with a multilayer perceptron structure. The gating network calculates the probability of a critical situation. The system-set threshold The probability of a critical situation was determined to be 0.65, which is less than the threshold of 0.85. The gating blocking and diversion module 40 did not output a blocking command to the monitoring and control node, and the flow direction parameter calibration step was executed.

[0112] In the parameter calibration step, the parameter degradation calibration module 50 adjusts the attention tensor along the visual space dimension. Performing max pooling, if the maximum cross-modal association weight extracted at a certain time step is 0.8, the downgrade weight at that time step is obtained as 1 after transformation by the inverse mapping function. 0.8 = 0.2. The parameter degradation calibration module 50 performs calculations for all time steps, resulting in a penalty vector with a time step dimension of 30. The original state features are obtained from continuous time series through channel mapping, time downsampling, and normalization. Number of state components The parameter degradation calibration module 50 will penalize the vector. Features along the original state The extended penalty vector is obtained by expanding the dimensions of the state components. The extended penalty vector is then multiplied element-wise with the original state features to obtain the first feature term. The parameter degradation calibration module 50 utilizes multimodal characterization sequences. Extracting multimodal mapping tensors The compensation coefficient is generated by performing a scalar subtraction and inversion operation on the extended penalty vector. The compensation coefficient is then multiplied element-wise with the multimodal mapping tensor to obtain the second feature term. The parameter downgrade calibration module 50 performs a vector addition operation between the first feature term and the second feature term, and summarizes and outputs the calibration parameters. .

[0113] The parameter degradation calibration module 50 decouples and extracts the parameters according to their physical meanings to obtain the initial state vector required by the digital twin model. The initial state vector includes the initial position of the underlying device. rad and initial velocity rad / s, which is assigned as the value of the state-space differential equation at time t. The initial state at that time. The parameter degradation calibration module 50 substitutes the control parameter vector into the mechanism equation of the digital twin model for forward iterative solution, extracts the compliance parameter set, and the multidimensional geometric region enclosed by this compliance parameter set in the mathematical space is the calculated safety control subspace. The safety control subspace indicates the allowable range of values ​​for the target joint torque. 15,15]Nm.

[0114] Reference Figure 3 The predicted trajectory curve represents the torque fluctuation trajectory obtained by the digital twin model through forward iteration based on the initial state; the upper limit curve (15 Nm) and lower limit curve (-15 Nm) of the torque threshold represent the physical boundaries set according to the material yield strength and mechanical constraints. The predicted trajectory curve is strictly limited between the upper limit curve and the lower limit curve of the torque threshold, and the region enclosed by these upper and lower limit curves intuitively reflects the target joint torque of […]. The safety control subspace of 15,15]Nm.

[0115] The mask constraint control module 60, based on this security control subspace, uses the multimodal large model decoder vocabulary (total capacity) Construct mask vector The mask constraint control module identified the 500th word as a control word after 60 iterations, and the target joint torque in the parsed physical control parameter set was 20 Nm. Because 20 [ 15,15], and as Figure 3 As indicated by the out-of-bounds control term value points, the set of physical control parameters exceeds the boundary range of the security control subspace. The mask constraint control module 60 assigns the 500th element of the mask vector to negative infinity. (∞). The mask constraint control module 60 performs tensor addition operations on the mask vector and the original log-probability distribution tensor in the corresponding dimension at the decoder probability output layer to obtain the constrained log-probability tensor. The mask constraint control module 60 performs normalized activation on the constrained log-probability tensor. Within the legal probability space, the generation probability of the control term that does not satisfy the security control subspace constraints approaches zero. Finally, the mask constraint control module 60 samples and generates a target joint torque of 12 Nm within the legal probability space (which conforms to the attached...). Figure 3 The sequence of control instructions (marked with the status of the compliance control instruction value point) is sent to the monitoring and control node to execute the control loop.

[0116] Experimental verification and effect comparison: To verify the effectiveness of an intelligent control method for industrial vision monitoring based on a multimodal large model, test samples were constructed based on the operating status of a multi-axis robotic arm. Three algorithmic schemes were then compared and analyzed in conjunction with the corresponding monitoring video streams. The three algorithmic schemes are as follows: Method A is an end-to-end mechanical control method based on a pure vision transducer. This method only calculates the probability of a crisis and generates a sequence of control instructions based on the visual word sequence, without introducing the continuous time sequence of the underlying equipment.

[0117] Method B is a behavior cloning control method based on a visual-temporal dual-stream network. This method integrates visual and temporal lexical sequences to calculate the crisis probability and generate a control instruction sequence, but does not include the steps of using the parameter degradation calibration module 50 to solve the digital twin model and the mask constraint control module 60 to generate a mask vector for constraint.

[0118] Method C is an intelligent control method for industrial visual monitoring based on a multimodal large model. This method generates control command sequences through visual lexical sequences, temporal lexical sequences, attention tensors, gating blocking and diversion modules 40, parameter degradation and calibration modules 50, digital twin models, and mask constraint control modules 60.

[0119] The experiment employed three statistical indicators. The instruction compliance rate represents the percentage of test cases in the generated control instruction sequence where the set of physical control parameters is within the allowable range of the safety control subspace; the unnecessary downtime rate represents the percentage of test cases where the probability of a crisis does not correspond to the actual crisis state but triggers a blocking instruction; and the single response time represents the time consumed from acquiring keyframe images to issuing the control instruction sequence.

[0120] The test data statistics are shown in the table below.

[0121] The data above shows that Method A, lacking a continuous temporal sequence from the underlying devices, struggles to adjust the control command sequence based on the dynamic state of these devices. Consequently, its command compliance rate is 76.5%, and its unnecessary downtime rate is 14.2%. Method B, by introducing visual and temporal lexical sequences, can reduce the unnecessary downtime rate. However, it fails to solve the safety control subspace using a digital twin model and does not constrain control lexical units through mask vectors at the decoder's probability output layer. Therefore, under boundary conditions, it may still generate control command sequences that exceed the safety control subspace.

[0122] Method C achieves a 99.7% compliance rate, a 0.6% unnecessary downtime rate, and a single response time of 121ms. Since the underlying equipment has a 150ms hysteresis period, Method C's single response time is shorter than this hysteresis period, meeting the control timing requirements of the target workstation. Method C utilizes a digital twin model to solve for the safety control subspace and performs tensor addition on the mask vector and the original logarithmic probability distribution tensor at the decoder's probability output layer to obtain a constrained logarithmic probability tensor. This numerically masks control terms that exceed the safety control subspace. Therefore, it improves the compliance of control command sequences while reducing the unnecessary downtime rate.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent control and management of industrial visual monitoring based on a multimodal large model, characterized in that, Includes the following steps: Keyframe images are acquired using image sensing devices, and continuous time sequences of underlying devices are obtained through monitoring and control nodes. The keyframe image is processed by a visual encoder to extract a visual word sequence, and the continuous temporal sequence is mapped into a temporal word sequence by a vector quantization encoder. The visual lexical sequence and the temporal lexical sequence are input into a cross-attention network, which outputs a multimodal representation sequence and extracts the attention tensor. The multimodal representation sequence is input into the attention tensor gating network to calculate the crisis probability. When the crisis probability is greater than a threshold, a blocking command is output to the monitoring and control node; otherwise, the parameter calibration step is entered. In the parameter calibration step, the attention tensor is converted into a penalty vector, and the original state features are calibrated using the penalty vector and the multimodal representation sequence to obtain calibration parameters. The calibration parameters are then input into the digital twin model to solve for the security control subspace. Based on the security control subspace, a mask vector is constructed in the decoder vocabulary of the multimodal large model. The mask vector is superimposed on the decoder probability output layer for constraint, and a control instruction sequence is sampled and sent to the monitoring and control node for execution.

2. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of acquiring keyframe images using an image sensing device and obtaining the continuous time sequence of the underlying device through a monitoring and control node includes: The image sensing device is used to acquire the monitoring video stream of the target workstation, and the key frame image is output by performing frame parsing operation on the monitoring video stream according to the preset detection interval. The monitoring and control node receives the digital conversion values ​​from the physical sensing elements and stores the digital conversion values ​​sequentially into a circular data queue. When the keyframe image is generated at the reference alignment point, the time observation window is traced back, and historical digital conversion values ​​whose time range falls within the time observation window interval are retrieved from the circular data queue. The retrieved historical digital values ​​are serialized and concatenated according to the chronological order of sampling time to obtain the continuous time sequence.

3. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The steps of extracting visual word sequences from the keyframe images using a visual encoder and mapping the continuous temporal sequence to a temporal word sequence using a vector quantization encoder include: The keyframe image is divided into two-dimensional image blocks without physical overlap. Each two-dimensional image block is flattened and mapped to a one-dimensional feature using the linear projection network layer configured inside the visual encoder. The visual word sequence is then serialized and output according to the spatial position order. The vector quantization encoder utilizes a one-dimensional convolutional network configured within it to perform sliding window convolution operations along the time step dimension of the continuous temporal sequence, and performs feature aggregation and downsampling in the channel dimension to output the temporal hidden layer feature sequence. Calculate the Euclidean distance between the temporal hidden layer feature sequence and each discrete feature in the preset discrete codebook space, and select the discrete feature with the smallest Euclidean distance to replace and generate the temporal lexical sequence.

4. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of inputting the visual lexical sequence and the temporal lexical sequence into a cross-attention network, outputting a multimodal representation sequence, and extracting the attention tensor includes: The visual word sequence is multiplied with the query weight matrix to output the query matrix, and the temporal word sequence is multiplied with the key weight matrix and the value weight matrix to output the key matrix and the value matrix respectively. Calculate the dot product of the query matrix and the transpose of the key matrix, divide the dot product by a scaling factor, and then apply a normalized exponential activation function to obtain the attention tensor. The attention tensor is multiplied by the value matrix to obtain a cross-modal weighted output. After performing dimensional back mapping on the cross-modal weighted output using the output projection matrix, a residual connection is performed with the visual lexical sequence, and a layer normalization operation is performed to obtain the multimodal representation sequence.

5. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of calculating the crisis probability by inputting the multimodal representation sequence and the attention tensor into the gating network, and outputting a blocking command to the monitoring and control node when the crisis probability is greater than a threshold, and otherwise proceeding to the parameter calibration step, includes: The mean of elements along the sequence length dimension of the multimodal representation sequence is calculated to output a one-dimensional visual temporal feature vector, and the attention tensor is flattened in two dimensions to generate a one-dimensional correlation distribution vector. The fused feature vector is obtained by concatenating the one-dimensional visual temporal feature vector and the one-dimensional correlation distribution vector along the feature dimension. The fused feature vector is input into the gated network for nonlinear mapping calculation to obtain the probability of the emergency. When the comparison determines that the probability of the emergency is greater than the threshold, the blocking command is output to the monitoring and control node via the bypass sending interface to cut off the servo driver power circuit of the underlying device. When the comparison determines that the probability of the emergency is not greater than the threshold, the parameter calibration step is entered.

6. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of converting the attention tensor into a penalty vector and using the penalty vector and the multimodal representation sequence to calibrate the original state features to obtain calibration parameters includes: Max pooling is performed on the attention tensor along the visual space dimension, and the output of the pooling operation is converted into the penalty vector by limiting it to a preset interval through an inverse mapping function; The penalty vector is extended along the state component dimension of the original state feature to obtain an extended penalty vector. The extended penalty vector is then multiplied element-wise with the original state feature to calculate the first feature term. The multimodal representation sequence is dimensionality reduced and projected to output a multimodal mapping tensor. A scalar 1 subtraction and inversion operation is performed on the extended penalty vector to generate a compensation coefficient. The compensation coefficient is then multiplied element-wise with the multimodal mapping tensor to calculate the second feature term. The calibration parameters are output by performing a vector addition operation on the first feature term and the second feature term.

7. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 6, characterized in that, The original state features are obtained from the continuous time sequence through channel mapping, time downsampling, and normalization. The state components in the original state features are arranged in a fixed order; Each state component in the original state feature is aligned with the number of time steps in the temporal lexical sequence in the time dimension.

8. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of inputting the calibration parameters into the digital twin model to solve for the safety control subspace includes: The calibration parameters are decoupled and extracted according to the physical meaning of the state components in the calibration parameters to obtain the initial state vector, and the initial state vector is assigned as the initial state of the state space differential equation. The extreme values ​​of the predicted trajectory of the underlying device in the next control cycle are calculated iteratively along the time axis using a numerical integration algorithm. Sampling is performed within the preset control parameter search range, and candidate control parameters are substituted into the digital twin model for prediction and deduction to extract a set of compliant parameters that do not trigger the restricted area and meet the physical boundary constraints. The multidimensional geometric region encompassed by the set of compliance parameters in the mathematical space is taken as the security control subspace.

9. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of constructing a mask vector from the decoder vocabulary of the multimodal large model based on the security control subspace includes: Initialize a one-dimensional tensor with all zeros as the mask vector, and traverse the decoder vocabulary of the multimodal large model to identify the attribute category of the word; If the term belongs to a semantic text term, the corresponding element of the mask vector remains zero. If the term belongs to a control term, the pre-maintained control term mapping table is called to parse the physical control parameter set corresponding to the control term, and the boundary relationship between the physical control parameter set and the security control subspace is verified. When the set of physical control parameters falls within the geometric boundary of the security control subspace, the corresponding element of the mask vector is assigned a value of zero; When the set of physical control parameters exceeds the boundary range of the security control subspace, the corresponding element of the mask vector is assigned a value of negative infinity.

10. The intelligent control method for industrial visual monitoring based on a multimodal large model according to claim 1, characterized in that, The step of superimposing the mask vector onto the decoder probability output layer for constraint, and sampling to generate a control command sequence to be sent to the monitoring and control node for execution includes: Extract the original log probability distribution tensor of the multimodal large model, and perform tensor addition operation of the corresponding dimension on the mask vector and the original log probability distribution tensor in the probability output layer of the decoder to obtain the constrained log probability tensor; The constrained log probability tensor is input into the activation function and subjected to exponential normalization to obtain the generated probability distribution, which constitutes the legal probability space. Within the legal probability space, autoregressive cyclic sampling is performed according to the set language model sampling strategy to output word units, and the output word units are concatenated to generate the control instruction sequence; The sequence of control instructions is encapsulated into an industrial control message and sent to the monitoring and control node for execution.

Images