Production operation scene intelligent monitoring method, device, equipment, medium and product

Abstract

The application relates to the technical field of industrial intelligent monitoring, and provides a production operation scene intelligent monitoring method, device, equipment, medium and product.The production operation scene intelligent monitoring method comprises the following steps: based on a preset production task process and a process specification, confidence degree distribution is calculated through multi-modal observation data, and a task belief field is generated; a nonlinear dynamic system model is established based on physical quantities in the multi-modal observation data, dimension is increased to a linear space, energy change is predicted in the linear space, and an energy state vector is generated; a bidirectional mapping relationship between energy change rate and task confidence degree is established, an energy belief coupler is constructed, and a joint deviation index is obtained; a free energy function is constructed based on the joint deviation index, an adjustment gradient direction is calculated under a free energy minimization criterion; a monitoring strategy is dynamically adjusted according to the adjustment gradient direction, and monitoring is performed in the production operation scene based on the monitoring strategy. The technical scheme provided by the application improves the real-time performance of abnormal detection.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent monitoring technology, and in particular to an intelligent monitoring method, device, equipment, medium and product for production operation scenarios. Background Technology

[0002] With the development of industrial intelligence and digital production, real-time monitoring and risk warning at production sites have become crucial for ensuring safety and improving efficiency. Existing production monitoring systems typically employ solutions based on image recognition, sensor data acquisition, and rule-based threshold judgment. They collect on-site information through devices such as cameras, temperature sensors, and current sensors, and then use traditional algorithms or shallow neural networks to identify and issue alerts regarding equipment status, personnel behavior, and environmental parameters.

[0003] Existing monitoring systems mostly operate in a single-modal or weakly fused multimodal manner, lacking temporal and semantic consistency among data sources. This results in monitoring results being significantly affected by noise and sampling asynchrony. Current algorithms can only perform surface recognition of objects or actions, failing to determine whether operational behavior conforms to the expected task state. Monitoring models are often static, leading to low resource utilization and poor real-time performance. Furthermore, at the energy and physical levels, existing prediction methods typically employ empirical regression or black-box neural network models, resulting in insufficient stability and interpretability of prediction results, hindering the achievement of coordinated information-driven and energy-driven monitoring. Summary of the Invention

[0004] This invention provides a method, device, equipment, medium, and product for intelligent monitoring of production operation scenarios, in order to overcome the deficiencies in the prior art.

[0005] This invention provides an intelligent monitoring method for production operation scenarios, comprising: Acquire multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario; Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated using the multimodal observation data to generate a task belief field. A nonlinear dynamic system model is established based on the physical quantities in the multimodal observation data. The nonlinear dynamic system model is then upgraded to a linear space. Energy changes are predicted within the linear space, and an energy state vector is generated. Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler. Based on the joint deviation index, a free energy function is constructed, and the adjustment gradient direction is calculated under the free energy minimization criterion. The monitoring strategy is dynamically adjusted according to the gradient direction, and monitoring is performed in the production operation scenario based on the monitoring strategy.

[0006] According to the present invention, an intelligent monitoring method for production operation scenarios is provided, wherein a bidirectional mapping relationship between energy change rate and task confidence is established based on the energy state vector and the task belief field, an energy belief coupler is constructed, and a joint deviation index is obtained based on the energy belief coupler, including: Each energy component in the energy state vector is mapped to the corresponding confidence node in the task belief field according to the work unit number and timestamp, forming an energy belief matching matrix; The time-delay correlation function is determined based on the energy belief matching matrix, and the dominant coupling delay is determined based on the time-delay correlation function. Under the constraint of the dominant coupling delay, an energy belief coupler is defined, comprising a forward mapping sub-model and a backward mapping sub-model; wherein the forward mapping sub-model takes the energy change rate as input and outputs the confidence update, and the backward mapping sub-model takes the confidence gradient as input and outputs the energy correction. The energy offset and belief offset are calculated based on the outputs of the forward mapping sub-model and the reverse mapping sub-model, and the joint bias index is calculated based on the energy offset and belief offset.

[0007] According to the present invention, an intelligent monitoring method for production operation scenarios is provided, wherein, under the constraint of the dominant coupling delay, an energy belief coupler comprising a forward mapping sub-model and a reverse mapping sub-model is defined, including: A parameter sharing layer and a residual channel are set between the forward mapping sub-model and the reverse mapping sub-model; wherein, the parameter sharing layer and the residual channel are used to synchronously update the common weights of the forward mapping sub-model and the reverse mapping sub-model and transmit cross-modal deviation information; During the training phase, the parameter-sharing layer is iteratively updated using a weighted minimization objective function until the convergence condition is met, resulting in the trained energy belief coupler.

[0008] According to the present invention, an intelligent monitoring method for production operation scenarios is provided, wherein the method generates a task belief field by calculating the confidence distribution of each task state based on the preset production task flow and process specifications and the multimodal observation data, including: Based on the preset production task flow and process specifications, a task status ontology is established, and a task tag set is generated; Modal features are extracted from the multimodal observation data, and the modal features and the task label set are input into the multimodal alignment encoder to generate the task feature vector of the job unit; Based on the task feature vector and the task state ontology, the confidence level of the task state is calculated, and a confidence matrix is ​​generated based on the confidence level. A temporal attention structure with process sequence masking is used to perform weighted fusion on the confidence matrix to generate a task confidence sequence; The task confidence sequence is stored as a confidence distribution according to time index and job unit number; Differentiate and normalize the confidence scores of adjacent time frames to obtain a gradient sequence representing the rate of change of the task state; The confidence distribution and the gradient sequence are jointly mapped to a unified spatiotemporal coordinate system to form the task belief field arranged by job unit index and timestamp.

[0009] According to the present invention, an intelligent monitoring method for production operation scenarios is provided, wherein the method involves establishing a nonlinear dynamic system model based on physical quantities in the multimodal observation data, upgrading the nonlinear dynamic system model to a linear space, predicting energy changes within the linear space, and generating an energy state vector, including: A set of state variables is established based on the physical quantities in the multimodal observation data, and a multidimensional state sequence is constructed according to the time index. A system evolution function is defined on the multidimensional state sequence, and the local changes of the system evolution function are fitted by a family of nonlinear functions to obtain the nonlinear dynamic system model. The nonlinear evolution function is mapped to a learnable Koopman feature space to extract the hidden state vector that can linearly describe the system dynamics; Within the Koopman feature space, based on the hidden state vector, the hidden state of subsequent time steps is predicted using a linear state transition operator; During the prediction process, the energy change rate of each state component is recorded, and the energy change rate is used to form the energy state vector.

[0010] According to the intelligent monitoring method for production operation scenarios provided by the present invention, the step of constructing a free energy function based on the joint deviation index and calculating the adjustment gradient direction under the free energy minimization criterion includes: Based on the joint bias index and multimodal observation error, a free energy input vector is formed under a unified time index; The second norm of the joint bias index and the probability density of the multimodal observation error are calculated in the free energy input vector, respectively. The second norm and the probability density are then weighted and summed to generate the free energy function. The partial derivative of the free energy function is calculated to form a gradient direction vector, and the adjustment gradient direction of the current monitoring strategy is determined based on the gradient direction vector.

[0011] According to the present invention, an intelligent monitoring method for a production operation scenario is provided, wherein after dynamically adjusting the monitoring strategy based on the adjustment gradient direction and performing monitoring in the production operation scenario based on the monitoring strategy, the method further includes: If the detected free energy or energy belief deviation is greater than the corresponding preset threshold, a control command is generated and the active inference parameters are incrementally updated.

[0012] According to the intelligent monitoring method for production operation scenarios provided by the present invention, the incremental update of the active inference parameters includes: At the beginning of each free energy iteration cycle, based on the adjustment gradient direction calculated in the previous iteration, the angle offset adjustment is performed on the viewpoint parameters and focal length parameters of the camera array, and the sampling rate and trigger threshold of the sound and vibration acquisition nodes are adjusted. In key areas, invoke a local enhanced sampling process, increase the temporal sampling frequency, and shorten the sliding window length; New multimodal observation data is obtained based on the adjusted camera array, sound and vibration acquisition nodes, and an observation matrix is ​​formed based on the new multimodal observation data; Based on the observation matrix, the updated multimodal observation error and the updated joint bias index are calculated and input into the free energy function to trigger the next free energy minimization iteration. After multiple iterations of minimizing free energy, when the rate of change of free energy is lower than the set convergence threshold, the current monitoring strategy parameters are determined, and the monitoring configuration corresponding to the state is recorded as a stable observation benchmark.

[0013] According to the intelligent monitoring method for production operation scenarios provided by the present invention, the step of acquiring multimodal observation data under a unified timestamp includes: Collect multimodal information from production operation scenarios and establish sampling time indexes for modal information from each data source; Based on the sampling time index, and with a preset unified sampling period as a benchmark, the modal information of each data source is subjected to time interpolation and linear resampling to generate a multimodal time series. In the multimodal time series, data points whose rate of change exceeds a set change threshold or whose sampling is lost are marked; For the labeled data points, anomaly removal and missing value compensation are performed by using nearest neighbor window reconstruction and weighted mean replacement. The multimodal time series, after anomaly removal and missing value compensation, are spliced ​​at the frame level according to a unified timestamp to generate the multimodal observation data.

[0014] The present invention also provides an intelligent monitoring device for production operation scenarios, comprising: The acquisition module is configured to acquire multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario; The first calculation module is configured to calculate the confidence distribution of each task state based on the preset production task process and process specifications, and generate a task belief field by using the multimodal observation data. The generation module is configured to establish a nonlinear dynamic system model based on the physical quantities in the multimodal observation data, upgrade the nonlinear dynamic system model to a linear space, predict energy changes in the linear space, and generate an energy state vector. The module is configured to establish a bidirectional mapping relationship between the energy change rate and the task confidence based on the energy state vector and the task belief field, construct an energy belief coupler, and obtain a joint bias index based on the energy belief coupler. The second calculation module is configured to construct a free energy function based on the joint deviation index, and calculate the adjustment gradient direction under the free energy minimization criterion. The monitoring module is configured to dynamically adjust the monitoring strategy based on the adjustment gradient direction, and perform monitoring in the production operation scenario based on the monitoring strategy.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intelligent monitoring method for production operation scenarios as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent monitoring method for production operation scenarios as described above.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the intelligent monitoring method for production operation scenarios as described above.

[0018] The intelligent monitoring method, device, equipment, medium, and product for production operation scenarios provided by this invention constructs a task belief field, mapping the production task process and technological specifications into a continuous confidence distribution of task states. This enables the identification of task state changes in each work unit based on multimodal observation data, thereby possessing task-semantic-oriented monitoring capabilities. By upgrading the nonlinear dynamic system model to a linear space for modeling, energy changes are predicted within the linear space, ensuring consistency between the prediction process and the law of energy conservation, thus improving the stability and interpretability of monitoring. Simultaneously, by setting up an energy belief coupler, a bidirectional mapping relationship is established between the energy change rate and task confidence, achieving dynamic collaboration between the physical and cognitive layers. This allows energy anomalies to drive belief updates, and belief deviations to inversely correct energy predictions, forming a cross-level information closed loop. A free energy function is constructed using a joint deviation index, and the adjustment gradient direction is calculated under the free energy minimization criterion. Based on the adjustment gradient direction, the monitoring strategy is dynamically adjusted, achieving a shift from passive detection to proactive perception, improving the real-time performance and resource utilization of anomaly detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the intelligent monitoring method for production operation scenarios provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the intelligent monitoring device for production operation scenarios provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0025] Figure 1 This is a flowchart illustrating an intelligent monitoring method for a production operation scenario according to an exemplary embodiment. For example... Figure 1 As shown in an exemplary embodiment, the intelligent monitoring method for production operation scenarios includes steps 110 to 160, which are described in detail below.

[0026] Step 110: Obtain multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario.

[0027] In this embodiment of the invention, multimodal information includes data from various data sources such as images, sound, temperature, current, voltage, and vibration. Since different sensors have varying sampling frequencies and physical characteristics, directly using raw data can lead to monitoring results being affected by sampling asynchrony and noise. Therefore, time synchronization is performed on the collected multimodal information to align all modal data to a basic clock frequency, for example, a uniform period of 50 milliseconds (ms). Data with frequencies higher than this can be downsampled, and data with frequencies lower than this can be compensated by interpolation. Furthermore, outlier data points caused by momentary sensor malfunctions, obstructions, or transmission packet loss are removed through anomaly filtering.

[0028] After the above time synchronization and anomaly filtering processes, the data from each modality are spliced ​​together at the frame level according to a unified timestamp to generate multimodal observation data.

[0029] In this embodiment of the invention, time synchronization and anomaly filtering processes effectively eliminate time and semantic inconsistencies between data sources, reducing the impact of noise and sampling asynchrony on monitoring results.

[0030] Step 120: Based on the preset production task process and process specifications, calculate the confidence distribution of each task state using the multimodal observation data to generate a task belief field.

[0031] In this embodiment of the invention, the preset production task flow and process specifications are predefined process logics based on actual industrial production scenarios. They define which work units are included in the production process, what task states each work unit has, for example, the task states of an assembly station can be defined as part picking, positioning, fastening, inspection, etc., as well as the logical relationships between these states, including sequential order and mutual exclusion. The task state represents the current logical node of the work unit.

[0032] Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated through multimodal observation data. The confidence distribution is a quantitative assessment of the probability that the current moment belongs to each task state after feature extraction from the multimodal observation data.

[0033] The confidence distribution and the gradient of the confidence distribution of all work units over a continuous time period are mapped to a spatiotemporal coordinate system to form a task belief field. The gradient of the confidence distribution represents the rate of change of the task status. The task belief field can reflect the completion degree and safety status of the current task. The local gradient of the task belief field represents the dynamic trend of the task status, that is, which task is being transitioned from to which task.

[0034] Step 130: Establish a nonlinear dynamic system model based on the physical quantities in the multimodal observation data, upgrade the nonlinear dynamic system model to a linear space, predict energy changes in the linear space, and generate an energy state vector.

[0035] In this embodiment of the invention, the physical quantities are data reflecting energy consumption and state, such as current, voltage, vibration, and temperature, contained in multimodal observation data. In the real physical world, energy evolution usually follows highly complex nonlinear laws, making it difficult to make accurate long-term predictions using simple regression models. Therefore, this embodiment of the invention establishes a nonlinear dynamic system model to characterize the evolutionary relationship of these physical quantities over time.

[0036] In this embodiment of the invention, a mapping mechanism using the Koopman operator is introduced. The system evolution trajectory, which is difficult to solve in a low-dimensional nonlinear space, is mapped to a high-dimensional feature space and then described by a linear state transition operator. Within this linear space, the energy change trajectory and rate of change for several future time steps can be predicted quickly and stably. Subsequently, the energy changes during the prediction process are recorded and structurally encapsulated to form an energy state vector. The fields of the energy state vector include the rate of change, direction of change, fluctuation amplitude, and prediction residual for each energy component. Through the established nonlinear dynamic system model, the prediction process is ensured to be consistent with the actual physical conservation laws.

[0037] Step 140: Based on the energy state vector and the task belief field, establish a bidirectional mapping relationship between the energy change rate and the task confidence, construct an energy belief coupler, and obtain a joint bias index based on the energy belief coupler.

[0038] In this embodiment of the invention, in traditional monitoring systems, data from physical sensors and image recognition results from cameras are often disconnected. Therefore, this embodiment establishes an energy belief coupler to bridge the gap between these two. Based on the energy state vector representing the underlying physical state and the task belief field representing the upper-level cognitive logic, which have already been obtained, a bidirectional mapping relationship is established to construct the energy belief coupler. In this bidirectional mapping relationship, changes in physical energy positively drive and increase the confidence of a certain task state, while the expected evolution of the task state negatively corrects and constrains the prediction of future energy parameters.

[0039] Based on the constructed energy belief coupler, a joint deviation index is obtained. This index measures whether there is a discrepancy between the expected physical energy and the actual cognitive task. For example, if a worker is identified as performing a tightening action, but the current and vibration energy do not increase accordingly, the joint deviation index will rise, suggesting a possible false action or equipment malfunction. The energy belief coupler enables dynamic coordination between the physical and cognitive layers, forming a closed loop between information-driven and energy-driven approaches, thus improving the recognition rate of complex, latent anomalies.

[0040] Step 150: Construct the free energy function based on the joint deviation index, and calculate the adjusted gradient direction under the free energy minimization criterion.

[0041] In this embodiment of the invention, a free energy function is constructed based on the joint bias index. This constructed free energy function is a mathematical function that comprehensively measures both the joint bias index and multimodal observation errors. A larger free energy function value indicates a more confused understanding of the current production scenario or a higher level of unknown risk. The goal of the monitoring system is to make its internal model's predictions as close to the real world as possible, i.e., to minimize the free energy. The partial derivative of the free energy function is calculated to determine the adjustment gradient direction, which explicitly points to the region of maximum information gain or the most severe deviation in the production operation scenario.

[0042] Step 160: Dynamically adjust the monitoring strategy according to the adjusted gradient direction, and perform monitoring in the production operation scenario based on the monitoring strategy.

[0043] In this embodiment of the invention, the monitoring strategy is dynamically adjusted based on the adjusted gradient direction. The monitoring strategy includes, but is not limited to, the camera's observation direction, camera focal length, and the sampling frequency of various sensors. The monitoring strategy is iteratively updated based on the calculated adjusted gradient direction. For example, if the adjusted gradient direction points to the region with the largest gradient in the task belief field, the camera's viewing angle may be automatically adjusted to align with that region, and the sampling frequency of relevant sensors may be increased to obtain more and more accurate information.

[0044] In this embodiment of the invention, by constructing a task belief field, the production task process and technological specifications are mapped to a continuous confidence distribution of task states. This enables the identification of task state changes in each work unit based on multimodal observation data, thereby providing task-semantic-oriented monitoring capabilities. By upgrading the nonlinear dynamic system model to a linear space for modeling, energy changes are predicted within the linear space, ensuring consistency between the prediction process and the law of energy conservation.

[0045] Simultaneously, by setting up an energy-belief coupler, a bidirectional mapping relationship is established between the energy change rate and task confidence, enabling dynamic collaboration between the physical and cognitive layers. This allows energy anomalies to drive belief updates, while belief biases can in turn correct energy predictions, forming a cross-layer information loop. A free energy function is constructed using a joint bias index, and the adjustment gradient direction is calculated under the free energy minimization criterion. Based on this adjustment gradient direction, the monitoring strategy is dynamically adjusted, achieving a shift from passive detection to proactive perception, improving the real-time performance and resource utilization of anomaly detection.

[0046] In an exemplary embodiment of the present invention, the step of establishing a bidirectional mapping relationship between the energy change rate and the task confidence level based on the energy state vector and the task belief field, constructing an energy belief coupler, and obtaining a joint bias index based on the energy belief coupler includes: Each energy component in the energy state vector is mapped to the corresponding confidence node in the task belief field according to the work unit number and timestamp, forming an energy belief matching matrix; The time-delay correlation function is determined based on the energy belief matching matrix, and the dominant coupling delay is determined based on the time-delay correlation function. Under the constraint of the dominant coupling delay, an energy belief coupler is defined, comprising a forward mapping sub-model and a backward mapping sub-model; wherein the forward mapping sub-model takes the energy change rate as input and outputs the confidence update, and the backward mapping sub-model takes the confidence gradient as input and outputs the energy correction. The energy offset and belief offset are calculated based on the outputs of the forward mapping sub-model and the reverse mapping sub-model, and the joint bias index is calculated based on the energy offset and belief offset.

[0047] In this embodiment of the invention, since the structures of energy data and task data are different, the timestamp and the work units in space are concatenated at corresponding nodes. Specifically, based on the common indices of the work unit number and the unified timestamp, each component in the energy state vector is paired with the confidence node at the corresponding spatiotemporal location in the task belief field to form an energy belief matching matrix. Each row or column of the energy belief matching matrix shows the parallel situation of physical energy state and task cognitive state at a specific time and a specific unit.

[0048] In actual industrial production, there may be a time delay between changes in physical quantities and their manifestation at the task level. For example, the current rises instantaneously after equipment startup, but the confidence level of the corresponding equipment startup task state may take a short stabilization period before reaching its peak. Therefore, by calculating the time-delay correlation function between the two and finding the peak value, the dominant coupling delay is determined. The dominant coupling delay represents the response delay between energy change and task belief change. In this embodiment, the energy change rate, confidence change rate, and the time-delay correlation function between the two are calculated on the energy belief matching matrix.

[0049] Under the constraint of the dominant coupling delay, a forward mapping sub-model and a backward mapping sub-model are defined. The forward mapping sub-model embodies the logic of physical changes driving task state changes; it takes the energy change rate as input and outputs the confidence update. Before inputting the energy change rate into this sub-model, it is time-aligned according to the determined dominant coupling delay. This sub-model can be a parameterized mapping structure, such as a small neural network.

[0050] The inverse mapping sub-model embodies the logic that changes in task state affect energy prediction. It takes the confidence gradient as input and outputs the energy correction. Similarly, the input also needs to undergo delayed alignment, and this sub-model also adopts a parameterized mapping structure.

[0051] The above forward and backward mapping sub-models together constitute the energy belief coupling structure.

[0052] The forward mapping sub-model predicts a required confidence update based on the current energy change, and determines the difference between the predicted confidence update and the actual confidence update in the task belief field to obtain the belief shift. Similarly, the backward mapping sub-model predicts a required energy correction based on the current confidence change, and determines the difference between the predicted energy correction and the actual energy change in the energy state vector to obtain the energy shift.

[0053] The energy offset and belief offset are weighted and combined to calculate the final joint bias index.

[0054] In this embodiment of the invention, a bidirectional mapping between energy and belief is established, and time delay analysis is introduced to make this mapping relationship closer to physical reality. As a result, the energy-belief coupler can accurately capture the dynamic correspondence between energy behavior and task behavior, thereby outputting a highly sensitive and reliable joint deviation index, providing a solid foundation for subsequent proactive reasoning and anomaly warning.

[0055] In an exemplary embodiment of the present invention, defining an energy belief coupler comprising a forward mapping sub-model and a backward mapping sub-model under the constraint of the dominant coupling delay includes: A parameter sharing layer and a residual channel are set between the forward mapping sub-model and the reverse mapping sub-model; wherein, the parameter sharing layer and the residual channel are used to synchronously update the common weights of the forward mapping sub-model and the reverse mapping sub-model and transmit cross-modal deviation information; During the training phase, the parameter-sharing layer is iteratively updated using a weighted minimization objective function until the convergence condition is met, resulting in the trained energy belief coupler.

[0056] In this embodiment of the invention, although the directions of the forward mapping (energy → belief) and the reverse mapping (belief → energy) are opposite, the correlation patterns they describe are interconnected. Therefore, in this invention, the two sub-models are not completely independent; several parameter-sharing layers are set between them, allowing the forward mapping sub-model and the reverse mapping sub-model to share a portion of the network weight parameters. Since energy fluctuations and task behavior reflect two aspects of the same production event at a deep semantic level, parameter sharing forces the model to capture the common change patterns between these two modalities. Through parameter sharing, the model can learn more fundamental correlations with fewer parameters, reducing the risk of overfitting and accelerating model convergence.

[0057] The residual channel is used to directly pass cross-modal difference information between two mapping sub-models, preventing deep networks from losing subtle bias features during backpropagation. In other words, the residual channel directly passes the output or intermediate layer features of one mapping sub-model and the bias information predicted by another mapping sub-model to the other mapping sub-model as an additional input or correction term, which helps the two sub-models to calibrate each other and work together during training and prediction.

[0058] During offline training of the model, a corresponding loss function needs to be set to guide the optimization of network parameters. The objective function, serving as the loss function, includes energy conservation residuals and belief consistency error. The energy conservation residuals are used to evaluate the bias in energy prediction; they are based on physical laws to calculate whether the predicted energy changes are reasonable, and this bias constitutes one loss term. The belief consistency error is used to evaluate the logical rationality of the confidence evolution; it is based on a pre-defined process flow to determine whether the transfer of task confidence conforms to the logical order and constraints, and its bias constitutes another loss term. The two loss terms are then combined according to pre-defined weights.

[0059] During training, optimization algorithms such as gradient descent are employed to iteratively update the non-shared parameters of the parameter-sharing layer and the two mapping sub-models based on the gradient of the objective function. Training continues until the objective function value falls below a preset threshold or a preset number of training epochs are reached. At this point, the model is considered to have reached convergence, and the energy belief coupler is considered fully constructed.

[0060] This invention introduces a parameter sharing layer and a residual channel, and designs a training objective that combines energy conservation residuals and belief consistency errors. This enables the energy belief coupler to deeply learn the inherent and stable coupling relationship between physical processes and task logic, thereby improving the accuracy of the coupler and giving it better generalization ability. It can adapt to monitoring tasks under different working conditions, thus providing a core guarantee for the stable operation of the entire system.

[0061] In an exemplary embodiment of the present invention, the step of calculating the confidence distribution of each task state based on the preset production task flow and process specifications, and generating a task belief field through the multimodal observation data, includes: Based on the preset production task flow and process specifications, a task status ontology is established, and a task tag set is generated; Modal features are extracted from the multimodal observation data, and the modal features and the task label set are input into the multimodal alignment encoder to generate the task feature vector of the job unit; Based on the task feature vector and the task state ontology, the confidence level of the task state is calculated, and a confidence matrix is ​​generated based on the confidence level. A temporal attention structure with process sequence masking is used to perform weighted fusion on the confidence matrix to generate a task confidence sequence; The task confidence sequence is stored as a confidence distribution according to time index and job unit number; Differentiate and normalize the confidence scores of adjacent time frames to obtain a gradient sequence representing the rate of change of the task state; The confidence distribution and the gradient sequence are jointly mapped to a unified spatiotemporal coordinate system to form the task belief field arranged by job unit index and timestamp.

[0062] In this embodiment of the invention, the task state ontology is a structured knowledge representation. It is not simply a list of actions, but rather a semantic model of the entire production process. The task state ontology includes: Task nodes define each key process in the production flow, such as part picking, positioning, fastening, and inspection. State transition relationships define the legal flow order between task nodes; for example, positioning must be done after material retrieval. The process constraint set defines more complex logical relationships, such as dependency relationships which define that execution of A must be completed before B can be completed, and mutual exclusion relationships which define that C and D cannot be executed simultaneously.

[0063] Each task node corresponds to a semantic description and execution conditions of a work unit. It defines specific work units and entity-action-process triples, clarifying their dependencies and mutual exclusions, thereby generating a set of task labels for subsequent classification and recognition by the artificial intelligence model. Examples of entity-action-process triples include worker, grasp, and part.

[0064] Due to the significant structural differences between different modalities, deep features most relevant to the task state are extracted from multimodal observation data after time synchronization and filtering. Different feature extraction methods can be used for different modalities, specifically: For image modalities, spatial features can be extracted using convolutional neural networks, and key pose features can be output using a skeleton detection model. For sound and vibration modes, spectral envelopes and pulse features can be extracted using signal processing techniques such as short-time Fourier transform and Mel spectrum analysis. For temperature, current, and voltage modes, their time difference and steady-state offset characteristics can be extracted using methods such as moving average filtering and first-order difference.

[0065] Subsequently, the modal features extracted from different modalities, along with the aforementioned task label set, are input into the multimodal alignment encoder. This multimodal alignment encoder is constructed based on a large-scale neural network. Through a cross-modal attention mechanism, it aligns and fuses the extracted modal features in terms of time and semantics, and generates a task feature vector at the fusion mapping layer that comprehensively represents the current work unit state, including action relevance, operation stage features, and process semantic cues.

[0066] The task feature vectors are mapped to the score space of the task label set using the task state projection head. After obtaining the initial score, the initial score is constrained and calibrated according to the execution order and mutual exclusion relationship defined in the task state ontology to form a frame-level confidence matrix.

[0067] Furthermore, a temporal attention structure with a process sequence mask is used to perform weighted fusion of the confidence matrix. This temporal attention structure adds a mask to the standard temporal attention mechanism. The mask is generated according to the process sequence in the task state ontology and is used to prevent the attention mechanism from paying attention to future or past states that do not conform to the process logic. For example, in the localization stage, the mask will enhance the model's attention to past material picking states and shield the influence of future detection states.

[0068] In addition, confidence smoothing and conflict redistribution are performed at the time window boundaries based on task dependencies to keep confidence consistent. For example, if the confidence of conflicting tasks increases at the same time, a forced redistribution is performed based on the dependency matrix, ultimately generating a window-level task confidence sequence.

[0069] The fused task confidence sequence is stored as a continuous confidence distribution according to time index and job unit number.

[0070] Differentiation and normalization are performed on the confidence scores of consecutive time frames to obtain a gradient sequence representing the rate of change of task state. The confidence score distribution and gradient sequence are mapped onto a unified spatiotemporal coordinate system to form a task belief field arranged by job unit index and timestamp.

[0071] In an exemplary embodiment of the present invention, the step of establishing a nonlinear dynamic system model based on the physical quantities in the multimodal observation data, upgrading the nonlinear dynamic system model to a linear space, predicting energy changes in the linear space, and generating an energy state vector includes: A set of state variables is established based on the physical quantities in the multimodal observation data, and a multidimensional state sequence is constructed according to the time index. A system evolution function is defined on the multidimensional state sequence, and the local changes of the system evolution function are fitted by a family of nonlinear functions to obtain the nonlinear dynamic system model. The nonlinear evolution function is mapped to a learnable Koopman feature space to extract the hidden state vector that can linearly describe the system dynamics; Within the Koopman feature space, based on the hidden state vector, the hidden state of subsequent time steps is predicted using a linear state transition operator; During the prediction process, the energy change rate of each state component is recorded, and the energy change rate is used to form the energy state vector.

[0072] In this embodiment of the invention, physical quantities are extracted from multimodal observation data and defined as a set of state variables. The extracted physical quantities are then combined into a multidimensional state sequence according to time indices. These data form the basis for characterizing the energy load of equipment operating at the work site.

[0073] Since the energy changes of industrial equipment during work are inherently nonlinear, a system evolution function is defined to describe the energy transfer relationship from the current time step to the next time step. The system evolution function takes the current value of the state vector as input and the energy vector of the next time step as output. During the training phase, a sliding time window is used to calculate the input-output pairs to represent the energy transfer relationship between adjacent time steps. A family of nonlinear functions, represented by a multilayer feedforward neural network, is used to fit the local changes of the system evolution function, thereby constructing a nonlinear dynamic system model.

[0074] This invention introduces the Koopman operator theory, mapping nonlinear evolution functions to a high-dimensional, learnable Koopman feature space. In this feature space, complex nonlinear dynamic processes can be described by simple linear transformations. Specifically, the Koopman feature space can be implemented by adding a feature transformation layer inside the nonlinear dynamic system model. The input of this feature transformation layer is an energy state vector, and the output is a set of high-dimensional observables, i.e., latent state vectors. During training, these observables are optimized by minimizing a joint objective function of prediction and reconstruction errors, extracting this latent state vector that linearly describes the system dynamics.

[0075] After upgrading to the Koopman feature space, a linear state transition operator is learned from the historical hidden state sequence through methods such as least squares estimation. In the prediction stage, the current hidden state vector is used as the initial input, and then by continuously multiplying it by this linear state transition operator, the hidden state sequence for multiple future time steps can be deduced, outputting the energy change trajectory in the continuous time domain.

[0076] The predicted hidden state sequence is deduced back into the physical space to obtain the predicted trajectory of future physical quantities. During this process, key dynamic indicators such as the rate of change, direction of change, fluctuation amplitude, and prediction residual of each state component are recorded. These indicators are combined to form a structured energy state vector, which is then associated with the corresponding timestamp and job unit number through an index table.

[0077] This invention, by introducing the Koopman operator theory, transforms the nonlinear prediction problem into a linear prediction problem, and utilizes the powerful feature fitting ability of neural networks to ensure the stability of the prediction process.

[0078] In an exemplary embodiment of the present invention, the step of constructing the free energy function based on the joint deviation index and calculating the adjusted gradient direction under the free energy minimization criterion includes: Based on the joint bias index and multimodal observation error, a free energy input vector is formed under a unified time index; The second norm of the joint bias index and the probability density of the multimodal observation error are calculated in the free energy input vector, respectively. The second norm and the probability density are then weighted and summed to generate the free energy function. The partial derivative of the free energy function is calculated to form a gradient direction vector, and the adjustment gradient direction of the current monitoring strategy is determined based on the gradient direction vector.

[0079] In this embodiment of the invention, the joint bias index quantifies the inherent inconsistency or difference between the physical layer model and the cognitive layer model. Multimodal observation error is the residual between the original sensor data and the model predictions, reflecting the uncertainty or noise level of the external observations themselves. The joint bias index is used as the physical-cognitive difference term, and the multimodal observation error is used as the observation residual term. These two indices are concatenated under a unified time index to form a free energy input vector, which serves as the original input for constructing the free energy function.

[0080] The free energy function is a scalar function used to quantify the overall uncertainty of a monitoring system. It is constructed as follows: For joint deviation index Calculate its second norm (L2 norm), which represents the degree of deviation from the model's prediction. The larger the L2 norm, the more severe the inherent conflict between the physical and cognitive models, and the higher the free energy.

[0081] Multimodal observation error Estimate its probability density function based on its historical samples And calculate the negative logarithmic probability. This item represents the inaccuracy of the data, specifically the multimodal observation error. The lower the probability of its occurrence, the greater its negative logarithmic probability, and the greater its contribution to free energy.

[0082] The free energy function is obtained by weighted summation of the above two terms, which can be expressed as: ; in, For joint deviation index, This is a diagonal weight matrix used for scaling and importance weighting of different bias components. This refers to the multimodal observation error obtained by stitching together the observation residuals from images, sounds, and various sensors. The probability density function obtained by estimating residual samples based on the nearest neighbor time window is in The value at that location, and These are the corresponding global weight coefficients, so that the relative impact of model deviation and observation noise can be flexibly adjusted under different operating conditions.

[0083] By taking the partial derivative of the free energy function using numerical differentiation or backpropagation, we obtain the gradient direction vector. In mathematical physics, the opposite direction of the gradient is the direction in which the function value decreases the fastest. Therefore, adjusting the gradient direction points towards the policy adjustment direction that maximizes the decrease in total uncertainty (free energy), i.e., the information gain.

[0084] In this embodiment of the invention, an adjustable parameter set including observation direction, camera focal length, sampling frequency and modal weights is established in the monitoring strategy parameter space. The parameter set is iteratively updated according to the gradient direction vector. After each parameter update, the gradient in the task belief field is recalculated, the region with the largest change in confidence is identified and marked as the region with the largest information gain, the region is preferentially allocated observation resources and key sampling is triggered, the multimodal observation error is recalculated with the updated observation results, and returned to the free energy input vector.

[0085] In an exemplary embodiment of the present invention, after dynamically adjusting the monitoring strategy according to the adjusted gradient direction and performing monitoring in the production operation scenario based on the monitoring strategy, the method further includes: If the detected free energy or energy belief deviation is greater than the corresponding preset threshold, a control command is generated and the active inference parameters are incrementally updated.

[0086] In this embodiment of the invention, free energy represents the total uncertainty currently faced by the system, while energy belief bias explicitly indicates the degree of conflict between physical layer prediction and cognitive layer understanding, i.e., the aforementioned joint bias index. Corresponding preset thresholds are set for these two indicators, which can be determined based on the safety standards of the actual production workshop or the standard deviation of historical operating data. When the above indicators exceed the preset thresholds, it means that an unknown anomaly or equipment failure may be occurring in the current production scenario. At this time, control commands are immediately generated, specifically including issuing audible and visual alarms to the site, issuing voice prompts, or, in emergency situations, directly issuing equipment shutdown control commands via the Programmable Logic Controller (PLC) bus to ensure production safety. Simultaneously, an incremental update mechanism is initiated. That is, based on the historical task belief evolution trajectory and energy response records, the active inference parameters are fine-tuned and updated online. The active inference parameters are the internal model parameters supporting the system's free energy minimization decision, including not only the weight coefficients of the free energy function (i.e., the aforementioned α and β), but also the trainable parameters in the model.

[0087] By leveraging newly captured anomalous data, such as historical mission belief evolution trajectories and energy response records, the aforementioned active inference parameters are fine-tuned online in small steps for incremental updates. This allows the model to learn new anomalous patterns or adapt to changes in the production environment without interrupting operation.

[0088] In an exemplary embodiment of the present invention, the incremental update of the active inference parameters includes: At the beginning of each free energy iteration cycle, based on the adjustment gradient direction calculated in the previous iteration, the angle offset adjustment is performed on the viewpoint parameters and focal length parameters of the camera array, and the sampling rate and trigger threshold of the sound and vibration acquisition nodes are adjusted. In key areas, invoke a local enhanced sampling process, increase the temporal sampling frequency, and shorten the sliding window length; New multimodal observation data is obtained based on the adjusted camera array, sound and vibration acquisition nodes, and an observation matrix is ​​formed based on the new multimodal observation data; Based on the observation matrix, the updated multimodal observation error and the updated joint bias index are calculated and input into the free energy function to trigger the next free energy minimization iteration. After multiple iterations of minimizing free energy, when the rate of change of free energy is lower than the set convergence threshold, the current monitoring strategy parameters are determined, and the monitoring configuration corresponding to the state is recorded as a stable observation benchmark.

[0089] In this embodiment of the invention, the active inference process is divided into multiple consecutive iterative cycles. At the start of each iteration, the pan-tilt control unit of the camera array adjusts the horizontal angle, pitch angle, and lens focal length according to the received adjustment gradient direction, so that the imaging center of the camera is physically closer to the high uncertainty region pointed to by the gradient. Simultaneously, the magnitude of the free energy gradient also serves as a basis for resource allocation; the larger the gradient magnitude, the higher the uncertainty in that region. Simultaneously, based on the magnitude of the gradient magnitude, the sampling rate of sensors such as sound and vibration sensors in that region is increased proportionally, or their trigger thresholds for acquisition are decreased, in order to capture weaker anomalous signals.

[0090] Within key regions, a local augmentation sampling process is initiated, increasing the temporal sampling frequency and shortening the sliding window length. These key regions represent the areas with the highest information gain. Local augmentation sampling is then activated within these regions to acquire higher-resolution multimodal observation data by increasing sampling density and shortening the analysis window. Subsequently, this newly acquired high-resolution multimodal observation data is concatenated with the original multimodal observation data on the time axis to update the observation matrix under a unified timestamp.

[0091] Based on this updated observation matrix, the joint bias index and multimodal observation error are recalculated. These updated values ​​are then substituted into the free energy function to calculate new free energy values ​​and new adjustment gradient directions, triggering the next round of iterative optimization. This iterative process continues until the free energy value no longer decreases significantly, i.e., the rate of change of free energy is below the set convergence threshold. This indicates that through proactive adjustment, the uncertainty of the current scenario has been sufficiently reduced, reaching a relatively stable cognitive state. At this point, the current monitoring strategy parameters are frozen and recorded as a stable observation benchmark for the current operating condition for subsequent routine monitoring.

[0092] In an exemplary embodiment of the present invention, acquiring multimodal observation data under a unified timestamp includes: Collect multimodal information from production operation scenarios and establish sampling time indexes for modal information from each data source; Based on the sampling time index, and with a preset unified sampling period as a benchmark, the modal information of each data source is subjected to time interpolation and linear resampling to generate a multimodal time series. In the multimodal time series, data points whose rate of change exceeds a set change threshold or whose sampling is lost are marked; For the labeled data points, anomaly removal and missing value compensation are performed by using nearest neighbor window reconstruction and weighted mean replacement. The multimodal time series, after anomaly removal and missing value compensation, are spliced ​​at the frame level according to a unified timestamp to generate the multimodal observation data.

[0093] In this embodiment of the invention, raw data streams are acquired from various sensors in the production operation scene, including images, sounds, temperature, current, voltage, and vibration. Each independent data source is assigned its own timestamp sequence, i.e., a sampling time index. Since the rhythms of the various data sources are not matched, a preset unified sampling period is used as the alignment benchmark. For modes with sampling rates lower than the benchmark frequency, time interpolation is used to generate smooth compensation data in the time dimension; for modes with extremely high sampling frequencies, linear resampling is used to compress the data, thereby ensuring that the data length and time rhythm of all sensors are completely consistent, generating an aligned multimodal time series.

[0094] In aligned multimodal time series data, momentary spikes or gaps may appear due to electromagnetic interference, network packet loss, or sensor jitter. A sliding window is used to calculate the instantaneous mean and variance, and the rate of change is monitored accordingly. If a data point's jump exceeds a set threshold determined based on the standard deviation of historical data, or if it is a null value, it is marked as an outlier.

[0095] For the marked outlier data points, a nearest-neighbor window reconstruction and weighted mean replacement are employed. Specifically, the observations within two healthy sliding windows before and after the outlier data point are extracted, and their weighted mean or interpolation is calculated to reconstruct a reasonable value to replace the outlier data point or fill in the missing value. All modal data columns, after interpolation alignment, filtering and denoising, and reconstruction compensation, are then frame-level stitched together along the feature dimension according to strictly aligned unified timestamps. For example, at time x, the image features, sound features, and temperature values ​​at that instant are assembled into a complete high-dimensional matrix, thereby generating the final multimodal observation data.

[0096] In this embodiment of the invention, through strict time alignment, dynamic anomaly detection, and precise data repair, a highly pure, well-organized, and lossless data foundation is provided for the upper-level model of the entire intelligent monitoring system, so as to facilitate subsequent task belief field construction, energy prediction, and active reasoning.

[0097] In this embodiment of the invention, in order to verify the feasibility of the invention in practice, the invention is applied to the assembly operation unit of a smart manufacturing workshop. This embodiment aims to demonstrate the application process and data processing method of the invention in a real-world scenario. The parameters, equipment models, and data volumes in the embodiment are merely examples and do not constitute a limitation on the scope of protection of the invention.

[0098] This embodiment describes a structural component assembly station where the work process includes four typical task states: part picking, positioning, fastening, and inspection. The station is equipped with three types of data acquisition devices: two industrial cameras, three sets of combined sound and vibration probes, and four sets of current, voltage, and temperature sensing units. The sampling frequencies are 20 FPS (Frames Per Second) for images, 48 ​​kHz for sound, 6 kHz for vibration, 1 kHz for electrical parameters, and 50 Hz for temperature. To verify the monitoring effect of the method of this invention, data was collected and analyzed over a continuous 30-minute work process.

[0099] Multimodal information was synchronized in time, with all modal information using a unified 50ms timestamp. A multimodal observation matrix containing 36,000 frames was generated through linear interpolation and missing data compensation. Keypoint pose and object interaction regions were extracted from image data. The spectral envelope and impact amplitude of sound and vibration signals were obtained through short-time analysis. The rate of change characteristics of current, voltage, and temperature signals were obtained through moving average and first-order difference.

[0100] The aforementioned features are input into a multimodal alignment encoder. Task feature vectors are generated based on task state definitions such as part grasping, positioning, fastening, and detection. State projection and constraint calibration are then performed according to the task state ontology to form a confidence matrix. A confidence sequence is generated using a sequential mask temporal attention structure, and smoothing and conflict redistribution are performed using a sliding window boundary to form a task belief field.

[0101] The confidence distribution of continuous time frames is shown in Table 1 below: Table 1

[0102] As shown in Table 1, as time progresses, the task status gradually transitions from material handling to positioning. The corresponding confidence curve changes smoothly without any inter-task conflicts, indicating that the sequence mask and constraint calibration mechanism of this invention correctly reflects the process flow logic. The confidence levels of the fastening and detection states remain low in the early stages, indicating that the time consistency constraints of this invention effectively prevent premature state transitions.

[0103] In the energy prediction stage, a set of state variables consisting of four types of physical quantities—current, voltage, vibration, and temperature—is constructed. After normalization, these variables are input into a nonlinear dynamic system model, and the energy evolution function is learned using a sliding window approach. Subsequently, a linearly predictable high-dimensional feature space is constructed using Koopman eigenmaps, and the energy change trajectory for the next 300ms is predicted using a linear state transition operator.

[0104] Record the rate of energy change, generate an energy state vector, and map it to the corresponding moment of the task belief field to establish an energy belief matching matrix.

[0105] The rate of energy change and the task state gradient are shown in Table 2 below: Table 2

[0106] As shown in Table 2, with gradual tightening during the positioning process, the vibration rate of change and the current rate of change increase synchronously, and show a significant positive correlation with the task state gradient. This indicates a direct coupling relationship between energy-level changes and task-level semantic changes. This invention calculates the dominant coupling delay using a time-delay correlation function and constructs forward and reverse mapping sub-models to achieve cross-modal logical association.

[0107] The results presented in Table 2 demonstrate that the energy belief coupler can accurately capture the correspondence between energy behavior and task behavior during process execution, enabling the joint deviation index to be used to reflect potential abnormal trends in the production process, such as insufficient fastening pressure, abnormal increase in vibration, and power deviation.

[0108] In the free energy minimization stage, the second norm of the joint deviation index and the negative logarithmic probability density of the multimodal observation error are combined to form the free energy function, and the adjustment gradient direction is calculated to guide the monitoring strategy update. When the adjustment gradient direction points to the end region of the robotic arm, the camera's viewing angle and focal length are automatically adjusted; when the energy fluctuation amplitude reflects an increase in risk in a certain direction, the sampling rate of the corresponding sensor is increased or the trigger threshold is decreased.

[0109] After multiple iterations, once the free energy drops to a stable range, the current monitoring strategy will be frozen as the new baseline configuration.

[0110] The intelligent monitoring device for production operation scenarios provided by this invention will be described below. The intelligent monitoring device for production operation scenarios described below and the intelligent monitoring method for production operation scenarios described above can be referred to in correspondence with each other. It should be noted that the device provided in the embodiments below and the method provided in the embodiments above belong to the same concept, and the specific way in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0111] In one exemplary embodiment of the present invention, please refer to Figure 2 , Figure 2 An intelligent monitoring device for production operation scenarios, as illustrated in an exemplary embodiment, includes the following modules: The acquisition module 210 is configured to acquire multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario; The first calculation module 220 is configured to calculate the confidence distribution of each task state based on the preset production task process and process specifications, and generate a task belief field by using the multimodal observation data. The generation module 230 is configured to establish a nonlinear dynamic system model based on the physical quantities in the multimodal observation data, upgrade the nonlinear dynamic system model to a linear space, predict energy changes in the linear space, and generate an energy state vector. Module 240 is configured to establish a bidirectional mapping relationship between energy change rate and task confidence based on the energy state vector and the task belief field, construct an energy belief coupler, and obtain a joint bias index based on the energy belief coupler; The second calculation module 250 is configured to construct a free energy function based on the joint deviation index and calculate the adjustment gradient direction under the free energy minimization criterion. The monitoring module 260 is configured to dynamically adjust the monitoring strategy according to the adjustment gradient direction, and perform monitoring in the production operation scenario based on the monitoring strategy.

[0112] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions from the memory 330 to execute an intelligent monitoring method for production operation scenarios. This method includes: acquiring multimodal observation data under a unified timestamp; wherein the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario. Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated using the multimodal observation data to generate a task belief field. A nonlinear dynamic system model is established based on the physical quantities in the multimodal observation data. The nonlinear dynamic system model is then upgraded to a linear space. Energy changes are predicted within the linear space, and an energy state vector is generated. Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler. Based on the joint deviation index, a free energy function is constructed, and the adjustment gradient direction is calculated under the free energy minimization criterion. The monitoring strategy is dynamically adjusted according to the gradient direction, and monitoring is performed in the production operation scenario based on the monitoring strategy.

[0113] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent monitoring method for production operation scenarios provided by the above methods. The method includes: acquiring multimodal observation data under a unified timestamp; wherein the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario. Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated using the multimodal observation data to generate a task belief field. A nonlinear dynamic system model is established based on the physical quantities in the multimodal observation data. The nonlinear dynamic system model is then upgraded to a linear space. Energy changes are predicted within the linear space, and an energy state vector is generated. Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler. Based on the joint deviation index, a free energy function is constructed, and the adjustment gradient direction is calculated under the free energy minimization criterion. The monitoring strategy is dynamically adjusted according to the gradient direction, and monitoring is performed in the production operation scenario based on the monitoring strategy.

[0115] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the intelligent monitoring method for production operation scenarios provided by the above methods. The method includes: acquiring multimodal observation data under a unified timestamp; wherein the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario. Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated using the multimodal observation data to generate a task belief field. A nonlinear dynamic system model is established based on the physical quantities in the multimodal observation data. The nonlinear dynamic system model is then upgraded to a linear space. Energy changes are predicted within the linear space, and an energy state vector is generated. Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler. Based on the joint deviation index, a free energy function is constructed, and the adjustment gradient direction is calculated under the free energy minimization criterion. The monitoring strategy is dynamically adjusted according to the gradient direction, and monitoring is performed in the production operation scenario based on the monitoring strategy.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the intelligent monitoring method for production operation scenarios described in various embodiments or some parts of embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent monitoring of production operation scenarios, characterized in that, include: Acquire multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario; Based on the preset production task process and process specifications, the confidence distribution of each task state is calculated using the multimodal observation data to generate a task belief field. A nonlinear dynamic system model is established based on the physical quantities in the multimodal observation data. The nonlinear dynamic system model is then upgraded to a linear space. Energy changes are predicted within the linear space, and an energy state vector is generated. Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler. Based on the joint deviation index, a free energy function is constructed, and the adjustment gradient direction is calculated under the free energy minimization criterion. The monitoring strategy is dynamically adjusted according to the gradient direction, and monitoring is performed in the production operation scenario based on the monitoring strategy.

2. The intelligent monitoring method for production operation scenarios according to claim 1, characterized in that, Based on the energy state vector and the task belief field, a bidirectional mapping relationship between the energy change rate and the task confidence is established, an energy belief coupler is constructed, and a joint bias index is obtained based on the energy belief coupler, including: Each energy component in the energy state vector is mapped to the corresponding confidence node in the task belief field according to the work unit number and timestamp, forming an energy belief matching matrix; The time-delay correlation function is determined based on the energy belief matching matrix, and the dominant coupling delay is determined based on the time-delay correlation function. Under the constraint of the dominant coupling delay, an energy belief coupler is defined, comprising a forward mapping sub-model and a backward mapping sub-model; wherein the forward mapping sub-model takes the energy change rate as input and outputs the confidence update, and the backward mapping sub-model takes the confidence gradient as input and outputs the energy correction. The energy offset and belief offset are calculated based on the outputs of the forward mapping sub-model and the reverse mapping sub-model, and the joint bias index is calculated based on the energy offset and belief offset.

3. The intelligent monitoring method for production operation scenarios according to claim 2, characterized in that, Under the constraint of the dominant coupling delay, an energy belief coupler comprising a forward mapping sub-model and a backward mapping sub-model is defined, including: A parameter sharing layer and a residual channel are set between the forward mapping sub-model and the reverse mapping sub-model; wherein, the parameter sharing layer and the residual channel are used to synchronously update the common weights of the forward mapping sub-model and the reverse mapping sub-model and transmit cross-modal deviation information; During the training phase, the parameter-sharing layer is iteratively updated using a weighted minimization objective function until the convergence condition is met, resulting in the trained energy belief coupler.

4. The intelligent monitoring method for production operation scenarios according to claim 1, characterized in that, The process, based on a preset production task flow and technological specifications, calculates the confidence distribution of each task state using the multimodal observation data to generate a task belief field, including: Based on the preset production task flow and process specifications, a task status ontology is established, and a task tag set is generated; Modal features are extracted from the multimodal observation data, and the modal features and the task label set are input into the multimodal alignment encoder to generate the task feature vector of the job unit; Based on the task feature vector and the task state ontology, the confidence level of the task state is calculated, and a confidence matrix is ​​generated based on the confidence level. A temporal attention structure with process sequence masking is used to perform weighted fusion on the confidence matrix to generate a task confidence sequence; The task confidence sequence is stored as a confidence distribution according to time index and job unit number; Differentiate and normalize the confidence scores of adjacent time frames to obtain a gradient sequence representing the rate of change of the task state; The confidence distribution and the gradient sequence are jointly mapped to a unified spatiotemporal coordinate system to form the task belief field arranged by job unit index and timestamp.

5. The intelligent monitoring method for production operation scenarios according to claim 1, characterized in that, The process of establishing a nonlinear dynamic system model based on physical quantities in the multimodal observation data, upgrading the nonlinear dynamic system model to a linear space, predicting energy changes within the linear space, and generating an energy state vector includes: A set of state variables is established based on the physical quantities in the multimodal observation data, and a multidimensional state sequence is constructed according to the time index. A system evolution function is defined on the multidimensional state sequence, and the local changes of the system evolution function are fitted by a family of nonlinear functions to obtain the nonlinear dynamic system model. The nonlinear evolution function is mapped to a learnable Koopman feature space to extract the hidden state vector that can linearly describe the system dynamics; Within the Koopman feature space, based on the hidden state vector, the hidden state of subsequent time steps is predicted using a linear state transition operator; During the prediction process, the energy change rate of each state component is recorded, and the energy change rate is used to form the energy state vector.

6. The intelligent monitoring method for production operation scenarios according to claim 1, characterized in that, The construction of the free energy function based on the joint deviation index, and the calculation of the adjusted gradient direction under the free energy minimization criterion, includes: Based on the joint bias index and multimodal observation error, a free energy input vector is formed under a unified time index; The second norm of the joint bias index and the probability density of the multimodal observation error are calculated in the free energy input vector, respectively. The second norm and the probability density are then weighted and summed to generate the free energy function. The partial derivative of the free energy function is calculated to form a gradient direction vector, and the adjustment gradient direction of the current monitoring strategy is determined based on the gradient direction vector.

7. The intelligent monitoring method for production operation scenarios according to claim 1, characterized in that, The method further includes: dynamically adjusting the monitoring strategy based on the gradient direction; and performing monitoring in the production operation scenario based on the monitoring strategy. If the detected free energy or energy belief deviation is greater than the corresponding preset threshold, a control command is generated and the active inference parameters are incrementally updated.

8. The intelligent monitoring method for production operation scenarios according to claim 7, characterized in that, The incremental update of the active inference parameters includes: At the beginning of each free energy iteration cycle, based on the adjustment gradient direction calculated in the previous iteration, the angle offset adjustment is performed on the viewpoint parameters and focal length parameters of the camera array, and the sampling rate and trigger threshold of the sound and vibration acquisition nodes are adjusted. In key areas, invoke a local enhanced sampling process, increase the temporal sampling frequency, and shorten the sliding window length; New multimodal observation data is obtained based on the adjusted camera array, sound and vibration acquisition nodes, and an observation matrix is ​​formed based on the new multimodal observation data; Based on the observation matrix, the updated multimodal observation error and the updated joint bias index are calculated and input into the free energy function to trigger the next free energy minimization iteration. After multiple iterations of minimizing free energy, when the rate of change of free energy is lower than the set convergence threshold, the current monitoring strategy parameters are determined, and the monitoring configuration corresponding to the state is recorded as a stable observation benchmark.

9. The intelligent monitoring method for production operation scenarios according to any one of claims 1 to 8, characterized in that, The acquisition of multimodal observation data under a unified timestamp includes: Collect multimodal information from production operation scenarios and establish sampling time indexes for modal information from each data source; Based on the sampling time index, and with a preset unified sampling period as a benchmark, the modal information of each data source is subjected to time interpolation and linear resampling to generate a multimodal time series. In the multimodal time series, data points whose rate of change exceeds a set change threshold or whose sampling is lost are marked; For the labeled data points, anomaly removal and missing value compensation are performed by using nearest neighbor window reconstruction and weighted mean replacement. The multimodal time series, after anomaly removal and missing value compensation, are spliced ​​at the frame level according to a unified timestamp to generate the multimodal observation data.

10. An intelligent monitoring device for production operation scenarios, characterized in that, include: The acquisition module is configured to acquire multimodal observation data under a unified timestamp; wherein, the multimodal observation data is generated after performing time synchronization and anomaly filtering processing on multimodal information collected in the production operation scenario; The first calculation module is configured to calculate the confidence distribution of each task state based on the preset production task process and process specifications, and generate a task belief field by using the multimodal observation data. The generation module is configured to establish a nonlinear dynamic system model based on the physical quantities in the multimodal observation data, upgrade the nonlinear dynamic system model to a linear space, predict energy changes in the linear space, and generate an energy state vector. The module is configured to establish a bidirectional mapping relationship between the energy change rate and the task confidence based on the energy state vector and the task belief field, construct an energy belief coupler, and obtain a joint bias index based on the energy belief coupler. The second calculation module is configured to construct a free energy function based on the joint deviation index, and calculate the adjustment gradient direction under the free energy minimization criterion. The monitoring module is configured to dynamically adjust the monitoring strategy based on the adjustment gradient direction, and perform monitoring in the production operation scenario based on the monitoring strategy.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent monitoring method for production operation scenarios as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent monitoring method for production operation scenarios as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent monitoring method for production operation scenarios as described in any one of claims 1 to 9.

Images