Preparation method of a multi-modal visual recognition-based intelligent verification method and system for a pre-prepared food production site

Abstract

The present application relates to the technical field of pre-prepared food production site intelligent verification, and more particularly to a pre-prepared food production site intelligent verification method based on multi-modal visual recognition, which establishes cross-modal feature correspondence through unsupervised clustering, calculates causal inevitability scores in combination with causal process maps, and identifies abnormal areas using physical constraint spaces, performs double-path abnormality determination of causal paths and physical paths, cross- validates abnormalities to generate verification reports with causal traceability and physical violation evidence, and improves the accuracy and interpretability of pre-prepared food production site anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent verification technology for pre-prepared food production sites, and in particular to an intelligent verification method and system for pre-prepared food production sites based on multimodal visual recognition. Background Technology

[0002] In the field of intelligent verification at pre-prepared food production sites, existing technologies mainly rely on single-modal visual monitoring or manual inspection to detect production processes. A common approach is to deploy high-definition cameras to capture images or video streams of the production line, and then identify apparent features such as ingredient status and equipment operation through pre-defined template matching or deep learning-based image classification models. Simultaneously, some systems integrate environmental sensors such as temperature and humidity, performing simple threshold comparisons between sensor data and visual information to determine if production conditions deviate from standard ranges. These methods typically design independent anomaly detection algorithms for a single observation modality (such as vision alone or sensor alone), outputting detection results as alarm notifications, lacking deep fusion modeling of multi-source heterogeneous data.

[0003] Conventional practices often lack effective spatial correspondence and semantic consistency verification between multimodal data. Sensors of different modalities (such as RGB cameras, infrared thermal imagers, and laser sensors) have heterogeneous data structures and spatial resolutions. Existing technologies often directly stitch together the original features of different modalities or simply superimpose detection results, ignoring the semantic alignment between modalities in their intrinsic space. When there are obstructions, changes in lighting, or equipment vibrations in the production environment, single-modal detection results are prone to false alarms or missed detections, while the weak cross-modal collaborative judgment capability makes it difficult to guarantee the accuracy of verification.

[0004] Conventional anomaly detection logic is often divorced from the causal chain and physical mechanisms of the production process. Existing methods typically rely on statistical thresholds or classifiers to determine anomalies in observed data; for example, a visual model might flag anomalies as occurring when it detects abnormal food color or when sensor readings exceed a preset range. This approach fails to distinguish between incidental observational noise and genuine process deviations, and it cannot trace prior operations or non-compliant behaviors that led to the anomaly. More importantly, the detection process neglects the constraints of physical laws on the production state (such as material conservation and thermodynamic equilibrium), resulting in the failure to identify some anomalies that violate physical laws (such as inflating raw material weight or falsifying temperature records), easily creating loopholes in the verification process. Summary of the Invention

[0005] This invention provides a method and system for intelligent verification of pre-prepared food production sites based on multimodal visual recognition, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides an intelligent verification method for pre-prepared food production sites based on multimodal visual recognition, comprising:

[0007] For the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse correspondences across modalities, resulting in anchor-aligned multimodal features.

[0008] A causal process graph for pre-prepared food production is constructed, and the multimodal features aligned with the anchor points are mapped to the node state space of the causal process graph. The causal necessity score of each operation event in the current production state is calculated using a causal inference algorithm.

[0009] A physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical laws is calculated, and abnormal modal response regions that violate physical consistency are identified.

[0010] The system performs dual-path anomaly determination, which includes causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region.

[0011] Cross-validate the process logic anomalies and the physical state anomalies, and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

[0012] For the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse correspondences across modalities, resulting in anchor-aligned multimodal features, including:

[0013] Each modal visual data in the multimodal visual data stream is divided into spatial grids, and each modal visual data is divided into multiple spatial grid units.

[0014] By constructing a cross-modal response consistency measurement mechanism, the response consistency scores between the spatial grid cells corresponding to different modes are calculated, and the spatial grid cells with response consistency scores higher than a preset consistency threshold are marked as candidate anchor point cells.

[0015] Spatial neighborhood aggregation is performed on the candidate anchor units. By calculating the response consistency propagation strength between the candidate anchor unit and other candidate anchor units in the spatial neighborhood, the candidate anchor units whose response consistency propagation strength satisfies the spatial continuity constraint are selected as the semantic anchors.

[0016] A cross-modal correspondence graph is established using the semantic anchors as nodes and the response similarity between semantic anchors of different modalities as edge weights.

[0017] Based on the cross-modal correspondence graph, feature alignment transformation is performed on the multimodal visual data stream. By using the spatial position of the semantic anchor point as a constraint, affine transformation correction is performed on the visual features of different modalities to obtain the anchor point aligned multimodal features.

[0018] Constructing a causal process diagram for pre-prepared food production includes:

[0019] Production operation event sequences are extracted from historical production data of pre-prepared dishes and decomposed into multiple operation event segments. An operation event co-occurrence matrix is ​​constructed for the multiple operation event segments. By performing a conditional independence test on the operation event co-occurrence matrix, direct causal dependencies and indirect causal dependencies between operation events are identified.

[0020] Based on the direct and indirect causal dependencies, an operational event causal dependency graph is constructed. Counterfactual inference is performed on each causal dependency path in the operational event causal dependency graph. By simulating the removal of predecessor operational events from the causal dependency path and calculating the change in the probability of subsequent operational events, causal dependency paths whose change exceeds a preset causal significance threshold are marked as valid causal edges. The valid causal edges are used as directed edges, and the causal process graph is constructed with the production operational events as nodes.

[0021] The multimodal features aligned with the anchor points are mapped to the node state space of the causal process graph. The causal inevitability score of each operational event in the current production state is calculated using a causal inference algorithm, including:

[0022] Each production operation event node in the causal process graph is associated with a predefined operation event semantic space, which includes the action type attribute, operation object attribute, and timing attribute of the operation event.

[0023] The anchor-aligned multimodal features are transformed by feature dimension mapping. By projecting the anchor-aligned multimodal features onto the operation event semantic space, a multimodal semantic feature vector is obtained. The similarity between the multimodal semantic feature vector and the production operation event node in terms of action type attribute, operation object attribute, and temporal sequence attribute is calculated, and a weighted fusion is performed to obtain the semantic matching degree. The multimodal semantic feature vector is assigned to the production operation event node with the highest semantic matching degree to form a node state representation.

[0024] Causal path reasoning is performed on the causal process graph. By propagating the node state representation along the directed edge and accumulating the node activation intensity on the causal dependency path, the causal path reachability score of each production operation event node is calculated. The causal path reachability score is compared with the prior probability of the expected operation event to obtain the causal inevitability score.

[0025] A physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical laws is calculated. Abnormal modal response regions that violate physical consistency are identified, including:

[0026] Physical constraint prior knowledge is extracted from the pre-cooked food production process specifications. Material input-output balance relationship is extracted based on material mass conservation constraint. Temperature gradient change law is extracted based on thermodynamic balance constraint. Velocity and acceleration boundary conditions of motion trajectory are extracted based on mechanical kinematic constraint. A set of physical constraint relationships is constructed based on the balance relationship, the change law and the boundary conditions.

[0027] The set of physical constraint relationships is converted into a constraint hyperplane of a multidimensional physical constraint space. The constraint hyperplane defines a feasible solution region that conforms to physical laws. The multimodal features of the anchor point alignment are calibrated with physical dimensions. The visual response values ​​are mapped to physical measurement values ​​and projected onto the multidimensional physical constraint space to obtain the distribution of observation points.

[0028] For each observation point in the observation point distribution, find the constraint-satisfied point on the constraint hyperplane that is closest to the observation point, calculate the deviation vector between the physical quantity measurement value of the observation point and the physical law prediction value of the constraint-satisfied point, and perform statistical analysis on the multimodal dimension components of the deviation vector of all observation points in the observation point distribution to obtain the residual distribution.

[0029] Anomaly detection is performed on the residual distribution to identify residual samples that deviate from the center of the normal distribution by more than the physical consistency threshold. The corresponding positions of the residual samples in the multimodal feature space aligned with the anchor point are spatially clustered and merged to obtain the abnormal modal response region.

[0030] Perform dual-path anomaly determination, including causal path and physical path anomalies. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold. The physical path identifies physical state anomalies based on the abnormal modal response region, including:

[0031] A threshold determination is made for the causal necessity score of each production operation event node in the causal process graph along the causal path. Nodes with causal necessity scores lower than the causal expectation threshold are marked as causal abnormal nodes. The upstream causal dependency path of the causal abnormal node in the causal process graph is extracted. By tracing back the changes in the causal necessity scores of each predecessor node on the upstream causal dependency path, the root cause node that leads to the decrease in causal necessity score is identified. The production operation event corresponding to the root cause node is marked as the process logic abnormality.

[0032] Physical constraint type attribution analysis is performed on the abnormal modal response region along the physical path. By calculating the constraint violation intensity of each observation point in the abnormal modal response region in the dimensions of material mass conservation constraint, thermodynamic equilibrium constraint, and mechanical kinematic constraint, the dominant constraint violation type of the abnormal modal response region is determined. Based on the dominant constraint violation type, a predefined physical state anomaly category library is associated with it, and the abnormal modal response region is mapped to the corresponding physical state anomaly category.

[0033] Cross-validation of the process logic anomalies and the physical state anomalies, and generation of a pre-prepared food production site inspection report with causal tracing paths and physical evidence of violations, includes:

[0034] Calculate the time delay difference and spatial distance difference between the root cause node corresponding to the process logic anomaly and the abnormal modal response region. When the time delay difference is within the causal propagation time window and the spatial distance difference is within the causal influence range, establish a cross-validation association between the root cause node and the abnormal modal response region.

[0035] Based on the cross-validation association, the consistency of the process logic anomaly and the physical state anomaly is verified. For anomaly pairs with cross-validation association, the complete causal propagation path from the root cause node to the downstream influencing node is extracted from the causal process graph as the causal tracing path. The deviation vector between the physical measurement value of the observation point corresponding to the abnormal modal response region and the constraint satisfaction point, as well as the dominant constraint violation type, are extracted from the physical constraint space as the physical violation evidence.

[0036] Record the anomaly type identifier and spatiotemporal information of the abnormal event, record the production operation event name and causal necessity score change of each node on the causal tracing path, record the spatial location label of the abnormal modal response area, the numerical distribution of the deviation vector and the constraint violation intensity of the dominant constraint violation type, and generate the pre-prepared vegetable production site verification report in combination with the physical violation evidence.

[0037] A second aspect of this invention provides an intelligent verification system for pre-prepared food production sites based on multimodal visual recognition, comprising:

[0038] Anchor point alignment unit is used to identify spatial regions with consistent responses in different modalities as semantic anchor points through unsupervised clustering of multimodal visual data streams from pre-prepared food production sites. The semantic anchor points are then used to establish sparse correspondences across modalities to obtain anchor point aligned multimodal features.

[0039] The causal inference unit is used to construct a causal process graph of pre-prepared food production, map the multimodal features aligned with the anchor points to the node state space of the causal process graph, and calculate the causal necessity score of each operation event in the current production state through a causal inference algorithm.

[0040] The physical constraint unit is used to introduce prior knowledge of physical constraints in the production site to construct a physical constraint space. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical law is calculated, and abnormal modal response regions that violate physical consistency are identified.

[0041] The dual-path determination unit is used to perform dual-path anomaly determination including causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region.

[0042] The verification report unit is used to cross-verify the process logic anomalies and the physical state anomalies and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

[0043] A third aspect of the present invention provides an electronic device, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0048] The beneficial effects of this application are as follows:

[0049] This method employs adaptive semantic anchor extraction based on multimodal visual data streams to automatically establish sparse cross-modal correspondences under unsupervised conditions, significantly reducing the computational complexity of modal alignment in multi-source sensor data fusion. The anchor alignment strategy bypasses the high computational cost of traditional pixel-by-pixel registration while preserving the intermodal collaborative response characteristics of key semantic regions, improving the efficiency of intermodal information complementarity by an order of magnitude. This approach is inherently robust to on-site interference factors such as illumination changes and equipment vibration, and can continuously optimize multimodal feature consistency without relying on labeled data.

[0050] A dual-dimensional embedding mechanism of causal process mapping and physical constraint space constructs a new paradigm for determining the state of operational events on the production floor. Causal necessity scores quantify the deviation of the current operation from the standard process from the perspective of the process logic chain, while physical residual distribution identifies subtle, imperceptible anomalies from the perspective of natural law constraints. Dual-path parallel computation avoids false positives caused by a single dimension and enhances the interpretability of anomaly determination through the complementary relationship between causal inference and physical verification, simultaneously improving the accuracy and recall of on-site verification.

[0051] The cross-validation process bidirectionally verifies the anomaly detection results of causal and physical paths, effectively suppressing false alarms and missed detections from single paths. The final verification report fully presents the causal tracing path and physical evidence of violations, allowing operators to directly locate the process node of the anomaly and the specific physical law violated. The entire process achieves anomaly determination through feature-level correlation rather than decision-level voting, significantly reducing the delay from data collection to report output and meeting the stringent timeliness requirements for real-time verification on pre-prepared food production lines. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating an intelligent verification method for pre-prepared food production sites based on multimodal visual recognition.

[0053] Figure 2 A flowchart illustrating the process for generating a pre-prepared food production site inspection report. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0056] Figure 1 This is a flowchart illustrating an embodiment of the intelligent verification method for pre-prepared food production sites based on multimodal visual recognition according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0057] For the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse correspondences across modalities, resulting in anchor-aligned multimodal features.

[0058] A causal process graph for pre-prepared food production is constructed, and the multimodal features aligned with the anchor points are mapped to the node state space of the causal process graph. The causal necessity score of each operation event in the current production state is calculated using a causal inference algorithm.

[0059] A physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical laws is calculated, and abnormal modal response regions that violate physical consistency are identified.

[0060] The system performs dual-path anomaly determination, which includes causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region.

[0061] Cross-validate the process logic anomalies and the physical state anomalies, and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

[0062] In one optional implementation, for the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse cross-modal correspondences, resulting in anchor-aligned multimodal features, including:

[0063] Each modal visual data in the multimodal visual data stream is divided into spatial grids, and each modal visual data is divided into multiple spatial grid units.

[0064] By constructing a cross-modal response consistency measurement mechanism, the response consistency scores between the spatial grid cells corresponding to different modes are calculated, and the spatial grid cells with response consistency scores higher than a preset consistency threshold are marked as candidate anchor point cells.

[0065] Spatial neighborhood aggregation is performed on the candidate anchor units. By calculating the response consistency propagation strength between the candidate anchor unit and other candidate anchor units in the spatial neighborhood, the candidate anchor units whose response consistency propagation strength satisfies the spatial continuity constraint are selected as the semantic anchors.

[0066] A cross-modal correspondence graph is established using the semantic anchors as nodes and the response similarity between semantic anchors of different modalities as edge weights.

[0067] Based on the cross-modal correspondence graph, feature alignment transformation is performed on the multimodal visual data stream. By using the spatial position of the semantic anchor point as a constraint, affine transformation correction is performed on the visual features of different modalities to obtain the anchor point aligned multimodal features.

[0068] After acquiring the multimodal visual data stream from the pre-prepared food production site, spatial meshing is performed on each modal visual data stream, including visible light image streams, near-infrared image streams, and thermal imaging image streams. Specifically, each frame of modal visual image is divided into several spatial mesh units according to a uniform mesh strategy. The granularity of the mesh is dynamically determined based on the image resolution of the production site and the typical size of the target object, ensuring that each mesh unit contains sufficient semantic information without losing local details due to an overly coarse mesh. After partitioning, the visual data of each modality is represented as a structured feature set composed of multiple spatial mesh units, providing a unified spatial indexing basis for subsequent cross-modal consistency measurement.

[0069] For mesh cell pairs from different modalities and spatial locations, their respective local feature descriptors are extracted. The construction of these feature descriptors comprehensively considers multi-dimensional information such as texture gradient, local statistical moments, and frequency domain energy distribution to improve the comparability of cross-modal features. Let the th... The position index in each modality is The eigenvector of the grid cell is , No. The eigenvectors of the corresponding grid cells in each modality are The response consistency score between the two is then calculated. The fusion weights are calculated using a weighted fusion method of normalized mutual information and cosine similarity, with the weights adaptively allocated based on the signal-to-noise ratio of each modality. Higher than the preset consistency threshold At that time, the mesh cell is marked as a candidate anchor point cell. A preset consistency threshold is set. The setting takes into account the range of light variation in the production site, the differences in response characteristics of different modal sensors, and the background noise level. It is usually determined by statistical analysis on a small number of labeled samples, or it can be dynamically estimated using the adaptive percentile method.

[0070] After initial screening, candidate anchor units are further validated for spatial continuity through spatial neighborhood aggregation to eliminate interference from isolated noise points on semantic anchor quality. For each candidate anchor unit, the response consistency propagation strength between it and other candidate anchor units in its spatial neighborhood is calculated. The response consistency propagation strength reflects the degree of clustering and consistency stability of candidate anchor units in their local spatial neighborhood. Let the candidate anchor unit... In its neighborhood set The propagation intensity in ,but ,in Candidate anchor point unit With neighboring units Consistency score of responses between them For indicator functions. Spatial continuity constraints require... The candidate anchor unit must meet the following constraints: the propagation intensity must be at least equal to the set minimum threshold, and the area of ​​the connected region containing the candidate anchor unit must be at least equal to the minimum anchor region threshold. Candidate anchor units that satisfy the above constraints are ultimately confirmed as semantic anchors. Semantic anchors represent spatial regions that have stable and consistent physical meaning in multiple modalities, such as the edge of the workbench on a production line, the outline of a food container, or the surface of a heating device, which are regions with clear physical attributes.

[0071] After obtaining the set of semantic anchors, a cross-modal correspondence graph is constructed using these anchors as nodes. Nodes in the graph correspond to semantic anchors identified in different modalities. Node attributes include the anchor's spatial coordinates, its modality identifier, and a feature descriptor. The edge weights between semantic anchors of different modalities are determined by response similarity. Response similarity comprehensively considers the distance between two anchors in the feature space and their geometric consistency. Geometric consistency is measured by calculating the relative positional deviation of the anchor pair in each modal coordinate system. Edges with higher weights indicate a high semantic correspondence between two cross-modal anchors and can serve as reliable cross-modal correspondence constraints. The sparsity of the cross-modal correspondence graph is controlled by the maximum neighbor count parameter. Each anchor node is connected only to a few cross-modal anchors with the highest response similarity, thus avoiding redundant computational overhead caused by dense connections while ensuring the accuracy of the correspondence.

[0072] Based on the constructed cross-modal correspondence graph, feature alignment transformation is performed on the multimodal visual data stream. Using the spatial location of semantic anchor points as geometric constraints, affine transformation correction is applied to the visual features of different modalities. The affine transformation parameters are estimated by minimizing the positional residuals of each modal anchor point in the common coordinate system, specifically solved using a robust least squares optimization method to suppress the influence of a small number of mismatched anchor points on the estimation of transformation parameters. Let the... The affine transformation matrix for each mode is: Then the coordinates in that mode are The feature points, whose alignment coordinates in the common coordinate system are: Affine transformation not only corrects the geometric deviations between different modal sensors caused by differences in installation position and focal length, but also compensates for the slight motion blur effect caused by imaging time difference to a certain extent. After affine transformation correction, the visual features of different modalities are aligned in a unified spatial coordinate system, and the feature maps of each modality achieve precise spatial correspondence in the semantic anchor point region, thus obtaining anchor point aligned multimodal features.

[0073] Anchor point alignment of multimodal features is of great significance in on-site verification of prepared food production. After alignment, surface texture features from the visible light mode, moisture distribution features from the near-infrared mode, and temperature field features from the thermal imaging mode can be jointly analyzed in the same spatial location, providing high-quality input for subsequent causal process mapping and physically constrained spatial projection. Alignment accuracy directly affects the reliability of cross-modal anomaly detection. Therefore, in actual deployment, the number and distribution density of semantic anchor points need to be adjusted according to the complexity of the production site to ensure sufficient semantic anchor point coverage in key operational areas of the production line (such as cutting, seasoning, and heating areas), thereby ensuring the consistency and accuracy of feature alignment transformation globally.

[0074] In one alternative implementation, constructing a causal process graph for pre-prepared food production includes:

[0075] Production operation event sequences are extracted from historical production data of pre-prepared dishes and decomposed into multiple operation event segments. An operation event co-occurrence matrix is ​​constructed for the multiple operation event segments. By performing a conditional independence test on the operation event co-occurrence matrix, direct causal dependencies and indirect causal dependencies between operation events are identified.

[0076] Based on the direct and indirect causal dependencies, an operational event causal dependency graph is constructed. Counterfactual inference is performed on each causal dependency path in the operational event causal dependency graph. By simulating the removal of predecessor operational events from the causal dependency path and calculating the change in the probability of subsequent operational events, causal dependency paths whose change exceeds a preset causal significance threshold are marked as valid causal edges. The valid causal edges are used as directed edges, and the causal process graph is constructed with the production operational events as nodes.

[0077] The historical production data for pre-prepared meals comes from various data sources, including production line sensor records, operator logs, video frame annotations, and equipment status change records. After temporal alignment of this heterogeneous data, the continuous temporal data is segmented according to operational semantic boundaries to obtain several operational event segments. The core of temporal decomposition lies in identifying the switching points of operational semantics: when the operation category label changes abruptly between consecutive frames, or when the equipment status parameters undergo a step change, that moment is marked as the segment boundary. Each operational event segment corresponds to a production action unit with clear semantics, such as typical operational steps in the pre-prepared meal production process, such as "heating," "stirring," "adding seasonings," and "cooling and settling." Segment boundaries are smoothed using a sliding time window to avoid over-segmentation caused by transient noise.

[0078] Construct an operation event co-occurrence matrix for multiple operation event fragments. Let the set of operation events contain a total of... Class operation events, co-occurrence matrix for A symmetric matrix, where the elements Indicates operation event Operation events The frequency statistics of events occurring sequentially within a preset time window. The width of the time window is set according to the longest operation interval in the pre-prepared food process, ensuring that operation event pairs with actual process correlation can be counted in the same window. When counting co-occurrence frequencies, the order of events is distinguished. In the preceding events Frequency and events following In the preceding events The frequencies were recorded separately to provide basic data for the subsequent identification of directed causal relationships.

[0079] Performing a conditional independence test on the co-occurrence matrix of operational events is a crucial step in distinguishing between direct and indirect causal dependencies. An independence test method based on conditional mutual information is employed for any pair of operational events. and In a given set of condition variables Under the condition of (i.e., the set of other intermediate operation events), calculate conditional mutual information. ,in and Representing operation events respectively and The occurrence state of the random variable. If Below the preset independence determination threshold Then it is considered that in the condition set Next event With the event They are conditionally independent; there is no direct causal dependency between them, but rather a causal relationship. The indirect causal dependency of intermediate events in the process. Conversely, if the conditional mutual information is significantly higher than the threshold... Then determine the event. With the event There is a direct causal dependency between them. By performing the above test on all event pairs, the statistical associations in the co-occurrence matrix can be systematically decomposed into two categories: direct causal dependency and indirect causal dependency.

[0080] Based on the results of the conditional independence test, a causal dependency graph of operation events is constructed. Each node in the graph represents a class of operation events, and direct causal dependencies correspond to directed edges in the graph. The direction of the edges is determined by the temporal order: if event... Prior to the event in terms of time sequence Furthermore, since there is a direct causal dependency between the two, then from the node... To the node Add a directed edge. Indirect causal dependencies are represented by directed paths through intermediate nodes in the graph, without adding separate edges to maintain the sparsity and interpretability of the graph structure. After the graph is constructed, it is checked for directed acyclicity. If a directed cycle (i.e., cyclic dependency) exists, the cycle is broken by removing the edge with the lowest statistical confidence in the cycle, ensuring that the causal dependency graph meets the structural requirements of a directed acyclic graph (DAG). This is a fundamental constraint of causal inference theory on causal graphs.

[0081] Counterfactual inference is performed on each causal dependency path in the causal dependency graph of operational events. The basic idea of ​​counterfactual inference is to simulate the assumption that the predecessor operation event in a certain causal dependency path has not occurred (i.e., it has been removed from the data or set to null), and observe whether the probability of the subsequent operation event occurs significantly. Specifically, for the nodes in the causal dependency graph... To the node A directed edge is used to filter historical production data to find edges containing operation events. A subset of samples, calculate the operation events in that subset. The probability of conditional occurrence Subsequently, intervention was simulated using the do operator to construct a removal operation event. The counterfactual scenario afterwards, calculating the operational events after intervention. probability of occurrence The difference between the two is defined as the causal effect size of the causal edge. .

[0082] When the causal effect size Exceeding the preset causal significance threshold At that time, determine from the node To the node A causal dependency path exhibits significant causal necessity; therefore, the directed edges corresponding to this path are marked as valid causal edges. A preset causal significance threshold is used. The threshold settings reference the key quality control points in the pre-prepared food production process. For causal edges involving critical food safety operations (such as sterilization and cold chain maintenance), stricter (lower) thresholds are adopted to improve the recall rate of effective causal edges. For causal edges between non-critical auxiliary operations, the thresholds can be appropriately relaxed. Through this differentiated threshold strategy, it is ensured that the causal process map can fully cover the key process constraints in the production process.

[0083] By integrating all directed edges marked as valid causal edges, and using production operation events as nodes and valid causal edges as directed edges, a causal process graph for pre-prepared food production is finally constructed. Each node in this graph not only records the category label of the operation event but also stores the typical characteristic distribution of that event in historical data (including duration distribution, associated sensor reading distribution, etc.), serving as a reference benchmark for subsequent node state space mapping. Each valid causal edge is accompanied by a causal effect magnitude. As edge weights, they reflect the strength of the causal dependency, providing a quantitative basis for the subsequent calculation of causal necessity scores. After the graph is constructed, it is topologically sorted to verify that the overall structure conforms to the temporal logic of the pre-prepared food production process, ensuring that the graph can accurately represent the complete causal chain from raw material pretreatment to finished product packaging, and providing structured process knowledge support for subsequent anomaly detection based on causal inference algorithms.

[0084] In one optional implementation, the anchor-aligned multimodal features are mapped to the node state space of the causal process graph, and the causal necessity score of each operational event occurring in the current production state is calculated using a causal inference algorithm, including:

[0085] Each production operation event node in the causal process graph is associated with a predefined operation event semantic space, which includes the action type attribute, operation object attribute, and timing attribute of the operation event.

[0086] The anchor-aligned multimodal features are transformed by feature dimension mapping. By projecting the anchor-aligned multimodal features onto the operation event semantic space, a multimodal semantic feature vector is obtained. The similarity between the multimodal semantic feature vector and the production operation event node in terms of action type attribute, operation object attribute, and temporal sequence attribute is calculated, and a weighted fusion is performed to obtain the semantic matching degree. The multimodal semantic feature vector is assigned to the production operation event node with the highest semantic matching degree to form a node state representation.

[0087] Causal path reasoning is performed on the causal process graph. By propagating the node state representation along the directed edge and accumulating the node activation intensity on the causal dependency path, the causal path reachability score of each production operation event node is calculated. The causal path reachability score is compared with the prior probability of the expected operation event to obtain the causal inevitability score.

[0088] Associating each production operation event node in the causal process graph with a predefined operation event semantic space is a crucial prerequisite for effectively connecting multimodal features with process knowledge. The operation event semantic space consists of three attribute axes: action type attribute describes the behavior category of the current operation, such as common process actions in pre-prepared food production like cutting, stirring, heating, marinating, and plating; operation object attribute describes the food or equipment object affected by the operation, such as meat raw materials, vegetable ingredients, seasonings, or heating equipment; and temporal sequence attribute describes the relative position of the operation event in the overall process flow, i.e., its predecessor and successor constraints in the process sequence. Each production operation event node is represented in the semantic space as a structured attribute vector. The three types of attributes are expressed in the form of one-hot encoding or continuous embedding vectors, and then concatenated to form a complete node semantic description vector. This structured semantic space design ensures that the alignment between subsequent multimodal features and node states has clear semantic interpretability, rather than relying solely on statistical correlation.

[0089] After obtaining the anchor-aligned multimodal features, they need to be projected onto the aforementioned operational event semantic space to obtain multimodal semantic feature vectors. Specifically, a feature dimension mapping transformation is performed on the anchor-aligned multimodal features. A learnable linear projection matrix is ​​used to compress or expand the high-dimensional features of each modality to a subspace with the same dimension as the operational event semantic space, so that features from different modalities such as visual images, infrared thermal images, and depth maps can be compared in the same semantic coordinate system. After projection, semantic feature vectors corresponding to each modality are obtained. These vectors are then weighted and summed according to modality weights to form the fused multimodal semantic feature vector. .

[0090] Calculate multimodal semantic feature vectors Extract the attribute similarity of each production operation event node across three categories. The projection components are plotted on the action type attribute subspace, the operation object attribute subspace, and the temporal sequence attribute subspace, and cosine similarity is calculated with the attribute vectors of the corresponding nodes. Let the action type attribute similarity be... The similarity of the operation objects' attributes is The temporal similarity of attributes is The three are weighted according to a preset weight. , , We perform weighted fusion to obtain the semantic matching degree between the multimodal semantic feature vector and the current node. ,Right now ,in Three weight values ​​are pre-set based on the sensitivity of the production scenario to different attributes. Semantic matching degree is calculated for each production operation event node in the causal process graph. The multimodal semantic feature vector is assigned to the node with the highest semantic matching degree, and this vector is used as the current state representation of that node, thus completing the mapping from perceptual features to process semantics. Once a node obtains its state representation, its activation intensity... Based on semantic matching degree Direct decision, The higher the value, the stronger the node activation intensity, indicating that the visual evidence observed at the current production site supports the operational event more strongly.

[0091] Causal path reasoning is performed on the causal process graph, propagating node state representations along directed edges in the graph and accumulating node activation strengths on causal dependency paths. Directed edges represent the causal sequence between operation events, i.e., if operation events... Operation event The direct causal precursor exists from point to The directed edges. During path propagation, starting from the initial node of the graph, each node is traversed sequentially according to topological sorting. The activation intensity of the predecessor node is passed to the current node after decay via directed edges, and then weighted and merged with the activation intensity of the current node itself. Let the node be... The causal path reachability score is The calculation method is to include all arriving nodes. The cumulative propagation value of the activation intensity of each predecessor node on the causal path and the node Self-activation strength Adding, that is ,in For nodes The set of all direct predecessor nodes, For the node To the node The pre-defined causal propagation attenuation coefficient on the directed edges, ranging from 0 to 1, reflects the strength of the causal relationship between two operational events. Through the above recursive calculation, each production operational event node can obtain a reachability score that comprehensively reflects its causal path support level in the current production state. .

[0092] Causal path reachability score Prior probability of the expected operational event occurring By comparing the results, the causal necessity scores of each operational event node are obtained. Prior probability Statistical analysis derived from historical production data reflects the expected probability of this operational event occurring within the standard process flow. Causality score. Defined as ,in This is a prior probability adjustment coefficient used to balance the relative weights between path reachability and prior expectation. When the score is significantly lower than the preset causal expectation threshold, it indicates that the observational evidence at the current production site at that operational event node is insufficient to support the causal inevitability that it should have occurred, i.e., there is a risk of process logic anomaly. This mechanism ensures that the causal inevitability score not only reflects the visual observation status at the current moment but also integrates the causal structure knowledge of the process flow and historical statistical priors, thus possessing stronger robustness and interpretability when judging process logic anomalies.

[0093] In actual pre-prepared food production scenarios, for example, a batch of products needs to undergo four main processes sequentially: raw material pretreatment, seasoning and marinating, high-temperature heating, and cooling and packaging. If visual monitoring data shows that the causal path reachability score of the high-temperature heating node is low, while the activation intensity of the seasoning and marinating node is normal, then the causal necessity score will reveal an anomaly that the high-temperature heating operation has been skipped or is out of sequence. This provides a clear causal tracing basis for subsequent cross-validation and verification report generation. The entire mapping and reasoning process is carried out under the topological constraints of the graph, ensuring that the anomaly judgment results are consistent with the actual logic of the production process.

[0094] In one optional implementation, a physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between each modal observation and the predicted value of the physical law is calculated, and abnormal modal response regions that violate physical consistency are identified, including:

[0095] Physical constraint prior knowledge is extracted from the pre-cooked food production process specifications. Material input-output balance relationship is extracted based on material mass conservation constraint. Temperature gradient change law is extracted based on thermodynamic balance constraint. Velocity and acceleration boundary conditions of motion trajectory are extracted based on mechanical kinematic constraint. A set of physical constraint relationships is constructed based on the balance relationship, the change law and the boundary conditions.

[0096] The set of physical constraint relationships is converted into a constraint hyperplane of a multidimensional physical constraint space. The constraint hyperplane defines a feasible solution region that conforms to physical laws. The multimodal features of the anchor point alignment are calibrated with physical dimensions. The visual response values ​​are mapped to physical measurement values ​​and projected onto the multidimensional physical constraint space to obtain the distribution of observation points.

[0097] For each observation point in the observation point distribution, find the constraint-satisfied point on the constraint hyperplane that is closest to the observation point, calculate the deviation vector between the physical quantity measurement value of the observation point and the physical law prediction value of the constraint-satisfied point, and perform statistical analysis on the multimodal dimension components of the deviation vector of all observation points in the observation point distribution to obtain the residual distribution.

[0098] Anomaly detection is performed on the residual distribution to identify residual samples that deviate from the center of the normal distribution by more than the physical consistency threshold. The corresponding positions of the residual samples in the multimodal feature space aligned with the anchor point are spatially clustered and merged to obtain the abnormal modal response region.

[0099] Extracting prior knowledge of physical constraints from pre-prepared food production process specifications, operation standard manuals, and historical production records is the starting point for constructing the physical constraint space. For the constraint of material mass conservation, the balance between material input and output in each process is extracted. For example, in the cutting and preparation process, the total weight of raw materials should be conserved, along with the sum of the weight of the cut materials and waste. In the marinating process, there is a mass balance between the amount of liquid seasoning absorbed and the amount absorbed by the materials. For the constraint of thermodynamic equilibrium, the gradient change law of temperature over time in processes such as heating and cooling is extracted, including the upper limit of the heating rate, the steady-state temperature range, and the slope boundary of the cooling curve. These laws are jointly determined by physical parameters such as material heat capacity, thermal conductivity, and equipment power. For the constraint of mechanical kinematics, the upper and lower limits of the speed and acceleration boundary conditions of equipment such as stirring paddles, conveyor belts, and robotic arms under normal operating conditions are extracted. Exceeding these boundaries indicates abnormal equipment operation or operational violations. These three types of physical constraints are integrated into a set of physical constraint relationships, each expressed in the form of mathematical inequalities or equations, forming the basis for subsequently constructing the constraint hyperplane.

[0100] When transforming a set of physical constraints into constraint hyperplanes in a multidimensional physical constraint space, each constraint corresponds to one or a set of hyperplanes, which together define a feasible solution region that conforms to physical laws. The feasible solution region is a convex polyhedron or non-convex region enclosed by the constraint hyperplanes. Observation points falling within this region are considered compatible with physical laws, while observation points falling outside the region are suspected of violating physical laws. Before projecting the anchor-aligned multimodal features onto this multidimensional physical constraint space, the visual response values ​​need to be calibrated using physical dimensions. Specifically, pixel brightness values ​​in RGB images are converted to actual illumination intensity or surface color feature values ​​using calibration parameters; grayscale response values ​​from infrared thermal imagers are converted to Celsius temperature using radiation calibration coefficients; distance response values ​​from depth sensors are converted to three-dimensional coordinates in millimeters using an intrinsic parameter matrix; and voltage signals from torque sensors are converted to torque values ​​in Newton-meter units using sensor sensitivity coefficients. After dimensional calibration, the physical measurements of each mode are assembled according to the dimensional definition of the constraint space to obtain the observation points of each anchor point in the multidimensional physical constraint space. The observation points of all anchor points together constitute the observation point distribution.

[0101] For each observation point in the observation point distribution, find the constraint-satisfied point on the constraint hyperplane that is closest to that observation point; that is, find the point on or inside the feasible solution region that minimizes the Euclidean distance. If the observation point itself is already inside the feasible solution region, the constraint-satisfied point is the observation point itself, and the deviation vector is the zero vector. If the observation point is outside the feasible solution region, the constraint-satisfied point is the projection point of the observation point on the boundary of the feasible solution region. In this case, the direction of the deviation vector points to the direction of violating the physical constraints, and its magnitude reflects the degree of violation. Let the physical quantity vector of the observation point be... The physical law prediction vector corresponding to the constraint-satisfied point is The deviation vector is defined as follows: ,in Each component corresponds to a residual value for a physical quantity dimension. The deviation vector for all observation points in the observation point distribution. The data is separated according to modal dimensions, and the residual components in each modal dimension are statistically analyzed. The mean, standard deviation, and quantiles of the residuals for each modal dimension are calculated to obtain the residual distribution. The residual distribution describes the statistical regularity of the deviation between the current multimodal observations in the production site and the predicted values ​​of physical laws, and is the core basis for subsequent anomaly detection.

[0102] Anomaly detection is performed on the residual distribution using a criterion based on the degree of deviation from the distribution center. Let a certain modal dimension be considered. The mean of the residuals is The standard deviation is The physical consistency threshold is Then, when the residual components of a certain observation point in that dimension satisfy At that time, the observation point is marked as a residual sample in that dimension. The value of is determined comprehensively based on the measurement accuracy and process tolerance of each physical quantity. For temperature constraints, a smaller value is usually chosen to ensure thermal safety, while for mass conservation constraints, a reasonable tolerance is set based on the weighing accuracy. The residual sample judgment results in the multimodal dimension are taken as a union, that is, any observation point judged as abnormal in any modal dimension is included in the residual sample set. For each sample in the residual sample set, based on its spatial coordinates in the multimodal feature space aligned with the anchor point, a density-adaptive spatial clustering algorithm is used to merge spatially adjacent residual samples into connected regions, forming several abnormal modal response regions. During the clustering and merging process, the spatial neighborhood radius of the anchor point is used as the search range, and residual samples whose distance to each other is less than this radius are grouped into the same cluster. Isolated single-point residual samples that do not meet the minimum cluster size requirement are regarded as noise points and filtered out to avoid false alarms caused by random noise from the sensor.

[0103] The resulting anomalous modal response regions carry multidimensional physical violation evidence, including the spatial location of the violation, the dimensions of the physical quantities involved, the magnitude and direction of the deviation vector, and the specific physical constraints violated. For example, if the residual in a region exceeds the threshold in the thermodynamic dimension, it indicates a significant deviation between the observed temperature and the predicted thermodynamic equilibrium value at that spatial location, corresponding to local overheating caused by heating equipment failure or abnormal material accumulation; if a region shows a negative residual in the mass conservation dimension, it indicates excessive material loss at that process location, corresponding to material leakage or weighing operation errors. These anomalous modal response regions with physical semantic annotations will serve as input for physical path anomaly determination, and will be cross-validated with the process logic anomaly results of the causal path to jointly support the generation of the pre-prepared food production site inspection report.

[0104] In one optional implementation, a dual-path anomaly determination is performed, comprising causal path and physical path anomalies. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are below a causal expectation threshold. The physical path identifies physical state anomalies based on the abnormal modal response region, including:

[0105] A threshold determination is made for the causal necessity score of each production operation event node in the causal process graph along the causal path. Nodes with causal necessity scores lower than the causal expectation threshold are marked as causal abnormal nodes. The upstream causal dependency path of the causal abnormal node in the causal process graph is extracted. By tracing back the changes in the causal necessity scores of each predecessor node on the upstream causal dependency path, the root cause node that leads to the decrease in causal necessity score is identified. The production operation event corresponding to the root cause node is marked as the process logic abnormality.

[0106] Physical constraint type attribution analysis is performed on the abnormal modal response region along the physical path. By calculating the constraint violation intensity of each observation point in the abnormal modal response region in the dimensions of material mass conservation constraint, thermodynamic equilibrium constraint, and mechanical kinematic constraint, the dominant constraint violation type of the abnormal modal response region is determined. Based on the dominant constraint violation type, a predefined physical state anomaly category library is associated with it, and the abnormal modal response region is mapped to the corresponding physical state anomaly category.

[0107] The core of the dual-path anomaly determination mechanism lies in decoupling process deviations at the causal logic level and measurement violations at the physical state level into two independent but complementary determination channels, thereby avoiding false alarms and missed alarms caused by modal noise or process flexibility when making determinations in a single dimension.

[0108] Along the causal path, the causal necessity score of each production operation event node in the causal process graph is compared against a threshold. Let the causal expectation threshold be... For any node in the graph If its causal necessity score is satisfy If so, then the node is marked as a causal anomaly node. The value is determined based on the statistical distribution of scores for similar nodes in historical normal production batches. It is typically taken as the quantile corresponding to the mean of a normal distribution minus 1.5 times the standard deviation, to achieve a balance between sensitivity and specificity. For key process nodes in pre-prepared food production, such as seasoning addition, sterilization and heat preservation, and cold chain packaging, the value can be determined according to the process importance level. Differentiated settings are implemented, with higher-importance nodes corresponding to stricter (i.e., larger values) thresholds to ensure that high-risk operational deviations are not overlooked.

[0109] After marking causal anomaly nodes, the upstream causal dependency path of each causal anomaly node in the causal process graph is extracted. The upstream causal dependency path consists of all directed edges in the graph pointing to the anomaly node and their recursive predecessor nodes, forming an ordered sequence of nodes extending from the source node to the current anomaly node. The causal necessity score of each predecessor node is backtracked along this path, and the score difference between adjacent predecessor nodes is calculated to identify locations where the score drops significantly. Let the i-th node on the path be... The causal necessity score of each predecessor node is , No. The score of each predecessor node is When the difference Exceeding the preset root cause identification sensitivity parameter At that time, the first Each node is identified as the starting point where the causal necessity score decreases, and the corresponding production operation event is marked as the root cause node. The root cause node represents the earliest operational link in the process logic chain that deviates from the expected causal relationship. Its corresponding production operation event is ultimately marked as a process logic anomaly, and the topological position of the anomaly in the graph, as well as the associated predecessor and successor node information, are recorded to provide a structured basis for the generation of subsequent causal tracing reports.

[0110] On the physical path, physical constraint type attribution analysis is performed on the identified abnormal modal response regions. The physical constraints involved in the pre-prepared food production site mainly include three categories: material mass conservation constraints, thermodynamic equilibrium constraints, and mechanical kinematic constraints. Material mass conservation constraints focus on the mass balance relationship between the amount of feed, the amount of output, and the amount of loss, and are suitable for visual inspection scenarios such as ingredient mixing and dispensing metering. Thermodynamic equilibrium constraints focus on the physical consistency between temperature field distribution, heat conduction rate, and heating power, and are suitable for heat treatment processes such as steaming, stir-frying, and sterilization. Mechanical kinematic constraints focus on whether the motion trajectory, velocity vector, and acceleration distribution of equipment conform to the rigid body kinematic equations, and are suitable for checking the motion state of mechanical components such as conveyor belts, stirring paddles, and cutting tools.

[0111] For each observation point in the abnormal modal response region, calculate its constraint violation strength across the three constraint dimensions mentioned above. Let the violation strength of the observation point in the material mass conservation constraint dimension be... The violation strength in the thermodynamic equilibrium constraint dimension is The violation strength in the mechanical kinematic constraint dimension is The violation intensity for each dimension is calculated based on the degree to which the residual component of the observation point exceeds the physical consistency threshold of that dimension; the greater the exceedance, the higher the violation intensity. The mean of the three types of violation intensities is taken for all observation points within the anomalous modal response region to obtain the region-level constraint violation intensity vector. .Pick The constraint type corresponding to the largest component is used as the dominant constraint violation type in the abnormal modal response region. When there are two components with a difference less than a preset discrimination threshold, the constraint type is considered as the dominant constraint violation type in the abnormal modal response region. In cases where this occurs, it is determined to be a composite constraint violation, and both dominant types are recorded.

[0112] After determining the dominant constraint violation type, this type is used as the index key to query a predefined physical state anomaly category library. The physical state anomaly category library is a structured mapping table where each record contains a constraint violation type identifier, the corresponding physical state anomaly category name, a typical manifestation description, and suggested verification measures. For example, when the dominant constraint violation type is a thermodynamic equilibrium constraint, the corresponding physical state anomaly categories in the library include specific categories such as "uniform heating temperature," "insufficient holding time," and "local overheating"; when the dominant constraint violation type is a material mass conservation constraint, the corresponding categories are "excessive deviation in material input" and "unqualified packaging weight." Through this mapping relationship, abstract physical residual anomalies are transformed into physical state anomaly categories with practical production significance, making the verification results directly readable and actionable for production site management personnel.

[0113] After mapping the abnormal modal response region to the corresponding physical state anomaly category, the spatial coordinate range of the region, the involved modal types, the specific intensity value of the dominant violation type, and the matching anomaly category label are recorded to form a structured physical state anomaly record entry. This entry, together with the process logic anomaly records generated on the causal path, constitutes the original evidence set for dual-path determination, providing complete input data for subsequent cross-validation and verification report generation. The dual-path design ensures that even if one path has a missing determination due to data quality issues, the other path can still independently provide valid anomaly evidence, thereby improving the robustness of the overall verification scheme under complex production site conditions.

[0114] In one optional implementation, cross-validating the process logic anomaly and the physical state anomaly to generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations includes:

[0115] Calculate the time delay difference and spatial distance difference between the root cause node corresponding to the process logic anomaly and the abnormal modal response region. When the time delay difference is within the causal propagation time window and the spatial distance difference is within the causal influence range, establish a cross-validation association between the root cause node and the abnormal modal response region.

[0116] Based on the cross-validation association, the consistency of the process logic anomaly and the physical state anomaly is verified. For anomaly pairs with cross-validation association, the complete causal propagation path from the root cause node to the downstream influencing node is extracted from the causal process graph as the causal tracing path. The deviation vector between the physical measurement value of the observation point corresponding to the abnormal modal response region and the constraint satisfaction point, as well as the dominant constraint violation type, are extracted from the physical constraint space as the physical violation evidence.

[0117] Record the anomaly type identifier and spatiotemporal information of the abnormal event, record the production operation event name and causal necessity score change of each node on the causal tracing path, record the spatial location label of the abnormal modal response area, the numerical distribution of the deviation vector and the constraint violation intensity of the dominant constraint violation type, and generate the pre-prepared vegetable production site verification report in combination with the physical violation evidence.

[0118] like Figure 2 As shown, the method includes:

[0119] After determining anomalies along both the causal and physical paths, cross-validation of process logic anomalies and physical state anomalies is necessary to confirm whether there is an inherent correlation between the two types of anomalous signals. This avoids misjudging independent noise interference as genuine production violations. The core basis for cross-validation is that genuine production anomalies often leave observable traces at both the process logic and physical state levels, and there is a temporal and spatial relationship between the two types of traces that conforms to the laws of physical causal propagation.

[0120] For each identified root cause node of process logic anomaly, its corresponding event occurrence time and spatial location annotation in the causal process graph are extracted; simultaneously, for each identified abnormal modal response region, its detection time and spatial coordinates in the multimodal visual data stream are extracted. The time delay difference between the two is calculated. and spatial distance difference Time delay difference This reflects the time interval required for the physical effects of a process operation event to propagate to the observable range of the sensor; spatial distance difference. This reflects the Euclidean distance between the operation location corresponding to the root cause node and the center location of the abnormal modal response region. The causal propagation time window is pre-set based on the technological characteristics of the pre-prepared food production site. and the scope of causal influence ,when and When both conditions are met, a cross-validation association is determined between the root cause node and the anomalous modal response region, and they are registered as an associated anomalous pair. Lower bound of the causal propagation time window. The upper limit is usually determined by the response delay of the production line sensors. The extent of causal influence is determined by the rate of diffusion of the physical effects of the operational event and the production cycle time. The calibration is based on the physical radius of action of various operating positions.

[0121] Consistency checks are performed on all registered associated anomaly pairs. Anomaly pairs with cross-validation correlations are considered high-confidence anomaly events with double confirmation. Anomaly signals that appear only in the causal path or only in the physical path and fail to establish cross-validation correlations are marked as single-path suspected anomalies and listed separately in the verification report with lower confidence for manual review. The purpose of consistency checks is to distinguish between two types of situations: one where process operations are indeed violated and have produced observable physical consequences, and the other where false alarms are caused by sensor false alarms or deviations in process pattern modeling.

[0122] For the linked anomaly pairs that pass the consistency check, causal origination paths and physical evidence of violations are extracted. The causal origination path is extracted as follows: starting from the root cause node in the causal process graph, traverse downstream along the directed causal edges, collecting all downstream nodes affected by the root cause node, forming a complete causal propagation chain. During traversal, the production operation event name and corresponding causal necessity score of each node on the path are recorded. The trend of score changes along the path is used to visually represent the gradual decay of causal influence. Physical violation evidence is extracted by reading the deviation vector of the observation point corresponding to the abnormal modal response region from the physical constraint space. (i.e., the difference between the observed value and the predicted value of the physical law), and at the same time, read the dominant constraint violation type and its corresponding constraint violation strength vector for this region. Deviation vector The values ​​of each component directly reflect the degree to which the observed value deviates from the predicted value of the physical law in different modal dimensions, and are quantitative evidence of physical violations; the dominant constraint violation type indicates the physical mechanism of the violation. For example, a violation of the material mass conservation constraint indicates a deviation in the amount of material fed, a violation of the thermodynamic equilibrium constraint indicates an abnormal heating temperature or duration, and a violation of the mechanical kinematics constraint indicates a deviation in the operating state of the equipment.

[0123] For each high-confidence correlated anomaly pair, the following information is recorded according to a unified structured format. The basic information section of the anomaly event records the anomaly type identifier (process logic anomaly, physical state anomaly, or dual-path confirmation anomaly), the time of occurrence, duration, and spatial coordinates of the location, forming a spatiotemporal information archive of the anomaly event. The causal tracing path section presents the complete propagation chain from the root cause node to each downstream influencing node in a directed graph structure. Each node on the path is labeled with the name of the production operation event (e.g., "heating operation," "stirring operation," "feeding operation," etc.) and its causal necessity score. The specific numerical values ​​allow inspectors to intuitively determine which step of the process logic has broken down. The physical evidence of violations is presented as a visual heatmap marking the spatial location of abnormal modal response areas, and as a numerical table showing the deviation vector. The component distribution across each modal dimension is presented as a histogram of the constraint violation strength vector. Material mass conservation violates strength Strength of thermodynamic equilibrium violation Mechanical kinematics violates strength A quantitative comparison of the three types of constraints is presented, and the types of violations of the dominant constraint and their physical meanings are described in textual form.

[0124] For suspected anomalies in a single path, a separate appendix for suspected anomalies is set up in the verification report, which records the anomaly type identifier, the spatiotemporal information of the occurrence, and the reason why cross-validation association could not be established. For example, the time delay difference exceeds the causal propagation time window or the spatial distance difference exceeds the causal influence range, providing sufficient clues for subsequent manual review.

[0125] The final verification report is output as a structured data file, along with a visual summary page suitable for on-site management personnel. The summary page displays the sequence of all anomalies during the verification period in a timeline format, overlays spatial location markers onto a production site floor plan, and presents the root cause propagation path of each anomaly using a cause-and-effect diagram. At the top of the page, the highest-risk anomalies requiring priority handling are listed in order of confidence level. By organically integrating causal tracing paths and physical evidence of violations into a single verification report, production management personnel not only know "where the anomaly occurred," but also understand "why the anomaly occurred" and "what the specific physical manifestation of the anomaly is," thereby supporting targeted corrective production decisions.

[0126] A second aspect of this invention provides an intelligent verification system for pre-prepared food production sites based on multimodal visual recognition, comprising:

[0127] Anchor point alignment unit is used to identify spatial regions with consistent responses in different modalities as semantic anchor points through unsupervised clustering of multimodal visual data streams from pre-prepared food production sites. The semantic anchor points are then used to establish sparse correspondences across modalities to obtain anchor point aligned multimodal features.

[0128] The causal inference unit is used to construct a causal process graph of pre-prepared food production, map the multimodal features aligned with the anchor points to the node state space of the causal process graph, and calculate the causal necessity score of each operation event in the current production state through a causal inference algorithm.

[0129] The physical constraint unit is used to introduce prior knowledge of physical constraints in the production site to construct a physical constraint space. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical law is calculated, and abnormal modal response regions that violate physical consistency are identified.

[0130] The dual-path determination unit is used to perform dual-path anomaly determination including causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region.

[0131] The verification report unit is used to cross-verify the process logic anomalies and the physical state anomalies and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

[0132] A third aspect of the present invention provides an electronic device, comprising:

[0133] processor;

[0134] Memory used to store processor-executable instructions;

[0135] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0136] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0137] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent verification of pre-prepared food production sites based on multimodal visual recognition, characterized in that: include: For the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse correspondences across modalities, resulting in anchor-aligned multimodal features. A causal process graph for pre-prepared food production is constructed, and the multimodal features aligned with the anchor points are mapped to the node state space of the causal process graph. The causal necessity score of each operation event in the current production state is calculated using a causal inference algorithm. A physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical laws is calculated, and abnormal modal response regions that violate physical consistency are identified. The system performs dual-path anomaly determination, which includes causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region. Cross-validate the process logic anomalies and the physical state anomalies, and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

2. The method according to claim 1, characterized in that, For the multimodal visual data stream of the pre-prepared food production site, unsupervised clustering is used to identify spatial regions with consistent responses in different modalities as semantic anchors. These semantic anchors are then used to establish sparse correspondences across modalities, resulting in anchor-aligned multimodal features, including: Each modal visual data in the multimodal visual data stream is divided into spatial grids, and each modal visual data is divided into multiple spatial grid units. By constructing a cross-modal response consistency measurement mechanism, the response consistency scores between the spatial grid cells corresponding to different modes are calculated, and the spatial grid cells with response consistency scores higher than a preset consistency threshold are marked as candidate anchor point cells. Spatial neighborhood aggregation is performed on the candidate anchor units. By calculating the response consistency propagation strength between the candidate anchor unit and other candidate anchor units in the spatial neighborhood, the candidate anchor units whose response consistency propagation strength satisfies the spatial continuity constraint are selected as the semantic anchors. A cross-modal correspondence graph is established using the semantic anchors as nodes and the response similarity between semantic anchors of different modalities as edge weights. Based on the cross-modal correspondence graph, feature alignment transformation is performed on the multimodal visual data stream. By using the spatial position of the semantic anchor point as a constraint, affine transformation correction is performed on the visual features of different modalities to obtain the anchor point aligned multimodal features.

3. The method according to claim 1, characterized in that, Constructing a causal process diagram for pre-prepared food production includes: Production operation event sequences are extracted from historical production data of pre-prepared dishes and decomposed into multiple operation event segments. An operation event co-occurrence matrix is ​​constructed for the multiple operation event segments. By performing a conditional independence test on the operation event co-occurrence matrix, direct causal dependencies and indirect causal dependencies between operation events are identified. Based on the direct and indirect causal dependencies, an operational event causal dependency graph is constructed. Counterfactual inference is performed on each causal dependency path in the operational event causal dependency graph. By simulating the removal of predecessor operational events from the causal dependency path and calculating the change in the probability of subsequent operational events, causal dependency paths whose change exceeds a preset causal significance threshold are marked as valid causal edges. The valid causal edges are used as directed edges, and the causal process graph is constructed with the production operational events as nodes.

4. The method according to claim 3, characterized in that, The multimodal features aligned with the anchor points are mapped to the node state space of the causal process graph. The causal inevitability score of each operational event in the current production state is calculated using a causal inference algorithm, including: Each production operation event node in the causal process graph is associated with a predefined operation event semantic space, which includes the action type attribute, operation object attribute, and timing attribute of the operation event. The anchor-aligned multimodal features are transformed by feature dimension mapping. By projecting the anchor-aligned multimodal features onto the operation event semantic space, a multimodal semantic feature vector is obtained. The similarity between the multimodal semantic feature vector and the production operation event node in terms of action type attribute, operation object attribute, and temporal sequence attribute is calculated, and a weighted fusion is performed to obtain the semantic matching degree. The multimodal semantic feature vector is assigned to the production operation event node with the highest semantic matching degree to form a node state representation. Causal path reasoning is performed on the causal process graph. By propagating the node state representation along the directed edge and accumulating the node activation intensity on the causal dependency path, the causal path reachability score of each production operation event node is calculated. The causal path reachability score is compared with the prior probability of the expected operation event to obtain the causal inevitability score.

5. The method according to claim 1, characterized in that, A physical constraint space is constructed by introducing prior knowledge of physical constraints from the production site. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical laws is calculated. Abnormal modal response regions that violate physical consistency are identified, including: Physical constraint prior knowledge is extracted from the pre-cooked food production process specifications. Material input-output balance relationship is extracted based on material mass conservation constraint. Temperature gradient change law is extracted based on thermodynamic balance constraint. Velocity and acceleration boundary conditions of motion trajectory are extracted based on mechanical kinematic constraint. A set of physical constraint relationships is constructed based on the balance relationship, the change law and the boundary conditions. The set of physical constraint relationships is converted into a constraint hyperplane of a multidimensional physical constraint space. The constraint hyperplane defines a feasible solution region that conforms to physical laws. The multimodal features of the anchor point alignment are calibrated with physical dimensions. The visual response values ​​are mapped to physical measurement values ​​and projected onto the multidimensional physical constraint space to obtain the distribution of observation points. For each observation point in the observation point distribution, find the constraint-satisfied point on the constraint hyperplane that is closest to the observation point, calculate the deviation vector between the physical quantity measurement value of the observation point and the physical law prediction value of the constraint-satisfied point, and perform statistical analysis on the multimodal dimension components of the deviation vector of all observation points in the observation point distribution to obtain the residual distribution. Anomaly detection is performed on the residual distribution to identify residual samples that deviate from the center of the normal distribution by more than the physical consistency threshold. The corresponding positions of the residual samples in the multimodal feature space aligned with the anchor point are spatially clustered and merged to obtain the abnormal modal response region.

6. The method according to claim 1, characterized in that, Perform dual-path anomaly determination, including causal path and physical path anomalies. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold. The physical path identifies physical state anomalies based on the abnormal modal response region, including: A threshold determination is made for the causal necessity score of each production operation event node in the causal process graph along the causal path. Nodes with causal necessity scores lower than the causal expectation threshold are marked as causal abnormal nodes. The upstream causal dependency path of the causal abnormal node in the causal process graph is extracted. By tracing back the changes in the causal necessity scores of each predecessor node on the upstream causal dependency path, the root cause node that leads to the decrease in causal necessity score is identified. The production operation event corresponding to the root cause node is marked as the process logic abnormality. Physical constraint type attribution analysis is performed on the abnormal modal response region along the physical path. By calculating the constraint violation intensity of each observation point in the abnormal modal response region in the dimensions of material mass conservation constraint, thermodynamic equilibrium constraint, and mechanical kinematic constraint, the dominant constraint violation type of the abnormal modal response region is determined. Based on the dominant constraint violation type, a predefined physical state anomaly category library is associated with it, and the abnormal modal response region is mapped to the corresponding physical state anomaly category.

7. The method according to claim 1, characterized in that, Cross-validation of the process logic anomalies and the physical state anomalies, and generation of a pre-prepared food production site inspection report with causal tracing paths and physical evidence of violations, includes: Calculate the time delay difference and spatial distance difference between the root cause node corresponding to the process logic anomaly and the abnormal modal response region. When the time delay difference is within the causal propagation time window and the spatial distance difference is within the causal influence range, establish a cross-validation association between the root cause node and the abnormal modal response region. Based on the cross-validation association, the consistency of the process logic anomaly and the physical state anomaly is verified. For anomaly pairs with cross-validation association, the complete causal propagation path from the root cause node to the downstream influencing node is extracted from the causal process graph as the causal tracing path. The deviation vector between the physical measurement value of the observation point corresponding to the abnormal modal response region and the constraint satisfaction point, as well as the dominant constraint violation type, are extracted from the physical constraint space as the physical violation evidence. Record the anomaly type identifier and spatiotemporal information of the abnormal event, record the production operation event name and causal necessity score change of each node on the causal tracing path, record the spatial location label of the abnormal modal response area, the numerical distribution of the deviation vector and the constraint violation intensity of the dominant constraint violation type, and generate the pre-prepared vegetable production site verification report in combination with the physical violation evidence.

8. A pre-prepared food production site intelligent inspection system based on multimodal visual recognition, used to implement the method as described in any one of claims 1-7, characterized in that, include: Anchor point alignment unit is used to identify spatial regions with consistent responses in different modalities as semantic anchor points through unsupervised clustering of multimodal visual data streams from pre-prepared food production sites. The semantic anchor points are then used to establish sparse correspondences across modalities to obtain anchor point aligned multimodal features. The causal inference unit is used to construct a causal process graph of pre-prepared food production, map the multimodal features aligned with the anchor points to the node state space of the causal process graph, and calculate the causal necessity score of each operation event in the current production state through a causal inference algorithm. The physical constraint unit is used to introduce prior knowledge of physical constraints in the production site to construct a physical constraint space. By projecting the multimodal features aligned with the anchor points onto the physical constraint space, the residual distribution between the observed values ​​of each mode and the predicted values ​​of the physical law is calculated, and abnormal modal response regions that violate physical consistency are identified. The dual-path determination unit is used to perform dual-path anomaly determination including causal path and physical path. The causal path identifies process logic anomalies based on nodes whose causal necessity scores are lower than the causal expectation threshold, and the physical path identifies physical state anomalies based on the abnormal modal response region. The verification report unit is used to cross-verify the process logic anomalies and the physical state anomalies and generate a pre-prepared food production site verification report with causal tracing paths and physical evidence of violations.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images