A method, system, device and storage medium for industrial safety scenario matching

Abstract

The application discloses a kind of industrial safety scene matching method, system, equipment and storage medium.The method first constructs the standardized triad data including hidden danger image, description text and rectification image;Through image feature matching module, feature matching proportion, internal point proportion and structure similarity index are extracted and fused, and quantitative result including comprehensive confidence is generated;Further, large language model analysis module according to quantitative result and hidden danger description, through structured prompt word guide carries out multi-step semantic reasoning, and outputs matching analysis result with grade;Finally, fusion visual and semantic evidence completes safety hidden danger rectification verification and decision.The application solves the problem of semantic gap and high false alarm rate of pure visual method through "visual matching-semantic reasoning" collaborative architecture, realizes intelligent and accurate identification of compliance rectification and real hidden danger.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence technology, and industrial security, and particularly to a method, system, device, and storage medium for matching industrial security scenarios. Background Technology

[0002] Current industrial hazard detection technologies primarily rely on computer vision methods, which face fundamental limitations when dealing with complex scenarios requiring deep semantic understanding. Mainstream technical approaches can be divided into two categories: one is traditional visual algorithms based on fixed rules (such as edge detection and template matching), which, while effective in structured scenes, suffers from weak generalization ability and is extremely sensitive to ambient lighting, viewpoint changes, and target occlusion; the other is deep learning-based target detection models (such as YOLO and Faster R-CNN), which, while capable of automatically learning features, lack interpretability in their decision-making process and heavily rely on large-scale labeled data. More critically, both traditional algorithms and deep learning models are limited to identifying and locating isolated targets of predefined categories (such as "safety helmet" and "fire extinguisher"), failing to understand the interaction relationships and contextual semantics between multiple elements in a scene. A typical example is the complex scenario of "a worker located within the danger zone of a mobile forklift"—accurately identifying such hazards requires not only detecting both the "worker" and the "forklift," but also understanding their spatial relationship, the forklift's movement, and determining whether a violation constitutes a violation based on safety regulations. Existing purely visual methods lack semantic reasoning capabilities, making it difficult to achieve effective early warnings for such tasks.

[0003] To improve the ability to detect scene changes, image feature matching techniques (such as SIFT and SURF) have been introduced. These techniques, through feature point extraction and spatial transformation, can achieve precise alignment of scene images and detect subtle visual differences, thus providing quantitative evidence for determining "whether it is the same scene" or "where the change occurred." However, this technique is essentially a geometric and texture-level computation, lacking semantic judgment regarding the reasons for the differences. For example, a significant image change caused by a compliance rectification (such as adding a protective cover) might be judged as low similarity from a feature matching perspective, leading to a false alarm by the system as "scene abnormality" or "match failure." This reveals the "semantic gap" inherent in purely visual methods: they can detect "differences," but cannot understand "why the differences exist."

[0004] Meanwhile, large language models have demonstrated powerful knowledge encoding and logical reasoning capabilities, and have proven their potential to understand semantic relationships from images in multimodal tasks, bringing new possibilities for solving the aforementioned problems. However, if used as a standalone image analyzer, it is difficult to directly achieve high-precision spatial alignment and pixel-level change detection.

[0005] Therefore, the current technological bottleneck lies in how to deeply integrate the precise geometric perception capabilities of image feature matching with the deep semantic reasoning capabilities of large language models. Specifically, there is an urgent need to design a collaborative framework that enables the quantitative results of visual matching (such as matching degree and difference regions) to structurally drive semantic reasoning, while allowing the reasoning results of large language models (such as the judgment that "the worker is located in the danger zone of the mobile forklift") to provide feedback and explain the visual findings, thereby achieving intelligent and accurate identification of real hidden dangers and compliance rectification in industrial scenarios. This is precisely the core technical problem that this invention aims to solve. Summary of the Invention

[0006] The overall objective of this invention is to address the technical shortcomings of existing industrial safety hazard detection methods in complex scenarios, such as insufficient semantic understanding, poor environmental adaptability, and high false alarm rates, by providing a method for industrial safety scenario matching. This method aims to construct a dual-track collaborative architecture of "precise visual matching - deep semantic reasoning." By combining the structured quantitative indicators output by the image feature matching module with common sense and safety standard knowledge from a large language model, it achieves intelligent and accurate identification of compliant rectification and real hazards in industrial scenarios, ultimately improving the automation level, accuracy, and decision interpretability of safety hazard verification.

[0007] Therefore, the first objective of this invention is to provide a method for matching industrial safety scenarios, which includes the following steps: Standardized hazard-rectification triplet data is constructed through a dataset preprocessing module. The triplet includes a hazard image before rectification, a standardized hazard description text, and a hazard image after rectification. The images of potential hazards before rectification and the images after rectification are calculated and fused with multi-dimensional visual matching quantitative indicators to generate a structured matching quantitative result that includes feature matching ratio, inlier ratio, structural similarity index and comprehensive confidence. Receive the structured matching quantification result and the standardized hidden danger description text, perform structured semantic reasoning and comprehensive judgment, and output the scene subject matching degree analysis result containing the matching level and reason; Based on the structured matching quantification results and the scene subject matching degree analysis results, a safety hazard rectification verification decision is made, and a structured decision report is output.

[0008] Preferably, the step of calculating and fusing multi-dimensional visual matching quantitative indicators between the pre-rectification hazard image and the post-rectification image includes: Extract key points and their feature descriptors that are scale- and rotation-invariant from the input images of potential hazards before and after rectification; The feature descriptor sets of the image before rectification and the image after rectification are matched, and the nearest neighbor distance ratio criterion is used for filtering to obtain an initial feature matching point set; Robustly estimate the homography matrix from the initial feature matching point set to describe the optimal planar projection transformation relationship between the image of hidden dangers before rectification and the image after rectification, and calculate the inlier ratio; Calculate the structural similarity index between the image of the hidden danger before rectification and the image after rectification; The feature matching ratio, the inlier ratio, and the structural similarity index are weighted and fused to generate the comprehensive confidence score, and the four indicators are encapsulated into a structured data object for output.

[0009] Preferably, the step of receiving the structured matching quantification result and the standardized hazard description text, and performing structured semantic reasoning and comprehensive judgment, includes: Based on preset rules, a preliminary judgment is made on the overall confidence level of the input to trigger different analysis modes, and a predefined five-level scene and subject matching degree judgment standard is loaded. Dynamically construct structured prompt words by filling the prompt word template with the structured matching quantification results, the standardized hazard description text, the decision diversion logic, and the judgment criteria to form structured prompt words; The structured prompts are submitted to the large language model, and the returned responses are received and parsed to obtain the scene subject matching degree analysis results.

[0010] Preferably, the weighted fusion calculation formula is a weighted arithmetic mean: Overall confidence score = (ω1 × feature matching ratio + ω2 × inlier ratio + ω3 × structural similarity index) / (ω1 + ω2 + ω3), Wherein, ω1, ω2, ω3 are preset weight coefficients.

[0011] Preferably, the structured prompts contain instructions for guiding the large language model to perform the following reasoning steps sequentially: Target scene comparison: Analyze whether the image of the hidden danger before rectification and the image after rectification describe the same core target scene; Key requirements extracted: Accurately extract the safety requirements that must be followed from the standardized hazard description text; Feature conformity check: For each extracted safety requirement, check whether the visual features of the image of the hidden danger before rectification and the image after rectification are consistent; Evaluation of rectification effectiveness: Compare the changes in characteristics before and after to assess whether the rectification is complete and effective.

[0012] The second objective of this invention is to provide a system for matching industrial safety scenarios, comprising the steps of any of the above-mentioned methods for matching industrial safety scenarios, including: The dataset preprocessing module is used to construct standardized hazard-rectification triplet data, wherein the triplet includes a hazard image before rectification, standardized hazard description text, and a hazard image after rectification. The image feature matching module is used to process the image of the hidden danger before rectification and the image after rectification in the triplet and output the structured matching quantification result; The large language model analysis module is used to perform semantic reasoning based on the structured matching quantification results and the standardized hidden danger description text, and output the scene subject matching degree analysis results; The decision and output module is used to generate a safety hazard rectification verification decision report based on fused evidence.

[0013] Preferably, the image feature matching module includes: The feature extraction unit is used to extract scale- and rotation-invariant key points and their feature descriptors from the input images before and after rectification. The feature matching and filtering unit is used to match and initially screen the feature descriptor sets of the input image before rectification and the image after rectification; The geometric verification and transformation estimation unit is used to robustly estimate the geometric transformation relationship between two images from the initial matching point set and calculate the inlier ratio; The structural similarity assessment unit is used to evaluate the overall consistency of images before and after rectification in terms of brightness, contrast, and structural information. The indicator fusion and structured output unit is used to integrate the outputs of the feature extraction unit, feature matching and filtering unit, geometric verification and transformation estimation unit and structural similarity evaluation unit to generate the final visual matching evidence.

[0014] Preferably, the large language model analysis module includes: The decision-making and standard initialization unit is used to perform preliminary decision-making based on the input comprehensive confidence score; A structured prompting word engine is used to dynamically construct instructions to guide large language models in expert-level reasoning. The Large Language Model Call and Inference Unit is used to perform multi-step inference of the large language model and generate decisions.

[0015] A third objective of this invention is to provide an electronic device comprising: Memory, used to store computer programs; A processor is used to execute a program stored in memory to implement any of the steps of the above-mentioned method for matching industrial safety scenarios.

[0016] A fourth objective of this invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described industrial safety scenario matching methods.

[0017] The beneficial effects of this invention are as follows:

[0018] 1. Significantly improved semantic understanding and environmental adaptability in complex industrial scenarios: By introducing a large language model analysis module, the system can understand hazard descriptions, safety regulations, and scenario contexts, thereby accurately determining whether image differences are due to unrectified safety hazards or compliance rectification, solving the problem of poor environmental adaptability of pure vision methods due to lack of semantic understanding.

[0019] 2. Achieved synergistic optimization of matching accuracy and decision reliability: The multi-dimensional feature fusion and weighted decision mechanism designed in this invention integrates feature matching ratio, inlier ratio, and structural similarity index into a comprehensive confidence score, providing quantifiable and interpretable visual evidence for high-level decision-making. This index is dynamically coupled with a structured semantic reasoning framework based on expert knowledge for joint decision-making, significantly improving the overall accuracy and reliability of scene matching and hazard verification.

[0020] 3. A transparent decision-making chain that is explainable and auditable has been constructed: the output of the entire process not only includes the matching level, but also includes detailed reasons based on quantitative indicators and semantic reasoning. The large language model analysis module analyzes the data according to preset, security-compliant reasoning steps and judgment criteria, making the decision-making process no longer a "black box" and meeting the stringent requirements of the industrial security field for process traceability and result auditability.

[0021] 4. A closed-loop management tool has been formed from hazard discovery to intelligent verification: This method deeply integrates expert experience (through hazard description and judgment criteria), low-level visual computing (through image feature matching) and high-level semantic reasoning (through large language models), which can automatically complete the entire process of "shooting-comparison-analysis-judgment", providing effective technical support for the intelligent and refined upgrading of industrial safety supervision.

[0022] 5. Effectively reduces the false alarm rate caused by compliance rectification: Traditional methods are prone to misreporting reasonable scenario changes as anomalies. This invention, through semantic understanding using a large language model, can accurately identify rectification behaviors and make correct judgments, thereby significantly reducing the false alarm rate while ensuring the detection rate and improving the practical value of the system. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 This is a flowchart illustrating a method for matching industrial safety scenarios according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of the sub-steps for performing multi-dimensional visual matching quantization calculation in an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of the sub-steps for performing structured semantic reasoning and comprehensive judgment in an embodiment of the present invention;

[0027] Figure 4 This is a before-and-after comparison diagram of the rectification of the specific case "electrical cable passageway without protective cover" used in Embodiment 1 of the present invention.

[0028] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0030] Terminology Explanation:

[0031] Gaussian scale space: refers to a set of images generated by convolving the input image with a series of Gaussian filters of increasing scale (degree of blur). The scale is controlled by the standard deviation parameter; the larger the value, the more blurred the image, representing macroscopic structure; the smaller the value, the clearer the image, representing microscopic details. This space is used to simulate the multi-scale characteristics of images and is the foundation for achieving scale-invariant feature extraction.

[0032] Gaussian difference scale space: refers to a set of images obtained by calculating the difference between two adjacent images of different scales in the Gaussian scale space. This operation approximates the calculation of the scale-normalized Laplacian response of the image, and can efficiently and stably detect the locations of potential keypoints that are not significantly affected by scale and rotation changes.

[0033] Keypoint Precision Localization and Screening: This refers to the process of refining the initially detected candidate keypoints. First, their positions and scales are corrected to sub-pixel accuracy using mathematical methods. Then, points with too low contrast (sensitive to noise) or located at the edges (unstable with changes in viewing angle) are removed, ultimately retaining a set of robust and stable feature points.

[0034] Nearest Neighbor Ratio Criterion: A screening method used to determine the reliability of a match between two feature descriptors. When searching for a match for a feature point, the two candidate points with the closest and second-closest distances are found. If the ratio of the nearest distance to the second-closest distance is less than a certain empirical threshold (such as 0.7), the nearest neighbor match is considered reliable; otherwise, it may be a fuzzy match or a mismatch and should be discarded.

[0035] Random Sample Consensus (RSC) is a robust model parameter estimation algorithm used to estimate the best mathematical model from a dataset containing a large number of erroneous data (outliers). Its basic idea is to fit the model by repeatedly randomly sampling the smallest subset of data and counting how many data points (called "interiors") agree with the model. Ultimately, the model with the largest number of supporting interiors is selected as the optimal estimate.

[0036] Symmetric transfer error: After estimating the geometric transformation model between two images, this error metric measures how well a pair of matching points conforms to the model. It simultaneously calculates the reprojection error from point A to point B through the model transformation, and the reprojection error from point B to point A through the inverse model transformation, and sums the two. This two-way measurement is more rigorous than unidirectional error and can better assess geometric consistency.

[0037] Structural similarity index: A metric used to measure the overall perceptual similarity between two images. It comprehensively compares images from three dimensions: brightness, contrast, and structural information, which is more in line with the judgment methods of the human visual system. Its value ranges from -1 to 1, with the closer the value is to 1, the more similar the two images are.

[0038] Weighted fusion: This invention employs a strategy that combines multiple visual matching metrics (feature matching ratio, inlier ratio, structural similarity index) across different dimensions into a single score (comprehensive confidence level). A preset weighting coefficient is assigned to each metric, and then their weighted average is calculated. This score comprehensively reflects the overall quality of the matching results in terms of local features, geometric consistency, and overall structure.

[0039] Example 1

[0040] This embodiment provides a method for matching industrial safety scenarios. This embodiment combines... Figure 4The specific case shown fully illustrates the implementation method of the present invention, "A Method for Matching Industrial Safety Scenarios." This case selects a set of before-and-after images of electrical wiring corridors in an industrial workshop for matching analysis to verify the effectiveness of the method of the present invention.

[0041] 1. Before rectification: The industrial workshop floor has exposed electrical wires laid directly on the ground without any protective measures, posing a significant safety hazard.

[0042] 2. Image after rectification: On the same workshop floor, the wires are now protected by metal protective covers.

[0043] 3. Hazard description: The electrical wiring passageway is not covered by protective covers.

[0044] An embodiment of the present invention provides a method for matching industrial safety scenarios, such as... Figure 1 As shown, it includes the following steps:

[0045] S1. Standardized hazard-rectification triplet data is constructed through the dataset preprocessing module. The triplet includes a hazard image before rectification, a standardized hazard description text, and a hazard image after rectification.

[0046] A high-quality, standardized dataset of industrial safety hazards was constructed to provide reliable and structured input for subsequent image feature matching and large language model analysis. The dataset preprocessing module strictly followed a standardized process of "expert photography - professional annotation - rectification verification" for data collection and construction.

[0047] First, experts in the field of production safety systematically photograph the potential hazards at various industrial sites (such as workshops, warehouses, and power distribution rooms) to ensure clear images and complete perspectives, thereby obtaining images of the hazard status before rectification.

[0048] Subsequently, the expert team, based on national and industry safety production standards and specifications, professionally analyzed and annotated each image of potential hazards before rectification, generating standardized hazard description text. This description text is not freely written but uses a standardized structure to ensure unambiguous information and effective machine parsing. A standardized hazard description typically includes the following core fields:

[0049] Hazard Category: Indicate the professional field to which the hazard belongs, such as "electrical safety", "mechanical equipment", "fire protection facilities" or "work environment", etc.

[0050] Detailed description: Objectively and in detail describe the on-site condition, location, and specific manifestations of the potential hazard.

[0051] Violation of regulations: Citing the specific safety standard clauses or internal company regulations that are violated to provide a legal basis for hazard identification.

[0052] Rectification requirements: Clearly and specifically list the rectification measures that must be taken and the qualified status that should be achieved. This is the direct criterion for subsequent verification of whether the rectified images are compliant.

[0053] Finally, after completing on-site rectification based on the aforementioned hazard description, the enterprise needs to take photos from the same location and perspective and upload the rectified images. The system then binds the pre-rectification hazard image, the standardized hazard description text, and the rectified image to form a complete, standardized triple (pre-rectification hazard image, description, rectified image), serving as a basic data unit for this method. The dataset constructed through this process covers major hazard categories such as electrical safety, mechanical equipment, fire protection facilities, and the working environment, ensuring the comprehensiveness of subsequent intelligent detection system training and its practicality in real-world applications.

[0054] In this embodiment of the invention, for Figure 4 The standardized triplet data generated and output by the dataset preprocessing module for the "electrical cable passageway not covered by protective covers" case shown is as follows:

[0055] Image showing potential hazards before rectification: An image showing exposed electrical wires laid on the floor of an industrial workshop.

[0056] Standardized hazard description text: This can be a structured JSON object, with key fields containing, for example: { Hazard Category: Electrical Safety Detailed description: "Black cables were laid directly exposed on the floor of the main passageway in the workshop, without any protection." "Violation of regulations": "Violates Article 7.1.2 of the 'Low-Voltage Power Distribution Design Code' GB 50054-2011, which requires cable laying to be protected." "Rectification Requirements": "Install metal protective covers to completely cover exposed cables." }

[0057] Image after rectification: An image showing the same location where the wires have been completely covered by a metal protective cover.

[0058] This triplet structure is clear, and the standardized description of potential hazards, especially the "rectification requirements" field, provides a precise and unambiguous semantic basis for the large language model analysis module in the subsequent step S3 to understand "what constitutes qualified rectification".

[0059] S2, calculate and fuse multi-dimensional visual matching quantitative indicators for the image of hidden danger before rectification and the image after rectification, and generate a structured matching quantitative result including feature matching ratio, inlier ratio, structural similarity index and comprehensive confidence.

[0060] This step receives the pre-rectification hazard image and the post-rectification image from step S1. Its core function is to accurately quantify the consistency between the pre-rectification hazard image and the post-rectification image in local features, geometric structure, and overall appearance through an algorithmic pipeline containing multiple consecutive sub-steps, and then fuse them to generate a comprehensive matching confidence score. Figure 2 As shown, the specific execution logic includes the following five sub-steps:

[0061] S21, extract key points and their feature descriptors with scale and rotation invariance from the input image of the hidden danger before rectification and the image after rectification.

[0062] Key points and their feature descriptors that are scale- and rotation-invariant are extracted from the input images of potential hazards before and after rectification. This forms the basis for subsequent accurate matching. Scale invariance is achieved by ensuring that the same physical point can still be stably detected even when it appears as different sizes in the image due to different shooting distances.

[0063] This step can be implemented using the SIFT algorithm, which systematically simulates the multi-scale features of an image by constructing a Gaussian scale space. Its core lies in applying a Gaussian blur to the image using a series of increasing standard deviations σ, each σ value corresponding to a specific observation "scale". The larger the σ value, the blurrier the image, and the coarser and more coarse the visible image structure; the smaller the σ value, the sharper the image, and the finer the details. By generating this series of images from sharp to blurry, the algorithm can search for stable feature points at different scale levels.

[0064] Specifically, for the input image I(x,y) Its Gaussian scale space L(x,y,σ) The image is coupled with a variable-scale Gaussian kernel. G(x,y,σ) Convolution yields:

[0065] in, The parameter σ is the scaling factor.

[0066] To efficiently detect keypoint locations that remain stable under scale changes, the algorithm does not directly search for extrema in the Gaussian scale space. Instead, it constructs a Difference of Gaussian (DoG) scale space. The DoG image is obtained by subtracting Gaussian images at adjacent scales (separated by a fixed multiple k, e.g., k=23).

[0067] D(x,y,σ) This represents the coordinates located in the image coordinates within the Difference of Gaussians (DoG) scale space. (x,y) Location, scale σ The pixel values ​​on this layer.

[0068] Subsequently, extremum detection is performed in the three-dimensional difference-of-Gaussian (DoG) scale space (i.e., two image dimensions x, y and one scale dimension σ). Each pixel is compared with its eight neighboring pixels at the same scale level (the DoG image), and with its corresponding nine neighboring pixels at the two adjacent scale levels above and below (more blurred and sharper) (a total of 26 neighbors). Only when the pixel's DoG response value is the maximum or minimum among all 26 of its neighbors is it initially labeled as a candidate keypoint.

[0069] To obtain the sub-pixel-level precise location and scale of keypoints and to filter out low-contrast points or edge response points, the algorithm... D(x,y,σ) At candidate point (x0,y0,σ0) Perform a three-dimensional Taylor expansion at this point:

[0070] in, This is the offset vector relative to the candidate point. The precise offset can be obtained by solving for the extreme points of this function. x^ This allows for the correction of the key point's position. (x0+x^,y0+y^) The scale is corrected to σ0+σ^. At the same time, if the function value |D(x^)| at this extreme point is too small (below the threshold, such as 0.03), it is considered to have insufficient contrast and is removed. In addition, by analyzing the principal curvature of the Hessian matrix of the DoG response at key points, unstable points located at the edges can be removed.

[0071] Finally, for each key point that passes the above refinement and selection, a feature descriptor with scale, rotation, and illumination robustness is generated. This descriptor is essentially a 128-dimensional feature vector, constructed following a standardized process:

[0072] 1. Determine the principal direction and rotation normalization: in the Gaussian scale space where the key points are located. L(x,y,σ) First, the gradient directions of all pixels in the neighborhood are calculated, and the peak direction of the statistical histogram is defined as the principal direction of the keypoint. Then, the coordinate system of the entire neighborhood is rotated to align with the principal direction. This step ensures that the subsequently extracted descriptors are rotation-invariant.

[0073] 2. Divide the local spatial grid: Within the neighborhood of the rotated and aligned keypoint (usually a 16×16 pixel area), divide it evenly into a 4×4 grid to obtain 16 spatial sub-regions.

[0074] 3. Calculate the gradient orientation histogram for each sub-region: For each 4×4 pixel sub-region, calculate the gradient magnitude and direction of all pixels within it. The 360° directional range is uniformly quantized into 8 directional intervals (each interval being 45°), and the gradient magnitude of each pixel is weighted and accumulated into the corresponding directional interval. This generates an 8-dimensional gradient orientation histogram for each sub-region.

[0075] 4. Concatenation and Normalization to Form the Final Vector: The 8-dimensional histograms generated from the 16 sub-regions are concatenated sequentially according to their spatial position to form an initial feature vector of 16 × 8 = 128 dimensions. Finally, this vector is normalized to further improve its stability to changes in illumination intensity. This normalized 128-dimensional vector is defined as the final feature descriptor for this keypoint.

[0076] Through the structured generation process described above, the resulting feature descriptors are not only invariant to image scaling and rotation, but also exhibit good robustness to a certain degree of affine deformation and illumination changes, laying the foundation for subsequent high-reliability feature matching.

[0077] Therefore, for each set of input images showing potential hazards before and after rectification, the output of this sub-step is a set of refined key points and their associated information. This set of information includes the coordinates of each key point. (x,y) ,scale σ The key points are the orientation and their corresponding feature descriptors. These descriptors together constitute a set of descriptors representing the visual features of the image, which will serve as the direct input to the FLANN matcher; while the coordinate information of the key points will be combined with the matching results as the basic input for the geometric verification of the RANSAC algorithm.

[0078] In this embodiment of the invention, for Figure 4 The example shown, "Electric wire passageway without protective covers," demonstrates SIFT feature extraction performed on both the pre-rectification and post-rectification images of the electric wire passageway. The pre-rectification image yielded 1523 stable keypoints and their corresponding feature descriptors; the post-rectification image yielded 1387 keypoints and descriptors. These descriptor sets form the basis for subsequent matching.

[0079] S22, Match the feature descriptor sets of the image before rectification and the image after rectification, and use the nearest neighbor distance ratio criterion to filter them to obtain an initial feature matching point set.

[0080] This sub-step matches the feature descriptor sets extracted in step S21 for the image of the hidden danger before rectification and the image after rectification, respectively. This step can accelerate the search process in high-dimensional space (128 dimensions in this embodiment of the invention) by using the Fast Approximate Nearest Neighbor Search Library (FLANN).

[0081] FLANN accelerates the search by constructing multiple randomized KD-tree indexes. In the configuration of this invention, the index parameter trees=5 (constructing 5 KD-trees) and the search depth parameter checks=50 are set for each query point to achieve high search efficiency within an acceptable approximation error.

[0082] For each feature descriptor in the image of potential hazards before rectification, it is regarded as a query vector in a high-dimensional feature space. q FLANN uses a high-dimensional vector dataset composed of feature descriptors from the reworked images. p 1 ,p 2 ,…,p N}, each of which p i ( 1 ≤i≤N This corresponds to the 128-dimensional feature vector of a specific key point in the image after rectification.

[0083] In the high-dimensional vector dataset, quickly find its nearest neighbor. p nn1 (i.e., the high-dimensional vector dataset and) q The vector with the smallest distance and its second nearest neighbor. p nn2 (That is, the vector in the high-dimensional vector dataset with the second smallest distance to q). To eliminate a large number of incorrect matches from the initial matching (e.g., due to texture duplication, noise, or excessive viewpoint variation), each of... p i (i = 1, 2, …, N) is a multidimensional real vector (e.g., 128-dimensional when using the SIFT algorithm) that encodes invariant visual features such as scale, rotation, and brightness of a local key point (such as a corner or edge) in the reworked image.

[0084] The nearest neighbor distance ratio (NNDR) criterion is used for screening: only if the nearest neighbor distance d 1 =‖ q - p nn1 ‖ Distance d to the next nearest neighbor 2 =‖ q - p nn2 The ratio of || r=d 1 / d2 Acceptance is only granted when the value is less than a preset empirical threshold (typically set to 0.7 to 0.8, 0.7 is used in this embodiment of the invention). q,p nn1 A valid match is defined as a pair of points M. This threshold is based on empirical observation: a correct match typically has a significantly closer nearest neighbor, while a vague or incorrect match often has two candidates that are close to each other. The final output is a filtered set of initial feature matching points M.

[0085] In this embodiment of the invention, for Figure 4 In the case of "electrical cable passageway not covered by protective cover", the FLANN matcher performs matching and NNDR filtering on the two sets of descriptors to obtain the initial set of matching pairs. M Its quantity | M |=145. This set of matching pairs will be used as input to the RANSAC algorithm.

[0086] S23, robustly estimate the homography matrix from the initial feature matching point set that describes the optimal planar projection transformation relationship between the image of hidden dangers before rectification and the image after rectification, and calculate the inlier ratio.

[0087] The initial feature matching point set output from step S22, which may contain mismatches. M In this process, the homography matrix describing the optimal planar projection transformation relationship between the image of the hidden danger before rectification and the image after rectification is robustly estimated. H This matrix is ​​the core of determining the geometric consistency between two graphs and provides the basis for calculating the scale of interior points.

[0088] To achieve this goal, this method can employ the Random Sampling Consensus (RANSAC) algorithm.

[0089] First, set a reprojection error threshold. δ (For example δ =5.0 pixels), this threshold defines the range of geometric errors within which a pair of matching points can still be considered to conform to the current estimate of the transformation model.

[0090] The core of the RANSAC algorithm is to find a homography matrix that maximizes the set of interior points. The optimization objective can be expressed as:

[0091] in, I(H) Indicates transformation H Under these conditions, all reprojection errors are less than the threshold. δ The set of matching point pairs is called the interior point set. Its definition is as follows:

[0092] in, M The set of all matching point pairs. r i ( H ) is the first i Reprojection error of matching points.

[0093] For a pair of matching points ( p i , p’ i ),in p i = ( x i , y i , 1 ) T The coordinates of the points in the image of the potential hazards before rectification are homogeneous. p’ i = ( x’ i , y’ i , 1 ) T The reprojection error of the corresponding point in the corrected image is calculated using the symmetric transfer error formula, which represents the homogeneous coordinates of that point.

[0094] Where the function π ([ u , v , w ]) T = ( u / w , v / v ) T This is a transformation from homogeneous coordinates to Euclidean coordinates. The symmetric transfer error formula simultaneously measures the point pair in the positive direction (…). H ) and reverse ( H -1 The projection consistency under transformation is more stringent than that of unidirectional error. Specifically, in this formula: HP i and H -1 p’ i Both are homogeneous coordinate vectors; π ( HP i )and π ( H-1 p’ The numbers 1 and 2 are Euclidean coordinates after the forward and reverse transformations, respectively.

[0095] The algorithm iteratively executes the following process: Random sampling: Randomly select 4 pairs of non-collinear matching points from the initial feature matching point set M (the minimum sample set required to solve the homography matrix).

[0096] Model computation: Using the Direct Linear Transformation (DLT) algorithm, a candidate homography matrix is ​​calculated based on these four point pairs. H tmp .

[0097] Interior point statistics: Calculate the interior point statistics for all matching point pairs. H tmp Symmetrical transfer error r i Statistical satisfaction r i (H) < δ Number of interior points .

[0098] Model update: If the number of inliers corresponding to the current candidate model is greater than the number of inliers of the historical best model, i.e. Then update the optimal model. And record its interior point set. .

[0099] Iteration Termination: The number of iterations in RANSAC N It is not fixed, but dynamically calculated based on the confidence probability p (usually set to 0.99) and the currently estimated outlier rate ε, to ensure that the probability of at least one sample consisting entirely of inliers is sufficiently high. The number of iterations is determined by the following formula:

[0100] in p = 0.99 is the confidence probability, and ε is the currently estimated outlier rate. The final output is determined by the maximum inlier set. Supported homography matrix estimation .

[0101] The algorithm reaches the number of iterations for dynamic calculation. N Then it terminated.

[0102] After the iteration is complete, the set of maximum interior points will be obtained. Supported optimal homography matrix estimation Interior point ratio R inlier The calculation is as follows: R inlier =∣ I ||M | This ratio intuitively reflects the proportion of correctly matched point pairs after geometric verification, and is a key indicator for measuring the geometric consistency and robustness of the matching results.

[0103] In this embodiment of the invention, for Figure 4 The example shown, "Electric wire passageway not covered by protective cover", yields the inlier ratio R after RANSAC estimation of the initial matching set of the electric wire passageway image pairs. inlier =0.875.

[0104] S24, calculate the structural similarity index between the image of the hidden danger before rectification and the image after rectification.

[0105] This sub-step can evaluate the overall consistency in brightness, contrast, and structural information between the pre-rectification and post-rectification images by calculating the Structural Similarity Index (SSIM), thus compensating for the locality of sparse feature point matching. The SSIM calculation is performed within a local sliding window to capture the similarity of local image structures. For image patches... x and y Its SSIM value is calculated as follows:

[0106] in: μ x , μ y These are image blocks x , y The pixel mean represents the brightness estimate.

[0107] σ x , σ y These are image blocks x , y The standard deviation represents the contrast estimate.

[0108] σ xy It is an image block x , y The covariance represents structural similarity.

[0109] C 1 , C 2 These are two constants used to avoid unstable calculation results when the denominator approaches zero. They are usually set as... C 1 =( k1 L ) 2 , C 2 =( k 2 L ) 2 .here, L It is the dynamic range of pixel values ​​(e.g., for an 8-bit grayscale image). L =255), k1 and k2 are constants much less than 1 (usually taken as 0.01 and 0.03 respectively).

[0110] In practical calculations, a fixed-size window (e.g., 11×11 pixels) is used as a local image patch, and the overlapping regions of the entire image are traversed with a step size (e.g., 1 pixel). A local SSIM value is calculated at each window location. After traversal, all local SSIM values ​​are averaged (or Gaussian weighted average can be used to give higher weight to the central region), ultimately yielding a global structural similarity index. The theoretical range of this structural similarity index is [-1, 1], where 1 represents two images that are completely identical.

[0111] In this embodiment of the invention, for Figure 4 The case of "electrical cable passageway without protective covers" was illustrated. The global structural similarity index was calculated to be 0.317 by comparing the images of the passageway before and after rectification. This result indicates that there are certain differences in the overall structure and appearance between the images before and after rectification, which is consistent with the fact that the addition of metal covers alters the visual appearance of the scene.

[0112] S25, the feature matching ratio, the inlier ratio and the structural similarity index are weighted and fused to generate the comprehensive confidence score, and the four indicators are encapsulated into a structured data object for output.

[0113] The underlying algorithm outputs of the preceding image matching sub-steps (steps S21 to S24) are transformed into a set of structured, quantifiable, and interpretable evaluation metrics, which are then fused to generate an overall matching quality score, providing intuitive and reliable input for subsequent high-level semantic decisions. Its execution logic comprises three core components: metric quantification, weighted fusion, and structured encapsulation.

[0114] First, the output of the preceding steps is standardized and quantified to generate three core evaluation metrics: 1. Feature matching ratio: Based on the set of effective matching pairs obtained in step S22 after FLANN matching and nearest neighbor distance ratio (NNDR) filtering. M The calculation method is the number of valid matching pairs | M| is the ratio of the smaller total number of SIFT keypoints extracted from the image before rectification and the image after rectification. This index takes a value in the range [0,1] and is used to measure the basic similarity and matching density between the image before rectification and the image after rectification at the local feature level.

[0115] 2. Interior point ratio: Based on the optimal homography matrix obtained by robust estimation using the RANSAC algorithm in step S23. and its corresponding interior point set I Its calculation method is based on the number of interior points. I Compared with the initial total number of matching pairs M The ratio of the two values. This index takes values ​​in the range [0,1] and is used to evaluate the geometric consistency of the matching results and the robustness of the algorithm. It effectively reflects the proportion of correct matches after the incorrect matches are removed.

[0116] 3. Structural Similarity Index: The global mean structural similarity (SSIM) score calculated in step S24 is directly used as the index. This index takes values ​​in the range [0,1] (theoretically [-1,1], but in practice, non-negative cases are more common), and is used to comprehensively evaluate the consistency between the image of the hidden danger before rectification and the image after rectification in terms of brightness, contrast, and overall structural information from a perceptual perspective.

[0117] Secondly, to provide a holistic quality assessment, a weighted fusion strategy is adopted to integrate the above three indicators into a single "comprehensive confidence" score. The weighted fusion calculation formula is the weighted arithmetic mean: Overall confidence score = (ω1 × feature matching ratio + ω2 × inlier ratio + ω3 × structural similarity index) / (ω1 + ω2 + ω3), Wherein, ω1, ω2, ω3 are preset weight coefficients.

[0118] ω1, ω2, and ω3 are used to adjust the importance of different dimensional indicators in the final evaluation. This formula compresses multi-dimensional matching information into a scalar score in the [0,1] interval, making it easier to set a uniform threshold for initial decision-making.

[0119] Finally, the quantification results are organized into a structured semantic description output. The output of this module is a data object containing four core fields, which realizes the transformation from low-level visual signals to high-level interpretable data:

[0120] 1. Feature matching ratio: A floating-point number in the range [0,1], reflecting the density of stable feature points matching between the image of potential hazards before rectification and the image after rectification.

[0121] 2. Interior point ratio: A floating-point number in the range [0,1], representing the proportion of correctly matched point pairs after robust geometric verification.

[0122] 3. Structural similarity index: a floating-point number in the range [0,1], which measures the overall consistency between the image of the hidden danger before rectification and the image after rectification in terms of brightness, contrast and structural information.

[0123] 4. Overall confidence score: A floating-point number in the range [0,1], which is the overall matching quality assessment score obtained by weighted fusion of the first three indicators.

[0124] In a preferred embodiment, a higher weight is given to the feature matching ratio (e.g., ω1=2) to emphasize the fundamental role of local feature matching; ω2 and ω3 can be set to 1 to balance the influence of geometric and structural information.

[0125] In this embodiment of the invention, for Figure 4 The example shown, "Electric wire passageway not covered by protective cover", is calculated and fused based on the output data from the previous steps:

[0126] From step S21: The total number of key points in the image of the hidden danger before rectification was 1523, and the total number of key points in the image after rectification was 1387.

[0127] From step S22: 145 valid initial matching pairs.

[0128] From step S23: The size of the interior point set is 121.

[0129] From step S24: Structural similarity index 0.317.

[0130] First, calculate the three core metrics:

[0131] Feature matching ratio = 145 / min(1523,1387) = 145 / 1387≈0.105.

[0132] The ratio of interior points is approximately 121 / 145 ≈ 0.875.

[0133] Structural similarity index = 0.317.

[0134] Subsequently, based on the preset weights ω 1 = 2, ω 2 = 1, calculate the overall confidence level: (2×0.105+1×0.875+1×0.317) / (2+1+1)=0.351.

[0135] The final output of the structured matching quantization result is: {"Feature Matching Ratio": 0.105, "Interior Point Ratio": 0.875, "Structural Similarity Index": 0.317, "Overall Confidence": 0.351}. This structured matching quantization result will serve as one of the core inputs for step S3.

[0136] S3, receive the structured matching quantification result and the standardized hidden danger description text, perform structured semantic reasoning and comprehensive judgment, and output the scene subject matching degree analysis result containing the matching level and reason.

[0137] It receives the structured matching quantification results from step S2 (including feature matching ratio, inlier ratio, structural similarity index, and overall confidence score) and the standardized hazard description text from step S1. Its core function is to guide a general-purpose large language model to simulate domain experts through carefully designed structured prompt word templates, performing multi-step, interpretable reasoning based on visual quantification evidence and safety hazard specifications, ultimately outputting a structured judgment result regarding the matching degree between the scene and the subject. The specific execution logic is as follows... Figure 3 As shown, it includes the following three sub-steps:

[0138] S31, based on preset rules, makes a preliminary judgment on the overall confidence level of the input to trigger different analysis modes, and loads the predefined five-level scene and subject matching degree judgment standard.

[0139] The system performs a preliminary assessment of the overall confidence level of the input based on preset rules, establishing a clear evaluation benchmark for the entire semantic reasoning process. First, the system sets an overall confidence level threshold (e.g., 0.4). If the overall confidence level is higher than or equal to this threshold, it is preliminarily considered that the image of the potential hazard before rectification and the image after rectification may belong to the same scene, and subsequent reasoning focuses on distinguishing them at a finer level. If the overall confidence level is lower than this threshold (e.g., 0.351 in this embodiment), a deep semantic analysis mode is triggered, focusing on determining whether the low matching degree is caused by compliance rectification.

[0140] At the same time, the system will use the clearly defined five-level scene and subject matching criteria (AE level) as the benchmark for the entire evaluation, as follows:

[0141] Grade A: The main subject of the scene is consistent, and the main subject of the scene and the core background are consistent.

[0142] Grade B: The main subject of the scene remains the same, but the background changes.

[0143] Grade C: The main subject of the scene is the same, but the angles are quite different.

[0144] Level D: The main subject of the scene is inconsistent.

[0145] Grade E: Cannot be determined.

[0146] This step, by combining threshold decision logic with refined judgment criteria, sets a clear task framework and evaluation basis for the subsequent targeted reasoning of the large language model.

[0147] S32, dynamically construct structured prompt words by filling the prompt word template with the structured matching quantification results, the standardized hazard description text, the decision diversion logic and judgment criteria, to form structured prompt words.

[0148] This sub-step is a crucial bridge connecting the visual quantification results and the semantic reasoning engine. Its function is to dynamically construct a precise, structured prompt word rich in domain knowledge by filling the prompt word template with the structured matching quantification results, the standardized hazard description text, the decision-making triage logic, and the judgment criteria. This prompt word, along with relevant data, is then submitted to the large language model. This process is specifically implemented through the following operations:

[0149] 1. Data Preparation and Assembly: The system first obtains input data from the upstream module, including: the structured matching quantification results (dictionary form) from step S25, such as {"Feature Matching Ratio": 0.105, "Interior Point Ratio": 0.875, "Structural Similarity Index": 0.317, "Overall Confidence": 0.351}, and the standardized hazard description text from step S1. These data are then parsed into internal variables.

[0150] 2. Dynamic Filling of Structured Prompt Template: The system maintains a predefined prompt template. The prompt template is a string containing placeholders. In this step, the system dynamically fills the corresponding positions in the template with the input data mentioned above, as well as the judgment logic and criteria inherited from step S31, generating a complete prompt for the current task. A simplified example of the core template structure is as follows:

[0151] “ You are a professional expert in industrial safety scenario and subject matching analysis;

Task and Input

Analysis steps that must be followed

Analysis Principles

[0152] Placeholders such as {quantitative_metrics}, {hazard_description}, {composite_confidence}, and {confidence_condition} will be replaced with actual data. Detailed explanations of each placeholder are as follows:

[0153] 1) {quantitative_metrics}: The final output of the image feature matching module (step S2), which is a structured data object (usually represented as a dictionary or JSON format) containing four core fields. Its specific content is as follows:

[0154] Feature matching ratio: a floating-point number, such as 0.105.

[0155] Interior point ratio: a floating-point number, such as 0.875.

[0156] Structural similarity index: a floating-point number, such as 0.317.

[0157] Overall confidence level: a floating-point number, such as 0.351.

[0158] {quantitative_metrics} objectively presents the quantitative analysis results of the underlying visual algorithm to the large language model, serving as one of the objective factual bases for its high-level semantic reasoning.

[0159] In this embodiment of the invention, for Figure 4 In the example shown, "Electric wire passageway without protective cover", this placeholder will be replaced with the string "Feature matching ratio: 0.105, Inner point ratio: 0.875, Structural similarity index: 0.317, Overall confidence: 0.351".

[0160] 2) {hazard_description}: The standardized hazard description text output by the dataset preprocessing module (step S1) contains a structured text description that clarifies the hazard category, specific phenomenon, violated regulations, and rectification requirements.

[0161] {hazard_description} provides the large language model with the specific professional domain rules and expected goals (i.e., "what needs to be rectified") for this analysis task, and serves as the benchmark for the large language model to "extract core requirements" and "evaluate the rectification effect".

[0162] In this embodiment of the invention, for Figure 4 The placeholder text shown in the example of "Electric wire passageway not covered by protective cover" will be replaced with a standardized description of the potential hazard of "Electric wire passageway not covered by protective cover".

[0163] 3) {composite_confidence}: This is also the output of the image feature matching module (step S2), and is the value of the comprehensive confidence in {quantitative_metrics}.

[0164] {quantitative_metrics} serves as the input value for the threshold judgment logic, directly triggering different inference path guidance. It is a key control variable connecting the visual quantification results and the semantic reasoning process.

[0165] In this embodiment of the invention, for Figure 4 For the example shown, "Electric wires are not covered by protective covers", directly enter the value 0.351.

[0166] 4) {confidence_condition}: A text command dynamically generated by the system after logically comparing the value of {composite_confidence} with a preset threshold (such as 0.4).

[0167] If {composite_confidence} is greater than or equal to the threshold, the generated text is: "This value reaches or exceeds the threshold, and it can be preliminarily considered to belong to the same scene. Please further determine the specific level (A, B, or C) based on the quantization results and image semantics."

[0168] If {composite_confidence} < threshold, the generated text reads: "This value is below the threshold, indicating a low degree of feature matching between images. You must conduct in-depth analysis combining image semantic information, focusing on determining whether this low matching degree is due to a reasonable change in the scene caused by the completion of hazard rectification, and give a level judgment accordingly."

[0169] This is the core embodiment of the "threshold-guided decision logic" in this invention. It is not a simple data replacement, but a logical judgment result containing business rules, used to clearly inform the large language model which analysis mode it should enter, thereby constraining and guiding its reasoning direction.

[0170] In this embodiment of the invention, for Figure 4 In the case of "electrical cable passageway without protective covers", the placeholder {confidence_condition} is replaced with "This value is below the threshold, indicating a low degree of feature matching between images. You must conduct in-depth analysis in conjunction with image semantic information, focusing on whether this low matching degree has caused a reasonable change in the scene due to the completion of the hazard rectification, and give a level judgment accordingly." because the overall confidence level is 0.351 < 0.4.

[0171] Through the precise replacement of each placeholder, a general prompt template is instantiated into a highly customized professional analysis instruction targeting specific hazards (such as electrical wiring corridors), incorporating specific visual evidence (such as quantitative indicators 0.105, 0.875, 0.317, 0.351), and a specific decision-making path. This ensures that the large language model can reason within a clear context and rules every time it is invoked.

[0172] 3. Call the Large Language Model API: Submit the assembled complete prompt text through the Application Programming Interface (API) provided by a large language model service provider (such as OpenAI, Anthropic, or a locally deployed LLM service). When calling, you usually need to specify the model name (such as gpt-4-vision-preview, claude-3-opus, etc.) and necessary parameters (such as setting the temperature to 0 to enhance determinism).

[0173] 4. Process and validate the response: Receive the text response returned by the large language model. The system will first attempt to parse it into a JSON object. Then, it will validate whether the JSON object contains the required fields (such as grade, reason), and whether the value of the grade field is within the preset range ["A", "B", "C", "D", "E"]. If format or content validation fails, the system can be configured with a retry mechanism or trigger an exception handling process.

[0174] In this embodiment of the invention, for Figure 4 The specific steps for handling the "electrical wiring passageway not covered by protective covers" case shown are as follows:

[0175] 1) Data assembly: The quantification result dictionary and hazard description text are accurately read.

[0176] 2) Template Filling: Fill the template with the overall confidence score of 0.351, the dictionary of quantitative indicators, and the specific text describing the potential hazard. Since 0.351 < 0.4, {confidence_condition} is set to "This value is below the threshold, indicating a low degree of feature matching between images. You must conduct in-depth analysis in conjunction with the semantic information of the images, focusing on determining whether this low matching degree has caused a reasonable change in the scene due to the completion of the hazard rectification, and give a level judgment accordingly."

[0177] API call: The generated complete prompt word of hundreds of words is sent to the configured large language model API endpoint via an HTTP request.

[0178] Response reception: Wait for and receive the text response returned by the API, and prepare it for processing in step S33.

[0179] Through the aforementioned mechanism, this sub-step dynamically transforms structured visual quantitative data and professional safety specification text into a set of executable structured instructions. These instructions precisely constrain and guide the reasoning process of the large language model, thereby ensuring the professionalism of its output and the traceability of its decision-making logic, thus completing a reliable mapping from low-level perceptual data to high-level domain decisions.

[0180] S33, Submit the structured prompt words to the large language model, receive and parse the returned response, and obtain the scene subject matching degree analysis result.

[0181] In this step, the large language model serves as the core inference engine. Based on the structured prompts rich in domain knowledge and task constraints constructed in step S32, it initiates its multimodal understanding and logical reasoning process. This process strictly follows a preset framework, deeply integrating and progressively analyzing underlying visual quantitative data, expert knowledge text, and image semantic content to ultimately generate a structured judgment result, namely, the scene and subject matching degree analysis result.

[0182] Specifically, the model first analyzes the input: clarifying its role as an "industrial safety scenario matching expert"; receiving and understanding the structured matching quantification results and their business implications output in step S2 (especially the key signal that the overall confidence level of 0.351 is lower than the threshold of 0.4); and accurately analyzing the professional requirements and rectification goals contained in the standardized hazard description.

[0183] Subsequently, the model strictly followed the "four-level reasoning steps" defined in the prompts, and based on its understanding of the relevant image content, performed the following progressive analysis:

[0184] 1. Target Scene Comparison: Analyze whether the image of the hidden danger before rectification and the image after rectification describe the same core target scene.

[0185] For the input images of potential hazards before rectification and images of hazard after rectification, determine whether the images of potential hazards before rectification and images of hazard after rectification capture the same core physical scene, such as "industrial workshop floor".

[0186] 2. Core Requirements Extraction: Accurately extract the safety requirements that must be followed from the standardized hazard description text.

[0187] From the structured description of potential hazards, the core safety regulations that must be followed are precisely extracted, such as "protective covers must be installed for electrical wiring passageways".

[0188] 3. Feature conformity check: For each extracted safety requirement, check whether the visual features of the image of the hidden danger before rectification and the image after rectification conform to each other.

[0189] The visual content of the images of potential hazards before and after rectification is compared with the aforementioned core requirements. For example, it is analyzed whether the images of potential hazards before rectification contain non-compliance items such as "exposed wires," and whether the images of potential hazards after rectification contain compliance items such as "wires being covered by metal covers."

[0190] 4. Evaluation of rectification effectiveness: Compare the changes in characteristics before and after to evaluate whether the rectification is complete and effective.

[0191] A comprehensive assessment is conducted to determine the characteristics of the shift from "non-compliant" to "compliant," such as whether corrective measures (e.g., installing a cover) have directly and completely eliminated the aforementioned hazard.

[0192] After completing the stepwise reasoning, the model will finally map the analysis conclusions to the "Level 5 Judgment Criteria (AE Level)" explicitly listed in the prompts. In this process, the model needs to comprehensively interpret the causal relationship between low-level visual evidence (such as low overall confidence) and high-level semantic understanding (compliance rectification leading to reasonable changes in appearance).

[0193] In this embodiment of the invention, for the case of "electrical cable passageway without protective covers," the specific input received by the large language model includes: quantitative indicators (overall confidence level of 0.351, etc.) and a description of the hazard. The model is guided into a deep analysis mode, and its reasoning process strictly follows the above steps. The model ultimately determines that, although the overall confidence level is low, this difference stems from the fully compliant and effective rectification action of "installing metal protective covers," and the main scene (electrical cable passageway) and the core background (workshop floor) are consistent. Therefore, according to the AE-level standard, the model outputs a matching level of A and the corresponding reasons.

[0194] The results of this embodiment highlight the core advantages of the invention. Relying solely on traditional pure visual matching methods, a comprehensive confidence level of 0.351 (below the threshold of 0.4) will directly lead to misclassification by the system as different scenarios (corresponding to level D or level E). However, this invention, by introducing a large language model analysis module, creatively constructs a dual-track collaborative architecture of "visual feature matching + semantic understanding reasoning." The system can accurately understand that the feature differences stem from the compliance rectification action of "installing protective covers," rather than a fundamental change in the scenario, thus making a correct judgment of level A matching. This strongly verifies that the invention is significantly superior to traditional methods in intelligently identifying reasonable scenario changes caused by compliance rectification, effectively reducing the false alarm rate, improving the accuracy and practicality of intelligent verification of industrial safety hazards, and providing reliable technical support for intelligent industrial safety supervision.

[0195] S4. Based on the structured matching quantification results and the scenario and subject matching degree analysis results, make a safety hazard rectification verification decision and output a structured decision report.

[0196] This step serves as the decision-making and output endpoint of this methodology. Its core function is to integrate and evaluate the evidence chain from previous steps, make a final judgment on the core safety issue of "whether the hidden danger has been effectively eliminated through rectification," and output structured decision results for execution. This step is a crucial link in realizing the value of the "visual-semantic" dual-track collaborative architecture of this invention.

[0197] Specifically, based on the objective visual matching evidence provided in step S2 and the deep semantic reasoning conclusions provided in step S3, the system executes the following decision logic:

[0198] 1. Comprehensive Evidence Assessment: The system does not consider the overall confidence level of visual matching in isolation, but rather analyzes its correlation with the matching level (AE level) of semantic judgment. The focus is on assessing whether low visual matching scores can be reasonably explained at the semantic level (e.g., compliance rectification).

[0199] 2. Security Rule Mapping: Maps the matching level of semantic determination to preset security business rules. For example, under the default rules:

[0200] A rating of A or B usually means that the main subject of the scenario is consistent and the rectification is effective, and the verification is passed.

[0201] If it is classified as Grade C, manual review may be required (due to significant differences in perspective).

[0202] A rating of D or E indicates that the subject of the scenario is inconsistent, the rectification may be ineffective or the object is wrong, and the verification will fail.

[0203] 3. Generate executable decision output: The system generates a structured decision object containing machine-readable decision conclusions, key evidence summaries, and visualized data, which can directly drive the industrial safety management system to execute subsequent processes (such as closing rectification work orders, triggering early warnings, or initiating review).

[0204] In this embodiment of the invention, for the verification task of "electrical cable passageway not covered by protective cover", the system makes a decision based on the comprehensive confidence level (0.351) output in step S2 and the A-level judgment and reasoning output in step S3:

[0205] 1. Evidence assessment: It is confirmed that the low overall confidence level (0.351) can be reasonably attributed at the semantic level to the compliance operation of "installing a protective cover".

[0206] 2. Rule Mapping: A-level judgments are directly mapped to the business rule of "effective rectification and elimination of hidden dangers".

[0207] 3. Decision Output: The system generates the final structured decision results, as shown in the example below: { "Verification Task ID": "UNIQUE_001" "Security Decisions":{ Conclusion: "The rectification was effective and the hidden dangers have been eliminated." "Suggested action": "Approved, rectification work order can be closed". Matching Rating: "A" "Key Evidence Summary":{ "Visual matching overall confidence level": 0.351 "Semantic Judgment Reason": "The difference stems from compliance rectification; the subject and background of the scenario are consistent." } } }

[0208] The decision result can be automatically pushed to the work order system to complete the intelligent closed-loop verification of hazard rectification.

[0209] This embodiment systematically demonstrates the end-to-end process of the present invention, from data input to safety decision-making, through a complete case of electrical wiring corridor renovation.

[0210] First, step S1 follows a standardized process of "expert photography - professional annotation - rectification verification" to construct structured triplet data containing standardized hazard descriptions. This ensures the professionalism and business alignment of the input information from the source, providing a reliable foundation for subsequent analysis.

[0211] Secondly, step S2 executes a complete visual computation pipeline: scale- and rotation-invariant feature extraction is achieved using the SIFT algorithm; efficient feature matching and filtering are performed using FLANN; geometric transformations are robustly estimated and interior point proportions are calculated based on the RANSAC algorithm; structural similarity is evaluated using SSIM; and finally, a weighted fusion strategy is innovatively adopted to integrate multi-dimensional indicators into a quantifiable and interpretable comprehensive confidence score (0.351 in this case). This transforms image differences into objective and structured visual evidence.

[0212] Furthermore, step S3 deeply integrates the aforementioned visual evidence with the hazard description through structured cue word engineering. It guides the large language model to act as a domain expert, rigorously executing a four-step reasoning process from "scene comparison" to "effect evaluation," and making a judgment based on a clearly defined five-level standard (AE level). In this case, the module accurately identified that the low overall confidence level (0.351) stemmed from the compliant rectification action of "installing protective covers," rather than a substantial change in the scene, thus outputting a semantic conclusion of A-level matching.

[0213] Finally, step S4 comprehensively analyzes visual evidence and semantic conclusions, mapping them to specific security business rules to generate a clear and actionable structured decision: "Rectification effective, hidden danger eliminated." This decision can directly drive the backend management system to complete the work order closure.

[0214] This case powerfully demonstrates the core advantages of this invention. Relying solely on traditional purely visual methods, a comprehensive confidence level of 0.351 (below the threshold of 0.4) will directly trigger a false alarm. However, this invention, through a dual-track collaborative architecture of "precise visual feature matching - deep domain semantic reasoning," enables the system to understand the compliance causes of low-level visual differences, thereby making correct security decisions. This not only technically solves the inherent defects of "semantic gap" and "high false alarm rate" pointed out in the background art, but also achieves a paradigm shift from passive alarms to intelligent verification at the business level, providing reliable technical support for the accurate investigation and closed-loop management of industrial safety hazards.

[0215] Example 2

[0216] This embodiment provides a system for matching industrial safety scenarios, which is used to implement a method for matching industrial safety scenarios.

[0217] The system constructs a dual-track collaborative architecture of "visual precision matching - semantic deep reasoning," which mainly includes: a dataset preprocessing module, an image feature matching module, a large language model analysis module, and a decision and output module. These modules are connected sequentially through structured data interfaces and work collaboratively.

[0218] 1. A dataset preprocessing module is used to construct standardized hazard-rectification triplet data, wherein the triplet includes a hazard image before rectification, standardized hazard description text, and a hazard image after rectification.

[0219] The dataset preprocessing module provides high-quality, structured input for the entire system. This includes:

[0220] 1) Image acquisition unit: used to acquire images of potential hazards before and after rectification taken by domain experts at the industrial site.

[0221] 2) Expert Annotation and Standardization Unit: This unit receives standardized hazard description text generated by experts based on safety regulations and processes it into a standardized structure. The standardized structure must include at least the fields of "Hazard Category," "Detailed Description," "Violation of Regulations," and "Rectification Requirements." This unit outputs the standardized hazard description text.

[0222] 3) Triplet Construction and Storage Unit: Used to bind the pre-rectification hazard image, the standardized hazard description text, and the post-rectification image to form a complete (pre-rectification hazard image, description, post-rectification image) triplet, and store it in the system database.

[0223] 2. An image feature matching module is used to process the triplet, which includes the image of the hidden danger before rectification and the image after rectification, and output a structured matching quantification result.

[0224] The image feature matching module receives image pairs from a triplet consisting of the pre-rectification hazard image and the post-rectification image from the dataset preprocessing module. Through a multi-level visual computing pipeline, it outputs structured matching quantification indicators, including:

[0225] 1) Feature extraction unit, used to extract key points and their feature descriptors with scale and rotation invariance from the input image of the hidden danger before rectification and the image after rectification.

[0226] The feature extraction unit can be implemented based on the SIFT algorithm. It performs extremum detection by constructing a Gaussian difference scale space and generates a robust descriptor for each stable key point that is robust to scale, rotation and illumination changes.

[0227] 2) Feature matching and filtering unit, used to match and initially screen the feature descriptor set of the input image before rectification and the image after rectification.

[0228] The feature matching and filtering unit can use fast approximate nearest neighbor search for matching, and filter out a reliable initial set of feature matching point pairs based on the nearest neighbor distance ratio criterion.

[0229] 3) Geometric verification and transformation estimation unit, used to robustly estimate the geometric transformation relationship between the image of hidden dangers before rectification and the image after rectification from the initial matching point set, and calculate the proportion of interior points.

[0230] The geometric verification and transformation estimation unit can be based on the RANSAC algorithm, through iterative sampling and consistency verification, to estimate the optimal homography matrix and count the number of interior points that conform to the geometric model.

[0231] 4) Structural similarity assessment unit, used to assess the overall consistency of the image of potential hazards before rectification and the image of rectification after rectification in terms of brightness, contrast and structural information.

[0232] The structural similarity assessment unit calculates the local structural similarity index through a sliding window and averages the results across the entire image to obtain a global structural similarity assessment value.

[0233] 5) The indicator fusion and structured output unit is used to integrate the outputs of the feature extraction unit, feature matching and filtering unit, geometric verification and transformation estimation unit and structural similarity evaluation unit to generate the final visual matching evidence.

[0234] The indicator fusion and structured output unit calculates the feature matching ratio and inlier ratio, and combines them with the structural similarity index, performing weighted fusion with preset weights to generate a comprehensive confidence score. Finally, this unit encapsulates the above four core indicators into a structured data object for output.

[0235] 3. The large language model analysis module is used to perform semantic reasoning based on the structured matching quantification results and the standardized hazard description text, and output the scene subject matching degree analysis results.

[0236] The large language model analysis module receives structured matching quantification results from the image feature matching module and standardized hazard description text from the dataset preprocessing module. It then guides the large language model to perform semantic reasoning and judgment using structured cue words. This includes:

[0237] 1) Decision splitting and standard initialization unit, used to perform preliminary decision splitting based on the input comprehensive confidence score.

[0238] The decision-making and standard initialization unit has a preset confidence threshold. If the score is lower than the threshold, a deep semantic analysis mode is triggered to determine whether the low matching degree is caused by compliance rectification. At the same time, this unit loads a predefined five-level scenario and subject matching degree judgment standard.

[0239] 2) A structured prompt word engine, used to dynamically construct instructions to guide large language models in expert-level reasoning.

[0240] The structured prompt engine maintains a predefined prompt template and fills it with specific matching quantitative indicators, hazard description text, decision-making and triage logic, and judgment criteria to generate a complete and task-clear prompt.

[0241] 3) Large Language Model Calling and Inference Unit, used to perform multi-step inference of the large language model and generate decisions.

[0242] The Large Language Model Call and Inference Unit submits the constructed prompt words to the Large Language Model API, receives the returned text response, parses it into structured JSON output, and verifies whether the matching level and reason contained therein meet the preset format and content requirements.

[0243] 4. Decision and output module, used to generate a safety hazard rectification verification decision report based on fused evidence.

[0244] The decision-making and output module is used for the final decision-making process regarding the verification of safety hazard rectification based on visual and semantic fusion evidence. It includes:

[0245] 1) Evidence comprehensive evaluation unit, used to correlate visual matching evidence (overall confidence level) with semantic reasoning conclusions (matching level and reasoning), assess the consistency between the two, and confirm whether low-level visual differences are reasonably explained at the high-level semantic level.

[0246] 2) The security rule mapping unit is used to map the matching level of semantic judgment to specific verification conclusions and handling suggestions based on preset business rules. For example, "Level A" is mapped to "rectification effective, hidden danger has been eliminated".

[0247] 3) A structured decision generation and output unit is used to integrate all analysis results and generate a final structured decision report. The structured decision report includes verification conclusions, handling recommendations, matching levels, and key evidence summaries, and can be directly output to the industrial safety management system to drive the closed loop of rectification work orders or trigger review processes.

[0248] After system startup, the dataset preprocessing module provides standardized triplet data. The image feature matching module performs feature extraction, matching, geometric verification, and similarity assessment on the pre-rectification and post-rectification images of potential hazards, and fuses them to generate structured quantitative indicators. The large language model analysis module receives these indicators and hazard descriptions, guides the large language model to perform multi-step reasoning through structured prompts, and outputs a semantic judgment level with justification. Finally, the decision and output module integrates visual and semantic evidence, generates the final safety verification decision based on business rules, and outputs it. The entire process achieves end-to-end automated processing from raw data to executable decisions.

[0249] Example 3

[0250] This embodiment proposes an electronic device, including: Memory, used to store computer programs; The processor is used to execute the program stored in the memory to implement the steps of the above-described method embodiment for matching industrial safety scenarios.

[0251] For details on the specific implementation of each step and related explanations, please refer to the aforementioned implementation method of industrial safety scenario matching, which will not be repeated here.

[0252] The memory of the electronic device mentioned in this embodiment may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0253] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0254] Example 4

[0255] This embodiment also proposes a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the above-described method embodiment for matching industrial safety scenarios. For details on the specific implementation and explanations of each step, please refer to the aforementioned method embodiment for matching industrial safety scenarios; further elaboration is not provided here.

[0256] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for matching industrial safety scenarios, characterized in that, include: Standardized hazard-rectification triplet data is constructed through a dataset preprocessing module. The triplet includes a hazard image before rectification, a standardized hazard description text, and a hazard image after rectification. The images of potential hazards before rectification and the images after rectification are calculated and fused with multi-dimensional visual matching quantitative indicators to generate a structured matching quantitative result that includes feature matching ratio, inlier ratio, structural similarity index and comprehensive confidence. Receive the structured matching quantification result and the standardized hidden danger description text, perform structured semantic reasoning and comprehensive judgment, and output the scene subject matching degree analysis result containing the matching level and reason; Based on the structured matching quantification results and the scene subject matching degree analysis results, a safety hazard rectification verification decision is made, and a structured decision report is output.

2. The method according to claim 1, characterized in that, The calculation and fusion of multi-dimensional visual matching quantitative indicators for the pre-rectification hazard image and the post-rectification image includes: Extract key points and their feature descriptors that are scale- and rotation-invariant from the input images of potential hazards before and after rectification; The feature descriptor sets of the image before rectification and the image after rectification are matched, and the nearest neighbor distance ratio criterion is used for filtering to obtain an initial feature matching point set; Robustly estimate the homography matrix from the initial feature matching point set to describe the optimal planar projection transformation relationship between the image of hidden dangers before rectification and the image after rectification, and calculate the inlier ratio; Calculate the structural similarity index between the image of the hidden danger before rectification and the image after rectification; The feature matching ratio, the inlier ratio, and the structural similarity index are weighted and fused to generate the comprehensive confidence score, and the four indicators are encapsulated into a structured data object for output.

3. The method according to claim 1, characterized in that, The step of receiving the structured matching quantification result and the standardized hazard description text, and performing structured semantic reasoning and comprehensive judgment, includes: Based on preset rules, a preliminary judgment is made on the overall confidence level of the input to trigger different analysis modes, and a predefined five-level scene and subject matching degree judgment standard is loaded. Dynamically construct structured prompt words by filling the prompt word template with the structured matching quantification results, the standardized hazard description text, the decision diversion logic, and the judgment criteria to form structured prompt words; The structured prompts are submitted to the large language model, and the returned responses are received and parsed to obtain the scene and subject matching degree analysis results.

4. The method according to claim 2, characterized in that, The weighted fusion calculation formula is a weighted arithmetic mean: Overall confidence score = (ω1 × feature matching ratio + ω2 × inlier ratio + ω3 × structural similarity index) / (ω1 + ω2 + ω3), Wherein, ω1, ω2, ω3 are preset weight coefficients.

5. The method according to claim 3, characterized in that, The structured prompts contain instructions for guiding the large language model to execute the following reasoning steps sequentially: Target scene comparison: Analyze whether the image of the hidden danger before rectification and the image after rectification describe the same core target scene; Key requirements extracted: Accurately extract the safety requirements that must be followed from the standardized hazard description text; Feature conformity check: For each extracted safety requirement, check whether the visual features of the image of the hidden danger before rectification and the image after rectification are consistent; Evaluation of rectification effectiveness: Compare the feature changes between the images of potential hazards before rectification and the images after rectification to evaluate whether the rectification is complete and effective.

6. A system for matching industrial safety scenarios, characterized in that, The system for implementing the method as described in any one of claims 1 to 5 includes: The dataset preprocessing module is used to construct standardized hazard-rectification triplet data, wherein the triplet includes a hazard image before rectification, standardized hazard description text, and a hazard image after rectification. The image feature matching module is used to process the image of the hidden danger before rectification and the image after rectification in the triplet and output the structured matching quantification result; The large language model analysis module is used to perform semantic reasoning based on the structured matching quantification results and the standardized hidden danger description text, and output the scene subject matching degree analysis results; The decision and output module is used to generate a safety hazard rectification verification decision report based on fused evidence.

7. The system according to claim 6, characterized in that, The image feature matching module includes: The feature extraction unit is used to extract key points and their feature descriptors that are scale- and rotation-invariant from the input image of the hidden danger before rectification and the image after rectification. The feature matching and filtering unit is used to match and initially screen the feature descriptor sets of the input image before rectification and the image after rectification; The geometric verification and transformation estimation unit is used to robustly estimate the geometric transformation relationship between the image of hidden dangers before rectification and the image after rectification from the initial matching point set, and to calculate the proportion of interior points; The structural similarity assessment unit is used to assess the overall consistency between the image of the hidden danger before rectification and the image after rectification in terms of brightness, contrast, and structural information. The indicator fusion and structured output unit is used to integrate the outputs of the feature extraction unit, feature matching and filtering unit, geometric verification and transformation estimation unit and structural similarity evaluation unit to generate the final visual matching evidence.

8. The system according to claim 6, characterized in that, The large language model analysis module includes: The decision-making and standard initialization unit is used to perform preliminary decision-making based on the input comprehensive confidence score; A structured prompting word engine is used to dynamically construct instructions to guide large language models in expert-level reasoning. The Large Language Model Call and Inference Unit is used to perform multi-step inference of the large language model and generate decisions.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to implement the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images