Industrial safety detection method combining time alignment and motion consistency

Abstract

The invention discloses an industrial safety detection method combining time alignment and motion consistency, and belongs to the technical field of industrial safety monitoring. According to the method, through seven steps of multi-camera data acquisition and preprocessing, target feature extraction, coordinate projection, instantaneous speed estimation, multi-dimensional similarity fusion, soft matching optimization and model training and target trajectory generation, the problem of space-time inconsistency during multi-camera cooperative detection in an industrial scene is solved. A camera clock offset parameter is innovatively introduced to correct a cross-camera time difference, appearance, geometry and time prior multi-dimensional similarity are fused, a multi-loss function optimization model is constructed, and cross-camera target accurate matching and long-time trajectory tracking are realized. High-precision technical support is provided for personnel safety monitoring and equipment abnormal behavior detection in an industrial scene, and the industrial production safety protection capability is improved.

Description

Technical Field

[0001] This invention relates to the field of industrial safety monitoring technology, and specifically to an industrial safety detection system that combines time alignment and motion consistency. Background Technology

[0002] In industrial production scenarios, real-time tracking and safety status monitoring of personnel, equipment, and other targets are crucial for ensuring production safety. Existing industrial safety detection systems often employ a multi-camera collaborative working mode. However, due to issues such as clock skew and differences in installation angles between different cameras, spatiotemporal inconsistencies arise during cross-camera target matching, affecting the continuity and accuracy of trajectory tracking. Furthermore, single-dimensional target matching methods (such as relying solely on appearance features) are susceptible to interference from factors such as lighting changes and occlusion, further reducing the reliability of the detection system and making it difficult to meet the high-precision requirements of safety detection in industrial settings. Therefore, there is an urgent need for an industrial safety detection technology that can solve the spatiotemporal inconsistency problem and integrate multi-dimensional matching information. Summary of the Invention

[0003] The purpose of this invention is to provide an industrial safety detection method that combines time alignment and motion consistency to solve the problems of spatiotemporal inconsistency and low matching accuracy in cross-camera target matching in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: An industrial safety inspection method combining time alignment and motion consistency includes the following steps: S1. Data Acquisition and Preprocessing: Acquire video streams from multiple cameras and ground homography matrices from each camera; detect targets from the video streams from multiple cameras and extract the target's two-dimensional bounding box; crop the target to obtain a cropped image patch; form a basic sample containing the target's two-dimensional bounding box and the target's cropped image patch; and automatically mine cross-camera positive and negative sample pairs based on spatiotemporal geometric constraints. S2. Target Feature Extraction: A learnable feature extraction network is used to map the target cropped image patch in step S1 to the unit spherical embedding space to obtain the target appearance embedding that satisfies the unit norm constraint. S3. Coordinate Projection: Based on the camera ground homography matrix obtained in step S1, the homogeneous coordinates of the "foot point" pixels corresponding to the target's two-dimensional bounding box are projected onto the ground plane coordinate system to obtain the target's ground plane coordinates; S4. Target instantaneous velocity estimation: Select a set of candidate points within a small time window adjacent to the target detection result under the same camera, and estimate the instantaneous velocity of the target using the least squares method based on the target's ground plane coordinates and corresponding timestamps in the set; S5. Multi-dimensional similarity fusion: Calculate appearance similarity, geometric similarity, and temporal prior similarity separately, and then fuse them using logarithmic space weighted summation combined with the Sigmoid function to obtain a unified matching confidence score; among which, the geometric similarity calculation is corrected for cross-camera time difference using camera clock offset parameters; S6. Soft Matching Optimization and Model Training: An optimal transmission algorithm with entropy regularization is adopted, and the soft matching matrix and the corresponding soft matching strength are solved by Sinkhorn iteration. At the same time, a multi-dimensional loss function is constructed to optimize the model parameters. The multi-dimensional loss function includes contrast loss, optimal transmission loss, loop consistency loss, calibration loss, dynamic smoothing loss and L1 loss. The weights of each loss term are adjusted by a balancing coefficient. S7. Target trajectory generation: Construct a graph structure using the soft matching strength obtained in step S6 as edge weights, and perform sparsification processing on the graph structure through threshold filtering or K-nearest neighbor algorithm; generate long-term target trajectories and assign global IDs using connected component, spectral clustering or minimum cost flow aggregation algorithms, and correct ID conflicts through cross-window optimal transmission or local graph cut method to obtain a globally consistent long-term target tracking trajectory.

[0005] This industrial safety detection method revolves around multi-camera collaborative target tracking, achieving accurate and continuous safety detection through seven steps: First, it acquires multi-camera video and a ground homography matrix, detects targets, and generates samples containing bounding boxes and cropped image patches, automatically filtering positive and negative samples across cameras; next, it uses a learnable network to extract target appearance features and map them to a specific space, ensuring feature standardization and comparability; then, it combines the homography matrix to convert the target bounding box "footpoint" coordinates into ground plane coordinates, unifying the spatial benchmark; subsequently, based on the ground plane coordinates and timestamps of adjacent detection points, it uses the least squares method to estimate the target's instantaneous velocity, capturing its motion state; then, it calculates similarity from three dimensions: appearance, geometry, and time, corrects for cross-camera time differences, and merges them into a unified matching reliability; next, it improves matching accuracy and robustness through a soft matching algorithm and a multi-loss function optimization model; finally, it constructs a graph structure based on the soft matching strength, generates trajectories through sparsification and aggregation algorithms, assigns global IDs, corrects ID conflicts, and ultimately obtains a globally consistent long-term target tracking trajectory, providing reliable support for industrial safety monitoring.

[0006] Preferably, the format of the basic sample in step S1 is represented as follows: ,in, Let be the cropped image patch detected by the c-th camera at time t. This is the corresponding two-dimensional bounding box.

[0007] Preferably, the target feature extraction in step S2 specifically involves: , For learnable feature extraction networks, The appearance embedding for the nth detection of the c-th camera at time t.

[0008] Preferably, the homogeneous pixel coordinates of the "foot point" corresponding to the target two-dimensional bounding box in step S3 are the homogeneous coordinates of the center of the bottom edge of the target two-dimensional bounding box. The formula for calculating the plane coordinates of the target location is: , Let c be the ground homography matrix of the c-th camera. This is a homogeneous coordinate normalization operation.

[0009] Preferably, the instantaneous velocity of the target in step S4 is calculated as follows: in, The set of candidate points of the same camera adjacent to detection i. Let t be the estimated instantaneous velocity, and t be the timestamp.

[0010] Preferably, in step S5: The formula for calculating appearance similarity is: S51. Calculate appearance distance based on Euclidean distance. S52. Obtain the appearance similarity by mapping the above appearance distances through an exponential function. in, For external temperature parameters; The formula for calculating geometric similarity is: S53. Calculate geometric distance based on the difference between observed and predicted locations. S54. Obtain the geometric similarity by mapping the above geometric distances through an exponential function. in, These are geometric temperature parameters; The time difference correction formula is: ,in, This refers to the clock offset parameter of the camera in the relevant location; The formula for calculating temporal prior similarity is: in, The average speed in the region. The time-prior temperature parameter; The formula for the confidence score of fusion matching is: in, This is the Sigmoid function, outputting to (0,1); These are learnable weight parameters.

[0011] Step S5 calculates similarity from three dimensions: appearance, geometry, and time. These are then fused to obtain a unified matching confidence score. The specific logic is clear and interconnected: In the appearance dimension, the difference between the appearance features of different targets is first calculated using Euclidean distance (i.e., appearance distance). Then, an exponential function is used to convert the distance into appearance similarity. The appearance temperature parameter adjusts the sensitivity of the similarity score to appearance differences, allowing targets with similar appearances to show higher similarity. In the geometry dimension, the geometric distance is calculated based on the difference between the actual observed position of the target and the predicted position inferred from the motion pattern. This distance is also mapped to geometric similarity using an exponential function. The geometric temperature parameter controls the magnitude of the similarity score's response to positional deviations. Simultaneously, the calculation uses a camera clock offset parameter to correct for cross-camera time differences, avoiding positional prediction errors caused by clock asynchrony. In the time dimension, the average movement speed of targets within the region is combined to calculate the temporal prior similarity using a formula. The temporal prior temperature parameter adjusts the weight of the similarity score in considering time differences, thus reflecting the rationality of the target's movement in the time dimension. Finally, the similarity scores of the three dimensions are first logarithmized and then weighted and summed according to the learnable weight parameters. The sum is then mapped to the 0-1 interval using the Sigmoid function to obtain the final unified matching confidence score. This confidence score can comprehensively measure the matching reliability of different targets and provide a comprehensive basis for subsequent target matching.

[0012] Preferably, the optimal transmission algorithm in step S6 is calculated as follows: The objective function of the entropy-regularized optimal transmission algorithm is: in, This is a soft-matching matrix; Row / column boundary constraint set The marginal "quality" includes the quality of unmatched virtual nodes; in, : Fusion matching confidence matrix, Cost matrix ; This represents the entropy regularity strength; the larger the value, the smoother and more robust the performance. Kullback–Leibler divergence.

[0013] In the soft matching optimization in step S6, the entropy-regularized optimal transmission algorithm is preferably used to calculate the soft matching matrix. Its core is to achieve a balance between the rationality and robustness of the matching through the objective function. This objective function optimizes the soft matching matrix. On the one hand, it needs to meet the requirements of the row / column marginal constraint set—ensuring that the matching results conform to the "marginal quality" of the targets under each camera (including the quality of unmatched virtual nodes, avoiding computational gaps due to some targets having no matching objects). On the other hand, the function incorporates a cost matrix based on the transformation of the fused matching confidence matrix (the cost matrix elements are obtained by taking the negative value of the logarithm of the matching confidence, meaning that the higher the matching confidence, the lower the corresponding cost). Simultaneously, an entropy regularization term is introduced through Kullback-Leibler divergence (KL divergence). Here, the entropy regularization strength is a key adjustment parameter; the larger the value, the smoother the distribution of the soft matching matrix, and the stronger the algorithm's resistance to interference such as target occlusion and feature noise in industrial scenarios. Ultimately, it can more stably solve for the soft matching matrix reflecting the degree of matching between targets and the corresponding soft matching strength.

[0014] Preferably, the calculation method for each multi-dimensional loss function in S6 is as follows: The formula for calculating the contrast loss is: in, : The set of positive sample pairs that satisfy , ; A set of samples that are difficult to bear, which are similar in appearance but geometrically / temporally impossible; : Compare the temperature parameters of the loss; Threshold; The optimal transmission loss calculation formula is: in, : OT soft matching strength, used as sample weight; The balance coefficient of the spacetime term, and Linkage; to Perform temperature recalibration , It is a power function, which increases the weight of high-confidence pairs; The formula for calculating loop consistency loss is: in, The camera loop edge set starting from detection i can be a short loop of length 2 to 3. The matching confidence from camera u to v is given by... Take values ​​along the edge and align them to the corresponding nodes; when optimizing the loss, use... Summation instead of product can result in a more stable numerical value. ; The formula for calculating the calibration loss is: in, : Pseudo-corresponding set, threshold ; : Detect the point i in world coordinates, by It can be obtained through substitution or 3D reconstruction; To detect the pixel of point j, you can use the foot point or the center of the bounding box; The camera's intrinsic / extrinsic parameters; adding sparse Huber loss to combat outliers; Dynamically set to within-batch quantiles; The formula for calculating dynamic smoothing loss is: in, Global trajectory ID index, obtained by aggregation of the soft-matched graph; : Time series points of the trajectory on the ground coordinates; Encourage sparse transfer matrices and suppress fake matches; The formula for calculating L1 loss is: Total loss function: in, The balancing coefficients for each loss term can be dynamically adjusted during the training phase.

[0015] Step S6 uses a multi-dimensional loss function to constrain model optimization from different dimensions: contrastive loss improves feature discrimination by comparing positive sample pairs with hard negative sample pairs; optimal transmission loss combines soft matching strength and spatiotemporal terms to optimize matching allocation; loop consistency loss ensures the consistency of cross-camera matching logic; calibration loss optimizes camera parameters and resists outliers by using pseudo-matching and sparse Huber loss; dynamic smoothing loss ensures the stability of the trajectory coordinate time series; L1 loss encourages the sparsity of the transmission matrix to suppress spurious matching; and finally, the total loss function is constructed by combining each loss term with dynamically scheduled balance coefficients to comprehensively optimize model parameters.

[0016] Some have beneficial effects: 1. By introducing a clock offset parameter to correct the time difference across cameras, the problem of spatiotemporal inconsistency caused by the asynchronous clocks of multiple cameras is solved, and the accuracy of cross-camera target matching is improved. 2. It integrates multi-dimensional similarity based on appearance, geometry, and temporal priors, comprehensively considering the target's appearance features, motion state, and temporal rationality, thereby reducing interference from factors such as lighting changes and occlusion; 3. A multi-dimensional loss function is used to optimize the model, and an entropy regularization optimal transmission algorithm is combined to achieve soft matching, which balances the accuracy and robustness of matching and ensures the continuity of trajectory tracking. 4. The process is clear and straightforward, relying on conventional hardware (ordinary network cameras) without requiring special equipment. It can be quickly deployed in various industrial scenarios, providing reliable technical support for personnel safety monitoring and equipment operation status monitoring, and has broad application prospects. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto. Example

[0019] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the overall process of the method of the present invention. An industrial safety detection method combining time alignment and motion consistency includes the following steps: S1. Data Acquisition and Preprocessing: Acquire video streams from multiple cameras and ground homography matrices from each camera; detect targets from the video streams from multiple cameras and extract the target's two-dimensional bounding box; crop the target to obtain a cropped image patch; form a basic sample containing the target's two-dimensional bounding box and the target's cropped image patch; and automatically mine cross-camera positive and negative sample pairs based on spatiotemporal geometric constraints. S2. Target Feature Extraction: A learnable feature extraction network is used to map the target cropped image patch in step S1 to the unit spherical embedding space to obtain the target appearance embedding that satisfies the unit norm constraint. S3. Coordinate Projection: Based on the camera ground homography matrix obtained in step S1, the homogeneous coordinates of the "foot point" pixels corresponding to the target's two-dimensional bounding box are projected onto the ground plane coordinate system to obtain the target's ground plane coordinates; S4. Target instantaneous velocity estimation: Select a set of candidate points within a small time window adjacent to the target detection result under the same camera, and estimate the instantaneous velocity of the target using the least squares method based on the target's ground plane coordinates and corresponding timestamps in the set; S5. Multi-dimensional similarity fusion: Calculate appearance similarity, geometric similarity, and temporal prior similarity separately, and then fuse them using logarithmic space weighted summation combined with the Sigmoid function to obtain a unified matching confidence score; among which, the geometric similarity calculation is corrected for cross-camera time difference using camera clock offset parameters; S6. Soft Matching Optimization and Model Training: An optimal transmission algorithm with entropy regularization is adopted, and the soft matching matrix and the corresponding soft matching strength are solved by Sinkhorn iteration. At the same time, a multi-dimensional loss function is constructed to optimize the model parameters. The multi-dimensional loss function includes contrast loss, optimal transmission loss, loop consistency loss, calibration loss, dynamic smoothing loss and L1 loss. The weights of each loss term are adjusted by a balancing coefficient. S7. Target trajectory generation: Construct a graph structure using the soft matching strength obtained in step S6 as edge weights, and perform sparsification processing on the graph structure through threshold filtering or K-nearest neighbor algorithm; generate long-term target trajectories and assign global IDs using connected component, spectral clustering or minimum cost flow aggregation algorithms, and correct ID conflicts through cross-window optimal transmission or local graph cut method to obtain a globally consistent long-term target tracking trajectory.

[0020] Preferably, the format of the basic sample in step S1 is represented as follows: ,in, Let be the cropped image patch detected by the c-th camera at time t. This is the corresponding two-dimensional bounding box.

[0021] Preferably, the target feature extraction in step S2 specifically involves: , For learnable feature extraction networks, The appearance embedding for the nth detection of the c-th camera at time t.

[0022] Preferably, the homogeneous pixel coordinates of the "foot point" corresponding to the target two-dimensional bounding box in step S3 are the homogeneous coordinates of the center of the bottom edge of the target two-dimensional bounding box. The formula for calculating the plane coordinates of the target location is: , Let c be the ground homography matrix of the c-th camera. This is a homogeneous coordinate normalization operation.

[0023] Preferably, the instantaneous velocity of the target in step S4 is calculated as follows: in, The set of candidate points of the same camera adjacent to detection i. Let t be the estimated instantaneous velocity, and t be the timestamp.

[0024] Preferably, in step S5: The formula for calculating appearance similarity is: S51. Calculate appearance distance based on Euclidean distance. S52. Obtain the appearance similarity by mapping the above appearance distances through an exponential function. in, For external temperature parameters; The formula for calculating geometric similarity is: S53. Calculate geometric distance based on the difference between observed and predicted locations. S54. Obtain the geometric similarity by mapping the above geometric distances through an exponential function. in, These are geometric temperature parameters; The time difference correction formula is: ,in, This refers to the clock offset parameter of the camera in the relevant location; The formula for calculating temporal prior similarity is: in, The average speed in the region. The time-prior temperature parameter; The formula for the confidence score of fusion matching is: in, This is the Sigmoid function, outputting to (0,1); These are learnable weight parameters.

[0025] Preferably, the optimal transmission algorithm in step S6 is calculated as follows: The objective function of the entropy-regularized optimal transmission algorithm is: in, This is a soft-matching matrix; Row / column boundary constraint set The marginal "quality" includes the quality of unmatched virtual nodes; in, : Fusion matching confidence matrix, Cost matrix ; This represents the entropy regularity strength; the larger the value, the smoother and more robust the performance. Kullback–Leibler divergence.

[0026] Preferably, the calculation method for each multi-dimensional loss function in step S6 is as follows: The formula for calculating the contrast loss is: in, : The set of positive sample pairs that satisfy , ; A set of samples that are difficult to bear, which are similar in appearance but geometrically / temporally impossible; : Compare the temperature parameters of the loss; Threshold; The optimal transmission loss calculation formula is: in, : OT soft matching strength, used as sample weight; The balance coefficient of the spacetime term, and Linkage; to Perform temperature recalibration , It is a power function, which increases the weight of high-confidence pairs; The formula for calculating loop consistency loss is: in, The camera loop edge set starting from detection i can be a short loop of length 2 to 3. The matching confidence from camera u to v is given by... Take values ​​along the edge and align them to the corresponding nodes; when optimizing the loss, use... Summation instead of product can result in a more stable numerical value. ; The formula for calculating the calibration loss is: in, : Pseudo-corresponding set, threshold ; : Detect the point i in world coordinates, by It can be obtained through substitution or 3D reconstruction; To detect the pixel of point j, you can use the foot point or the center of the bounding box; The camera's intrinsic / extrinsic parameters; adding sparse Huber loss to combat outliers; Dynamically set to within-batch quantiles; The formula for calculating dynamic smoothing loss is: in, Global trajectory ID index, obtained by aggregation of the soft-matched graph; : Time series points of the trajectory on the ground coordinates; Encourage sparse transfer matrices and suppress fake matches; The formula for calculating L1 loss is: Total loss function: in, The balancing coefficients for each loss term can be dynamically adjusted during the training phase.

[0027] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An industrial safety detection method combining time alignment and motion consistency, characterized in that, Includes the following steps: S1. Data Acquisition and Preprocessing: Acquire video streams from multiple cameras and ground homography matrices from each camera; detect targets from the video streams from multiple cameras and extract the target's two-dimensional bounding box; crop the target to obtain a cropped image patch; form a basic sample containing the target's two-dimensional bounding box and the target's cropped image patch; and automatically mine cross-camera positive and negative sample pairs based on spatiotemporal geometric constraints. S2. Target Feature Extraction: A learnable feature extraction network is used to map the target cropped image patch in step S1 to the unit spherical embedding space to obtain the target appearance embedding that satisfies the unit norm constraint. S3. Coordinate Projection: Based on the camera ground homography matrix obtained in step S1, the homogeneous coordinates of the "foot point" pixels corresponding to the target's two-dimensional bounding box are projected onto the ground plane coordinate system to obtain the target's ground plane coordinates; S4. Target instantaneous velocity estimation: Select a set of candidate points within a small time window adjacent to the target detection result under the same camera, and estimate the instantaneous velocity of the target using the least squares method based on the target's ground plane coordinates and corresponding timestamps in the set; S5. Multi-dimensional similarity fusion: Calculate appearance similarity, geometric similarity, and temporal prior similarity separately, and then fuse them using logarithmic space weighted summation combined with the Sigmoid function to obtain a unified matching confidence score; among which, the geometric similarity calculation is corrected for cross-camera time difference using camera clock offset parameters; S6. Soft Matching Optimization and Model Training: An optimal transmission algorithm with entropy regularization is adopted, and the soft matching matrix and the corresponding soft matching strength are solved by Sinkhorn iteration. At the same time, a multi-dimensional loss function is constructed to optimize the model parameters. The multi-dimensional loss function includes contrast loss, optimal transmission loss, loop consistency loss, calibration loss, dynamic smoothing loss and L1 loss. The weights of each loss term are adjusted by a balancing coefficient. S7. Target trajectory generation: Construct a graph structure using the soft matching strength obtained in step S6 as the edge weight, and perform sparsification processing on the graph structure through threshold filtering or K-nearest neighbor algorithm; Long-term target trajectories are generated and global IDs are assigned using connected component analysis, spectral clustering, or minimum cost flow aggregation algorithms. ID conflicts are corrected by cross-window optimal transmission or local graph cut methods to obtain globally consistent long-term target tracking trajectories.

2. The industrial safety detection method combining time alignment and motion consistency according to claim 1, characterized in that, The format of the basic sample mentioned in step S1 is as follows: ,in, Let be the cropped image patch detected by the c-th camera at time t. This is the corresponding two-dimensional bounding box.

3. The industrial safety detection method combining time alignment and motion consistency according to claim 2, characterized in that, The target feature extraction in step S2 is specifically as follows: , For learnable feature extraction networks, The appearance embedding for the nth detection of the c-th camera at time t.

4. The industrial safety detection method combining time alignment and motion consistency according to claim 3, characterized in that, The homogeneous pixel coordinates of the "foot point" corresponding to the target 2D bounding box in step S3 are the homogeneous coordinates of the center of the bottom edge of the target 2D bounding box. The formula for calculating the plane coordinates of the target location is: , Let c be the ground homography matrix of the c-th camera. This is a homogeneous coordinate normalization operation.

5. The industrial safety detection method combining time alignment and motion consistency according to claim 4, characterized in that, The formula for calculating the instantaneous velocity of the target in step S4 is as follows: in, The set of candidate points of the same camera adjacent to detection i. Let t be the estimated instantaneous velocity, and t be the timestamp.

6. The industrial safety detection method combining time alignment and motion consistency according to claim 5, characterized in that, In step S5: The formula for calculating appearance similarity is: S51. Calculate appearance distance based on Euclidean distance. S52. Obtain the appearance similarity by mapping the above appearance distances through an exponential function. in, For external temperature parameters; The formula for calculating geometric similarity is: S53. Calculate geometric distance based on the difference between observed and predicted locations. S54. Obtain the geometric similarity by mapping the above geometric distances through an exponential function. in, These are geometric temperature parameters; The time difference correction formula is: ,in, This refers to the clock offset parameter of the camera in the relevant location; The formula for calculating temporal prior similarity is: in, The average speed in the region. The time-prior temperature parameter; The formula for the confidence score of fusion matching is: in, This is the Sigmoid function, outputting to (0,1); These are learnable weight parameters.

7. The industrial safety detection method combining time alignment and motion consistency according to claim 6, characterized in that, The optimal transmission algorithm in step S6 is calculated as follows: The objective function of the entropy-regularized optimal transmission algorithm is: in, This is a soft-matching matrix; Row / column boundary constraint set The marginal "quality" includes the quality of unmatched virtual nodes; in, : Fusion matching confidence matrix, Cost matrix ; This represents the entropy regularity strength; the larger the value, the smoother and more robust the performance. Kullback–Leibler divergence.

Images