Method and system for monitoring safety of operations at different heights using multi-view image fusion

Abstract

The present application relates to the field of image processing and construction operation monitoring technology, and provides a method and system for monitoring the safety of operations at different heights using multi-view image fusion. The method comprises: using monitoring cameras at different angles to simultaneously shoot the operation space to obtain multiple monitoring images with different views; performing nonlinear image correction on the monitoring images with different views; integrating the images after the nonlinear correction of the multiple images with different views into the coordinate system of the fused image to generate a fused image; performing operation target detection on the fused image; the operation targets include operation equipment and operators; for operation targets in the upper and lower spaces, calculating the correlation coefficient of the upper and lower spaces, and determining the operation risk level based on the correlation coefficient; determining the corresponding early warning method based on the operation risk level and issuing an early warning according to the corresponding early warning method. The present invention achieves comprehensive coverage and precise monitoring of operation areas at different heights, improves early warning accuracy and response efficiency, and ensures construction safety.

Description

Technical Field

[0001] The present application relates to the technical fields of image processing and construction operation monitoring, and in particular to a method and system for monitoring safety of operations at different heights using multi-view image fusion. Background Art

[0002] In construction sites, due to construction requirements, there are often work areas at different heights, forming a scene where upper and lower spaces are working simultaneously. Although this work mode improves construction efficiency, it also brings significant safety hazards, especially objects in the upper work area may fall, posing a threat to workers and equipment below. Traditional security monitoring systems mostly use cameras with a single field of view. Due to the limitations of height and viewing angle, it is difficult to fully cover the work areas at different heights, resulting in monitoring blind spots and the inability to effectively identify and warn of potential safety risks. In addition, from a technical perspective, current visual monitoring has the following shortcomings: (1) Limited field of view: The height of a camera with a single field of view is limited, making it difficult to accurately capture targets at different heights. (2) Difficulty in target recognition. In complex construction environments, traditional image processing technology has difficulty in accurately identifying and tracking target feature points at different heights. (3) Insufficient image fusion. The lack of effective multi-field of view image fusion technology makes it impossible to comprehensively utilize the field of view information of multiple cameras for accurate monitoring. Summary of the Invention

[0003] In response to the above technical problems, the purpose of this application is to provide a method and system for monitoring the safety of operations at different heights using multi-view image fusion, aiming to solve the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a method for safely monitoring operations at different heights using multi-view image fusion, the method comprising:

[0005] Use surveillance cameras at different angles to simultaneously capture the workspace and obtain multiple surveillance images with different fields of view;

[0006] performing nonlinear image correction on the monitoring images of the different fields of view;

[0007] Perform salient object detection on the image after nonlinear image correction and extract the feature information of the salient object;

[0008] Perform salient feature matching on the salient target feature information of multiple surveillance images with different fields of view, and obtain the salient feature point with the highest matching degree. Use this point as a reference to construct the coordinate system of the fused image.

[0009] Integrating the images obtained by nonlinear correction of multiple images of different fields of view into the coordinate system of the fused image to generate a fused image;

[0010] Performing operation target detection on the fused image to obtain the location of the operation target; the operation target includes the operation equipment and the operation personnel;

[0011] Based on the position of the operation target, the correlation coefficient of the upper and lower spaces of the operation targets is calculated, and the operation risk level is determined according to the correlation coefficient;

[0012] Determine the corresponding warning method according to the operation risk level and issue a warning according to the corresponding warning method.

[0013] Furthermore, the step of performing nonlinear image correction on the monitoring image includes:

[0014] According to the vertical targets and horizontal targets in the monitoring image, second-order nonlinear polynomials in the vertical and horizontal directions are constructed respectively;

[0015] Detect vertical and horizontal edges in surveillance images using Hough transform;

[0016] Feature points are selected on the vertical and horizontal edges of the monitoring image, and the coefficients of the second-order nonlinear polynomial are fitted using the least squares method using the selected feature points;

[0017] Nonlinear correction is performed on the monitoring image using coefficients of a second-order nonlinear polynomial.

[0018] Furthermore, the second-order nonlinear polynomials in the horizontal and vertical directions are:

[0019] ;

[0020] ;

[0021] in, are polynomial coefficients, x, y are original pixel coordinates, 、 are the corrected coordinates.

[0022] Furthermore, the steps of performing salient object detection on the image after nonlinear image correction and extracting feature information of the salient objects include:

[0023] Build an end-to-end deep learning model to detect salient objects in images after nonlinear image correction and extract the feature information of salient objects;

[0024] The loss function used in the end-to-end deep learning model is:

[0025] ;

[0026] in is the anchor feature, is a positive sample, is a negative sample, is the interval threshold, and d is the Cartesian distance between two feature points.

[0027] Furthermore, the sample data construction step of the deep learning model includes:

[0028] Salient objects are annotated in construction site surveillance video frames. The annotation information is the bounding box of the salient object location, and the IDs of the same salient object in different field of view images are associated.

[0029] Geometric enhancement and photometric enhancement are performed on the construction site monitoring video frames; wherein the geometric enhancement includes one or more of rotation, translation and scaling; and the photometric enhancement includes one or more of brightness adjustment, contrast adjustment and adding Gaussian noise.

[0030] Furthermore, the feature information of the salient target is extracted by an end-to-end deep learning model, and the step of performing salient feature matching on the feature information of the salient targets of the plurality of surveillance images with different fields of view includes:

[0031] Adding the feature information of the salient target output by the deep learning model to a convolutional layer to obtain a feature vector output by the convolutional layer;

[0032] The feature vectors of different views are matched by calculating the cosine similarity.

[0033] Furthermore, the correlation coefficient is calculated as follows:

[0034] ;

[0035] in, represents the correlation coefficient;

[0036] Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target;

[0037] X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target;

[0038] k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard;

[0039] m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

[0040] Furthermore, the steps of integrating the corrected nonlinear images of multiple different fields of view into the coordinate system of the fused image to generate the fused image include:

[0041] According to the positional relationship between the image after nonlinear image correction and the significant feature points, the image information after nonlinear image correction is reconstructed in the coordinate system of the fused image through affine transformation to generate a fused image.

[0042] The present application also provides a multi-view image fusion safety monitoring system for operations at different heights, the system comprising:

[0043] An image acquisition module is used to simultaneously capture the workspace using surveillance cameras at different angles to obtain multiple surveillance images with different fields of view;

[0044] An image correction module, configured to perform nonlinear image correction on the monitoring images of different fields of view;

[0045] The target detection and extraction module is used to detect significant targets in the image after nonlinear image correction and extract the feature information of significant targets;

[0046] The fusion image coordinate system construction module is used to perform significant feature matching on the significant target feature information of multiple surveillance images with different fields of view, obtain the significant feature point with the highest matching degree, and use this point as the reference to construct the coordinate system of the fusion image;

[0047] A fusion image generation module, used for integrating the images after nonlinear correction of multiple images of different fields of view into the coordinate system of the fusion image to generate a fusion image;

[0048] The operation detection module is used to perform operation detection on the fused image and obtain the operation position range;

[0049] The risk level determination module is used to calculate the correlation coefficient of the upper and lower spaces for the operation targets based on the operation position range, and determine the operation risk level according to the correlation coefficient;

[0050] The early warning module is used to determine the corresponding early warning method according to the operation risk level and issue an early warning according to the corresponding early warning method.

[0051] Furthermore, the correlation coefficient is calculated as follows:

[0052] ;

[0053] in, represents the correlation coefficient;

[0054] Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target;

[0055] X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target;

[0056] k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard;

[0057] m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

[0058] This invention utilizes the field of view information of multiple cameras to eliminate blind spots in single-view monitoring, achieving all-around, blind-spot-free monitoring coverage. By performing nonlinear image correction, wide-angle distortion is eliminated, providing high-quality image data for subsequent processing and improving the accuracy of subsequent target detection, feature matching, and image fusion. By identifying common target feature points from multiple images with different fields of view and performing image fusion, the multi-field information is integrated, enriching target features and enhancing the accuracy and spatial correlation of target recognition. By detecting operations and calculating the correlation coefficient between upper and lower space targets, the correlation coefficient is determined based on the correlation coefficient. The corresponding early warning method is then determined and issued according to the corresponding early warning method. This achieves comprehensive coverage and precise monitoring of work areas at different heights, effectively reducing the safety risks posed by falling objects and ensuring construction safety. This invention provides a full-link high- and low-altitude operation safety monitoring system, from multi-source data fusion, precise target correlation, to intelligent decision-making, effectively eliminating blind spots, improving early warning accuracy and response efficiency, and ensuring construction safety. This invention has important application value in the field of construction site safety monitoring, significantly improving construction safety, reducing the occurrence of safety accidents, and enhancing the intelligent level of construction management. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 This is a flowchart of a method for safely monitoring operations at different heights using multi-view image fusion provided by an embodiment of the present application;

[0061] Figure 2 This is a structural diagram of a safety monitoring system for operations at different heights using multi-view image fusion provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0064] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as such, will not be interpreted in an idealized or overly formal sense.

[0065] See also Figure 1 The embodiment of the present application provides a method for monitoring safety of operations at different heights using multi-view image fusion, including steps S1-S8:

[0066] S1. Use surveillance cameras at different angles to simultaneously shoot the workspace and obtain multiple surveillance images with different fields of view;

[0067] S2. performing nonlinear image correction on the monitoring images of different fields of view;

[0068] S3, performing salient target detection on the image after nonlinear image correction, and extracting feature information of the salient target;

[0069] S4. Perform salient feature matching on salient target feature information of multiple surveillance images with different fields of view, and obtain the salient feature point with the highest matching degree, and use this point as a reference to construct a coordinate system for the fused image;

[0070] S5, integrating the images obtained by nonlinear correction of the plurality of images of different fields of view into the coordinate system of the fused image to generate a fused image;

[0071] S6. Detecting operating targets on the fused image to obtain operating target locations; wherein the operating targets include operating equipment and operating personnel;

[0072] S7. Based on the operation target position, for the operation targets in the upper and lower spaces, calculate the correlation coefficient of the upper and lower spaces, and determine the operation risk level according to the correlation coefficient;

[0073] S8. Determine a corresponding warning method according to the operation risk level and issue a warning according to the corresponding warning method.

[0074] Step S1 is the multi-angle video acquisition and image acquisition stage. In this stage, surveillance cameras are installed at different heights and orientations on the construction site. Specifically, wide-angle surveillance cameras, such as fisheye cameras, are installed to cover the upper and lower working areas. During the video acquisition process, it is necessary to ensure camera time synchronization and transmit the video stream to the central server via a local area network or 5G. Thereafter, the image is pre-processed, that is, the video stream is frame extracted to obtain multi-field images with the same timestamp. It should be noted that multi-field images refer to multiple images with different fields of view, or can also be understood as multiple images with different perspectives. The image must be at least 1080p, and 4K is recommended to retain more details. At the same time, in order to effectively implement data traceability, during the data storage process, the original image and timestamp information are stored in the format of .png or .jpg, and the surveillance camera number and spatial position parameters are marked.

[0075] Step S2 is the nonlinear image correction stage. Nonlinear image correction is performed to eliminate wide-angle distortion, unify image features from different perspectives, ensure spatial consistency of images from different fields of view, and improve the accuracy of subsequent target detection, feature matching, and image fusion.

[0076] Images with different fields of view may contain the same objects (such as workers, equipment, and construction materials). However, due to differences in visual perception and position, these objects are represented inconsistently in different images. Direct fusion can lead to information redundancy or loss. This problem can be solved through steps S3-S5. In step S3, salient objects are detected in the image after nonlinear image correction. Salient objects are designated, easily detected objects such as workers, equipment, and construction materials. Feature information of these objects, such as edges, texture, and color, is extracted. Specifically, an end-to-end deep learning model is constructed to implement salient object detection and feature extraction.

[0077] Step S4 involves matching and associating salient feature information from different fields of view (i.e., surveillance images from different fields of view). This step primarily involves applying the salient target feature information output by the deep learning model to a convolutional layer, setting the output of this layer as feature information, and then constructing a feature vector using this feature information. Feature vectors from different fields of view are matched by calculating cosine similarity. Furthermore, for fusion image reference point selection, the salient feature point with the highest matching score (i.e., the feature point of the salient target) is selected and used as the coordinate origin. The pixel coordinates are then converted to world coordinates using the PNP algorithm to construct the coordinate system for the fused image. Specifically, this coordinate system is a Cartesian coordinate system.

[0078] In step S5, the pixel position of the first image is expanded above the fused image coordinate system, and the pixel position of the second image is expanded below the coordinate system. Repeat the above process, integrate the pixel position information of multiple images into the same coordinate system, and generate a fused image. The specific method will be explained later and will not be described in detail here. The present invention eliminates the information inconsistency caused by the difference in viewing angle by integrating the target information in different fields of view into the same coordinate system. Through multi-field fusion, the features of the significant targets are richer, and the accuracy and robustness of target recognition are improved. The fused image can more intuitively reflect the positional relationship of the target in space, and provide data support for the correlation analysis of upper and lower space operations.

[0079] In step S6, simultaneous work in upper and lower spaces at a construction site can pose safety hazards (such as falling objects). It is necessary to detect work targets in both upper and lower spaces in real time and obtain their locations for correlation analysis. Work targets include workers and equipment. Specifically, an object detection algorithm (such as the YOLO algorithm) is used to analyze the fused image and identify the locations of work targets such as workers and equipment.

[0080] In step S7, based on the operation target position, the correlation coefficient of the upper and lower spaces of the operation targets is calculated according to the correlation coefficient calculation formula, and the operation risk level is determined according to the correlation coefficient.

[0081] In step S8, a correspondence between the operation risk level and the warning method is pre-established. For example, a high-risk operation will be alerted through sound, light, text messages, etc., and the operation will be forced to suspend. A medium-risk operation will receive a pop-up window prompt. A low-risk operation will not take any action. Therefore, in application, the corresponding warning method can be determined according to the operation risk level and the warning can be issued according to the corresponding warning method.

[0082] This invention utilizes the field of view information of multiple cameras to eliminate blind spots in single-view monitoring, achieving all-around, blind-spot-free monitoring coverage. By performing nonlinear image correction, wide-angle distortion is eliminated, providing high-quality image data for subsequent processing and improving the accuracy of subsequent target detection, feature matching, and image fusion. By identifying common target feature points from multiple images with different fields of view and performing image fusion, the multi-field information is integrated, enriching target features and enhancing the accuracy and spatial correlation of target recognition. By detecting operations and calculating the correlation coefficient between upper and lower space targets, the correlation coefficient is determined based on the correlation coefficient. The corresponding early warning method is then determined and issued according to the corresponding early warning method. This achieves comprehensive coverage and precise monitoring of work areas at different heights, effectively reducing the safety risks posed by falling objects and ensuring construction safety. This invention provides a full-link high- and low-altitude operation safety monitoring system, from multi-source data fusion, precise target correlation, to intelligent decision-making, effectively eliminating blind spots, improving early warning accuracy and response efficiency, and ensuring construction safety. This invention has important application value in the field of construction site safety monitoring, significantly improving construction safety, reducing the occurrence of safety accidents, and enhancing the intelligent level of construction management.

[0083] In one embodiment, the step of performing nonlinear image correction on the monitoring image includes:

[0084] According to the vertical targets and horizontal targets in the monitoring image, second-order nonlinear polynomials in the vertical and horizontal directions are constructed respectively;

[0085] Detect vertical and horizontal edges in surveillance images using Hough transform;

[0086] Feature points are selected on the vertical and horizontal edges of the monitoring image, and the coefficients of the second-order nonlinear polynomial are fitted using the least squares method using the selected feature points;

[0087] Nonlinear correction is performed on the monitoring image using coefficients of a second-order nonlinear polynomial.

[0088] In an embodiment of the present application, a method for performing nonlinear image correction on surveillance images primarily constructs second-order nonlinear polynomials in the vertical and horizontal directions based on vertical targets (such as building edges and cranes) and horizontal targets (such as the ground and scaffolding) in the surveillance images. The method then uses the Hough transform to detect vertical edges (in one scenario, the vertical edge of a curtain wall) and horizontal edges (in another scenario, the horizontal edge of a curtain wall) in the surveillance images. Feature points are selected on the vertical and horizontal edges of the image, and the coefficients of the second-order nonlinear polynomials are fitted using the least squares method. Finally, the image is nonlinearly corrected using the coefficients of the second-order nonlinear polynomials to eliminate distortion caused by wide-angle lenses and restore linear features in the image in both vertical and horizontal directions. This nonlinear image correction method effectively improves image quality, eliminates wide-angle distortion, and makes the image closer to the real scene, facilitating subsequent target feature extraction and matching. Furthermore, this operation unifies feature standards. After correction, target features in images from different fields of view are spatially consistent, laying the foundation for multi-field image fusion. Finally, recognition accuracy is effectively improved. The corrected image can more accurately reflect the actual position and shape of the target, improving the accuracy of subsequent target detection and association.

[0089] In one embodiment, the second-order nonlinear polynomials in the horizontal and vertical directions are:

[0090] ;

[0091] ;

[0092] in, are polynomial coefficients, x, y are original pixel coordinates, 、 are the corrected coordinates.

[0093] In the embodiment of the present application, feature points are selected on the vertical and horizontal edges of the monitoring image, and the coefficients of a second-order nonlinear polynomial are fitted using the least squares method using the selected feature points. Specifically, the coefficients of the second-order nonlinear polynomial are fitted according to the following formula:

[0094] ;

[0095] in, , is the cth position value of the point taken by the real horizontal edge and vertical edge, and is the cth position value corrected by the algorithm. The straightness error of vertical and horizontal lines in the corrected image is less than 1 pixel (RMS), and C is the number of feature points.

[0096] In one embodiment, the steps of performing salient object detection on the image after nonlinear image correction and extracting feature information of the salient object include:

[0097] Build an end-to-end deep learning model to detect salient objects in images after nonlinear image correction and extract the feature information of salient objects;

[0098] The loss function used in the end-to-end deep learning model is:

[0099] ;

[0100] in is the anchor feature, is a positive sample, is a negative sample, is the interval threshold, and d is the Cartesian distance between two feature points.

[0101] In an embodiment of the present application, the end-to-end deep learning model, that is, the input is the corrected image, the backbone network adopts a lightweight and efficient convolutional neural network such as the EfficientNet network, the detection and feature extraction branch modules output the target detection results (bounding box and category) and high-dimensional feature vectors in parallel, and finally outputs the target position information and feature vectors in the output layer.

[0102] In terms of loss function, in order to force the feature vectors of the same target to be closer under different vision conditions and the distance between different targets to be farther, the algorithm chooses Triplet Loss, that is,

[0103] ;

[0104] in is the anchor feature, is a positive sample (same target), is a negative sample (different target), is the interval threshold (usually set to 0.5), and d is the Cartesian distance between two feature points.

[0105] In one embodiment, the step of constructing sample data of the deep learning model includes:

[0106] Salient objects are annotated in construction site surveillance video frames. The annotation information is the bounding box of the salient object location, and the IDs of the same salient object in different field of view images are associated.

[0107] Geometric enhancement and photometric enhancement are performed on the construction site monitoring video frames; wherein the geometric enhancement includes one or more of rotation, translation and scaling; and the photometric enhancement includes one or more of brightness adjustment, contrast adjustment and adding Gaussian noise.

[0108] In this embodiment of the present application, the deep learning model training process first constructs sample data. Specifically, salient objects (such as workers, equipment, and construction materials) are annotated in video frames from construction site surveillance. This annotation information includes bounding boxes for the target locations, and the IDs of the same object in images from different fields of view are associated to form cross-visual association annotations. Next, the data is subjected to geometric and photometric enhancement. Geometric enhancement includes operations such as rotation (±15°), translation (±10%), and scaling (0.8-1.2x). Photometric enhancement includes brightness adjustment (±20%), contrast adjustment (±15%), and the addition of Gaussian noise. Model training also requires setting hyperparameters. In this regard, the training process uses an initial learning rate of 1e-4, a cosine annealing strategy, a batch size of 16, and an NVIDIA 3090 GPU with 24GB of video memory. This embodiment of the present application uses rotation, translation, brightness changes, and contrast adjustment to expand the training data and improve the model's generalization capabilities. Geometric enhancement and photometric enhancement of data can be performed after or before labeling, but are generally performed after labeling.

[0109] In one embodiment, the feature information of the salient target is extracted by an end-to-end deep learning model, and the step of performing salient feature matching on the feature information of the salient targets of the plurality of surveillance images with different fields of view includes:

[0110] Adding the feature information of the salient target output by the deep learning model to a convolutional layer to obtain a feature vector output by the convolutional layer;

[0111] The feature vectors of different views are matched by calculating the cosine similarity.

[0112] In the embodiment of the present application, the feature information of the salient target output by the deep learning model is added to a convolutional layer, and the output of the layer is set as the feature information, and then the feature vector is constructed using this feature information. The feature vectors of different views are matched by calculating the cosine similarity, that is: ;

[0113] in, and It is the feature vector output by the convolution layer for the same target in different fields of view.

[0114] When the matching similarity is greater than 0.85, it is determined to be the same target. If there is a multi-target matching conflict, the Hungarian Algorithm is used to resolve the multi-target matching conflict.

[0115] In one embodiment, the correlation coefficient is calculated as follows:

[0116] ;

[0117] in, represents the correlation coefficient;

[0118] Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target;

[0119] X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target;

[0120] k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard;

[0121] m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

[0122] In the embodiment of the present application, Y represents the vertical distance (Y axis), X represents the horizontal distance (X axis), k>0, and m>0.

[0123] When X=0, the function simplifies to , when Y=0, , which is the baseline safety value. When Y>0, As the Y exponential increases, it reflects a nonlinear increase in risk.

[0124] When Y=0, the above function is simplified to , when X=0, , is the baseline safety value. When X>0, It decays as X increases, reflecting the buffering effect of horizontal distance.

[0125] In summary, when Y>0 and X>0, the vertical hazard is amplified by the exponential and the horizontal hazard is suppressed by the denominator.

[0126] In actual scenarios, the danger value is defined based on the image's feedback on distance. Assuming a pixel value of Y=5 meters, then:

[0127] ;

[0128] Define the safety buffer range of the horizontal distance. If X = 10m, when the risk is reduced to 10% of the baseline value, then:

[0129] =>m=0.9;

[0130] Then set the job risk level classification:

[0131] When H < 1.5, the system determines it as a low risk; when 1.5 < H < 3, it is determined as a medium risk; when H > 3, it is a high risk.

[0132] Finally, determine the corresponding warning method according to the operation risk level and issue a warning according to the corresponding warning method. For example, for high risks, use methods such as sound and light, text messages, etc. for reminder and force the operation to pause; for medium risks, display a pop-up prompt; for low risks, take no measures, etc. Therefore, during application, the corresponding warning method can be determined according to the operation risk level and a warning can be issued according to the corresponding warning method.

[0133] The embodiment of the present application dynamically evaluates the operation risk level by integrating the distances between different operation targets in the vertical and horizontal directions. This operation quantifies the danger coefficient of construction site operations (the correlation coefficient is used to measure danger, so it can also be called the danger coefficient), realizing the transformation from passive monitoring to active prevention.

[0134] In a scenario, the equipment is above the operation space and the personnel are below the operation space. Then, Y represents the vertical height difference between the upper operation equipment and the lower operation personnel; X represents the horizontal offset between the upper operation equipment and the lower operation personnel.

[0135] The present invention can achieve real-time warning of potential dangerous operations. Through an automated detection and warning mechanism, potential safety hazards in upper and lower space operations can be discovered in a timely manner, reducing the probability of accidents. Secondly, it improves management efficiency: the warning information is directly sent to relevant personnel, reducing the workload of manual inspections and improving safety management efficiency. Finally, it ensures operation safety: through accurate calculation and warning of the correlation coefficient, potential dangers in upper and lower space operations are effectively avoided, ensuring the safety of operation personnel and equipment.

[0136] In real time, the steps of integrating the corrected images of multiple non-linear images with different fields of view into the coordinate system of the fusion image include:

[0137] According to the positional relationship between the corrected image of the non-linear image and the significant feature points, through affine transformation, reconstruct the information of the corrected image of the non-linear image in the coordinate system of the fusion image to generate the fusion image.

[0138] In the embodiments of the present application, the positional relationship between the image after nonlinear image correction and the salient feature points (generally four) is defined as the positional relationship between the pixels of the image after nonlinear image correction and the salient feature points. The image information after nonlinear image correction is reconstructed in the coordinate system, mapping the pixels of the image after nonlinear image correction to the coordinate system of the fused image. In OpenCV, this mapping is primarily performed using cv2.warpPerspective. During the mapping process, the method of the present invention primarily employs bilinear interpolation to fill in and fuse missing pixel locations.

[0139] See also Figure 2 The present application also provides a multi-view image fusion system for monitoring safety of operations at different heights, the system comprising:

[0140] Image acquisition module 1, used to simultaneously shoot the work space using surveillance cameras at different angles to obtain multiple surveillance images with different fields of view;

[0141] An image correction module 2 is used to perform nonlinear image correction on the monitoring images of different fields of view;

[0142] The target detection and extraction module 3 is used to detect significant targets in the image after nonlinear image correction and extract feature information of the significant targets;

[0143] The fusion image coordinate system construction module 4 is used to perform significant feature matching on the significant target feature information of multiple surveillance images with different fields of view, obtain the significant feature point with the highest matching degree, and use this point as a reference to construct the coordinate system of the fusion image;

[0144] A fused image generation module 5 is used to integrate the images after nonlinear correction of multiple images of different fields of view into the coordinate system of the fused image to generate a fused image;

[0145] The operation detection module 6 is used to perform operation detection on the fused image and obtain the operation position range;

[0146] The risk level determination module 7 is used to calculate the correlation coefficient of the upper and lower spaces of the operation targets based on the operation position range, and determine the operation risk level according to the correlation coefficient;

[0147] The early warning module 8 is used to determine a corresponding early warning method according to the operation risk level and issue an early warning according to the corresponding early warning method.

[0148] In one embodiment, the correlation coefficient is calculated as follows:

[0149] ;

[0150] in, represents the correlation coefficient;

[0151] Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target;

[0152] X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target;

[0153] k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard;

[0154] m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

[0155] In one embodiment, performing nonlinear image correction on the surveillance image includes:

[0156] According to the vertical targets and horizontal targets in the monitoring image, second-order nonlinear polynomials in the vertical and horizontal directions are constructed respectively;

[0157] Detect vertical and horizontal edges in surveillance images using Hough transform;

[0158] Feature points are selected on the vertical and horizontal edges of the monitoring image, and the coefficients of the second-order nonlinear polynomial are fitted using the least squares method using the selected feature points;

[0159] Nonlinear correction is performed on the monitoring image using coefficients of a second-order nonlinear polynomial.

[0160] In one embodiment, the second-order nonlinear polynomials in the horizontal and vertical directions are:

[0161] ;

[0162] ;

[0163] in, are polynomial coefficients, x, y are original pixel coordinates, 、 are the corrected coordinates.

[0164] In one embodiment, performing salient object detection on the image after nonlinear image correction and extracting feature information of the salient object includes:

[0165] Build an end-to-end deep learning model to detect salient objects in images after nonlinear image correction and extract the feature information of salient objects;

[0166] The loss function used in the end-to-end deep learning model is:

[0167] ;

[0168] in is the anchor feature, is a positive sample, is a negative sample, is the interval threshold, and d is the Cartesian distance between two feature points.

[0169] In one embodiment, the step of constructing sample data of the deep learning model includes:

[0170] Salient objects are annotated in construction site surveillance video frames. The annotation information is the bounding box of the salient object location, and the IDs of the same salient object in different field of view images are associated.

[0171] Geometric enhancement and photometric enhancement are performed on the construction site monitoring video frames; wherein the geometric enhancement includes one or more of rotation, translation and scaling; and the photometric enhancement includes one or more of brightness adjustment, contrast adjustment and adding Gaussian noise.

[0172] In one embodiment, the feature information of the salient target is extracted through an end-to-end deep learning model, and the salient feature matching of the feature information of the salient targets in the plurality of surveillance images with different fields of view includes:

[0173] Adding the feature information of the salient target output by the deep learning model to a convolutional layer to obtain a feature vector output by the convolutional layer;

[0174] The feature vectors of different views are matched by calculating the cosine similarity.

[0175] It should be noted that the above-mentioned method for safely monitoring operations at different heights using multi-view image fusion is applicable to the system for safely monitoring operations at different heights using multi-view image fusion of the present invention. Therefore, the embodiments of the present invention will not be described in detail here.

[0176] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0177] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0178] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for monitoring safety of operations at different heights using multi-view image fusion, characterized in that: The method comprises: Use surveillance cameras at different angles to simultaneously capture the workspace and obtain multiple surveillance images with different fields of view; performing nonlinear image correction on the monitoring images of the different fields of view; Perform salient object detection on the image after nonlinear image correction and extract the feature information of the salient object; Perform salient feature matching on the salient target feature information of multiple surveillance images with different fields of view, and obtain the salient feature point with the highest matching degree. Use this point as a reference to construct the coordinate system of the fused image. Integrating the images obtained by nonlinear correction of multiple images of different fields of view into the coordinate system of the fused image to generate a fused image; Performing operation target detection on the fused image to obtain the location of the operation target; the operation target includes the operation equipment and the operation personnel; Based on the position of the operation target, the correlation coefficient of the upper and lower spaces of the operation targets is calculated, and the operation risk level is determined according to the correlation coefficient; Determine the corresponding warning method according to the operation risk level and issue a warning according to the corresponding warning method; The calculation method of the correlation coefficient is: ； in, represents the correlation coefficient; Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target; X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target; k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard; m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

2. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 1, characterized in that: The steps of performing nonlinear image correction on the monitoring image include: According to the vertical targets and horizontal targets in the monitoring image, second-order nonlinear polynomials in the vertical and horizontal directions are constructed respectively; Detect vertical and horizontal edges in surveillance images using Hough transform; Feature points are selected on the vertical and horizontal edges of the monitoring image, and the coefficients of the second-order nonlinear polynomial are fitted using the least squares method using the selected feature points; Nonlinear correction is performed on the monitoring image using coefficients of a second-order nonlinear polynomial.

3. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 2, characterized in that: The second-order nonlinear polynomials in the horizontal and vertical directions are: ； ； in, are polynomial coefficients, x, y are original pixel coordinates, 、 are the corrected coordinates.

4. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 1, characterized in that: The steps of detecting salient objects on the image after nonlinear image correction and extracting feature information of the salient objects include: Build an end-to-end deep learning model to detect salient objects in images after nonlinear image correction and extract the feature information of salient objects; The loss function used in the end-to-end deep learning model is: ； in is the anchor feature, is a positive sample, is a negative sample, is the interval threshold, and d is the Cartesian distance between two feature points.

5. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 4, characterized in that: The sample data construction steps of the deep learning model include: Salient objects are annotated in construction site surveillance video frames. The annotation information is the bounding box of the salient object location, and the IDs of the same salient object in different field of view images are associated. Geometric enhancement and photometric enhancement are performed on the construction site monitoring video frames; wherein the geometric enhancement includes one or more of rotation, translation and scaling; and the photometric enhancement includes one or more of brightness adjustment, contrast adjustment and adding Gaussian noise.

6. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 1, characterized in that: The feature information of the salient target is extracted by an end-to-end deep learning model. The step of performing salient feature matching on the feature information of the salient targets of the plurality of surveillance images with different fields of view includes: Adding the feature information of the salient target output by the deep learning model to a convolutional layer to obtain a feature vector output by the convolutional layer; The feature vectors of different views are matched by calculating the cosine similarity.

7. The method for monitoring safety of operations at different heights using multi-view image fusion according to claim 1, characterized in that: The steps of integrating the images obtained by nonlinear correction of the plurality of images of different fields of view into the coordinate system of the fused image to generate the fused image include: According to the positional relationship between the image after nonlinear image correction and the significant feature points, the image information after nonlinear image correction is reconstructed in the coordinate system of the fused image through affine transformation to generate a fused image.

8. A multi-view image fusion safety monitoring system for operations at different heights, characterized in that: The system comprises: An image acquisition module is used to simultaneously capture the workspace using surveillance cameras at different angles to obtain multiple surveillance images with different fields of view; An image correction module, configured to perform nonlinear image correction on the monitoring images of different fields of view; The target detection and extraction module is used to detect significant targets in the image after nonlinear image correction and extract the feature information of significant targets; The fusion image coordinate system construction module is used to perform significant feature matching on the significant target feature information of multiple surveillance images with different fields of view, obtain the significant feature point with the highest matching degree, and use this point as the reference to construct the coordinate system of the fusion image; A fusion image generation module, used for integrating the images after nonlinear correction of multiple images of different fields of view into the coordinate system of the fusion image to generate a fusion image; The operation detection module is used to perform operation detection on the fused image and obtain the operation position range; The risk level determination module is used to calculate the correlation coefficient of the upper and lower spaces for the operation targets based on the operation position range, and determine the operation risk level according to the correlation coefficient; An early warning module is used to determine a corresponding early warning method according to the operation risk level and issue an early warning according to the corresponding early warning method; The calculation method of the correlation coefficient is: ； in, represents the correlation coefficient; Y: Indicates the vertical distance, which means the vertical height difference between the upper working target and the lower working target; X: Horizontal distance, indicating the horizontal offset between the upper working target and the lower working target; k: vertical hazard growth coefficient, which controls the nonlinear effect of vertical distance on hazard; m: Horizontal hazard factor attenuation coefficient, which controls the inhibitory effect of horizontal distance on hazard.

Images