Method, device and storage medium for warning personnel approaching a foundation pit
The dual-branch structure and semantic segmentation technology of the foundation pit detection network solved the problem of reduced accuracy of the foundation pit early warning system under construction vibration, achieved efficient and accurate early warning operations, and simplified the management process.
Patent Information
- Application Number
- CN202510006791.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2045-01-03
AI Technical Summary
In the existing technology, the foundation pit early warning system has a reduced accuracy in virtual range detection and low early warning efficiency due to the deviation of the camera under construction vibration at the construction site, which requires frequent manual marking of the virtual range.
A dual-branch structure of the foundation pit detection network is adopted. The first branch performs coarse detection, and the second branch performs fine detection. The distance between the person and the foundation pit is calculated by combining semantic segmentation, and detection is performed layer by layer to avoid relying on virtual range markers.
It improves the accuracy and efficiency of foundation pit early warning, reduces the sensitivity to camera offset, simplifies the operation process, and reduces resource consumption.
Smart Images
Figure CN120014539B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application belong to the technical field of computer vision, and in particular, relate to a method, device, and storage medium for warning a foundation pit of approaching personnel. Background Art
[0002] With the continuous improvement of economy and technology, the requirements for the quality of the entire engineering construction are also higher. Among them, foundation pits face certain risks, such as collapse, burial, falling from heights, mechanical injuries and electric shock, and their safety management work is also receiving more and more attention.
[0003] At present, the management of foundation pits mainly involves deploying cameras, calibrating a virtual range of the foundation pit in the video data collected by the cameras, and issuing an early warning when a person is detected entering the virtual range.
[0004] However, there are vibrations during construction on the construction site, which will be transmitted to the camera. The camera will continuously accumulate tiny offsets, causing the virtual range to gradually deviate from the foundation pit, resulting in a decrease in the accuracy of whether the inspection personnel have entered the virtual range, and the accuracy of the early warning will also be reduced accordingly.
[0005] At this time, the administrator needs to re-mark the virtual range and repeatedly mark the virtual range, which is a cumbersome operation and results in low efficiency of early warning. Summary of the Invention
[0006] In view of this, embodiments of the present application provide a method, device, and storage medium for warning a foundation pit of personnel approaching, so as to improve the accuracy and efficiency of warning a foundation pit of personnel approaching.
[0007] A first aspect of an embodiment of the present application provides a method for warning a foundation pit of a person approaching, comprising:
[0008] Collect multiple frames of image data from the foundation pit at the construction site;
[0009] Loading a foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first trunk network and a classification head structure; the second branch structure has a second trunk network and a segmentation head structure;
[0010] Inputting the image data into the first branch structure to extract multi-level first target image features;
[0011] Inputting the first target image feature of the final level into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit;
[0012] If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold, inputting the image data and the non-final-level first target image features into the second backbone network for interactively extracting second target image features;
[0013] Inputting the second target image feature into the segmentation head structure to segment a first pixel point having a semantic meaning of a person and a second pixel point having a semantic meaning of the foundation pit;
[0014] Calculating a second distance between the person and the foundation pit based on the first pixel point and the second pixel point;
[0015] If the second distance is less than or equal to a second distance threshold, a warning operation is performed on the person; and the second distance threshold is less than the first distance threshold.
[0016] A second aspect of an embodiment of the present application provides a device for warning a person approaching a foundation pit, comprising:
[0017] An image data acquisition module is used to collect multiple frames of image data of the foundation pit at the construction site;
[0018] A foundation pit detection network loading module is used to load a foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first trunk network and a classification head structure; the second branch structure has a second trunk network and a segmentation head structure;
[0019] A first target image feature extraction module, configured to input the image data into the first branch structure to extract multi-level first target image features;
[0020] A position relationship classification module, configured to input the final-level first target image features into the classification head structure for binary classification, thereby obtaining a position relationship between the personnel and the foundation pit;
[0021] a second target image feature extraction module configured to input the image data and the non-final-level first target image feature into the second backbone network to interactively extract a second target image feature if the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold;
[0022] A semantic segmentation module, configured to input the second target image feature into the segmentation head structure to segment a first pixel point having a semantic meaning of a person and a second pixel point having a semantic meaning of the foundation pit;
[0023] a distance calculation module, configured to calculate a second distance between the person and the foundation pit based on the first pixel point and the second pixel point;
[0024] An early warning operation execution module is used to execute an early warning operation on the person if the second distance is less than or equal to a second distance threshold; the second distance threshold is less than the first distance threshold.
[0025] A third aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for warning personnel approaching a foundation pit as described in the first aspect above is implemented.
[0026] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for warning personnel approaching a foundation pit as described in the first aspect above is implemented.
[0027] A fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method for warning a person approaching a foundation pit as described in the first aspect.
[0028] In this embodiment, multiple frames of image data are collected from the foundation pit at the construction site; a foundation pit detection network is loaded; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first backbone network and a classification head structure; the second branch structure has a second backbone network and a segmentation head structure; the image data is input into the first branch structure to extract multi-level first target image features; the final first target image features are input into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; if the positional relationship is that the first distance between the personnel and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final first target image features are input into the second backbone network to interactively extract the second target image features; the second target image features are input into the segmentation head structure to segment a first pixel point with the semantics of a person and a second pixel point with the semantics of a foundation pit; the second distance between the personnel and the foundation pit is calculated based on the first pixel point and the second pixel point; if the second distance is less than or equal to the second distance threshold, an early warning operation is performed on the personnel; the second distance threshold is less than the first distance threshold. This embodiment uses classification at the first level to detect whether a person is close to the foundation pit, and uses semantic segmentation at the second level to detect whether a person is close to the foundation pit, step by step. It not only ensures the accuracy of detection, thereby alarming the accuracy of early warning, but also optimizes the resource consumption of the foundation pit detection network. This process does not rely on a pre-marked virtual range. Even if the camera is offset due to the vibration of construction on the construction site, it can still detect whether a person is close to the foundation pit normally. There is no need for the administrator to mark the virtual range, which greatly improves the ease of operation and effectively improves the efficiency of early warning. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0030] Figure 1 Schematic diagram of a method for warning a person approaching a foundation pit provided by an embodiment of the present application;
[0031] Figure 2 is a schematic diagram of a foundation pit detection network provided in an embodiment of the present application;
[0032] Figure 3 This is a schematic diagram of a device for warning a person approaching a foundation pit provided by an embodiment of the present application;
[0033] Figure 4 This is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.
[0035] The technical solution of this application is described below through specific embodiments.
[0036] Reference Figure 1 , which shows a schematic diagram of a method for warning a person approaching a foundation pit provided by an embodiment of the present application, which may specifically include the following steps:
[0037] Step 101: Collect multiple frames of image data of a foundation pit at a construction site.
[0038] In this embodiment, one or more cameras may be deployed on the construction site, and each camera may collect video data facing the foundation pit, wherein the video data includes multiple frames of image data.
[0039] Step 102: Load the foundation pit detection network.
[0040] The camera can transmit video data to the local edge computing node or cloud, and the edge computing node or cloud deploys and loads the foundation pit detection network to detect whether people are approaching the foundation pit.
[0041] Among them, the foundation pit detection network has a first branch structure and a second branch structure; Figure 2 As shown, the first branch structure has a first backbone network Backbone_1 and a classification head structure Head_1; the second branch structure has a second backbone network Backbone_2 and a segmentation head structure Head_2.
[0042] The first branch structure is used for coarse detection of whether personnel are approaching the foundation pit, and the second branch structure is used for fine detection of whether personnel are approaching the foundation pit.
[0043] When training the foundation pit detection network, the first branch structure is trained first, and the second branch structure is trained while keeping the parameters of the first branch structure unchanged.
[0044] When training the first branch structure, the first backbone network Backbone_1 and the classification head structure Head_1 are trained separately as a binary classification network. The first distance between the positive sample labeled person and the edge of the foundation pit is less than or equal to the first distance threshold, and the first distance between the negative sample labeled person and the edge of the foundation pit is greater than the first distance threshold. The loss function can be selected as cross entropy.
[0045] When training the second branch structure, the first backbone network Backbone_1, the second backbone network Backbone_2 and the segmentation head structure Head_2 are trained separately as semantic segmentation networks. The sample labels are the first pixel point with the semantic meaning of a person and the second pixel point with the semantic meaning of a foundation pit. The loss function can be selected from cross entropy, Dice coefficient, etc. The parameters of the first backbone network Backbone_1 remain unchanged.
[0046] Step 103: Input the image data into the first branch structure to extract multi-level first target image features.
[0047] In this embodiment, the image data Image may be input into the first branch structure Backbone_1 to sequentially extract multi-level first target image features according to its hierarchical structure.
[0048] In the specific implementation, Figure 2 As shown, the first backbone network Backbone_1 includes a first convolution module ConvModule_1, a second convolution module ConvModule_2 and a third convolution module ConvModule_3; accordingly, the multi-level first target image features include a first classification image feature, a second classification image feature and a third classification image feature.
[0049] Among them, the first convolution module ConvModule_1, the second convolution module ConvModule_2 and the third convolution module ConvModule_3 have the same structure, all including a convolutional layer (Convolutional Layer), a BN (Batch Normalization) function, and a ReLU (Rectified Linear Unit) function. The convolutional layer provides convolution operations, the BN function provides batch normalization processing, and the ReLU provides activation processing under linear rectification.
[0050] In this first backbone network Backbone_1, the image data Image is input into the first convolution module ConvModule_1, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the first classification image feature.
[0051] The first classification image features are input into the second convolution module ConvModule_2, and convolution operation, batch normalization processing and activation operation under linear rectification are performed in sequence to obtain the second classification image features.
[0052] The second classification image features are input into the third convolution module ConvModule_3, and convolution operation, batch normalization processing and activation operation under linear rectification are performed in sequence to obtain the third classification image features.
[0053] Step 104: Input the final first target image feature into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit.
[0054] In this embodiment, if Figure 2 As shown, the first target image feature of the final level (i.e., the last level) is input into the classification head structure Head_1 for binary classification to obtain the positional relationship between the personnel and the foundation pit.
[0055] When the multi-level first target image features include first classification image features, second classification image features and third classification image features, the final-level first target image features refer to the third classification image features, and the non-final-level first target image features refer to the first classification image features and the second classification image features.
[0056] In the specific implementation, Figure 2 As shown, the classification head structure Head_1 includes a first fully connected layer FC_1, a second fully connected layer FC_2 and a third fully connected layer FC_3.
[0057] In this classification head structure Head_1, the third classification image feature is input into the first fully connected layer FC_1 and mapped into the fourth classification image feature.
[0058] The fourth classification image feature is input into the second fully connected layer FC_2 and mapped into the fifth classification image feature.
[0059] The fifth classification image feature is input into the third fully connected layer FC_3 and mapped into the sixth classification image feature.
[0060] Use activation functions such as Sigmoid (S-type function) to activate the sixth classification image feature to obtain the position probability.
[0061] If the position probability is greater than or equal to the probability threshold, the position relationship Class between the person and the foundation pit is determined to be that the first distance between the person and the edge of the foundation pit is less than or equal to the first distance threshold.
[0062] If the position probability is less than the probability threshold, the position relationship Class between the person and the foundation pit is determined to be that the first distance between the person and the edge of the foundation pit is greater than the first distance threshold.
[0063] Step 105: If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final-level first target image features are input into the second backbone network to interactively extract the second target image features.
[0064] If the position relationship between the person and the foundation pit is that the first distance between the person and the edge of the foundation pit is less than or equal to the first distance threshold, it means that the person is likely to be close to the foundation pit. Figure 2 As shown, the image data Image and the non-final first target image features are input into the second backbone network Backbone_2 for interaction, and the second target image features are extracted therefrom.
[0065] The multi-level first target image features all contain features of personnel and foundation pits. The final-level first target image features are differentiated into classifications, while the non-final-level first target image features are not differentiated obviously, which helps to improve the quality of the second target image features.
[0066] When resources are sufficient, such as when the foundation pit detection network is deployed in the cloud, the second branch structure can be operated sustainably.
[0067] In resource-constrained situations, such as when the foundation pit detection network is deployed on edge computing nodes, the second branch structure can be selectively operated.
[0068] In this case, a first number of times the first distance is less than or equal to the first distance threshold and a second number of times the second distance is less than or equal to the second distance threshold within a previous period of time (ie, a recent period of time, such as the last 2 hours) can be counted respectively.
[0069] Statistical analysis is performed based on the first quantity and the second quantity to identify the operation pattern of the personnel.
[0070] Exemplarily, the ratio between the second number and the first number is calculated to obtain the warning ratio.
[0071] If the first number is greater than or equal to the first confidence threshold and the warning ratio is greater than the second confidence threshold, it means that the warnings are more frequent, and the personnel's working mode is determined to be working at the edge of the foundation pit.
[0072] If the first number is less than the first confidence threshold and / or the warning ratio is less than the second confidence threshold, it is determined that the personnel's working mode is not working at the edge of the foundation pit.
[0073] Furthermore, when the second distance is less than or equal to the second distance threshold, an early warning operation will be performed on the personnel. Under normal circumstances, if the personnel are not working at the edge of the foundation pit, they will follow the early warning and stay away from the foundation pit or be driven away by the supervisor. Therefore, if the early warning is frequent but the personnel do not stay away from the foundation pit, it is usually because they are temporarily working near the foundation pit and are recognized by the supervisor, and the early warning operation will be processed silently.
[0074] If the operation mode is to operate at the edge of the foundation pit, it is prohibited to input the image data and the non-final first target image features into the second backbone network Backbone_2 to interactively extract the second target image features.
[0075] If the operation mode is not at the edge of the foundation pit, the image data and the non-final first target image features are allowed to be input into the second backbone network Backbone_2 to interactively extract the second target image features, thereby reducing resource consumption.
[0076] In the specific implementation, Figure 2 As shown, the second backbone network Backbone_2 includes a fourth convolution module ConvModule_4, a fifth convolution module ConvModule_5, a sixth convolution module ConvModule_6, a seventh convolution module ConvModule_7 and an eighth convolution module ConvModule_8.
[0077] Among them, the fourth convolution module ConvModule_4, the fifth convolution module ConvModule_5, the sixth convolution module ConvModule_6, the seventh convolution module ConvModule_7 and the eighth convolution module ConvModule_8 have the same structure, all including convolution layers, BN functions, and ReLU functions. The convolution layers provide convolution operations, the BN functions provide batch normalization processing, and the ReLUs provide activation processing under linear rectification.
[0078] In this second backbone network Backbone_2, the image data Image is input into the fourth convolution module ConvModule_4, where convolution operation, batch normalization processing, and activation processing under linear rectification are performed in sequence to obtain the first semantic image feature.
[0079] The first semantic image feature is input into the fifth convolution module ConvModule_5, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second semantic image feature.
[0080] The second semantic image feature is input into the sixth convolution module ConvModule_6, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the third semantic image feature.
[0081] Downsample the second classification image feature to obtain the first transformed image feature.
[0082] The third semantic image feature and the first transformed image feature are concatenated to form a fourth semantic image feature.
[0083] The fourth semantic image feature is input into the seventh convolution module ConvModule_7, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the fifth semantic image feature.
[0084] Downsample the first classification image feature to obtain a second transformed image feature.
[0085] The fifth semantic image feature and the second transformed image feature are concatenated to form a sixth semantic image feature.
[0086] The sixth semantic image feature is input into the eighth convolution module ConvModule_8, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second target image feature.
[0087] Step 106: Input the second target image feature into the segmentation head structure to segment the first pixel point with the semantic meaning of a person and the second pixel point with the semantic meaning of a foundation pit.
[0088] In this embodiment, if Figure 2 As shown, the second target image feature is input into the segmentation head structure Head_2 to segment the first pixel point whose semantic Semantic is a person and the second pixel point whose semantic Semantic is a foundation pit.
[0089] In the specific implementation, Figure 2As shown, the segmentation head structure Head_2 includes a first convolutional layer Conv_1, a second convolutional layer Conv_2 and a third convolutional layer Conv_3.
[0090] In this segmentation head structure Head_2, the second target image feature is input into the first convolutional layer Conv_1 to perform a convolution operation to obtain the seventh semantic image feature.
[0091] The seventh semantic image feature is input into the second convolutional layer Conv_2 to perform a convolution operation to obtain the eighth semantic image feature.
[0092] The eighth semantic image feature is input into the third convolutional layer Conv_3 to perform a convolution operation to obtain a ninth semantic image feature.
[0093] An activation function such as Sigmoid is used to perform an activation operation on the ninth semantic image feature to obtain a first pixel point whose semantic semantic is a person and a second pixel point whose semantic semantic is a pit.
[0094] Step 107: Calculate a second distance between the person and the foundation pit based on the first pixel point and the second pixel point.
[0095] In this embodiment, the first pixel point and the second pixel point may be measured to calculate the second distance between the person and the foundation pit.
[0096] Considering that the first pixel point with the semantics of a person and the second pixel point with the semantics of a foundation pit are both irregular shapes, in order to improve computing efficiency, this embodiment can regularize the first pixel point with the semantics of a person and the second pixel point with the semantics of a foundation pit.
[0097] In a specific implementation, a minimum bounding rectangle is added to the first pixel point, where the minimum bounding rectangle has four corner points.
[0098] The arc is fitted using the second pixel point according to a polynomial or other method.
[0099] Calculate the shortest distance from each corner point in the minimum enclosing rectangle to the arc respectively. That is, generate a line segment through each corner point in the minimum enclosing rectangle. The line segment intersects the arc at an intersection point. The tangent of the arc at the intersection point is perpendicular to the line segment. The length of the line segment is the shortest distance from each corner point in the minimum enclosing rectangle to the arc.
[0100] Compare the shortest distances corresponding to each corner point, multiply the smallest shortest distance by a preset conversion coefficient to obtain the second distance between the personnel and the foundation pit.
[0101] The conversion coefficient is the actual distance corresponding to the unit pixel that is pre-calibrated for the camera.
[0102] Step 108: If the second distance is less than or equal to the second distance threshold, perform a warning operation on the person.
[0103] In this embodiment, the second distance between the person and the foundation pit is compared with the second distance threshold, and the second distance threshold is smaller than the first distance threshold.
[0104] If the second distance is less than or equal to the second distance threshold, an early warning operation is performed on the person (i.e., the first pixel point with the semantics of a person), such as highlighting the first pixel point with the semantics of a person, playing a warning sound, etc., to prompt the supervisor to pay attention to whether the person is illegally approaching the foundation pit.
[0105] In this embodiment, multiple frames of image data are collected from the foundation pit at the construction site; a foundation pit detection network is loaded; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first backbone network and a classification head structure; the second branch structure has a second backbone network and a segmentation head structure; the image data is input into the first branch structure to extract multi-level first target image features; the final first target image features are input into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; if the positional relationship is that the first distance between the personnel and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final first target image features are input into the second backbone network to interactively extract the second target image features; the second target image features are input into the segmentation head structure to segment a first pixel point with the semantics of a person and a second pixel point with the semantics of a foundation pit; the second distance between the personnel and the foundation pit is calculated based on the first pixel point and the second pixel point; if the second distance is less than or equal to the second distance threshold, an early warning operation is performed on the personnel; the second distance threshold is less than the first distance threshold. This embodiment uses classification at the first level to detect whether a person is close to the foundation pit, and uses semantic segmentation at the second level to detect whether a person is close to the foundation pit, step by step. It not only ensures the accuracy of detection, thereby alarming the accuracy of early warning, but also optimizes the resource consumption of the foundation pit detection network. This process does not rely on a pre-marked virtual range. Even if the camera is offset due to the vibration of construction on the construction site, it can still detect whether a person is close to the foundation pit normally. There is no need for the administrator to mark the virtual range, which greatly improves the ease of operation and effectively improves the efficiency of early warning.
[0106] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0107] Reference Figure 3 , which shows a schematic diagram of a device for warning of approaching personnel in a foundation pit provided by an embodiment of the present application, and may specifically include the following modules:
[0108] The image data acquisition module 301 is used to collect multiple frames of image data of the foundation pit at the construction site;
[0109] The foundation pit detection network loading module 302 is used to load the foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first backbone network and a classification head structure; the second branch structure has a second backbone network and a segmentation head structure;
[0110] A first target image feature extraction module 303 is configured to input the image data into the first branch structure to extract multi-level first target image features;
[0111] A position relationship classification module 304 is configured to input the final-level first target image features into the classification head structure for binary classification to obtain the position relationship between the personnel and the foundation pit;
[0112] A second target image feature extraction module 305 is configured to input the image data and the non-final-level first target image feature into the second backbone network to interactively extract a second target image feature if the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold;
[0113] A semantic segmentation module 306 is configured to input the second target image feature into the segmentation head structure to segment a first pixel point with a semantic meaning of a person and a second pixel point with a semantic meaning of the foundation pit;
[0114] a distance calculation module 307, configured to calculate a second distance between the person and the foundation pit based on the first pixel point and the second pixel point;
[0115] The warning operation execution module 308 is configured to execute a warning operation on the person if the second distance is less than or equal to a second distance threshold; the second distance threshold is less than the first distance threshold.
[0116] In one embodiment of the present application, the first backbone network includes a first convolution module, a second convolution module, and a third convolution module; the multi-level first target image features include a first classification image feature, a second classification image feature, and a third classification image feature;
[0117] The first target image feature extraction module 303 is further configured to:
[0118] Inputting the image data into the first convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first classification image feature;
[0119] Inputting the first classification image feature into the second convolution module, sequentially performing convolution operation, batch normalization processing, and activation operation under linear rectification to obtain a second classification image feature;
[0120] The second classification image feature is input into the third convolution module to perform convolution operation, batch normalization processing and activation operation under linear rectification in sequence to obtain the third classification image feature.
[0121] In one embodiment of the present application, the classification head structure includes a first fully connected layer, a second fully connected layer and a third fully connected layer;
[0122] The position relationship classification module 304 is further configured to:
[0123] Inputting the third classification image feature into the first fully connected layer to map it into a fourth classification image feature;
[0124] Inputting the fourth classification image feature into the second fully connected layer to map it into a fifth classification image feature;
[0125] Inputting the fifth classification image feature into the third fully connected layer to map it into a sixth classification image feature;
[0126] activating the sixth classification image feature to obtain a position probability;
[0127] If the position probability is greater than or equal to a probability threshold, determining the positional relationship between the person and the foundation pit as a first distance between the person and an edge of the foundation pit is less than or equal to a first distance threshold;
[0128] If the position probability is less than the probability threshold, the position relationship between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being greater than a first distance threshold.
[0129] In one embodiment of the present application, the second backbone network includes a fourth convolution module, a fifth convolution module, a sixth convolution module, a seventh convolution module, and an eighth convolution module;
[0130] The second target image feature extraction module 305 is further configured to:
[0131] Inputting the image data into the fourth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first semantic image feature;
[0132] Inputting the first semantic image feature into the fifth convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a second semantic image feature;
[0133] Inputting the second semantic image feature into the sixth convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a third semantic image feature;
[0134] downsampling the second classified image features to obtain first transformed image features;
[0135] splicing the third semantic image feature and the first transformed image feature into a fourth semantic image feature;
[0136] Inputting the fourth semantic image feature into the seventh convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a fifth semantic image feature;
[0137] downsampling the first classified image features to obtain second transformed image features;
[0138] splicing the fifth semantic image feature and the second transformed image feature into a sixth semantic image feature;
[0139] The sixth semantic image feature is input into the eighth convolution module, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second target image feature.
[0140] In one embodiment of the present application, the segmentation head structure includes a first convolutional layer, a second convolutional layer, and a third convolutional layer;
[0141] The semantic segmentation module 306 is further configured to:
[0142] Inputting the second target image feature into the first convolutional layer to perform a convolution operation to obtain a seventh semantic image feature;
[0143] Inputting the seventh semantic image feature into the second convolutional layer to perform a convolution operation to obtain an eighth semantic image feature;
[0144] Inputting the eighth semantic image feature into the third convolutional layer to perform a convolution operation to obtain a ninth semantic image feature;
[0145] An activation operation is performed on the ninth semantic image feature to obtain a first pixel point with the semantic meaning of a person and a second pixel point with the semantic meaning of the foundation pit.
[0146] In one embodiment of the present application, the second target image feature extraction module 305 includes:
[0147] Counting a first number of times that the first distance is less than or equal to a first distance threshold, and a second number of times that the second distance is less than or equal to a second distance threshold within a previous period of time;
[0148] identifying the operating mode of the personnel based on the first number and the second number;
[0149] If the operation mode is to operate at the edge of the foundation pit, it is prohibited to input the image data and the non-final-level first target image features into the second backbone network to interactively extract the second target image features;
[0150] If the operation mode is not operating at the edge of the foundation pit, the image data and the non-final-level first target image features are allowed to be input into the second backbone network to interactively extract the second target image features.
[0151] In one embodiment of the present application, the second target image feature extraction module 305 includes:
[0152] Calculating a ratio between the second number and the first number to obtain a warning ratio;
[0153] If the first number is greater than or equal to a first confidence threshold and the warning ratio is greater than a second confidence threshold, determining that the working mode of the personnel is working at the edge of the foundation pit;
[0154] If the first number is smaller than a first confidence threshold, and / or the warning ratio is smaller than a second confidence threshold, it is determined that the working mode of the personnel is not working at the edge of the foundation pit.
[0155] In one embodiment of the present application, the distance calculation module 307 is further configured to:
[0156] Adding a minimum bounding rectangle to the first pixel point;
[0157] Fitting an arc using the second pixel point;
[0158] Calculate the shortest distance from each corner point in the minimum circumscribed rectangle to the arc respectively;
[0159] The smallest of the shortest distances is multiplied by a preset conversion coefficient to obtain a second distance between the person and the foundation pit.
[0160] An embodiment of the present application provides a device for warning a foundation pit of approaching personnel. By using this device, each step in the aforementioned method embodiments can be implemented.
[0161] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.
[0162] Reference Figure 4 , shows a schematic diagram of a terminal device provided by an embodiment of the present application. Figure 4As shown, the terminal device 400 in the embodiment of the present application includes: a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. When the processor 410 executes the computer program 421, the steps of each embodiment of the method for warning a person approaching a foundation pit are implemented. Alternatively, when the processor 410 executes the computer program 421, the functions of each module / unit in the above-mentioned device embodiments are implemented.
[0163] Exemplarily, the computer program 421 may be divided into one or more modules / units, which are stored in the memory 420 and executed by the processor 410 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments may be used to describe the execution process of the computer program 421 in the terminal device 400.
[0164] The terminal device 400 may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will understand that Figure 4 It is only an example of the terminal device 400 and does not constitute a limitation of the terminal device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 400 may also include input and output devices, network access devices, buses, etc.
[0165] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0166] The memory 420 may be an internal storage unit of the terminal device 400, such as a hard disk or memory of the terminal device 400. The memory 420 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 400. Furthermore, the memory 420 may include both an internal storage unit of the terminal device 400 and an external storage device. The memory 420 is used to store the computer program 421 and other programs and data required by the terminal device 400. The memory 420 may also be used to temporarily store data that has been output or is about to be output.
[0167] An embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for warning personnel approaching a foundation pit as described in the aforementioned embodiments is implemented.
[0168] An embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for warning a person approaching a foundation pit as described in the aforementioned embodiments is implemented.
[0169] An embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is caused to execute the method for warning a person approaching a foundation pit as described in the aforementioned embodiments.
[0170] The above embodiments are intended only to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application and should be included within the scope of protection of the present application.
Claims
1. A method for warning a foundation pit of approaching personnel, characterized in that: include: Collect multiple frames of image data from the foundation pit at the construction site; Load the foundation pit detection network; The foundation pit detection network has a first branch structure and a second branch structure; The first branch structure has a first backbone network and a classification head structure; The second branch structure has a second backbone network and a split head structure; Inputting the image data into the first branch structure to extract multi-level first target image features; Inputting the first target image feature of the final level into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold, inputting the image data and the non-final-level first target image features into the second backbone network for interactively extracting second target image features; Inputting the second target image feature into the segmentation head structure to segment a first pixel point having a semantic meaning of a person and a second pixel point having a semantic meaning of the foundation pit; Calculating a second distance between the person and the foundation pit based on the first pixel point and the second pixel point; If the second distance is less than or equal to a second distance threshold, a warning operation is performed on the person; and the second distance threshold is less than the first distance threshold.
2. The method according to claim 1, characterized in that The first backbone network includes a first convolution module, a second convolution module, and a third convolution module; the multi-level first target image features include a first classification image feature, a second classification image feature, and a third classification image feature; The step of inputting the image data into the first branch structure to extract multi-level first target image features includes: Inputting the image data into the first convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first classification image feature; Inputting the first classification image feature into the second convolution module, sequentially performing convolution operation, batch normalization processing, and activation operation under linear rectification to obtain a second classification image feature; The second classification image feature is input into the third convolution module to perform convolution operation, batch normalization processing and activation operation under linear rectification in sequence to obtain the third classification image feature.
3. The method according to claim 2, characterized in that The classification head structure includes a first fully connected layer, a second fully connected layer and a third fully connected layer; The inputting the final-level first target image feature into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit includes: Inputting the third classification image feature into the first fully connected layer to map it into a fourth classification image feature; Inputting the fourth classification image feature into the second fully connected layer to map it into a fifth classification image feature; Inputting the fifth classification image feature into the third fully connected layer to map it into a sixth classification image feature; activating the sixth classification image feature to obtain a position probability; If the position probability is greater than or equal to a probability threshold, determining the positional relationship between the person and the foundation pit as a first distance between the person and an edge of the foundation pit is less than or equal to a first distance threshold; If the position probability is less than the probability threshold, the position relationship between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being greater than a first distance threshold.
4. The method according to claim 2, characterized in that The second backbone network includes a fourth convolution module, a fifth convolution module, a sixth convolution module, a seventh convolution module and an eighth convolution module; The step of inputting the image data and the non-final-level first target image features into the second backbone network to interactively extract the second target image features comprises: Inputting the image data into the fourth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first semantic image feature; Inputting the first semantic image feature into the fifth convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a second semantic image feature; Inputting the second semantic image feature into the sixth convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a third semantic image feature; downsampling the second classified image features to obtain first transformed image features; splicing the third semantic image feature and the first transformed image feature into a fourth semantic image feature; Inputting the fourth semantic image feature into the seventh convolution module, sequentially performing convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a fifth semantic image feature; downsampling the first classified image features to obtain second transformed image features; splicing the fifth semantic image feature and the second transformed image feature into a sixth semantic image feature; The sixth semantic image feature is input into the eighth convolution module, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second target image feature.
5. The method according to claim 4, characterized in that The segmentation head structure includes a first convolutional layer, a second convolutional layer and a third convolutional layer; The step of inputting the second target image feature into the segmentation head structure to segment a first pixel point having a semantic meaning of a person and a second pixel point having a semantic meaning of the foundation pit comprises: Inputting the second target image feature into the first convolutional layer to perform a convolution operation to obtain a seventh semantic image feature; Inputting the seventh semantic image feature into the second convolutional layer to perform a convolution operation to obtain an eighth semantic image feature; Inputting the eighth semantic image feature into the third convolutional layer to perform a convolution operation to obtain a ninth semantic image feature; An activation operation is performed on the ninth semantic image feature to obtain a first pixel point with the semantic meaning of a person and a second pixel point with the semantic meaning of the foundation pit.
6. The method according to any one of claims 1 to 5, characterized in that The step of inputting the image data and the non-final-level first target image features into the second backbone network to interactively extract the second target image features comprises: Counting a first number of times that the first distance is less than or equal to a first distance threshold, and a second number of times that the second distance is less than or equal to a second distance threshold within a previous period of time; identifying the operating mode of the personnel based on the first number and the second number; If the operation mode is to operate at the edge of the foundation pit, it is prohibited to input the image data and the non-final-level first target image features into the second backbone network to interactively extract the second target image features; If the operation mode is not operating at the edge of the foundation pit, the image data and the non-final-level first target image features are allowed to be input into the second backbone network to interactively extract the second target image features.
7. The method according to claim 6, characterized in that The identifying the operation mode of the personnel according to the first quantity and the second quantity includes: Calculating a ratio between the second number and the first number to obtain a warning ratio; If the first number is greater than or equal to a first confidence threshold and the warning ratio is greater than a second confidence threshold, determining that the working mode of the personnel is working at the edge of the foundation pit; If the first number is smaller than a first confidence threshold, and / or the warning ratio is smaller than a second confidence threshold, it is determined that the working mode of the personnel is not working at the edge of the foundation pit.
8. The method according to any one of claims 1 to 5, characterized in that The calculating the second distance between the person and the foundation pit based on the first pixel point and the second pixel point includes: Adding a minimum bounding rectangle to the first pixel point; Fitting an arc using the second pixel point; Calculate the shortest distance from each corner point in the minimum circumscribed rectangle to the arc respectively; The smallest of the shortest distances is multiplied by a preset conversion coefficient to obtain a second distance between the person and the foundation pit.
9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for warning a foundation pit of approaching personnel as described in any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for warning a foundation pit of approaching personnel as described in any one of claims 1 to 8 is implemented.