Foundation pit early warning personnel approaching method and device and storage medium

Abstract

The invention provides a method and equipment for early warning personnel approaching a foundation pit, and a storage medium. The method comprises the following steps: collecting multi-frame image data of the foundation pit of a construction site; inputting the image data into the first branch structure to extract multi-stage first target image features; the last-stage first target image features are input into a classification head structure for binary classification, and the position relation between the personnel and the foundation pit is obtained; inputting the image data and a non-last-stage first target image feature into a second backbone network to interactively extract a second target image feature; inputting the second target image features into a segmentation head structure, and segmenting first pixel points of which the semantics is personnel and second pixel points of which the semantics is foundation pits; calculating a second distance between the personnel and the foundation pit according to the first pixel point and the second pixel point; and if the second distance is smaller than or equal to the second distance threshold value, executing early warning operation on the personnel. The embodiment does not depend on the pre-marked virtual range, the operation convenience is greatly improved, and the early warning efficiency is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of computer vision, and in particular, relate to a method, device, and storage medium for warning a foundation pit of approaching personnel. Background Art

[0002] With the continuous improvement of economy and technology, higher requirements are placed on the quality of the entire engineering construction. Among them, foundation pits face certain risks, such as collapse, burial, falling from heights, mechanical injuries and electric shock, and their safety management is receiving more and more attention.

[0003] At present, the management of foundation pits mainly involves deploying cameras, marking a virtual range for the foundation pit in the video data collected by the cameras, and issuing an early warning when a person is detected entering the virtual range.

[0004] However, there are vibrations during construction on the construction site, which will be transmitted to the camera. The camera will continuously accumulate tiny offsets, causing the virtual range to gradually deviate from the foundation pit, resulting in a decrease in the accuracy of whether the inspection personnel have entered the virtual range, and the accuracy of the early warning will also be reduced accordingly.

[0005] At this time, the administrator needs to re-mark the virtual range and repeatedly mark the virtual range, which is cumbersome and leads to low efficiency of early warning. Summary of the invention

[0006] In view of this, an embodiment of the present application provides a method, device and storage medium for warning a foundation pit of personnel approaching, so as to improve the accuracy and efficiency of warning a personnel approaching a foundation pit.

[0007] A first aspect of an embodiment of the present application provides a method for warning a foundation pit of a person approaching, comprising:

[0008] Collect multiple frames of image data from the foundation pit at the construction site;

[0009] Loading a foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first trunk network and a classification head structure; the second branch structure has a second trunk network and a segmentation head structure;

[0010] Inputting the image data into the first branch structure to extract multi-level first target image features;

[0011] Inputting the first target image feature of the final level into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit;

[0012] If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold, inputting the image data and the non-final first target image features into the second backbone network to interactively extract second target image features;

[0013] Inputting the second target image feature into the segmentation head structure to segment a first pixel point with a semantic meaning of a person and a second pixel point with a semantic meaning of the foundation pit;

[0014] Calculating a second distance between the person and the foundation pit according to the first pixel point and the second pixel point;

[0015] If the second distance is less than or equal to a second distance threshold, a warning operation is performed on the person; and the second distance threshold is less than the first distance threshold.

[0016] A second aspect of an embodiment of the present application provides a device for warning a foundation pit of a person approaching, comprising:

[0017] An image data acquisition module is used to collect multiple frames of image data of the foundation pit at the construction site;

[0018] A foundation pit detection network loading module, used for loading a foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first trunk network and a classification head structure; the second branch structure has a second trunk network and a segmentation head structure;

[0019] A first target image feature extraction module, used for inputting the image data into the first branch structure to extract multi-level first target image features;

[0020] A position relationship classification module, used for inputting the first target image feature of the final level into the classification head structure for binary classification to obtain the position relationship between the personnel and the foundation pit;

[0021] A second target image feature extraction module, configured to input the image data and the non-final first target image feature into the second backbone network to interactively extract a second target image feature if the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold;

[0022] A semantic segmentation module, used for inputting the second target image feature into the segmentation head structure to segment a first pixel point with a semantic meaning of a person and a second pixel point with a semantic meaning of the foundation pit;

[0023] a distance calculation module, configured to calculate a second distance between the person and the foundation pit according to the first pixel point and the second pixel point;

[0024] A warning operation execution module is used to execute a warning operation on the person if the second distance is less than or equal to a second distance threshold; the second distance threshold is less than the first distance threshold.

[0025] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for warning a person approaching a foundation pit as described in the first aspect above is implemented.

[0026] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for warning a person approaching a foundation pit as described in the first aspect above is implemented.

[0027] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the method for warning a foundation pit of approaching personnel as described in the first aspect.

[0028] In this embodiment, multiple frames of image data are collected from the foundation pit at the construction site; a foundation pit detection network is loaded; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first backbone network and a classification head structure; the second branch structure has a second backbone network and a segmentation head structure; the image data is input into the first branch structure to extract multi-level first target image features; the final first target image features are input into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; if the positional relationship is that the first distance between the personnel and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final first target image features are input into the second backbone network to interactively extract the second target image features; the second target image features are input into the segmentation head structure to segment the first pixel point with the semantics of the personnel and the second pixel point with the semantics of the foundation pit; the second distance between the personnel and the foundation pit is calculated based on the first pixel point and the second pixel point; if the second distance is less than or equal to the second distance threshold, an early warning operation is performed on the personnel; the second distance threshold is less than the first distance threshold. This embodiment uses classification at the first level to detect whether a person is close to a foundation pit, and uses semantic segmentation at the second level to detect whether a person is close to a foundation pit, layer by layer, which not only ensures the accuracy of the detection, thereby alarming the accuracy of the early warning, but also optimizes the resource consumption of the foundation pit detection network. This process does not rely on a pre-marked virtual range. Even if the camera is offset due to the vibration of the construction site, it can still detect whether a person is close to the foundation pit normally. There is no need for the administrator to mark the virtual range, which greatly improves the ease of operation and effectively improves the efficiency of the early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 It is a schematic diagram of a method for warning a person approaching a foundation pit provided by an embodiment of the present application;

[0031] Figure 2 is a schematic diagram of a foundation pit detection network provided in an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of a device for warning a person approaching a foundation pit provided by an embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0035] The technical solution of the present application is described below through specific embodiments.

[0036] Reference Figure 1 , showing a schematic diagram of a method for warning a person approaching a foundation pit provided by an embodiment of the present application, which may specifically include the following steps:

[0037] Step 101: collect multiple frames of image data of a foundation pit at a construction site.

[0038] In this embodiment, one or more cameras may be deployed on the construction site, and each camera may collect video data facing the foundation pit, wherein the video data includes multiple frames of image data.

[0039] Step 102: Load the foundation pit detection network.

[0040] The camera can transmit video data to the local edge computing node or cloud, and the edge computing node or cloud deploys and loads the foundation pit detection network to detect whether people are approaching the foundation pit.

[0041] Among them, the foundation pit detection network has a first branch structure and a second branch structure; Figure 2 As shown, the first branch structure has a first backbone network Backbone_1 and a classification head structure Head_1; the second branch structure has a second backbone network Backbone_2 and a segmentation head structure Head_2.

[0042] The first branch structure is used for coarse detection of whether personnel are close to the foundation pit, and the second branch structure is used for fine detection of whether personnel are close to the foundation pit.

[0043] When training the foundation pit detection network, the first branch structure is trained first, and the second branch structure is trained while keeping the parameters of the first branch structure unchanged.

[0044] When training the first branch structure, the first backbone network Backbone_1 and the classification head structure Head_1 are trained separately as binary classification networks. The first distance between the positive sample labeled person and the edge of the foundation pit is less than or equal to the first distance threshold, and the first distance between the negative sample labeled person and the edge of the foundation pit is greater than the first distance threshold. The loss function can select cross entropy.

[0045] When training the second branch structure, the first backbone network Backbone_1, the second backbone network Backbone_2 and the segmentation head structure Head_2 are trained separately as semantic segmentation networks. The samples are labeled with the first pixel point whose semantics is a person and the second pixel point whose semantics is a pit. The loss function can be selected from cross entropy, Dice coefficient, etc. The parameters of the first backbone network Backbone_1 remain unchanged.

[0046] Step 103: Input the image data into the first branch structure to extract multi-level first target image features.

[0047] In this embodiment, the image data Image may be input into the first branch structure Backbone_1 to extract multiple levels of first target image features in sequence according to its hierarchical structure.

[0048] In the specific implementation, Figure 2 As shown, the first backbone network Backbone_1 includes a first convolution module ConvModule_1, a second convolution module ConvModule_2 and a third convolution module ConvModule_3; accordingly, the multi-level first target image features include a first classification image feature, a second classification image feature and a third classification image feature.

[0049] Among them, the first convolution module ConvModule_1, the second convolution module ConvModule_2 and the third convolution module ConvModule_3 have the same structure, all of which include a convolutional layer (Convolutional Layer), a BN (Batch Normalization) function, and a ReLU (Rectified Linear Unit) function. The convolutional layer provides convolution operations, the BN function provides batch normalization processing, and the ReLU provides activation processing under linear rectification.

[0050] In this first backbone network Backbone_1, the image data Image is input into the first convolution module ConvModule_1, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the first classification image feature.

[0051] The first classification image features are input into the second convolution module ConvModule_2 to perform convolution operations, batch normalization processing and activation operations under linear rectification in sequence to obtain the second classification image features.

[0052] The second classification image features are input into the third convolution module ConvModule_3, and convolution operation, batch normalization processing and activation operation under linear rectification are performed in sequence to obtain the third classification image features.

[0053] Step 104: Input the final first target image feature into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit.

[0054] In this embodiment, if Figure 2 As shown, the first target image feature of the final level (i.e., the last level) is input into the classification head structure Head_1 for binary classification to obtain the positional relationship between the personnel and the foundation pit.

[0055] When the multi-level first target image features include first classification image features, second classification image features and third classification image features, the final-level first target image features refer to the third classification image features, and the non-final-level first target image features refer to the first classification image features and the second classification image features.

[0056] In the specific implementation, Figure 2 As shown, the classification head structure Head_1 includes a first fully connected layer FC_1, a second fully connected layer FC_2 and a third fully connected layer FC_3.

[0057] In this classification head structure Head_1, the third classification image feature is input into the first fully connected layer FC_1 and mapped into the fourth classification image feature.

[0058] The fourth classification image feature is input into the second fully connected layer FC_2 and mapped into the fifth classification image feature.

[0059] The fifth classification image feature is input into the third fully connected layer FC_3 and mapped into the sixth classification image feature.

[0060] The activation function such as Sigmoid (S-type function) is used to activate the sixth classification image feature to obtain the position probability.

[0061] If the position probability is greater than or equal to the probability threshold, the position relationship Class between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being less than or equal to a first distance threshold.

[0062] If the position probability is less than the probability threshold, the position relationship Class between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being greater than a first distance threshold.

[0063] Step 105: If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final first target image features are input into the second backbone network to interactively extract the second target image features.

[0064] If the position relationship between the person and the foundation pit is that the first distance between the person and the edge of the foundation pit is less than or equal to the first distance threshold, it means that the person is likely to approach the foundation pit. Figure 2 As shown, the image data Image and the non-final first target image features are input into the second backbone network Backbone_2 for interaction, and the second target image features are extracted therefrom.

[0065] There are features of personnel and foundation pits in the multi-level first target image features. The final first target image features are differentiated into classifications, while the non-final first target image features are not differentiated obviously, which helps to improve the quality of the second target image features.

[0066] When resources are sufficient, such as when the foundation pit detection network is deployed in the cloud, the second branch structure can be operated sustainably.

[0067] In cases where resources are limited, such as when the foundation pit detection network is deployed on edge computing nodes, the second branch structure can be selectively operated.

[0068] In this case, a first number of first distances less than or equal to a first distance threshold and a second number of second distances less than or equal to a second distance threshold in a previous period of time (ie, a recent period of time, such as the last 2 hours) may be counted respectively.

[0069] A statistical analysis is performed based on the first quantity and the second quantity to identify the operation mode of the personnel.

[0070] Exemplarily, the ratio between the second number and the first number is calculated to obtain the warning ratio.

[0071] If the first number is greater than or equal to the first confidence threshold and the warning ratio is greater than the second confidence threshold, it means that the warning is more frequent, and it is determined that the working mode of the personnel is working at the edge of the foundation pit.

[0072] If the first number is less than the first confidence threshold, and / or the warning ratio is less than the second confidence threshold, it is determined that the working mode of the personnel is not working at the edge of the foundation pit.

[0073] Furthermore, when the second distance is less than or equal to the second distance threshold, an early warning operation will be performed on the personnel. Under normal circumstances, if the personnel are not working at the edge of the foundation pit, they will follow the early warning and stay away from the foundation pit or be driven away by the supervisor. Therefore, if the early warning is frequent but the personnel do not stay away from the foundation pit, they are usually temporarily working near the foundation pit and are recognized by the supervisor, and the early warning operation will be processed silently.

[0074] If the operation mode is to work at the edge of the foundation pit, it is prohibited to input the image data and the non-final first target image features into the second backbone network Backbone_2 to interactively extract the second target image features.

[0075] If the operation mode is not to work at the edge of the foundation pit, the image data and the non-final first target image features are allowed to be input into the second backbone network Backbone_2 to interactively extract the second target image features, thereby reducing resource consumption.

[0076] In the specific implementation, Figure 2 As shown, the second backbone network Backbone_2 includes a fourth convolution module ConvModule_4, a fifth convolution module ConvModule_5, a sixth convolution module ConvModule_6, a seventh convolution module ConvModule_7 and an eighth convolution module ConvModule_8.

[0077] Among them, the fourth convolution module ConvModule_4, the fifth convolution module ConvModule_5, the sixth convolution module ConvModule_6, the seventh convolution module ConvModule_7 and the eighth convolution module ConvModule_8 have the same structure, all of which include convolution layers, BN functions, and ReLU functions. The convolution layer provides convolution operations, the BN function provides batch normalization processing, and the ReLU provides activation processing under linear rectification.

[0078] In this second backbone network Backbone_2, the image data Image is input into the fourth convolution module ConvModule_4, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the first semantic image feature.

[0079] The first semantic image feature is input into the fifth convolution module ConvModule_5, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second semantic image feature.

[0080] The second semantic image feature is input into the sixth convolution module ConvModule_6, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the third semantic image feature.

[0081] DownSample is performed on the second classification image feature to obtain the first transformed image feature.

[0082] The third semantic image feature and the first transformed image feature are concatenated to obtain a fourth semantic image feature.

[0083] The fourth semantic image feature is input into the seventh convolution module ConvModule_7, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the fifth semantic image feature.

[0084] DownSample the first classification image feature to obtain a second transformed image feature.

[0085] The fifth semantic image feature and the second transformed image feature are concatenated to obtain a sixth semantic image feature.

[0086] The sixth semantic image feature is input into the eighth convolution module ConvModule_8, and convolution operation, batch normalization processing and activation processing under linear rectification are performed in sequence to obtain the second target image feature.

[0087] Step 106: Input the second target image feature into the segmentation head structure to segment the first pixel point with the semantics of a person and the second pixel point with the semantics of a foundation pit.

[0088] In this embodiment, if Figure 2 As shown, the second target image feature is input into the segmentation head structure Head_2 to segment a first pixel point whose semantics is a person and a second pixel point whose semantics is a pit.

[0089] In the specific implementation, Figure 2As shown, the segmentation head structure Head_2 includes a first convolutional layer Conv_1, a second convolutional layer Conv_2 and a third convolutional layer Conv_3.

[0090] In this segmentation head structure Head_2, the second target image feature is input into the first convolution layer Conv_1 to perform a convolution operation to obtain the seventh semantic image feature.

[0091] The seventh semantic image feature is input into the second convolutional layer Conv_2 to perform a convolution operation to obtain an eighth semantic image feature.

[0092] The eighth semantic image feature is input into the third convolutional layer Conv_3 to perform a convolution operation to obtain a ninth semantic image feature.

[0093] An activation operation is performed on the ninth semantic image feature using an activation function such as Sigmoid to obtain a first pixel point whose semantic semantic is a person and a second pixel point whose semantic semantic is a pit.

[0094] Step 107: Calculate the second distance between the person and the foundation pit based on the first pixel point and the second pixel point.

[0095] In this embodiment, the first pixel point and the second pixel point may be measured to calculate the second distance between the personnel and the foundation pit.

[0096] Considering that the first pixel point with the semantics of a person and the second pixel point with the semantics of a pit are both irregular shapes, in order to improve calculation efficiency, this embodiment can regularize the first pixel point with the semantics of a person and the second pixel point with the semantics of a pit.

[0097] In a specific implementation, a minimum bounding rectangle is added to the first pixel point, and the minimum bounding rectangle has four corner points.

[0098] The arc is fitted using the second pixel point according to a polynomial or other method.

[0099] The shortest distance from each corner point in the minimum circumscribed rectangle to the arc is calculated respectively, that is, a line segment is generated through each corner point in the minimum circumscribed rectangle, the line segment intersects the arc at an intersection point, the tangent of the arc at the intersection point is perpendicular to the line segment, and the length of the line segment is the shortest distance from each corner point in the minimum circumscribed rectangle to the arc.

[0100] Compare the shortest distances corresponding to each corner point, multiply the smallest shortest distance by a preset conversion coefficient, and obtain the second distance between the personnel and the foundation pit.

[0101] The conversion coefficient is the actual distance corresponding to the unit pixel that is pre-calibrated for the camera.

[0102] Step 108: If the second distance is less than or equal to the second distance threshold, a warning operation is performed on the person.

[0103] In this embodiment, the second distance between the person and the foundation pit is compared with a second distance threshold, and the second distance threshold is smaller than the first distance threshold.

[0104] If the second distance is less than or equal to the second distance threshold, a warning operation is performed on the person (i.e., the first pixel point with the semantics of a person), such as highlighting the first pixel point with the semantics of a person, playing a warning sound, etc., to prompt the supervisor to pay attention to whether the person is illegally approaching the foundation pit.

[0105] In this embodiment, multiple frames of image data are collected from the foundation pit at the construction site; a foundation pit detection network is loaded; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first backbone network and a classification head structure; the second branch structure has a second backbone network and a segmentation head structure; the image data is input into the first branch structure to extract multi-level first target image features; the final first target image features are input into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; if the positional relationship is that the first distance between the personnel and the edge of the foundation pit is less than or equal to the first distance threshold, the image data and the non-final first target image features are input into the second backbone network to interactively extract the second target image features; the second target image features are input into the segmentation head structure to segment the first pixel point with the semantics of the personnel and the second pixel point with the semantics of the foundation pit; the second distance between the personnel and the foundation pit is calculated based on the first pixel point and the second pixel point; if the second distance is less than or equal to the second distance threshold, an early warning operation is performed on the personnel; the second distance threshold is less than the first distance threshold. This embodiment uses classification at the first level to detect whether a person is close to a foundation pit, and uses semantic segmentation at the second level to detect whether a person is close to a foundation pit, layer by layer, which not only ensures the accuracy of the detection, thereby alarming the accuracy of the early warning, but also optimizes the resource consumption of the foundation pit detection network. This process does not rely on a pre-marked virtual range. Even if the camera is offset due to the vibration of the construction site, it can still detect whether a person is close to the foundation pit normally. There is no need for the administrator to mark the virtual range, which greatly improves the ease of operation and effectively improves the efficiency of the early warning.

[0106] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0107] Reference Figure 3 , shows a schematic diagram of a device for warning a person approaching a foundation pit provided by an embodiment of the present application, which may specifically include the following modules:

[0108] The image data acquisition module 301 is used to acquire multiple frames of image data of the foundation pit at the construction site;

[0109] The foundation pit detection network loading module 302 is used to load the foundation pit detection network; the foundation pit detection network has a first branch structure and a second branch structure; the first branch structure has a first trunk network and a classification head structure; the second branch structure has a second trunk network and a segmentation head structure;

[0110] A first target image feature extraction module 303, used for inputting the image data into the first branch structure to extract multi-level first target image features;

[0111] A position relationship classification module 304, used for inputting the final first target image feature into the classification head structure for binary classification to obtain the position relationship between the personnel and the foundation pit;

[0112] A second target image feature extraction module 305 is configured to input the image data and the non-final first target image feature into the second backbone network to interactively extract the second target image feature if the position relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold;

[0113] A semantic segmentation module 306, configured to input the second target image feature into the segmentation head structure to segment a first pixel point with a semantic meaning of a person and a second pixel point with a semantic meaning of the foundation pit;

[0114] A distance calculation module 307, configured to calculate a second distance between the person and the foundation pit according to the first pixel point and the second pixel point;

[0115] The warning operation execution module 308 is used to execute a warning operation on the person if the second distance is less than or equal to a second distance threshold; the second distance threshold is less than the first distance threshold.

[0116] In one embodiment of the present application, the first backbone network includes a first convolution module, a second convolution module and a third convolution module; the multi-level first target image features include a first classification image feature, a second classification image feature and a third classification image feature;

[0117] The first target image feature extraction module 303 is also used for:

[0118] Inputting the image data into the first convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first classification image feature;

[0119] Inputting the first classification image feature into the second convolution module to sequentially perform convolution operation, batch normalization processing and activation operation under linear rectification to obtain a second classification image feature;

[0120] The second classification image feature is input into the third convolution module to sequentially perform convolution operation, batch normalization processing and activation operation under linear rectification to obtain the third classification image feature.

[0121] In one embodiment of the present application, the classification head structure includes a first fully connected layer, a second fully connected layer and a third fully connected layer;

[0122] The position relationship classification module 304 is also used for:

[0123] Inputting the third classification image feature into the first fully connected layer to map it into a fourth classification image feature;

[0124] Inputting the fourth classification image feature into the second fully connected layer to map it into a fifth classification image feature;

[0125] Inputting the fifth classification image feature into the third fully connected layer to map it into a sixth classification image feature;

[0126] activating the sixth classification image feature to obtain a position probability;

[0127] If the position probability is greater than or equal to the probability threshold, determining the position relationship between the person and the foundation pit as a first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold;

[0128] If the position probability is less than the probability threshold, the position relationship between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being greater than a first distance threshold.

[0129] In one embodiment of the present application, the second backbone network includes a fourth convolution module, a fifth convolution module, a sixth convolution module, a seventh convolution module and an eighth convolution module;

[0130] The second target image feature extraction module 305 is also used for:

[0131] Inputting the image data into the fourth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first semantic image feature;

[0132] Inputting the first semantic image feature into the fifth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a second semantic image feature;

[0133] Inputting the second semantic image feature into the sixth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a third semantic image feature;

[0134] Downsampling the second classified image feature to obtain a first transformed image feature;

[0135] splicing the third semantic image feature and the first transformed image feature into a fourth semantic image feature;

[0136] Inputting the fourth semantic image feature into the seventh convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a fifth semantic image feature;

[0137] Downsampling the first classified image features to obtain second transformed image features;

[0138] splicing the fifth semantic image feature and the second transformed image feature into a sixth semantic image feature;

[0139] The sixth semantic image feature is input into the eighth convolution module to sequentially perform convolution operation, batch normalization processing and activation processing under linear rectification to obtain a second target image feature.

[0140] In one embodiment of the present application, the segmentation head structure includes a first convolutional layer, a second convolutional layer and a third convolutional layer;

[0141] The semantic segmentation module 306 is also used for:

[0142] Inputting the second target image feature into the first convolutional layer to perform a convolution operation to obtain a seventh semantic image feature;

[0143] Inputting the seventh semantic image feature into the second convolutional layer to perform a convolution operation to obtain an eighth semantic image feature;

[0144] Inputting the eighth semantic image feature into the third convolutional layer to perform a convolution operation to obtain a ninth semantic image feature;

[0145] An activation operation is performed on the ninth semantic image feature to obtain a first pixel point with the semantics of a person and a second pixel point with the semantics of the foundation pit.

[0146] In one embodiment of the present application, the second target image feature extraction module 305 includes:

[0147] Respectively counting a first number of times that the first distance is less than or equal to a first distance threshold and a second number of times that the second distance is less than or equal to a second distance threshold in a previous period of time;

[0148] identifying the operation mode of the personnel according to the first number and the second number;

[0149] If the operation mode is to operate at the edge of the foundation pit, it is prohibited to input the image data and the non-final first target image features into the second backbone network to interactively extract the second target image features;

[0150] If the operation mode is not to operate at the edge of the foundation pit, the image data and the non-final first target image features are allowed to be input into the second backbone network to interactively extract the second target image features.

[0151] In one embodiment of the present application, the second target image feature extraction module 305 includes:

[0152] Calculating a ratio between the second number and the first number to obtain a warning ratio;

[0153] If the first number is greater than or equal to a first confidence threshold and the warning ratio is greater than a second confidence threshold, determining that the working mode of the personnel is working at the edge of the foundation pit;

[0154] If the first number is smaller than a first confidence threshold, and / or the warning ratio is smaller than a second confidence threshold, it is determined that the working mode of the personnel is not working at the edge of the foundation pit.

[0155] In one embodiment of the present application, the distance calculation module 307 is further used to:

[0156] Adding a minimum bounding rectangle to the first pixel point;

[0157] Fitting an arc using the second pixel point;

[0158] Calculate the shortest distance from each corner point in the minimum circumscribed rectangle to the arc respectively;

[0159] The smallest of the shortest distances is multiplied by a preset conversion coefficient to obtain a second distance between the person and the foundation pit.

[0160] An embodiment of the present application provides a device for warning a foundation pit of approaching personnel. By using the device, each step in the aforementioned method embodiments can be implemented.

[0161] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0162] Reference Figure 4 , shows a schematic diagram of a terminal device provided in an embodiment of the present application. Figure 4As shown, the terminal device 400 in the embodiment of the present application includes: a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. When the processor 410 executes the computer program 421, the steps in each embodiment of the above-mentioned method for warning personnel approaching the foundation pit are implemented. Alternatively, when the processor 410 executes the computer program 421, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0163] Exemplarily, the computer program 421 may be divided into one or more modules / units, which are stored in the memory 420 and executed by the processor 410 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 421 in the terminal device 400.

[0164] The terminal device 400 may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 400 and does not constitute a limitation on the terminal device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 400 may also include input and output devices, network access devices, buses, etc.

[0165] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] The memory 420 may be an internal storage unit of the terminal device 400, such as a hard disk or memory of the terminal device 400. The memory 420 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 400. Further, the memory 420 may also include both an internal storage unit of the terminal device 400 and an external storage device. The memory 420 is used to store the computer program 421 and other programs and data required by the terminal device 400. The memory 420 may also be used to temporarily store data that has been output or is to be output.

[0167] An embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for warning a person approaching a foundation pit as described in the aforementioned embodiments is implemented.

[0168] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for warning a person approaching a foundation pit as described in the aforementioned embodiments is implemented.

[0169] The embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method for warning a person approaching a foundation pit as described in the aforementioned embodiments.

[0170] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for warning a foundation pit of approaching personnel, characterized in that: include: Collect multiple frames of image data from the foundation pit at the construction site; Load the foundation pit detection network; The foundation pit detection network has a first branch structure and a second branch structure; The first branch structure has a first backbone network and a classification head structure; The second branch structure has a second backbone network and a split head structure; Inputting the image data into the first branch structure to extract multi-level first target image features; Inputting the first target image feature of the final level into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit; If the positional relationship is that the first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold, inputting the image data and the non-final first target image features into the second backbone network to interactively extract second target image features; Inputting the second target image feature into the segmentation head structure to segment a first pixel point with a semantic meaning of a person and a second pixel point with a semantic meaning of the foundation pit; Calculating a second distance between the person and the foundation pit according to the first pixel point and the second pixel point; If the second distance is less than or equal to a second distance threshold, a warning operation is performed on the person; and the second distance threshold is less than the first distance threshold.

2. The method according to claim 1, characterized in that The first backbone network includes a first convolution module, a second convolution module and a third convolution module; the multi-level first target image features include a first classification image feature, a second classification image feature and a third classification image feature; The step of inputting the image data into the first branch structure to extract multi-level first target image features comprises: Inputting the image data into the first convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first classification image feature; Inputting the first classification image feature into the second convolution module to sequentially perform convolution operation, batch normalization processing and activation operation under linear rectification to obtain a second classification image feature; The second classification image feature is input into the third convolution module to sequentially perform convolution operation, batch normalization processing and activation operation under linear rectification to obtain the third classification image feature.

3. The method according to claim 2, characterized in that The classification head structure includes a first fully connected layer, a second fully connected layer and a third fully connected layer; The inputting the final-level first target image feature into the classification head structure for binary classification to obtain the positional relationship between the personnel and the foundation pit includes: Inputting the third classification image feature into the first fully connected layer to map it into a fourth classification image feature; Inputting the fourth classification image feature into the second fully connected layer to map it into a fifth classification image feature; Inputting the fifth classification image feature into the third fully connected layer to map it into a sixth classification image feature; activating the sixth classification image feature to obtain a position probability; If the position probability is greater than or equal to the probability threshold, determining the position relationship between the person and the foundation pit as a first distance between the person and the edge of the foundation pit is less than or equal to a first distance threshold; If the position probability is less than the probability threshold, the position relationship between the person and the foundation pit is determined as a first distance between the person and the edge of the foundation pit being greater than a first distance threshold.

4. The method according to claim 2, characterized in that: The second backbone network includes a fourth convolution module, a fifth convolution module, a sixth convolution module, a seventh convolution module and an eighth convolution module; The step of inputting the image data and the non-final first target image features into the second backbone network to interactively extract the second target image features comprises: Inputting the image data into the fourth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a first semantic image feature; Inputting the first semantic image feature into the fifth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a second semantic image feature; Inputting the second semantic image feature into the sixth convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a third semantic image feature; Downsampling the second classified image feature to obtain a first transformed image feature; splicing the third semantic image feature and the first transformed image feature into a fourth semantic image feature; Inputting the fourth semantic image feature into the seventh convolution module to sequentially perform convolution operation, batch normalization processing, and activation processing under linear rectification to obtain a fifth semantic image feature; Downsampling the first classified image features to obtain second transformed image features; splicing the fifth semantic image feature and the second transformed image feature into a sixth semantic image feature; The sixth semantic image feature is input into the eighth convolution module to sequentially perform convolution operation, batch normalization processing and activation processing under linear rectification to obtain a second target image feature.

5. The method according to claim 4, characterized in that The segmentation head structure includes a first convolutional layer, a second convolutional layer and a third convolutional layer; The step of inputting the second target image feature into the segmentation head structure to segment the first pixel point with the semantic meaning of a person and the second pixel point with the semantic meaning of the foundation pit comprises: Inputting the second target image feature into the first convolutional layer to perform a convolution operation to obtain a seventh semantic image feature; Inputting the seventh semantic image feature into the second convolutional layer to perform a convolution operation to obtain an eighth semantic image feature; Inputting the eighth semantic image feature into the third convolutional layer to perform a convolution operation to obtain a ninth semantic image feature; An activation operation is performed on the ninth semantic image feature to obtain a first pixel point with the semantics of a person and a second pixel point with the semantics of the foundation pit.

6. The method according to any one of claims 1 to 5, characterized in that The step of inputting the image data and the non-final first target image features into the second backbone network to interactively extract the second target image features comprises: Respectively counting a first number of times that the first distance is less than or equal to a first distance threshold and a second number of times that the second distance is less than or equal to a second distance threshold in a previous period of time; identifying the operation mode of the personnel according to the first number and the second number; If the operation mode is to operate at the edge of the foundation pit, it is prohibited to input the image data and the non-final first target image features into the second backbone network to interactively extract the second target image features; If the operation mode is not to operate at the edge of the foundation pit, the image data and the non-final first target image features are allowed to be input into the second backbone network to interactively extract the second target image features.

7. The method according to claim 6, characterized in that The identifying the operation mode of the personnel according to the first number and the second number includes: Calculating a ratio between the second number and the first number to obtain a warning ratio; If the first number is greater than or equal to a first confidence threshold and the warning ratio is greater than a second confidence threshold, determining that the working mode of the personnel is working at the edge of the foundation pit; If the first number is smaller than a first confidence threshold, and / or the warning ratio is smaller than a second confidence threshold, it is determined that the working mode of the personnel is not working at the edge of the foundation pit.

8. The method according to any one of claims 1 to 5, characterized in that The calculating the second distance between the person and the foundation pit according to the first pixel point and the second pixel point includes: Adding a minimum bounding rectangle to the first pixel point; Fitting an arc using the second pixel point; Calculate the shortest distance from each corner point in the minimum circumscribed rectangle to the arc respectively; The smallest of the shortest distances is multiplied by a preset conversion coefficient to obtain a second distance between the person and the foundation pit.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for warning a foundation pit of approaching personnel is implemented as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for warning a foundation pit of approaching personnel as described in any one of claims 1 to 8 is implemented.

Images