Risk early warning method and device for underground pipe network and storage medium

Abstract

The invention relates to the technical field of risk early warning, and discloses a risk early warning method and device for an underground pipe network and a storage medium. The method comprises the steps that the actual distance between a bucket of an excavator and the corresponding ground position of an underground pipeline is obtained; under the condition that the actual distance is smaller than a set threshold value, relevant video data of a bucket of the excavator, sound data during construction operation and vibration data during bucket operation are obtained; determining the state of the excavator according to the related video data of the bucket of the excavator, the sound data during construction operation and the vibration data during bucket operation; and under the condition that the excavator is in the working state, an alarm mechanism is triggered. According to the scheme, the alarm mechanism can be triggered immediately when the excavator is in the working state possibly endangering the safety of the underground pipeline. Therefore, an operator is reminded to adjust the operation mode in time, and the accident risk caused by human negligence is effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of risk warning technology, for example, to a risk warning method, device and storage medium for underground pipe networks. Background Art

[0002] At this stage, with the continuous development of urbanization, infrastructure construction and maintenance have become an important cornerstone to support social and economic development. Specifically, as a large-scale construction machinery, excavators play an indispensable role in earth excavation, road construction, pipeline laying and other projects. However, during the operation of the excavator, due to the complex layout and strong concealment of underground pipelines, accidents of digging up underground pipelines occur from time to time. These pipelines include but are not limited to key infrastructure such as water supply, gas, electricity, and communications. Their destruction will not only lead to waste of resources and environmental pollution, but may also cause serious safety accidents.

[0003] At present, the relevant technology discloses a risk warning method for underground pipelines, including: setting up monitoring cameras in the construction area or using infrared sensors to detect the excavator, and triggering the alarm system when the excavator is detected entering the preset area.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] Although the relevant technology can achieve a certain degree of risk warning, it relies on only a single type of sensor for risk monitoring and cannot comprehensively and accurately judge the working status of the excavator and its specific position relationship with the underground pipeline, so it is prone to false alarms or missed alarms.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a risk warning method, device and storage medium for underground pipeline networks, which can improve the accuracy of risk warning for underground pipeline networks.

[0009] In some embodiments, the risk warning method for underground pipelines includes: obtaining the actual distance between the excavator bucket and the ground position corresponding to the underground pipeline; when the actual distance is less than a set threshold, obtaining the excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operations; determining the state of the excavator based on the excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operations; and triggering an alarm mechanism when the excavator is in a working state.

[0010] In some embodiments, the risk warning method for underground pipelines includes: determining multimodal features based on excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operations; and determining the state of the excavator based on the multimodal features.

[0011] In some embodiments, the risk warning method for underground pipelines includes: obtaining video features, sound features, and vibration features based on the excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operations; performing feature projection on the video features, sound features, and vibration features to obtain projected video features, projected sound features, and projected vibration features; respectively determining the weights of the video features, the weights of the sound features, and the weights of the vibration features; determining the fused multimodal features based on the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features.

[0012] In some embodiments, the risk warning method for underground pipelines includes: inputting the excavator bucket related video data into a first convolutional neural network to extract video features in the video data; and, inputting the sound data during construction work into a second convolutional neural network to extract sound features in the sound data; and, inputting the vibration data during bucket operation into a third convolutional neural network to extract vibration features in the vibration data.

[0013] In some embodiments, the risk warning method for underground pipe network includes: fusion =a v ×F v '+a a ×F a '+a z ×F z '; Among them, F fusion is the fused multimodal feature, a v is the weight of the video feature, F v ' is the video feature after projection, a a is the weight of the sound feature, F a' is the sound feature after projection, a z is the weight of the vibration feature, F z ' is the vibration characteristic after projection.

[0014] In some embodiments, the risk warning method for underground pipeline networks includes: inputting multimodal features into a multi-task learning network to use multiple task heads to process the multimodal features in parallel; determining the state of the excavator based on the classification results output by the independent output layer of each task head; wherein the multi-task learning network includes multiple task heads, each task head has an independent output layer, and each task head is used to classify different states of the excavator.

[0015] In some embodiments, the risk warning method for underground pipe networks includes: if the classification results of more than half of the independent output layers indicate that the excavator is in a working state, determining that the state of the excavator is a working state.

[0016] In some embodiments, the risk warning device for underground pipelines includes: a first acquisition module, configured to obtain the actual distance between the bucket of the excavator and the ground position corresponding to the underground pipeline; a second acquisition module, configured to obtain the excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operation when the actual distance is less than a set threshold; a determination module, configured to determine the state of the excavator based on the excavator bucket-related video data, sound data during construction operations, and vibration data during bucket operation; a trigger module, configured to trigger an alarm mechanism when the excavator is in a working state.

[0017] In some embodiments, the risk warning device for underground pipeline network includes: a processor and a memory storing program instructions, and the processor is configured to execute the aforementioned risk warning method for underground pipeline network when running the program instructions.

[0018] In some embodiments, the storage medium stores program instructions, and when the program instructions are run, the aforementioned risk warning method for underground pipe networks is executed.

[0019] The risk warning method, device and storage medium for underground pipe networks provided by the embodiments of the present disclosure can achieve the following technical effects: it can combine video, sound and vibration data to more accurately judge the actual working state of the excavator. Furthermore, when it is determined that the excavator is in a working state that may endanger the safety of the underground pipeline, the alarm mechanism will be triggered immediately. In this way, the operator can be reminded in time to adjust the working method or stop the operation, effectively reducing the risk of accidents caused by human negligence. It realizes the refined monitoring and risk assessment of the excavator operation process, significantly improving the safety and management efficiency of the construction site.

[0020] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0022] Figure 1 is a schematic diagram of a risk warning method for an underground pipe network provided by an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a method for determining the state of an excavator provided by an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a method for determining multimodal features provided by an embodiment of the present disclosure;

[0025] Figure 4 is another schematic diagram of a method for determining a state of an excavator provided by an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a risk warning device for an underground pipe network provided by an embodiment of the present disclosure;

[0027] Figure 6 It is a schematic diagram of another risk warning device for underground pipe networks provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0029] The terms "first", "second", etc. in the description of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0030] Unless otherwise stated, the term "plurality" means two or more.

[0031] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0032] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0033] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0034] Figure 1 is a schematic diagram of a risk warning method for underground pipe networks provided by an embodiment of the present disclosure; Figure 1 As shown, the embodiment of the present disclosure provides a risk warning method for an underground pipe network, comprising:

[0035] S11, the early warning system obtains the actual distance between the bucket of the excavator and the ground position corresponding to the underground pipeline.

[0036] S12, when the actual distance is less than the set threshold, the early warning system obtains the excavator bucket related video data, the sound data during the construction operation, and the vibration data during the bucket operation.

[0037] S13, the early warning system determines the state of the excavator based on the video data related to the excavator bucket, the sound data during the construction operation, and the vibration data during the bucket operation.

[0038] S14, when the excavator is in the working state, the early warning system triggers the alarm mechanism.

[0039] In this solution, the early warning system can use target detection technology for on-site detection. Here, the target detection technology can be selected according to the material of the underground pipeline network and the surrounding environment. Among them, the target detection technology is geological radar detection or electromagnetic induction detection. On-site detection includes field detection in the target area, operating geological radar or ground detection equipment, and recording the density and electromagnetic properties of underground materials. Furthermore, the early warning system can analyze the data obtained from the on-site detection and use professional software to process the reflected signals of radar or electromagnetic waves to identify pipeline information. In one example, the pipeline information includes the location, depth and direction of the pipeline. Optionally, the detection results are marked on the ground to provide a reference for subsequent steps.

[0040] Furthermore, the early warning system stores the pipeline information in GIS (Geographic Information System). Specifically, the early warning system can digitally organize the pipeline information and store the organized data in GIS to create a spatial database of the pipeline. In this way, it is convenient to use the spatial analysis function of GIS to associate the pipeline data with other geographic spatial data. In an optimized solution, GIS can verify the input pipeline data to ensure the accuracy and completeness of the data.

[0041] Optionally, the geographic location coordinates can be obtained through the Beidou positioning system. Specifically, the Beidou positioning system can be operated at the ground position marked according to the detection results to determine the longitude and latitude coordinates of the position. And the geographic coordinates obtained by Beidou positioning are converted into the coordinate system used by GIS to ensure data consistency. In one example, the spatial query function can be used in GIS to extract the corresponding geographic location coordinates based on the pipeline information, and the distance between the corresponding ground position of the underground pipeline and the RGB-D camera is calculated according to the Haversine formula. With this solution, accurate determination of data can be achieved.

[0042] In this embodiment, an RGB-D depth camera is installed on the excavator. Here, the image of the ground position corresponding to the underground pipeline can be captured by the RGB-D depth camera. Specifically, the early warning system can control the excavator to move to the target area, and use the RGB-D depth camera to collect the image and depth data of the ground position corresponding to the underground pipeline. In addition, by analyzing the depth data collected by the RGB-D camera, the early warning system can calculate the actual distance from the excavator bucket to the ground position corresponding to the underground pipeline. As a preferred solution, the early warning system can calibrate the RGB-D camera to ensure its measurement accuracy. With this solution, the actual distance between the excavator bucket and the ground position corresponding to the underground pipeline can be accurately obtained.

[0043] Furthermore, when the actual distance is less than the set threshold, the early warning system can obtain the excavator's bucket-related video data, the sound data during construction operations, and the vibration data during bucket operations. Among them, the set threshold can be pre-stored in the early warning system. As an example, the set threshold can be determined in combination with the safety construction requirements of the pipeline. Specifically, the early warning system can capture the video of the excavator operation through an RGB-D camera to obtain the excavator's bucket-related video data. The early warning system can use the sound recorded by the sound sensor during the excavator operation as the sound data during construction operations. At the same time, the early warning system can also collect the vibration of the excavator during operation through an accelerometer or other sensing equipment to obtain the vibration data during bucket operation. With this solution, it is possible to accurately obtain the excavator's bucket-related video data, the sound data during construction operations, and the vibration data during bucket operations.

[0044] Furthermore, the early warning system can determine the state of the excavator based on the video data related to the excavator bucket, the sound data during construction work, and the vibration data during bucket operation. Specifically, the early warning system determines the state of the excavator based on the video data related to the excavator bucket, the sound data during construction work, and the vibration data during bucket operation, including: the early warning system determines the multimodal features based on the video data related to the excavator bucket, the sound data during construction work, and the vibration data during bucket operation. The early warning system determines the state of the excavator based on the multimodal features. With this solution, based on the fusion of multimodal features, the state of the excavator can be judged more accurately.

[0045] Furthermore, when the excavator is in a working state, the early warning system can trigger an alarm mechanism.

[0046] The risk warning method for underground pipe networks provided by the embodiment of the present disclosure can combine video, sound and vibration data to more accurately judge the actual working state of the excavator. Furthermore, when it is determined that the excavator is in a working state that may endanger the safety of the underground pipeline, the alarm mechanism will be triggered immediately. In this way, the operator can be reminded in time to adjust the working method or stop the operation, effectively reducing the risk of accidents caused by human negligence. Refined monitoring and risk assessment of the excavator operation process are achieved, significantly improving the safety and management efficiency of the construction site.

[0047] Figure 2 is a schematic diagram of a method for determining the state of an excavator provided by an embodiment of the present disclosure; Figure 2 As shown, optionally, in S13, the early warning system determines the state of the excavator according to the video data related to the excavator bucket, the sound data during the construction operation, and the vibration data during the bucket operation, including:

[0048] S21, the early warning system determines the multimodal features based on the video data related to the excavator bucket, the sound data during the construction operation, and the vibration data during the bucket operation.

[0049] S22, the early warning system determines the state of the excavator based on the multimodal characteristics.

[0050] In this solution, the early warning system can determine the multimodal features by combining the video data related to the bucket of the excavator, the sound data during the construction operation, and the vibration data during the bucket operation. Specifically, the early warning system determines the multimodal features based on the video data related to the bucket of the excavator, the sound data during the construction operation, and the vibration data during the bucket operation, including: the early warning system obtains the video features, the sound features, and the vibration features based on the video data related to the bucket of the excavator, the sound data during the construction operation, and the vibration data during the bucket operation. The early warning system performs feature projection on the video features, the sound features, and the vibration features to obtain the projected video features, the projected sound features, and the projected vibration features. The early warning system determines the weight of the video features, the weight of the sound features, and the weight of the vibration features, respectively. The early warning system determines the fused multimodal features based on the projected video features, the projected sound features, the projected vibration features, the weight of the video features, the weight of the sound features, and the weight of the vibration features. With this solution, we can integrate multi-source data such as the excavator bucket video, operation sound and vibration, and use feature projection technology to optimize the feature information of each dimension after feature extraction. We can also assign corresponding weights based on the importance of the features so that the early warning system can obtain multi-modal features after weighted fusion to ensure the accuracy and comprehensiveness of the monitoring signal.

[0051] Furthermore, the early warning system can determine the state of the excavator in combination with multimodal features. Specifically, the early warning system determines the state of the excavator based on the multimodal features, including: the early warning system inputs the multimodal features into the multi-task learning network so that the multi-task learning network processes multiple classification tasks in parallel. Upon receiving the output of the multi-task learning network, the early warning system determines the state of the excavator based on the classification results provided by the independent output layers of each task head. With this solution, the current operating state of the excavator can be accurately determined by integrating and analyzing multimodal features such as video, sound, and vibration from the excavator bucket, providing strong technical support and decision-making basis for safety management at the construction site.

[0052] Figure 3 is a schematic diagram of a method for determining multimodal features provided by an embodiment of the present disclosure; Figure 3 As shown, optionally, in S21, the early warning system determines multimodal features based on the video data related to the bucket of the excavator, the sound data during the construction operation, and the vibration data during the bucket operation, including:

[0053] S31, the early warning system obtains video features, sound features, and vibration features based on the excavator bucket-related video data, the sound data during construction operations, and the vibration data during bucket operations.

[0054] S32, the early warning system performs feature projection on the video features, the sound features, and the vibration features to obtain projected video features, projected sound features, and projected vibration features.

[0055] S33, the early warning system determines the weight of the video feature, the weight of the sound feature, and the weight of the vibration feature respectively.

[0056] S34, the early warning system determines the fused multimodal features according to the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features.

[0057] In this solution, the early warning system obtains video features, sound features, and vibration features based on the excavator bucket-related video data, the sound data during construction, and the vibration data during bucket operation, including: the early warning system inputs the excavator bucket-related video data into the first convolutional neural network to extract video features from the video data; and the early warning system inputs the sound data during construction into the second convolutional neural network to extract sound features from the sound data; and the early warning system inputs the vibration data during bucket operation into the third convolutional neural network to extract vibration features from the vibration data. The early warning system uses three convolutional neural networks to process video, sound, and vibration data respectively, accurately extracts key features from each data, and provides an accurate data basis for judging the state of the excavator.

[0058] Furthermore, the early warning system can perform feature projection on the video features, sound features, and vibration features to obtain projected video features, projected sound features, and projected vibration features. Specifically, the early warning system projects each feature into the same feature space for weighting and fusion. In one example, linear transformation is usually used for feature projection. In the fully connected layer, each feature can be linearly transformed by a corresponding weight matrix. Let these three weight matrices be W, respectively. v , W a and W z , corresponding to video, sound and vibration modes respectively. Thus, for the video feature F v , sound feature F a and vibration characteristics F z , and then multiply it with the corresponding weight matrix to obtain the projected features:

[0059] Video characteristics after projection: F v '=Wv ×F v ;

[0060] Sound characteristics after projection: F a '=W a ×F a ;

[0061] Vibration characteristics after projection: F z '=W s ×F z ;

[0062] With this solution, it is possible to obtain the video features after projection, the sound features after projection, and the vibration features after projection.

[0063] Further, the early warning system may determine the weight of the video feature, the weight of the sound feature, and the weight of the vibration feature, respectively. Specifically, the early warning system may calculate the attention score of each feature. In one example, the early warning system may perform a dot product operation on the projected features, and the dot product operation is used to measure the similarity of two vectors. For example, a scoring function may be used to calculate the attention score of each feature. Optionally, the attention score of each feature is calculated using a scoring function, including:

[0064] e v = score(F v ', F a ', F z ')

[0065] e a = score(F a ', F v ', F z ')

[0066] e z = score(F z ', F v ', F a ')

[0067] Among them, the scoring function score can be a simple dot product or a more complex function.

[0068] In one example,

[0069] Among them, here Represents a concatenation operation. The concatenation operation refers to concatenating feature vectors of multiple modalities into a new feature vector. It can be understood that the concatenation operation is to splice multiple feature vectors together along a specific dimension to form a longer feature vector, thereby merging the information of each modality together. h represents the weight or attention score calculated by the attention mechanism. F i' represents the data characteristics of a certain mode. For example, F v ' is the video feature after projection, F a ' is the sound feature after projection, F z ' is the vibration characteristic after projection.

[0070] Among them, W e is the weight matrix used to calculate the attention score, which is used to linearly transform the concatenated features. As an example, if F i 'The dimension is d, the concatenated features The dimension of is 2d, then W e The dimension of is d×2d.

[0071] Optionally, W e It can be shown as:

[0072]

[0073] Furthermore, the early warning system can normalize the attention score of each feature through a softmax function to obtain the weight of each feature.

[0074] Optionally, the weight α of the video feature v This can be shown in the following way:

[0075]

[0076] Optionally, the weight α of the sound feature a This can be shown in the following way:

[0077]

[0078] Optionally, the weight α of the sound feature z This can be shown in the following way:

[0079]

[0080] It can be understood that the Softmax function can convert any real-valued vector into a probability distribution so that all output values ​​are non-negative and the sum is 1. In this way, the weight α of the obtained frequency feature can be v , the weight of the sound feature α a and the weight α of the sound feature z It can reflect the importance of each feature, and make the sum of their weights 1. With this solution, the weights of video features, sound features, and vibration features can be accurately determined.

[0081] Furthermore, the early warning system determines the fused multimodal features based on the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features. With this solution, feature data fusion can be achieved by using an adaptive weighting method, so that data of different modalities can complement and enhance each other, improve the richness and discrimination of feature representation, and provide an accurate data basis for judging the state of the excavator.

[0082] Optionally, S31, the early warning system obtains video features, sound features and vibration features based on the excavator bucket-related video data, the sound data during construction operations and the vibration data during bucket operations.

[0083] The early warning system inputs the video data related to the bucket of the excavator into the first convolutional neural network to extract video features in the video data; and,

[0084] The early warning system inputs the sound data during the construction work into the second convolutional neural network to extract the sound features in the sound data; and,

[0085] The early warning system inputs the vibration data of the bucket during operation into the third convolutional neural network to extract the vibration features in the vibration data.

[0086] In this solution, the early warning system can use the first convolutional neural network to process the video data related to the bucket of the excavator to extract video features in the video data. Here, the video features include but are not limited to the position features, direction features, and movement speed features of the bucket. Specifically, the early warning system can input the video data related to the bucket of the excavator into the first convolutional neural network to extract video features in the video data. With this solution, accurate video features can be obtained.

[0087] In this solution, the early warning system can use the second convolutional neural network to process the sound data during construction work to extract the sound features in the sound data. Here, the sound features refer to the spectrum and time domain features of the sound, such as frequency distribution and amplitude changes. Specifically, the early warning system can input the sound data during construction work into the second convolutional neural network to extract the sound features in the sound data. With this solution, accurate sound features can be obtained.

[0088] In this solution, the early warning system can use the third convolutional neural network to process the vibration data during bucket operation to extract vibration features in the vibration data. Here, the vibration features include but are not limited to the frequency and intensity features of the vibration. Specifically, the early warning module inputs the vibration data during bucket operation into the third convolutional neural network to extract vibration features in the vibration data. With this solution, accurate vibration features can be obtained.

[0089] Optionally, in S34, the early warning system determines the fused multimodal features according to the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features, including:

[0090] F fusion =a v ×F v '+a a ×F a '+a z ×F z '

[0091] Among them, F fusion is the fused multimodal feature, a v is the weight of the video feature, F v ' is the video feature after projection, a a is the weight of the sound feature, F a ' is the sound feature after projection, a z is the weight of the vibration feature, F z ' is the vibration characteristic after projection.

[0092] In this solution, the early warning system can apply the calculated attention weights to the corresponding features and perform weighted summation on the features to obtain the fused multimodal features. With this solution, the accuracy of judging the working status of the excavator is effectively improved by integrating the three aspects of video, sound and vibration. The data of different modes complement each other and can effectively reduce the misjudgment caused by insufficient single mode data or noise interference.

[0093] Figure 4 is another schematic diagram of a method for determining the state of an excavator provided by an embodiment of the present disclosure; Figure 4 As shown, optionally, in S22, the early warning system determines the state of the excavator according to the multimodal features, including:

[0094] S41, the early warning system inputs the multimodal features into the multi-task learning network to utilize multiple task heads to process the multimodal features in parallel.

[0095] S42, the early warning system determines the state of the excavator according to the classification results output by the independent output layers of each task head.

[0096] The multi-task learning network includes multiple task heads, each task head has an independent output layer, and each task head is used to classify different states of the excavator.

[0097] In this solution, the early warning system can input multimodal features into a multi-task learning network to process the multimodal features in parallel using multiple task heads. Specifically, after the multi-task learning network receives the multimodal features, all task heads including the network for analyzing the motion trajectory of the shovel arm in the video, the network for sound judgment, and the network for vibration judgment run simultaneously, thereby realizing parallel processing of different tasks. It can be understood that the multimodal features can be input into a task head part composed of a fully connected layer. The fully connected layer can map the features to an output space related to a specific task. At the same time, the network design allows multiple tasks to be processed simultaneously. In this network, each task head has an output layer, each output layer is independent of other task heads, and can be designed specifically for a specific task. It can be understood that the output layer of each task head uses a softmax function for classification. In this embodiment, the output layer can divide the state of the excavator into different categories based on the characteristics of the input data. In this case, the multi-task learning network can process different tasks in parallel, for example, analyzing the motion trajectory of the shovel arm, sound and vibration signals at the same time.

[0098] Furthermore, the early warning system can combine the classification results of the independent output layers of each task head to determine the state of the excavator. In this scheme, the tasks are decomposed by multiple task heads, and the shovel arm posture in the video, the sound during construction, the shovel arm vibration and other tasks are processed separately, making full use of the shared information between different tasks to improve the overall performance and reliability of the model.

[0099] Optionally, in S42, upon receiving the output of the multi-task learning network, the early warning system determines the state of the excavator according to the classification results output by the independent output layers of each task head, including:

[0100] If the classification results of more than half of the independent output layers indicate that the excavator is in a working state, the early warning system determines that the state of the excavator is a working state.

[0101] In this solution, the early warning system determines the state of the excavator based on the classification results provided by the independent output layers of each task head. In this embodiment, the multi-task learning network includes three task heads. Then, in the task learning network, if the classification results of the independent output layers of two or three of the task heads indicate that the excavator is in a working state, the early warning system determines that the state of the excavator is in a working state. With this solution, the judgment of the working state of the excavator can be made more comprehensive and accurate.

[0102] Figure 5 is a schematic diagram of a risk warning device for an underground pipe network provided by an embodiment of the present disclosure; Figure 5As shown, the embodiment of the present disclosure provides a risk warning device 200 for underground pipe network, including a first acquisition module 51, a second acquisition module 52, a determination module 53 and a trigger module 54. The first acquisition module 51 is configured to obtain the actual distance between the bucket of the excavator and the ground position corresponding to the underground pipeline; the second acquisition module 52 is configured to obtain the video data related to the bucket of the excavator, the sound data during the construction operation and the vibration data during the bucket operation when the actual distance is less than the set threshold; the determination module 53 is configured to determine the state of the excavator according to the video data related to the bucket of the excavator, the sound data during the construction operation and the vibration data during the bucket operation; the trigger module 54 is configured to trigger the alarm mechanism when the state of the excavator is in the working state.

[0103] The risk warning device 200 for underground pipe networks provided by the embodiment of the present disclosure can combine video, sound and vibration data to more accurately judge the actual working state of the excavator. Furthermore, when it is determined that the excavator is in a working state that may endanger the safety of the underground pipeline, the alarm mechanism will be triggered immediately. In this way, the operator can be reminded in time to adjust the working method or stop the operation, effectively reducing the risk of accidents caused by human negligence. Refined monitoring and risk assessment of the excavator operation process are achieved, significantly improving the safety and management efficiency of the construction site.

[0104] Figure 6 is another schematic diagram of a risk warning device for an underground pipe network provided by an embodiment of the present disclosure; Figure 6 As shown, the embodiment of the present disclosure provides a risk warning device 300 for an underground pipe network, including a processor 301 and a memory 302. Optionally, the device 300 may also include a communication interface 303 and a bus 304. The processor 301, the communication interface 303, and the memory 302 may communicate with each other through the bus 304. The communication interface 303 may be used for information transmission. The processor 301 may call the logic instructions in the memory 302 to execute the risk warning method for an underground pipe network of the above embodiment.

[0105] In addition, the logic instructions in the memory 302 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0106] The memory 302 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 301 executes the function application and data processing by running the program instructions / modules stored in the memory 302, that is, the risk warning method for underground pipe networks in the above embodiment is implemented.

[0107] The memory 302 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 302 may include a high-speed random access memory and may also include a non-volatile memory.

[0108] The embodiment of the present disclosure provides a storage medium storing program instructions, which, when run, execute the aforementioned risk warning method for underground pipe networks.

[0109] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0110] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments. As used in the description of the embodiments, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or including (comprising) refer to the existence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or these groups. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0111] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0112] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0113] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A risk early warning method for underground pipe networks, characterized in that: include: Get the actual distance between the excavator bucket and the corresponding ground position of the underground pipeline; When the actual distance is less than the set threshold, obtain the excavator bucket related video data, the sound data during the construction operation, and the vibration data during the bucket operation; Determine the state of the excavator based on the video data related to the excavator bucket, the sound data during the construction operation, and the vibration data during the bucket operation; When the excavator is in working state, the alarm mechanism is triggered.

2. The risk warning method according to claim 1, characterized in that: Based on the excavator bucket-related video data, the sound data during construction work, and the vibration data during bucket operation, the status of the excavator is determined, including: Determine multimodal features based on video data related to the excavator bucket, sound data during construction work, and vibration data during bucket operation; Based on the multimodal features, the state of the excavator is determined.

3. The risk warning method according to claim 2 is characterized in that: Based on the video data related to the excavator bucket, the sound data during construction work, and the vibration data during bucket operation, multimodal features are determined, including: According to the video data related to the bucket of the excavator, the sound data during the construction operation and the vibration data during the bucket operation, the video features, the sound features and the vibration features are obtained; Performing feature projection on the video features, the sound features, and the vibration features to obtain projected video features, projected sound features, and projected vibration features; Determine the weight of the video feature, the weight of the sound feature, and the weight of the vibration feature respectively; The fused multimodal features are determined according to the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features.

4. The risk warning method according to claim 3 is characterized in that: Based on the video data related to the excavator bucket, the sound data during construction and the vibration data during bucket operation, the video features, sound features and vibration features are obtained, including: Inputting the video data related to the bucket of the excavator into the first convolutional neural network to extract video features in the video data; and, Inputting the sound data during the construction work into the second convolutional neural network to extract the sound features in the sound data; and, The vibration data of the bucket during operation is input into the third convolutional neural network to extract the vibration features in the vibration data.

5. The risk warning method according to claim 3 is characterized in that: According to the projected video features, the projected sound features, the projected vibration features, the weights of the video features, the weights of the sound features, and the weights of the vibration features, the fused multimodal features are determined, including: F fusion =a v ×F v ’+a a ×F a ’+a z ×F z ’ Among them, F fusion is the fused multimodal feature, a v is the weight of the video feature, F v ' is the video feature after projection, a a is the weight of the sound feature, F a ' is the sound feature after projection, a z is the weight of the vibration feature, F z ' is the vibration characteristic after projection.

6. The risk early warning method according to claim 2, characterized in that: Determine the state of the excavator based on multimodal features, including: Input the multimodal features into the multi-task learning network to process the multimodal features in parallel using multiple task heads; Determine the state of the excavator based on the classification results output by the independent output layers of each task head; The multi-task learning network includes multiple task heads, each task head has an independent output layer, and each task head is used to classify different states of the excavator.

7. The risk early warning method according to claim 6, characterized in that: According to the classification results output by the independent output layer of each task head, the state of the excavator is determined, including: If the classification results of more than half of the independent output layers indicate that the excavator is in a working state, the state of the excavator is determined to be a working state.

8. A risk warning device for underground pipe network, characterized in that: include: A first acquisition module is configured to acquire an actual distance between a bucket of the excavator and a ground position corresponding to the underground pipeline; The second acquisition module is configured to acquire video data related to the bucket of the excavator, sound data during construction work, and vibration data during bucket operation when the actual distance is less than a set threshold; A determination module configured to determine the state of the excavator based on video data related to the excavator bucket, sound data during construction work, and vibration data during bucket operation; The trigger module is configured to trigger an alarm mechanism when the excavator is in a working state.

9. A risk warning device for an underground pipe network, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the risk warning method for underground pipe networks as described in any one of claims 1 to 7 when running the program instructions.

10. A storage medium storing program instructions, characterized in that: When the program instructions are run, the risk warning method for underground pipe networks as described in any one of claims 1 to 7 is executed.

Images