A hidden danger identification method and device for tower crane inspection, an electronic device, and a storage medium

Abstract

The application discloses a hidden danger identification method and device for tower crane inspection, electronic equipment and storage medium, the method comprises the following steps: acquiring tower crane inspection data, wherein the tower crane inspection data comprises inspection images and / or inspection videos of key parts of the tower crane; identifying the tower crane inspection data according to a pre-trained hidden danger identification model, and determining tower crane hidden danger data, wherein the tower crane hidden danger data comprises hidden danger images with hidden dangers, the hidden danger images are frames in the inspection images or the inspection videos with visualized marks and hidden danger parameters, and the hidden danger parameters comprise a hidden danger type and a hidden danger confidence. The application can visually mark the hidden danger positions in the corresponding frames in the inspection images or the inspection videos, and simultaneously display the hidden danger type and the hidden danger confidence of the corresponding hidden dangers, so that it can be intuitively seen whether there are hidden dangers at various inspection positions and the hidden danger confidence, and the possible hidden dangers can be displayed as much as possible, thereby reducing the probability of missing identification of hidden dangers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent detection of construction equipment, and in particular to a hidden danger identification method and device for tower crane inspection, electronic equipment and a storage medium. BACKGROUND

[0002] At present, in traditional construction sites, as one of the large equipment, the tower crane needs to be particularly noticed to prevent the occurrence of safety hazards. At present, its inspection mainly relies on manual means, and there are problems such as difficult to reach high altitude, low inspection frequency, and untimely monitoring, which makes it difficult to achieve comprehensive monitoring of the tower crane and the detection result is not high in accuracy. In recent years, with the development of robot and unmanned aerial vehicle technology, an unmanned inspection technology for inspecting and detecting the tower crane by using robots or unmanned aerial vehicles has appeared in the prior art. This technology can effectively avoid the safety hazards existing in manual inspection, and can also better inspect the positions that are difficult for manual inspection. In these technologies, robots and unmanned aerial vehicles can replace manual inspection to the corresponding positions and shoot corresponding images or videos, but how to identify whether there is a safety hazard according to the images or videos shot is a more important part of the technology, which determines the accuracy of unmanned inspection. SUMMARY

[0003] The embodiments of the present application provide a hidden danger identification method and device for tower crane inspection, electronic equipment and a storage medium to solve the problem of insufficient accuracy in the prior art unmanned inspection technology.

[0004] According to a first aspect of the present application, a hidden danger identification method for tower crane inspection is provided, comprising:

[0005] acquiring tower crane inspection data, wherein the tower crane inspection data includes inspection images and / or inspection videos of key parts of the tower crane;

[0006] identifying the tower crane inspection data according to a pre-trained hidden danger identification model to determine tower crane hidden danger data, wherein the tower crane hidden danger data includes hidden danger images with hidden dangers, the hidden danger images are frames in the inspection images or the inspection videos with hidden danger visual markers and hidden danger parameters, and the hidden danger parameters include hidden danger types and hidden danger confidence levels.

[0007] The hidden danger identification method for tower crane inspection of the present application can distinguish the inspection images or frames in the inspection videos with hidden dangers by identifying the acquired tower crane inspection data using the hidden danger identification model, and visually mark the hidden danger positions in the corresponding inspection images or frames in the inspection videos, while displaying the hidden danger types and hidden danger confidence levels of the corresponding hidden dangers, so that it can be more intuitive to see whether there is a possibility of hidden danger at each inspection position, and the confidence level of the hidden danger is high or low, so as to display the hidden dangers as much as possible and reduce the probability of missing the identification of hidden dangers.

[0008] In some embodiments, the pre-trained hidden danger identification model is a model based on a YOLOv11 model for image recognition, wherein in the model,

[0009] The LDConv module is used to replace all 3x3 down-sampling convolution layers in the original Backbone and Neck.

[0010] Thus, by such a setting, the adaptability to irregular defects and components during identification can be improved, and the feature recognition ability in a complex background can be enhanced. Since the crane components and defect shapes are irregular, for example, the structural components of the crane (such as a hoist arm, a steel structure joint, and a weld) and potential defects (such as irregular cracks, sheet corrosion, and twisted deformation) often have diversified and irregular geometric shapes, when the unmanned aerial vehicle or robot is photographed at different angles and distances, these targets will have significant perspective deformation in the image. The fixed sampling grid of the standard convolution in the YOLOv11 model cannot effectively adapt to these irregular shapes and changes in the angle, resulting in insufficient feature extraction and easy to miss or misidentify shape-specific defects. Meanwhile, in the obtained inspection image, the metal surface reflection of the crane, the complex background of the work environment (such as the sky, other buildings, and ground debris), and other factors reduce the distinguishability of the defect features and the background. The standard convolution in the YOLOv11 model is not sensitive enough to the subtle feature changes in such complex scenes. Therefore, the YOLOv11 model is optimized as described above.

[0011] In some embodiments, the pre-trained hidden danger identification model is a model based on a YOLOv11 model for image recognition, wherein in the model,

[0012] In the last C3 module of the Backbone, the original BottleNeck structure is replaced with an RFEM to form a C3-RFEM module.

[0013] Thus, by such a setting, the feature receptive field can be significantly expanded, the feature expression of a small defect can be strengthened, and the understanding of a structure-related defect can be improved. Since in the crane inspection, many key defects are small in size, such as an early crack, a loosened or missing single bolt, and a local small corrosion point. In the deep network of the standard YOLOv11, with the reduction of the feature map resolution, the feature information of these small target defects is easily diluted and even completely lost, which leads to difficulty in detection. Moreover, the crane structure is complex, and the importance of the same defect in different structural backgrounds can be different. For example, a small crack is more dangerous on a main load-bearing component than on a non-load-bearing guard plate. The standard backbone network has limited ability to perceive the deep context association. Therefore, the YOLOv11 model is optimized as described above.

[0014] In some embodiments, the pre-trained hidden danger identification model is a model based on a YOLOv11 model for image recognition, wherein in the model,

[0015] In the Backbone output stage, the P3, P4 and P5 features output by the Backbone are respectively sent to the corresponding CAA modules to obtain new weighted features which are then input into the Neck.

[0016] Thus, by such setting, long-distance spatial context relationship can be efficiently modeled, the overall perception of large and slender components and defects thereof can be enhanced, and the recognition degree of defects under poor imaging conditions can be improved. Since the hoisting machinery often includes large-span and slender components such as a boom and a cable. When the unmanned aerial vehicle is patrolled, these components may occupy a large part of the picture, but the defects (such as slight bending of the boom and local damage of the cable) thereon can be very subtle or distributed over a long distance. The standard convolutional network is difficult to effectively establish the dependence relationship between long-distance pixels when processing such targets, which can lead to insufficient perception of overall morphological abnormalities or distributed small defects. Moreover, due to the reflection of the metal surface, the shadow and the change of the visual angle caused by the tilting of the unmanned aerial vehicle, the defect features on the hoisting machinery are not prominent enough and are easily overwhelmed by background noise. Therefore, the YOLOv11 model is optimized as described above.

[0017] In some embodiments, the pre-trained hidden danger identification model is a model based on a YOLOv11 model for image recognition, wherein in the model,

[0018] In the training stage, the loss calculation module is modified, and the original CIoU / GIoU bounding box regression loss is replaced by a Unified-IoU (UIoU) loss function.

[0019] Therefore, by setting in this way, the loss weight of different quality prediction boxes can be balanced through a dynamic attention mechanism, and the confidence information of the prediction boxes is combined to achieve more accurate positioning of the crane defects and improve the regression quality of key and difficult defects. It can be understood that for the safety evaluation of the crane, the accurate position and size of the defect are crucial. For example, the length of the crack, the area of the corrosion, and the degree of deformation directly affect the safety judgment. The standard IoU class loss function may have a bottleneck in optimizing high-precision positioning, especially in the case of defects with diverse shapes and large size differences. At the same time, in the crane inspection data, there may be a large number of backgrounds or easy-to-detect defects, while the number of key, subtle, or irregularly shaped defects (difficult example samples) is small. The standard loss function may not be sufficient for optimizing these difficult example samples in the later training stage, resulting in less than ideal regression accuracy of the model for such key defects. Therefore, the YOLOv11 model is optimized as described above.

[0020] In some embodiments, the tower crane inspection data is identified according to the pre-trained hidden danger identification model to determine the tower crane hidden danger data, comprising:

[0021] The tower crane inspection data is preprocessed to obtain first image data corresponding to the frames in the inspection image or inspection video;

[0022] The first image data is processed using the hidden danger identification model to obtain a first detection result, the first detection result including a plurality of potential hidden danger parameters, the potential hidden danger parameters including a potential hidden danger location in the first image data corresponding to the potential hidden danger, a potential hidden danger first confidence, and a probability of the potential hidden danger belonging to each hidden danger type;

[0023] The retained potential hidden danger parameters are determined according to the first detection result to form a hidden danger parameter;

[0024] The visual markers are formed in the frames in the inspection image or inspection video corresponding to the first image data according to the hidden danger parameter to form a hidden danger image.

[0025] Therefore, by setting in this way, all potential hidden dangers can be identified and detected first, thereby effectively improving the completeness of hidden danger identification, and then the potential hidden dangers are screened according to the probability of each hidden danger type of each potential hidden danger and the potential hidden danger confidence, thereby retaining hidden dangers with higher confidence, thereby improving the identification accuracy of hidden danger identification, to solve the problem of insufficient accuracy in the prior art of unmanned inspection technology.

[0026] In some embodiments, the retained potential hidden danger parameters are determined according to the first detection result to form a hidden danger parameter, comprising:

[0027] determine the confidence of each potential hazard parameter belonging to each hazard type according to the potential hazard first confidence in each potential hazard parameter and the probability of the potential hazard belonging to each hazard type;

[0028] determine a first potential hazard parameter in each potential hazard parameter according to the confidence of each potential hazard parameter belonging to each hazard type and a preset threshold value;

[0029] determine whether the potential hazard positions of each potential hazard parameter in the first potential hazard parameter overlap,

[0030] when there is overlap, only keep one potential hazard parameter with the highest confidence among the potential hazard parameters with overlapping potential hazard positions to obtain a retained potential hazard parameter, and form a hazard parameter.

[0031] Therefore, by such a setting, the identification accuracy of each hazard retained can be further improved by using the confidence of each potential hazard parameter belonging to each hazard type and the preset threshold value, and the detection accuracy of the unmanned inspection technology can be improved.

[0032] According to a second aspect of the present application, a hazard identification device for tower crane inspection is provided, comprising:

[0033] an inspection data acquisition module, configured to acquire tower crane inspection data, wherein the tower crane inspection data comprises inspection images and / or inspection videos of key parts of the tower crane;

[0034] a hazard identification module, configured to identify the tower crane inspection data according to a pre-trained hazard identification model, and determine tower crane hazard data, wherein the tower crane hazard data comprises hazard images with hazards, the hazard images are frames in the inspection images or the inspection videos with hazard visual markers and hazard parameters, and the hazard parameters comprise hazard types and hazard confidences.

[0035] According to a third aspect of the present application, an electronic device is provided, comprising at least one processor and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the hazard identification method for tower crane inspection of the first aspect.

[0036] According to a fourth aspect of the present application, a storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the steps of the hazard identification method for tower crane inspection of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and all other drawings obtained by those skilled in the art without creative efforts based on these drawings also belong to the protection scope of the present application.

[0038] Figure 1 A flowchart of the whole process of the hidden danger identification method for tower crane inspection according to an embodiment of the present application;

[0039] Figure 2 A model structure diagram for the tower crane hidden danger identification scene optimized based on YOLOv11 according to the hidden danger identification method for tower crane inspection according to an embodiment of the present application;

[0040] Figure 3 A step flowchart when the tower crane training data is identified according to the pre-trained hidden danger identification model in step S12 of the hidden danger identification method for tower crane inspection according to an embodiment of the present application;

[0041] Figure 4 A step flowchart of step S23 in step S12 of the hidden danger identification method for tower crane inspection according to an embodiment of the present application;

[0042] Figure 5 A principle block diagram of the hidden danger identification device for tower crane inspection according to an embodiment of the present application;

[0043] Figure 6 A structure diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0045] It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict.

[0046] The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0047] In the present application, "module", "device", "system" and the like refer to relevant entities applied to computers, such as hardware, combination of hardware and software, software or software in execution, etc. In detail, for example, the element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Also, an application or a script running on a server, a server can be an element. One or more elements can be in an execution process and / or thread, and the element can be localized on one computer and / or distributed between two or more computers, and can be run by various computer readable media. The element can also communicate through a local and / or remote process according to a signal having one or more data packets, for example, a signal from data of a local system, another element in a distributed system, and / or a network through a signal with other systems over the Internet.

[0048] Finally, it should also be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including", "containing", "consisting" and the like, mean that the elements listed after the term encompass the elements recited and do not exclude other elements from the process, method, article, or apparatus that includes the recited elements. Without more limitations, the element defined by the statement "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the recited element.

[0049] The hidden danger identification method for tower crane inspection in the embodiments of the present application can be applied in hidden danger identification devices for tower crane inspection, so that users can use the hidden danger identification devices for tower crane inspection to identify hidden dangers in the acquired tower crane inspection data. The hidden danger identification devices for tower crane inspection include, but are not limited to, smart phones, smart tablets, personal PCs, computers, cloud servers, etc.

[0050] The application will be described in further detail below with reference to the drawings.

[0051] Figure 1 The overall flow of the hidden danger identification method for tower crane inspection according to an embodiment of the present application is schematically shown, and the method comprises the following steps: Figure 1 The hidden danger identification method for tower crane inspection according to the present application comprises the following steps:

[0052] Step S11: Obtain tower crane inspection data, wherein the tower crane inspection data comprises inspection images and / or inspection videos of key parts of the tower crane;

[0053] Step S12: Identify the tower crane inspection data according to a pre-trained hidden danger identification model to determine tower crane hidden danger data, wherein the tower crane hidden danger data comprises hidden danger images with hidden dangers, and the hidden danger images are frames in the inspection images or inspection videos with hidden danger visual markers and hidden danger parameters, and the hidden danger parameters comprise hidden danger types and hidden danger confidence levels.

[0054] Step S11 is a step of obtaining tower crane inspection data. Specifically, the obtained tower crane inspection data can be a plurality of image data or video data of key parts of the tower crane obtained by a camera module carried on a UAV or a robot, or data comprising both of the above. The data can be obtained in real time through wireless communication transmission with the UAV or the robot, or the data can be transmitted through wireless communication or wired communication with the UAV or the robot after the UAV or the robot completes the inspection.

[0055] Step S12 involves using a pre-trained hazard identification model to identify tower crane inspection data. Specifically, the pre-trained hazard identification model in this step is a neural network model. This model is based on the YOLOv11 model and optimized for tower crane hazard identification scenarios. The model includes a pre-built hazard database containing several hazard samples, including positive and negative hazard samples. Positive hazard samples are used to assist the model in identifying hazards, while negative hazard samples are used to assist the model in determining whether misidentification has occurred. The input to the hazard identification model is the acquired tower crane inspection data, which can be the inspection images acquired in step S11, inspection videos, or frames from processed inspection videos. Its output is hazard images showing the presence of hazards. These hazard images are inspection images or frames from inspection videos with hazard visualization markers and hazard parameters. Hazard visualization markers are labels placed within frames of inspection images or videos to indicate the location of hazards within that image. These markers can be arrows or bounding boxes surrounding the hazard. Hazard parameters are data representing hazards in the current inspection image, including hazard type and hazard confidence level. They indicate the hazard type indicated by the visualization marker and the confidence level of that type. Additionally, hazard parameters may include the location of the visualization marker within the frames of the inspection image or video, thus better illustrating the location indicated by the marker.

[0056] Specifically, during training, the hazard identification model uses a training dataset containing several inspection images and corresponding inspection images with visually marked hazards, i.e., hazard images. By training the hazard identification model, it becomes more applicable to hazard identification scenarios during tower crane inspections.

[0057] The hazard identification model itself is based on YOLOv11 and optimized for tower crane hazard identification scenarios. Figure 2 This diagram schematically illustrates the overall architecture of the hazard identification model in a hazard identification method for tower crane inspection according to an embodiment of the present invention. Specifically, refer to... Figure 2 As shown, this invention optimizes several parts of the YOLOv11 model.

[0058] The structural components of the hoisting machinery (such as the jib, steel structure node, weld) and potential defects (such as irregular cracks, sheet corrosion, twisted deformation) often present diversified and irregular geometric shapes. When the unmanned aerial vehicle or robot takes pictures at different angles and distances, these targets will produce significant perspective deformation in the image. The fixed sampling grid of the standard convolution in YOLOv11 is difficult to effectively adapt to these irregular shapes and changes in perspective, resulting in insufficient feature extraction and easy to miss or misidentify shape-specific defects or hazards. At the same time, in the obtained inspection images, the metal surface reflection of the hoisting machinery, the complex background of the working environment (such as the sky, other buildings, ground debris), and other factors reduce the distinguishability of the features of defects or hazards from the background. The standard convolution in YOLOv11 is not sensitive enough to subtle feature changes in such complex scenes. Therefore, in some possible embodiments, the present application replaces all 3x3 down-sampling convolution layers (convolution layers with stride = 2) in the original Backbone and Neck with LDConv modules. These convolution layers exist at the beginning of each stage of the backbone network (used to reduce the resolution of the feature map) and in the feature fusion path of the Neck. When replacing, the convolution kernel size and stride are kept unchanged, only the convolution operator is replaced with LDConv to enable it to learn irregular sampling. The specific method can be: in the CSP / C3 module of the Backbone, the 3x3 convolution layer originally used for downscaling is replaced with LDConv; in the PAN fusion layer of the Neck, the convolution responsible for down-sampling or fusion is also replaced with LDConv. Through such integration, LDConv achieves the ability to extract features of any shape convolution kernel by generating flexible initial sampling coordinates and cooperating with learnable offsets. At the same time, for the common linear cracks (may be curved, bifurcated) on hoisting machinery, irregularly shaped rusted areas or structural deformations, LDConv can also adaptively adjust the sampling points to more accurately capture the outlines and internal texture features of these targets. Even at the variable shooting angles of the unmanned aerial vehicle, it can effectively deal with the geometric deformation of parts and defects, significantly improving the recognition accuracy. Moreover, through a more flexible sampling mechanism, LDConv can better focus on the hoisting machinery and its defect areas, reducing the interference of background noise. For example, under the conditions of reflection or shadow, LDConv can more effectively extract key defect features such as small cracks and loose bolts from complex background textures.

[0059] Further, in the inspection of hoisting machinery, many key defects are small in size, such as early-stage cracks, loosening or absence of individual bolts, and local small corrosion spots. In the deep network of the standard YOLOv11, with the reduction of the resolution of the feature map, the feature information of these small target defects is easily diluted and even completely lost, making it difficult to detect. Moreover, hoisting machinery has a complex structure, and the importance of the same defect in different structural backgrounds may be different. For example, a small crack is more dangerous on a main load-bearing component than on a non-load-bearing guard plate. The standard backbone network of YOLOv11 has very limited ability to perceive the context association at this deep level. Therefore, in some possible embodiments, the present application embeds an RFEM module into the Backbone of YOLOv11 at a suitable level to enhance the perception of small targets. The specific integration methods include two kinds: one is to replace the bottleneck convolution of the backbone module - in the last C3 module (output P5 feature layer) of Backbone, replace the original BottleNeck structure with RFEM. In this way, a new C3-RFEM module is formed: the input is first passed through the CSP split path, where the backbone part is no longer a normal convolution sequence, but an RFEM multi-branch hollow convolution, and then the features are fused and output through the short connection. Through this replacement, the receptive field of the deepest feature map is greatly improved, which is beneficial to the detection of small objects and distant targets. The second is to insert an RFEM module between different C3 modules: for example, an RFEM layer is added after the output of the middle and high layer features and before the downsampling, and the features are enhanced before continuing to propagate. In this way, small target features can be appropriately "supplemented" at each scale during the process of step-by-step downsampling. No matter which way is adopted, the output dimension of RFEM remains consistent with that of the original convolution layer, so it can be seamlessly replaced or inserted without affecting the subsequent structure. In the specific embodiment, the first method described above is adopted. Through such a setting, for the cracks as fine as hair, the slight displacement or absence of fasteners (bolts, etc.) on hoisting machinery, the multi-scale hollow convolution of RFEM can capture the context information of these small targets without significantly increasing the amount of calculation. This means that even if the defect itself has a very small pixel ratio, the model can combine its location and environmental features on larger structural components to make judgments, thereby effectively preventing missed detection. Moreover, by enhancing the receptive field, the model can better understand the relationship between the defect and the hoisting machinery component it is in, and even the larger range of structure. For example, when judging the corrosion of a weld area, RFEM can help the model associate to the stress state information of the entire weld and even the connecting piece (through learning a large amount of data), so as to more accurately assess the potential risk level of the corrosion, rather than just identifying a corrosion spot in isolation.

[0060] Further, the hoisting machinery often contains large-span, slender components such as the jib and the cable. When the unmanned aerial vehicle or the robot inspects, these components may occupy a large part of the picture, but the defects (such as slight bending of the jib, local damage of the cable) thereon can be very subtle or distributed over a long distance. The standard convolutional network in YOLOv11 has difficulty in effectively establishing the dependence relationship between long-distance pixels when processing such targets, which can lead to insufficient perception of overall morphological abnormalities or distributed small defects. At the same time, due to the reflection of the metal surface, the shadow and the change of the visual angle caused by the oblique shooting of the unmanned aerial vehicle, the defect features on the hoisting machinery are not prominent enough and are easily overwhelmed by background noise. Therefore, in some possible implementation manners, the CAA attention mechanism is fused in the back end of the backbone network in YOLOv11. Backbone back-end fusion: at the output stage of the backbone network (after the last CSP module and before being sent to the neck), a CAA module is added to recalibrate the feature maps of each scale globally. Specifically, for the P3, P4 and P5 features output by the backbone, they can be respectively sent to the corresponding CAA module to obtain the weighted new features and then input to the neck for PAN fusion. In this way, each layer of features has combined the cross-scale context information before entering the detection head, which is beneficial to small targets not being overwhelmed in the fusion process. As an attention unit parallel to the conventional convolution operation, CAA does not change the original feature map dimension, but only recalibrates the feature values. Therefore, it is relatively independent in model implementation and easy to plug and play. In this implementation manner, CAA efficiently models the long-distance spatial context relationship through global average pooling and orthogonal strip convolution. For slender structures such as the jib and the steel wire rope, CAA can aggregate the global context information of the entire component. This enables the model to better understand the relationship between the defect and the overall morphology of the component, the stress distribution when detecting defects on such components (for example, detecting the linear crack or the multi-point corrosion of the jib over a long distance), thereby improving the detection accuracy and the sensitivity to subtle deformation. At the same time, in the common metal reflection, strong shadow or complex operation background of the hoisting machinery, CAA can recalibrate the feature map through global context information, effectively suppress the background noise, enhance the feature expression of the low-contrast defects such as the rust, the oil stain covered crack, and make them "highlight" in the complex environment, thereby improving the robustness of the detection.

[0061] Further, for the safety assessment of hoisting machinery, the precise location and size of defects are crucial. For example, the length of a crack, the area of corrosion, and the degree of deformation directly affect the safety judgment. The standard IoU loss function may have bottlenecks in optimizing high-precision positioning, especially in cases where defect morphology is diverse and size varies greatly. Moreover, in hoisting machinery inspection data, there may be a large number of backgrounds or easy-to-detect defects, while the number of key, subtle, or irregularly shaped defects (difficult example samples) is relatively small. The standard loss function may not adequately optimize these difficult example samples in the later training stage, resulting in less than ideal regression accuracy of the model for these key defects. Therefore, in some possible implementations, during the training phase of the YOLOv11 model, the original CIoU / GIoU bounding box regression loss of YOLOv11 is replaced with a Unified-IoU (UIoU) loss function. In the output of the detection head of YOLOv11, instead of calculating the CIoU / GIoU loss of each predicted box and the ground truth box, we calculate it according to the UIoU formula. Specifically, it includes calculating the IoU of the predicted box and the ground truth box, as well as their relative position relationship and scale difference, and then obtaining the loss value according to the unified penalty function defined by UIoU. The formula of UIoU contains the combination of dynamic weight term and IoU term, so when implementing, we need to track the IoU quality of the current predicted box and the training progress (epoch) and other information to determine the loss weight assigned to this sample. After enabling UIoU, the gradient obtained by backpropagation will pay more attention to high-quality predicted boxes, which will slightly adjust the update direction of the entire model parameters. It needs to be emphasized that in this embodiment, the replacement of the loss function only affects the training process and does not increase the inference calculation amount. Through such a setting, for the positioning needs of hoisting machinery defects (such as the start and end points of a crack, and the precise boundary of a corrosion area), the dynamic weight mechanism of UIoU can quickly correct the predicted box with large deviation in the early training stage, and focus on fine-tuning high-quality predicted boxes in the later training stage. This enables the model to more accurately frame the defects, providing a more reliable basis for subsequent defect size measurement and risk level assessment. At the same time, UIoU can more effectively handle various complex defects encountered in hoisting machinery inspection by focusing on the IoU quality and confidence of the predicted box. For predictions with high confidence but slightly deviated positioning (e.g., a crack is initially identified but the boundary is not accurate), UIoU will increase the punishment to promote accurate positioning; for low-confidence noise predictions, the impact is relatively reduced. This mechanism helps the model better learn and optimize defects that are difficult to accurately enclose, small, or irregular in shape, thereby improving the safety and reliability of the overall system for inspection.

[0062] In this embodiment, each optimization module can be applied individually or in combination in the YOLOv11 model. In addition to the above optimizations, the other structures are consistent with the YOLOv11 model.

[0063] Figure 3 The step flow of identifying the tower crane training data according to the pre-trained hidden danger identification model in step S12 is schematically shown in an embodiment of the present application, and the flow can be implemented by including the following steps, as shown in Figure 3

[0064] Step S21: Preprocessing the tower crane inspection data to obtain first image data corresponding to the frames in the inspection image or inspection video;

[0065] Step S22: Processing the first image data by using the hidden danger identification model to obtain a first detection result, the first detection result including a plurality of potential hidden danger parameters, the potential hidden danger parameters including a potential hidden danger position corresponding to the potential hidden danger in the first image data, a potential hidden danger first confidence, and a probability of the potential hidden danger belonging to each hidden danger type;

[0066] Step S23: Determining the retained potential hidden danger parameters according to the first detection result to form hidden danger parameters;

[0067] Step S24: Forming a visual mark in the frame of the inspection image or inspection video corresponding to the first image data according to the hidden danger parameters to form a hidden danger image.

[0068] Specifically, step S21 is a step of preprocessing the obtained tower crane inspection data. The tower crane inspection data contains patrol images and / or inspection videos of key parts of the tower crane. Since the output of the hidden danger identification model is a hidden danger image, which is image data, the frames in the inspection video need to be intercepted for subsequent processing. In step S21, in addition to the format processing of the obtained tower crane inspection data, it can also include image size adjustment processing, normalization processing, color space conversion processing, dimension adjustment processing, and batch processing. These processes can make the input image better meet the input requirements of the model and help improve the model performance. Specifically, the size adjustment processing is to adjust the image to a fixed size (such as 640x640 pixels) used in model training. In this process, in order to maintain the object scale, equal proportion scaling is performed, and the insufficient area is filled with gray or other colors. The normalization processing is to scale the pixel value from the original range (such as 0-255) to the range expected by the model (usually 0.0 to 1.0), which is usually achieved by dividing each pixel value by 255.0. The color space conversion processing is to convert the image from the common BGR format (OpenCV default) to the RGB format. The dimension adjustment processing is to adjust the array dimension order to match the requirements of the model input tensor, from (H, W, C) to (C, H, W). After the final processing, the first image data is formed.

[0069] ​Exemplarily, taking the input image tower_crane_frame_1688.jpg (1920x1080, BGR, uint8) as an example, it is first scaled to the length of the longest side to be 640, becoming 640x360. Then, a gray area of (640-360) / 2=140 pixels is filled in the height direction on the top and bottom to obtain an image of 640x640. Then, the pixel value is converted from [0, 255] to [0.0, 1.0] (float32). The color difference format is converted from BGR to RGB. The dimension is adjusted from (640, 640, 3) to (3, 640, 640). Finally, the batch dimension is increased to become (1, 3, 640, 640). The data format of the first image data output by the processed final image before the input model is: a tensor (Tensor) conforming to the model requirements. Shape: [batch_size, channels, height, width], [1, 3, 640, 640]. Data type: float32. Numerical range: [0.0, 1.0].

[0070] Step S22 is a step of processing the first image data after preprocessing to obtain a first detection result. The first detection result is a detection result containing all potential hazards. In step S22, the hazards are identified to the maximum extent regardless of the confidence level of the detected hazards, which can effectively avoid the situation of missing hazards. In this step, the preprocessed image tensor is sent to the pre-trained model. The model performs a forward propagation (Forward Pass) calculation and outputs a set of first prediction results. These first prediction results include a large number of potential hazards and their potential hazard positions in the first image data (such as bounding box coordinates), a potential hazard first confidence score (Objectness Score) contained in each potential hazard, and the probability (Class Probabilities) of each potential hazard belonging to each hazard type (each predefined safety hazard category).

[0071] Exemplarily, the tensor input of (1, 3, 640, 640) described above is obtained one or more tensors, the sum of which encodes a large amount of prediction information. For a model of a 640x640 input, a tensor of a shape similar to [batch_size, num_predictions, 5+num_classes] ([1, 8400, 5+4]) is output. Among them, num_predictions is the total number of potential bounding boxes generated by the model (in the model of the example, it is the sum of predictions based on three different scale feature maps, 80x80+40x40+20x20=8400). 5+num_classes is the information contained in each prediction, including cx, cy, w, h, objectness_score, class1_prob, class2_prob,..., classN_prob. Among them, (cx, cy) is the center point coordinate of the bounding box (relative to the normalized image size or grid unit). (w, h) is the width and height of the bounding box (relative to the normalized image size or grid unit). objectness_score is the confidence of any known object existing in the bounding box. classN_prob is the probability that the object in the bounding box belongs to the Nth class (assuming there are N safety hazard classes). The data format of the first detection result of the final output is one or more tensors (Tensor). It contains a large number of predictions, [batch_size, num_predictions, box_params + objectness + num_classes]. These data contain relative coordinates, object confidence and class probability floating point values.

[0072] After obtaining the first detection data, step S23 is performed to determine the retained potential hazard parameters to form the hazard parameters. In this step, the potential hazard parameters to be retained are obtained after screening the potential hazard parameters. Figure 4 The step flow of step S23 in the hazard identification method for tower crane inspection in an embodiment of the present application is schematically shown, referring to Figure 4 which can be specifically implemented to include the steps of:

[0073] Step S31: determining the confidence of each potential hazard parameter belonging to each hazard type according to the potential hazard first confidence in each potential hazard parameter and the probability of the potential hazard belonging to each hazard type;

[0074] Step S32: determining the first potential hazard parameter in each potential hazard parameter according to the confidence of each potential hazard parameter belonging to each hazard type and a preset threshold;

[0075] Step S33: judging whether the potential hidden danger positions of each potential hidden danger parameter in the first potential hidden danger parameter exist overlap,

[0076] Step S34: when the overlap exists, only keeping the one with the highest confidence in the potential hidden danger parameters with the overlapping potential hidden danger positions, to obtain the kept potential hidden danger parameter, forming the hidden danger parameter.

[0077] In step S31, the confidence of each potential hidden danger belonging to each hidden danger type is first determined. Since the potential hidden danger parameter obtained in step S22 contains the potential confidence of each potential hidden danger and the probability of its belonging to each hidden danger type, the confidence of each potential hidden danger belonging to each hidden danger type can be determined based on the above two parameters. Specifically, the confidence of each potential hidden danger belonging to each hidden danger type can be obtained by multiplying the potential confidence of each potential hidden danger and the probability of its belonging to each hidden danger type. It can be understood that in this step S31, each potential hidden danger corresponds to a probability value of N hidden danger types, and each potential hidden danger accordingly has N confidences corresponding to N hidden danger types.

[0078] Step S32 is to determine the first potential hidden danger parameter in the potential hidden danger parameters based on the confidence of each potential hidden danger belonging to each hidden danger type obtained in step S31. Specifically, it is to determine based on the pre-set preset threshold and the confidence of each potential hidden danger belonging to each hidden danger type obtained in step S31. The preset threshold can be set according to the experience in the training and use process of the actual model, and in this embodiment, the preset threshold is set to 0.4. Then, when the confidence of each potential hidden danger belonging to each hidden danger type obtained in step S31 is lower than 0.4, it is considered to be filtered, and thus the potential hidden danger parameter to be kept is determined.

[0079] For example, it is assumed that there are two prediction boxes for the same "crack" in the first detection result of the model: Box A: [cx1, cy1, w1, h1], objectness=0.9, prob(crack)=0.95 =>Final Confidence = 0.9* 0.95 = 0.855; Box B: [cx2, cy2, w2, h2], objectness=0.88, prob(crack)=0.92 =>Final Confidence = 0.88 * 0.92 = 0.8096. Since both values are greater than the preset threshold 0.4, both potential hidden danger parameters are kept.

[0080] After step S32 is completed, since different potential hazards can be identified for the same location, and even for the same potential hazard, there can be multiple different hazard types corresponding to the confidence values greater than the preset threshold, step S33 is performed to determine whether the potential hazard locations of the potential hazard parameters overlap. Specifically, since in the first detection result, there are potential hazard locations of each potential hazard in the first image data, the overlap of the potential hazard locations can be determined by calculating the Non-Maximum Suppression (NMS) of the potential hazard locations. When the overlap is higher than the preset overlap threshold, it is considered that the potential hazard locations of the two overlap. Specifically, in this embodiment, the overlap threshold can be set to 0.5. After determination, step S34 is performed to retain the potential hazard parameter with the highest confidence in the potential hazard parameters with overlapping potential hazard locations, and then the retained potential hazard parameter is obtained to form the hazard parameter.

[0081] For example, continue to use the above example for illustration. After determining that the confidence of both is greater than the preset threshold, the NMS threshold is calculated. Specifically, the IoU of Box A and Box B is calculated. If IoU>0.5 (NMS threshold), the Box A (0.855) with higher confidence is retained and the Box B is discarded.

[0082] After steps S31 to S34 are completed, step S23 is completed, and the hazard parameter is formed. Finally, step S24 is performed to form a visual marker in the frame of the inspection image or inspection video corresponding to the first image data according to the hazard parameter. Specifically, for example, continue to use the above example for illustration. The normalized coordinates [cx1, cy1, w1, h1] of Box A are converted back to pixel coordinates [xmin, ymin, xmax, ymax] on tower_crane_frame_1688.jpg (1920x1080), for example, [750, 320, 850, 420], and a visual marker is generated to form a hazard image. The hazard parameter corresponding to the visual marker includes the hazard type and the hazard confidence, which can be the data content determined in step S23. These hazard parameters can also be visually displayed on one side of the visual marker in the hazard image, or can be displayed as background data without being displayed in the hazard image.

[0083] The hidden danger identification method for tower crane inspection provided by the present application can identify the acquired tower crane inspection data by using a hidden danger identification model, so as to distinguish the inspection image or the frame in the inspection video in which the hidden danger exists, and visually mark the hidden danger position in the corresponding inspection image or the frame in the inspection video, while displaying the hidden danger type and the hidden danger confidence of the corresponding hidden danger, so that it can be more intuitively seen whether there is a possibility of hidden danger at each inspection position, and the confidence of the hidden danger is high or low, so as to display the hidden danger as much as possible and reduce the probability of missing the hidden danger.

[0084] Figure 5 The principle block diagram of the hidden danger identification device for tower crane inspection in an embodiment of the present application is schematically shown, referring to Figure 5 The hidden danger identification device for tower crane inspection provided by the present application is composed of the following parts:

[0085] The inspection data acquisition module 1 is used to acquire the tower crane inspection data, wherein the tower crane inspection data includes the inspection image and / or the inspection video of the key parts of the tower crane.

[0086] The hidden danger identification module 2 is used to identify the tower crane inspection data according to the pre-trained hidden danger identification model, and determine the tower crane hidden danger data, wherein the tower crane hidden danger data includes the hidden danger image in which the hidden danger exists, the hidden danger image is the frame in the inspection image or the inspection video with the hidden danger visual mark and the hidden danger parameter, and the hidden danger parameter includes the hidden danger type and the hidden danger confidence.

[0087] It should be noted that the implementation process and implementation principle of the hidden danger identification device for tower crane inspection in the embodiment of the present application can be specifically referred to the corresponding description of the above method embodiment, for example, the corresponding description of the acquisition of the inspection data, the design and optimization of the hidden danger identification model, the identification of the tower crane hidden danger data, etc. in the method embodiment part, so it will not be described here. For example, the hidden danger identification device for tower crane inspection in the embodiment of the present application can be any intelligent device with a processor, including but not limited to a computer, a smart phone, a personal computer, a robot, a cloud server, etc.

[0088] In some embodiments, the present application provides a non-volatile computer readable storage medium, wherein the storage medium stores one or more programs including execution instructions, the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the hidden danger identification method for tower crane inspection in any one of the above embodiments of the present application.

[0089] In some embodiments, the present application also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the method for identifying hidden dangers in tower crane inspection according to any one of the above embodiments.

[0090] In some embodiments, the present application also provides an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for identifying hidden dangers in tower crane inspection according to any one of the above embodiments.

[0091] In some embodiments, the present application also provides a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method for identifying hidden dangers in tower crane inspection according to any one of the above embodiments.

[0092] Figure 6 is a hardware structure schematic diagram of an electronic device for executing the method for identifying hidden dangers in tower crane inspection according to another embodiment of the present application, as shown in Figure 6 The device comprises:

[0093] one or more processors 610 and a memory 620, Figure 6 In an example, the processor 610 is taken as an example.

[0094] The device for executing the method for identifying hidden dangers in tower crane inspection can further comprise an input device 630 and an output device 640.

[0095] The processor 610, the memory 620, the input device 630 and the output device 640 can be connected through a bus or other means, Figure 6 In an example, the connection through the bus is taken as an example.

[0096] The memory 620 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for identifying hidden dangers in tower crane inspection according to the embodiments of the present application. The processor 610 performs various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, implements the method for identifying hidden dangers in tower crane inspection according to the above method embodiments.

[0097] The memory 620 can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required by at least one function, and the data storage area can store data created according to the use of the tower crane inspection hidden danger identification method and the like. In addition, the memory 620 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, flash memory device, or other nonvolatile solid-state memory device. In some embodiments, the memory 620 can optionally include a memory disposed remotely with respect to the processor 610, which can be connected to the electronic device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The input device 630 can receive input digital or character information, and generate signals related to user settings and function control of the image processing device. The output device 640 can include a display device such as a display screen.

[0099] The one or more modules are stored in the memory 620 and, when executed by the one or more processors 610, perform the tower crane inspection hidden danger identification method in any of the above method embodiments.

[0100] The above product can perform the method provided in the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0101] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0102] (1) Mobile communication device: This type of device is characterized by having mobile communication function, and providing voice and data communication as the main target. This type of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0103] (2) Ultra-mobile personal computer device: This type of device belongs to the category of personal computers, has computing and processing functions, and generally also has mobile Internet characteristics. This type of terminal includes: PDA, MID and UMPC devices, such as iPad.

[0104] (3) Portable entertainment device: This type of device can display and play multimedia content. This type of device includes: audio and video player (such as iPod), handheld game console, electronic book, and smart toy and portable car navigation device.

[0105] (4) Server: a device providing computing services, the server is composed of a processor, a hard disk, a memory, a system bus, etc., the server is similar to a general computer architecture, but since it needs to provide high-reliable services, it has higher requirements in processing capability, stability, reliability, security, scalability, manageability, etc.

[0106] (5) Other electronic devices with data interaction function.

[0107] The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a general hardware platform, or by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hidden danger identification method for tower crane inspection, characterized in that, The method comprises the following steps: acquiring tower inspection data, wherein the tower inspection data comprises inspection images and / or inspection videos of key parts of a tower crane; identifying the tower inspection data according to a pre-trained hidden danger identification model to determine tower hidden danger data, wherein the tower hidden danger data comprises hidden danger images, the hidden danger images are inspection images or frames in inspection videos with hidden danger visual markers and hidden danger parameters, and the hidden danger parameters comprise hidden danger types and hidden danger confidence levels; the pre-trained hidden danger identification model is a model based on a YOLOv11 model for image recognition, wherein in the model, all 3*3 down-sampling convolutional layers in the original Backbone and Neck are replaced by LDConv modules; in the last C3 module of the Backbone, a BottleNeck structure is replaced by an RFEM to form a C3-RFEM module; in the output stage of the Backbone, P3, P4 and P5 features output by the Backbone are respectively sent to corresponding CAA modules to obtain new weighted features which are then input into the Neck; in the training stage, a loss calculation module is modified, and original CIoU / GIoU bounding box regression loss is replaced by a Unified-IoU (UIoU) loss function.

2. The hidden danger identification method for tower crane inspection according to claim 1, characterized in that, The method of identifying the tower inspection data according to the pre-trained hidden danger identification model to determine the tower hidden danger data comprises the following steps: preprocessing the tower inspection data to obtain first image data corresponding to the inspection images or frames in the inspection videos; processing the first image data by using the hidden danger identification model to obtain first detection results, wherein the first detection results comprise a plurality of potential hidden danger parameters, the potential hidden danger parameters comprise potential hidden danger positions corresponding to the potential hidden dangers in the first image data, potential hidden danger first confidence levels and probabilities of the potential hidden dangers belonging to each hidden danger type; determining retained potential hidden danger parameters according to the first detection results to form hidden danger parameters; forming visual markers in the inspection images or frames in the inspection videos corresponding to the first image data according to the hidden danger parameters to form hidden danger images.

3. The hidden danger identification method for tower crane inspection according to claim 2, characterized in that, The method of determining the retained potential hidden danger parameters according to the first detection results to form the hidden danger parameters comprises the following steps: determining confidence levels of each potential hidden danger parameter belonging to each hidden danger type according to the potential hidden danger first confidence levels and the probabilities of the potential hidden dangers belonging to each hidden danger type in each potential hidden danger parameter; determining a first potential hidden danger parameter in each potential hidden danger parameter according to the confidence levels of each potential hidden danger parameter belonging to each hidden danger type and a preset threshold; judging whether the potential hidden danger positions of each potential hidden danger parameter in the first potential hidden danger parameter overlap or not, when the potential hidden danger positions overlap, only retaining one potential hidden danger parameter with the highest confidence level among the potential hidden danger parameters with overlapping potential hidden danger positions to obtain the retained potential hidden danger parameter to form the hidden danger parameter.

4. A hidden danger identification device for tower inspection, comprising: an inspection data acquisition module configured to acquire tower inspection data, wherein the tower inspection data comprises inspection images and / or inspection videos of key parts of a tower crane; The hidden danger identification module is configured to identify tower crane inspection data according to a pre-trained hidden danger identification model, and determine tower crane hidden danger data, wherein the tower crane hidden danger data includes hidden danger images with hidden dangers, and the hidden danger images are frames in inspection images or inspection videos with hidden danger visual markers and hidden danger parameters, and the hidden danger parameters include hidden danger types and hidden danger confidence levels. The pre-trained hidden danger identification model takes a YOLOv11 model as a basis for image recognition, wherein in the model, All 3*3 down-sampling convolutional layers in the Backbone and the Neck are replaced by LDConv modules; In the last C3 module of the Backbone, a BottleNeck structure is replaced by an RFEM to form a C3-RFEM module; In the output stage of the Backbone, P3, P4 and P5 features output by the Backbone are respectively sent to corresponding CAA modules to obtain new weighted features, which are then input into the Neck; In the training stage, a loss calculation module is modified, and original CIoU / GIoU bounding box regression loss is replaced by a Unified-IoU (UIoU) loss function.

5. An electronic device comprising: At least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1 to 3.

6. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of the method of any one of claims 1 to 3.

Images