Method and device for detecting personnel under suspension arm based on monocular image and computer equipment

Abstract

The invention relates to a method and a device for detecting personnel under a suspension arm based on a monocular image and computer equipment. The method comprises the following steps: acquiring a monocular image acquired by a camera installed on a suspension arm construction site, and inputting the monocular image into a yolov8 target segmentation model for segmentation detection to obtain a target area map comprising at least one suspension arm detection area and at least one target personnel detection area, then inputting the target area map into a DehanythingV2 depth estimation model for depth estimation to obtain a target area depth map, based on the target area depth map, obtaining the total map distance of the target area map, the suspension arm depth of each suspension arm detection area and the personnel depth of each target personnel detection area, and finally, according to the total map distance, the suspension arm depth and the personnel depth, determining the personnel depth of each target personnel according to the total map distance, the suspension arm depth and the personnel depth. And generating a detection result of personnel detection under the suspension arm. By means of the combination mode, the data labeling complexity and computing power resource consumption are greatly reduced, and then the detection efficiency of personnel under the suspension arm is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent management of production operation environments, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for detecting personnel under a crane arm based on monocular images. Background Art

[0002] With the development of science and technology, the safety standards of construction sites are constantly improving. But even so, it is absolutely forbidden for people to stay under the crane. Safety regulations must be strictly followed to ensure that there are no people in the working area, prevent accidents, and ensure the safety of every worker.

[0003] The existing method for detecting people under the crane arm based on binocular images combines a deep neural network to detect the image position coordinates of the crane arm and the person; based on the pixel difference of the same target in the binocular image, the three-dimensional coordinate points of the target are restored, and the three-dimensional coordinate points of the crane arm and the person are compared to determine whether the person is in the area under the crane arm.

[0004] However, the current method for detecting people under a crane arm based on binocular images has the problem of low detection efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for detecting people under a boom based on monocular images, which can improve detection efficiency, in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting people under a boom based on a monocular image, comprising:

[0007] Obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0008] Input the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0009] Based on the target area depth map, the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained;

[0010] The detection results of personnel under the boom are generated based on the full map distance, boom depth and the depth of each person.

[0011] In one embodiment, a detection result of a person under the boom is generated according to the full image distance, the boom depth and the depth of each person, including:

[0012] Get the depth distance between the boom depth and the depth of each person;

[0013] Based on the full-image distance and each depth distance, the detection results of personnel detection under the boom are generated.

[0014] In one embodiment, based on the full image distance and each depth distance, a detection result of a person under the boom is generated, including:

[0015] A rule is constructed based on the full image distance and a preset depth distance threshold to obtain a depth distance threshold;

[0016] When the depth distance is less than the depth distance threshold, it is determined that the boom and the person are at the same depth, and a corresponding detection result is generated and returned to the user terminal;

[0017] If the depth distance is equal to or greater than the depth distance threshold, it is determined that the boom and the person are not at the same depth, and the corresponding detection result is generated and returned to the user terminal.

[0018] In one embodiment, the monocular image is input into the pre-built yolov8 target segmentation model for segmentation detection to obtain a target area map, including:

[0019] The monocular image is input into the yolov8 target segmentation model for segmentation detection. The yolov8 target segmentation model is used to filter out candidate monocular images with boom detection areas and personnel detection areas from the monocular image.

[0020] According to the vertical positional relationship between the boom detection area and the personnel detection area, the candidate monocular image is cropped to obtain a target area map.

[0021] In an exemplary embodiment, based on the target area depth map, the full image distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained, including:

[0022] Traverse the target area depth map to obtain the candidate boom depth of each boom detection area and the personnel depth of each target personnel detection area;

[0023] Determine the maximum boom depth from each candidate boom depth, and determine the minimum personnel depth from each personnel depth, and determine the full map distance of the target area map according to the maximum boom depth and the minimum personnel depth;

[0024] A mean boom depth is obtained based on each candidate boom depth, and the mean boom depth is determined as the boom depth.

[0025] In one embodiment, obtaining a monocular image captured by a camera installed at a crane construction site includes:

[0026] Obtain the original monocular image captured by the camera installed at the crane construction site;

[0027] The original monocular image is resized according to a preset image size to obtain a monocular image.

[0028] In a second aspect, the present application also provides a device for detecting people under a boom based on a monocular image, comprising:

[0029] The area map acquisition module is used to acquire the monocular image collected by the camera installed at the crane construction site, input the monocular image into the pre-built yolov8 target segmentation model for segmentation detection, and obtain the target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0030] A depth map acquisition module is used to input the target area map into a pre-built DephanythingV2 depth estimation model for depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0031] A depth data acquisition module is used to obtain the full-image distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area based on the target area depth map;

[0032] The personnel detection module is used to generate the detection results of personnel under the boom according to the full map distance, boom depth and the depth of each person.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0035] Input the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0036] Based on the target area depth map, the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained;

[0037] The detection results of personnel under the boom are generated based on the full map distance, boom depth and the depth of each person.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0040] Input the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0041] Based on the target area depth map, the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained;

[0042] The detection results of personnel under the boom are generated based on the full map distance, boom depth and the depth of each person.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0044] Obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0045] Input the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0046] Based on the target area depth map, the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained;

[0047] The detection results of personnel under the boom are generated based on the full map distance, boom depth and the depth of each person.

[0048] The above-mentioned method, device, computer equipment, computer-readable storage medium and computer program product for detecting people under the boom based on monocular images obtain a monocular image collected by a camera installed at the construction site of the boom, input the monocular image into the pre-built yolov8 target segmentation model for segmentation detection, obtain a target area map including at least one boom detection area and at least one target person detection area, and then input the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, obtain the target area depth map through the DephanythingV2 depth estimation model, and obtain the full-image distance of the target area map, the boom depth of each boom detection area and the person depth of each target person detection area based on the target area depth map, and finally generate the detection result of the person under the boom according to the full-image distance, the boom depth and the depth of each person. The monocular image is used for target segmentation detection, and then the depth estimation is performed. The position problem of the two-dimensional image in the three-dimensional space is effectively solved by the depth estimation of the detection target, and a more accurate target position relationship is obtained. The above-mentioned combination method greatly reduces the complexity of data annotation and the consumption of computing resources, thereby improving the detection efficiency of the person under the boom detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] FIG1 is a diagram showing an application environment of a method for detecting people under a boom based on a monocular image in one embodiment;

[0051] FIG2 is a schematic diagram of a flow chart of a method for detecting personnel under a boom based on a monocular image in one embodiment;

[0052] FIG3 is a schematic flow chart of a method for detecting a person under a boom based on a monocular image in another embodiment;

[0053] FIG4 is a schematic diagram of an original monocular image in one embodiment;

[0054] FIG5 is a schematic diagram of original monocular image reasoning in another embodiment;

[0055] FIG6 is a block diagram of a target area in one embodiment;

[0056] FIG7 is a schematic diagram of cutting out a target area map in another embodiment;

[0057] FIG8 is a schematic diagram of a depth map of a target area in one embodiment;

[0058] FIG9 is a schematic diagram of depth relationship and detection frame calculation results in another embodiment;

[0059] FIG10 is a block diagram of a device for detecting personnel under a boom based on a monocular image in one embodiment;

[0060] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The method for detecting people under the boom based on a monocular image provided in an embodiment of the present application can be applied in an application environment as shown in FIG1. ​​Among them, the camera 102 is installed at the construction site of the boom project, and pictures of the boom area and the personnel area can be taken. The camera 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the monocular image collected by the camera 102, inputs the monocular image into the pre-built yolov8 target segmentation model for segmentation detection, and obtains a target area map including at least one boom detection area and at least one target personnel detection area, and then inputs the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtains the target area depth map through the DephanythingV2 depth estimation model. Based on the target area depth map, the full image distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained. Finally, according to the full image distance, the boom depth and the depth of each personnel, the detection result of the detection of people under the boom is generated. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0063] In an exemplary embodiment, as shown in FIG2 , a method for detecting people under a crane arm based on a monocular image is provided, and the method is applied to the server 104 in FIG1 as an example for explanation, including the following steps S201 to S204. Among them:

[0064] Step S201, obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area.

[0065] Among them, monocular images can be understood as images taken with a single camera (not a stereo camera or binocular camera). Monocular images can only provide information from one perspective, so they contain less information than stereo images and are easier to obtain than stereo images; the yolov8 target segmentation model is an advanced target detection and segmentation model, and is the latest version in the yolo (you only look once) series. Yolov8 uses deep learning technology and is mainly used for real-time object detection and instance segmentation; the boom detection area can be understood as the boom target detection box, and similarly, the target person detection area can be understood as the person target detection box.

[0066] Optionally, the server 104 obtains a monocular image captured by a camera 102 installed at the crane construction site, and then inputs the monocular image into a pre-built yolov8 target segmentation model for segmentation detection to obtain a target area map including at least one crane target detection frame and at least one personnel target detection frame. The server 104 obtains a monocular image and inputs the monocular image into a yolov8 target segmentation model for segmentation detection to obtain a target area map, which lays a data foundation for the subsequent acquisition of the depth of each target area in the target area map. At the same time, the monocular image does not require excessive data annotation and data processing, which reduces the cost of computing resources.

[0067] Step S202 , inputting the target area map into a pre-built DephanythingV2 depth estimation model for depth estimation, and obtaining a target area depth map through the DephanythingV2 depth estimation model.

[0068] Among them, the DephanythingV2 depth estimation model is a model used for depth estimation, which can infer depth information from a monocular image. The target area depth map can be understood as a depth display map containing the regional depth of each detection area.

[0069] For example, the server 104 inputs the target area map into the pre-built DephanythingV2 depth estimation model for depth estimation, and obtains the target area depth map containing the regional depth values ​​of each detection area through the DephanythingV2 depth estimation model. By inputting the target area map into the DephanythingV2 model, the depth value of each specific area or object can be obtained, which lays a data foundation for the subsequent acquisition of the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area. The accuracy of the depth value in the target area depth map is improved by using the DephanythingV2 model, thereby improving the detection accuracy of personnel detection under the boom.

[0070] Step S203, based on the target area depth map, obtain the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area.

[0071] The full-image distance can be understood as the difference between the maximum and minimum regional depths in the target region depth map, and the boom depth can be understood as the average of the depth values ​​of the boom region.

[0072] Optionally, based on the target area depth map, the server 104 obtains the personnel depth of each target personnel detection area and the candidate boom depth of each boom detection area, obtains the boom depth mean based on each candidate boom depth, and determines the boom depth mean as the boom depth; and traverses the target area depth map to determine the maximum area depth and the minimum area depth, and determines the difference between the maximum area depth and the minimum area depth as the full map distance of the target area map. The full map distance, boom depth, and multiple personnel depths are obtained using the target area depth map, which speeds up the acquisition of each depth data, thereby improving the detection speed of personnel detection under the boom.

[0073] Step S204, generating a detection result of the person under the boom according to the full map distance, the boom depth and the depth of each person.

[0074] For example, the server 104 uses the full-map distance to determine the corresponding depth distance threshold, and compares the depth distance between the boom depth and the depth of each person with the depth distance threshold. When the depth distance is less than the depth distance threshold, it is determined that the boom and the person below are at the same depth, and further danger detection is required, and the boom is temporarily not started; when the depth distance is greater than or equal to the depth distance threshold, it is determined that the boom and the person below are not at the same depth, and no further danger detection is required, and the boom can be started for construction. By judging the depth relationship between the boom and the person below through the full-map distance, potential dangerous situations can be effectively identified, accidents that may be caused by misoperation of the boom can be prevented, and the safety of on-site operations can be ensured. In addition, the depth distance threshold is set according to the current application scenario rather than a fixed value, thereby improving the reliability of the detection results.

[0075] In the above-mentioned method for detecting people under the boom based on monocular images, by obtaining a monocular image collected by a camera installed at the construction site of the boom, the monocular image is input into the pre-built yolov8 target segmentation model for segmentation detection, and a target area map including at least one boom detection area and at least one target person detection area is obtained, and then the target area map is input into the pre-built DephanythingV2 depth estimation model for depth estimation, and the target area depth map is obtained by the DephanythingV2 depth estimation model. Based on the target area depth map, the full image distance of the target area map, the boom depth of each boom detection area, and the person depth of each target person detection area are obtained. Finally, according to the full image distance, the boom depth and the depth of each person, the detection result of the person under the boom is generated. The monocular image is used for target segmentation detection, and then the depth estimation is performed. The position problem of the two-dimensional image in the three-dimensional space is effectively solved by the depth estimation of the detection target, and a more accurate target position relationship is obtained. The above-mentioned combination method greatly reduces the complexity of data annotation and the consumption of computing resources, thereby improving the detection efficiency of the person under the boom.

[0076] In one embodiment, a detection result of personnel detection under the boom is generated according to the full-image distance, the boom depth and the depth of each person, including: obtaining the depth distance between the boom depth and the depth of each person; and generating a detection result of personnel detection under the boom based on the full-image distance and the depth distances.

[0077] Optionally, the server 104 uses the boom depth to subtract the depth of each person to obtain the corresponding depth distance, and then generates the detection result of the person detection under the boom according to the depth distance threshold obtained from the full map distance and the comparison result between the depth distances. By calculating the difference between the boom depth and the depth of the person below, the system can obtain an accurate spatial relationship, thereby determining the relative position between the boom and the person, and generating a corresponding detection result by comparing the distance of the relative position and the depth distance threshold, responding to the detected dangerous situation in real time, thereby immediately deciding whether to suspend the boom operation and enhancing safety protection during the operation.

[0078] In one of the embodiments, based on the full-image distance and each depth distance, a detection result of a person under the boom is generated, including: constructing a rule according to the full-image distance and a pre-set depth distance threshold to obtain the depth distance threshold; when the depth distance is less than the depth distance threshold, it is determined that the boom and the person are at the same depth, and a corresponding detection result is generated and returned to the user terminal; if the depth distance is equal to or greater than the depth distance threshold, it is determined that the boom and the person are not at the same depth, and a corresponding detection result is generated and returned to the user terminal.

[0079] The depth distance threshold construction rule can be understood as determining X% of the full image distance as the depth distance threshold.

[0080] For example, the server 104 constructs rules based on the full map distance and a preset depth distance threshold, and determines X% of the full map distance as the depth distance threshold. When the depth distance is less than the depth distance threshold, it is determined that the boom and the personnel are at the same depth, and a corresponding detection result is generated and returned to the user terminal. The user receives the detection result and makes an instruction to further perform danger detection; if the depth distance is equal to or greater than the depth distance threshold, it is determined that the boom and the personnel are not at the same depth, and a corresponding detection result is generated and returned to the user terminal. After receiving this detection result, the user can directly start the boom for construction. The depth distance threshold construction rule is set according to the actual situation on site, that is, X% can be adjusted according to the actual situation. In this way, the corresponding safety protection level can be adjusted better, thereby improving the safety of user operations.

[0081] In an exemplary embodiment, a monocular image is input into a pre-built yolov8 target segmentation model for segmentation detection to obtain a target area map, including: inputting the monocular image into the yolov8 target segmentation model for segmentation detection, and screening out candidate monocular images with a boom detection area and a personnel detection area from the monocular image through the yolov8 target segmentation model; and cropping the candidate monocular image according to the positional relationship between the boom detection area and the personnel detection area in the vertical direction to obtain a target area map.

[0082] Optionally, the server 104 inputs the monocular image into the yolov8 target segmentation model for segmentation detection, and the yolov8 target segmentation model is used to screen out candidate monocular images with boom detection areas and personnel detection areas from the monocular image, and then, according to the vertical positional relationship between the boom detection area and the personnel detection area, the boom detection area and the personnel detection area are cropped together in the same vertical direction to obtain a target area map. The screening of the monocular image avoids the interference of invalid monocular images, thereby speeding up the detection speed of personnel detection under the boom. In addition, by cropping the monocular image, the interference of the full image is avoided, thereby ensuring the detection accuracy of personnel detection under the boom.

[0083] In one embodiment, based on the target area depth map, the full-map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area are obtained, including: traversing the target area depth map, obtaining candidate boom depths of each boom detection area, and the personnel depth of each target personnel detection area; determining the maximum boom depth from each candidate boom depth, and determining the minimum personnel depth from each personnel depth, and determining the full-map distance of the target area map according to the maximum boom depth and the minimum personnel depth; obtaining the average boom depth based on each candidate boom depth, and determining the average boom depth as the boom depth.

[0084] Among them, the candidate boom depths can be understood as the depths of different boom positions. Since the boom has a shape, the corresponding boom depths are different.

[0085] For example, the server 104 traverses the target area depth map, obtains the candidate boom depths of each boom detection area, and the personnel depths of each target personnel detection area, and determines the maximum boom depth from each candidate boom depth, and determines the minimum personnel depth from each personnel depth, and uses the maximum boom depth to subtract the minimum personnel depth to obtain the full map distance of the target area map, and obtains the mean boom depth based on each candidate boom depth, and determines the mean boom depth as the boom depth. Using the mean depth as a representation of the boom depth can effectively smooth the fluctuations introduced by single-point data, enhance overall stability, and determine the full map distance based on the maximum boom depth and the minimum personnel depth, thereby ensuring the adaptability of the full map distance to the current application.

[0086] In one of the embodiments, obtaining a monocular image captured by a camera installed at a crane construction site includes: obtaining an original monocular image captured by the camera installed at the crane construction site; and resizing the original monocular image according to a preset image size to obtain a monocular image.

[0087] Optionally, the server 104 obtains the original monocular image captured by the camera installed at the crane construction site through the communication network, and then resizes the original monocular image according to a preset image size to obtain a monocular image. Adjusting the image size enables the system to adapt to different image processing algorithms and model input requirements, ensures compatibility, speeds up the response speed of the yolov8 target segmentation model for segmentation detection, and also improves the accuracy of segmentation detection.

[0088] In an exemplary embodiment, in order to solve the current problem of detecting people staying under the boom, as shown in FIG3 , a specific algorithm flow chart of a method for detecting people under the boom based on a monocular image is proposed, wherein the specific operation of the algorithm detection is as follows (the following data are all specific embodiments, and the solution of the present application can be realized only in this case):

[0089] 1. Image detection: first resize the image to the target size (as shown in Figure 4), and send it to the trained yolov8-seg model for segmentation detection. The personnel detection area is obtained (as shown in Figure 5), where the red box is the detected boom, the green box is the operator detection box, and the yellow box is the boom platform operator. The two green areas below the boom in the red area are considered to be below the boom by judging directly through the vertical coordinates.

[0090] 2. Cropping the target area. Due to the uncertainty of the shooting angle and the complexity of the photographed area, the upper and lower thresholds of depth estimation will be affected to a certain extent. Calculate the largest target area of ​​the person frame (as shown in Figure 6), and crop the largest target area (as shown in Figure 7) to avoid interference from the full image.

[0091] 3. Monocular depth estimation algorithm: After cropping the largest target area, it is sent to the DepthanythingV2 depth estimation model for estimation, and the full depth value of the cropped image area is returned (as shown in Figure 8).

[0092] 4. Depth threshold calculation method: First, calculate the maximum and minimum depth values ​​of the target image area as the total distance a of the whole image, and then take the average depth threshold of the segmented area for each target as the depth of the target. Calculate the distance difference b between the crane M and the person N. If b accounts for less than 10% of the total distance a of the whole image, the crane M and the person N are considered to be at the same depth.

[0093] 5. Calculate the depth of the target area to determine whether it belongs to a depth range (as shown in Figure 8). The model detected by the segmentation algorithm. The corrected target detection frame of the person staying under the boom (as shown in Figure 9) can effectively eliminate the false detection of the spatial position of the boom and the person.

[0094] Compared with the prior art, this application has the following advantages:

[0095] 1. By resizing the original monocular image captured by the camera and then inputting the resized monocular image into the trained yolov8-seg model for segmentation detection, the detection accuracy of segmentation detection is improved, thereby improving the detection accuracy of personnel detection under the boom.

[0096] 2. By combining target detection with a depth estimation algorithm, the position relationship of the two-dimensional image in the three-dimensional space is effectively converted through the depth estimation of the detection target, which greatly reduces the consumption of computing resources and speeds up the detection of people under the crane arm.

[0097] 3. Using two-dimensional monocular images to judge the positional relationship in three-dimensional space reduces the amount of data processing, thereby improving the detection efficiency of personnel under the boom.

[0098] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting people under the boom based on a monocular image for realizing the method for detecting people under the boom based on a monocular image. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more devices for detecting people under the boom based on a monocular image provided below can refer to the limitations of the method for detecting people under the boom based on a monocular image above, and will not be repeated here.

[0100] In an exemplary embodiment, as shown in FIG10 , a device for detecting people under a boom based on a monocular image is provided, comprising: an area map acquisition module 1001, a depth map acquisition module 1002, a depth data acquisition module 1003 and a people detection module 1004, wherein:

[0101] The area map acquisition module 1001 is used to acquire a monocular image collected by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area;

[0102] The depth map acquisition module 1002 is used to input the target area map into the pre-built DephanythingV2 depth estimation model to perform depth estimation, and obtain the target area depth map through the DephanythingV2 depth estimation model;

[0103] The depth data acquisition module 1003 is used to obtain the full-image distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area based on the target area depth map;

[0104] The personnel detection module 1004 is used to generate a detection result of personnel detection under the boom according to the full-image distance, the boom depth and the depth of each person.

[0105] In one embodiment, the personnel detection module 1004 further includes:

[0106] The depth distance acquisition submodule is used to obtain the depth distance between the boom depth and the depth of each person.

[0107] The personnel detection submodule is used to generate the detection results of personnel detection under the boom based on the full image distance and the depth distance.

[0108] In one of the embodiments, the personnel detection submodule is also used to construct rules based on the full-image distance and a pre-set depth distance threshold to obtain the depth distance threshold; when the depth distance is less than the depth distance threshold, it is determined that the boom and the personnel are at the same depth, and a corresponding detection result is generated and returned to the user terminal; if the depth distance is equal to or greater than the depth distance threshold, it is determined that the boom and the personnel are not at the same depth, and a corresponding detection result is generated and returned to the user terminal.

[0109] In an exemplary embodiment, the region map acquisition module 1001 is also used to input the monocular image into the yolov8 target segmentation model for segmentation detection, and screen out candidate monocular images with boom detection areas and personnel detection areas from the monocular image through the yolov8 target segmentation model; according to the positional relationship between the boom detection area and the personnel detection area in the vertical direction, the candidate monocular image is cropped to obtain a target region map.

[0110] In one embodiment, the depth data acquisition module 1003 is also used to traverse the target area depth map, obtain the candidate boom depths of each boom detection area, and the personnel depths of each target personnel detection area; determine the maximum boom depth from each candidate boom depth, and determine the minimum personnel depth from each personnel depth, and determine the full map distance of the target area map based on the maximum boom depth and the minimum personnel depth; obtain the average boom depth based on each candidate boom depth, and determine the average boom depth as the boom depth.

[0111] In one embodiment, the area map acquisition module 1001 also includes a monocular image preprocessing submodule, which is used to obtain the original monocular image captured by a camera installed at the crane construction site; the original monocular image is resized according to a preset image size to obtain a monocular image.

[0112] Each module in the above-mentioned monocular image-based under-arm personnel detection device can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0113] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store monocular images, target area maps, target area depth maps, full map distances, boom depths, personnel depths and detection results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting personnel under a boom based on a monocular image is implemented.

[0114] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method for detecting people under a boom based on a monocular image in the above embodiment is implemented.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting people under a boom based on a monocular image in the above embodiment is implemented.

[0117] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for detecting people under a boom based on a monocular image in the above embodiment.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting people under a boom based on a monocular image, characterized in that: The method comprises: Obtain a monocular image captured by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area; Inputting the target area map into a pre-built DephanythingV2 depth estimation model for depth estimation, and obtaining a target area depth map through the DephanythingV2 depth estimation model; Based on the target area depth map, obtaining the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area; A detection result of the person detection under the boom is generated according to the full-image distance, the boom depth and each of the depths.

2. The method according to claim 1, characterized in that The generating of the detection result of the person under the boom according to the full map distance, the boom depth and the depth of each person comprises: Obtaining the depth distance between the boom depth and each of the personnel depths; Based on the full-image distance and each of the depth distances, a detection result of the person detection under the boom is generated.

3. The method according to claim 2, characterized in that The generating a detection result of the person detection under the boom based on the full image distance and each of the depth distances includes: A rule is constructed according to the full image distance and a preset depth distance threshold to obtain a depth distance threshold; When the depth distance is less than the depth distance threshold, it is determined that the boom and the person are at the same depth, and a corresponding detection result is generated and returned to the user terminal; If the depth distance is equal to or greater than the depth distance threshold, it is determined that the boom and the person are not at the same depth, and a corresponding detection result is generated and returned to the user terminal.

4. The method according to claim 1, characterized in that The monocular image is input into a pre-built yolov8 target segmentation model for segmentation detection to obtain a target area map, including: The monocular image is input into the yolov8 target segmentation model for segmentation detection, and the candidate monocular images with boom detection areas and personnel detection areas are screened out from the monocular image by the yolov8 target segmentation model; According to the positional relationship between the boom detection area and the personnel detection area in the vertical direction, the candidate monocular image is cropped to obtain the target area map.

5. The method according to claim 1, characterized in that The obtaining, based on the target area depth map, the full map distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area includes: Traversing the target area depth map, obtaining the candidate boom depths of each boom detection area and the personnel depths of each target personnel detection area; Determine a maximum boom depth from each of the candidate boom depths, and determine a minimum personnel depth from each of the personnel depths, and determine the full map distance of the target area map according to the maximum boom depth and the minimum personnel depth; A boom depth average is obtained based on each of the candidate boom depths, and the boom depth average is determined as the boom depth.

6. The method according to claim 1, characterized in that The obtaining of a monocular image collected by a camera installed at the crane construction site includes: Obtain the original monocular image captured by the camera installed at the crane construction site; The original monocular image is resized according to a preset image size to obtain the monocular image.

7. A device for detecting people under a boom based on a monocular image, characterized in that: The device comprises: The area map acquisition module is used to acquire a monocular image collected by a camera installed at the crane construction site, input the monocular image into a pre-built yolov8 target segmentation model for segmentation detection, and obtain a target area map; the target area map includes at least one crane detection area and at least one target person detection area; A depth map acquisition module, used for inputting the target area map into a pre-built DephanythingV2 depth estimation model for depth estimation, and obtaining a target area depth map through the DephanythingV2 depth estimation model; A depth data acquisition module, for obtaining the full-image distance of the target area map, the boom depth of each boom detection area, and the personnel depth of each target personnel detection area based on the target area depth map; The personnel detection module is used to generate a detection result of personnel detection under the boom according to the full map distance, the boom depths and the personnel depths.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images