Method, device and system for identifying personnel under crane jib and related equipment

Abstract

The application discloses a personnel identification method, device and system under a crane boom and related equipment. The original image is input into a segmentation network model to segment the personnel and the boom, and personnel boom mask data is obtained. The original image is input into a depth estimation network to obtain depth data of the image. The personnel boom mask data and the depth data of the image are combined to obtain personnel boom depth data. According to the personnel boom depth data, the overlapping degree of the personnel and the boom in horizontal, vertical and depth directions is judged, and a first probability that the personnel is located below the boom is determined. Position coding information of the personnel boom depth data and model prompt words are input into a large language model to obtain a second probability that the personnel is located below the boom. According to the first probability and the second probability, a comprehensive judgment result that the personnel is located in a dangerous area below the boom is obtained. The personnel under the boom can be accurately identified, the device has low deployment cost, strong space perception ability, and the identification process is more efficient and convenient.

Description

Technical Field

[0001] This invention relates to the field of industrial safety monitoring technology, and in particular to a method, device, system and related equipment for identifying personnel under a crane boom. Background Technology

[0002] In oil and gas production and construction scenarios, tower cranes (referred to as "cranes") are widely used as heavy equipment in equipment installation, pipeline laying, and other operations. However, oil and gas production and construction sites are complex environments with numerous obstacles and visual interference, and crane operators have limited visibility, especially with blind spots below the boom, making them highly susceptible to collisions, crushing accidents, and other safety incidents, seriously threatening the lives of construction workers. Traditional safety monitoring methods mainly rely on manual supervision and simple sensor alarms, which suffer from low efficiency, high false alarm rates, and the inability to proactively identify dangerous behaviors, making it difficult to meet the high safety requirements of oil and gas production and construction.

[0003] In the field of industrial safety monitoring, especially in oil and gas production and construction scenarios, determining whether a person is standing under a crane boom is a critical issue. Currently, deep learning-based object detection algorithms are the mainstream solution, typically using RGB (red, green, and blue) images as input data. These methods can accurately detect the positions of the boom and workers in the image, but they detect the two-dimensional spatial positions of the boom and workers in the image. Due to the lack of perception of scene spatial information, they cannot accurately determine whether a person is actually under the boom, thus limiting their reliability in practical applications. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a method, apparatus, system and related equipment for identifying personnel under a crane boom that overcomes or at least partially solves the above problems.

[0005] This invention provides a method for identifying personnel under a crane boom, comprising:

[0006] The original image of the crane construction site is input into the segmentation network model to segment out the personnel and crane boom in the original image, and the personnel and crane boom mask data is obtained.

[0007] The original image is input into the depth estimation network to obtain the depth data of the original image;

[0008] The personnel boom mask data and the depth data of the original image are combined to obtain the personnel boom depth data;

[0009] Based on the personnel boom depth data, determine the degree of overlap between the personnel and the boom in the horizontal, vertical and depth directions, and determine the first probability that the personnel are located in the danger zone below the boom based on the degree of overlap;

[0010] The personnel boom depth data is encoded to obtain location coding information. The location coding information and the pre-built large language model prompts are input into the pre-built large language model to obtain the second probability that the personnel are located in the danger zone below the boom.

[0011] Based on the first probability and the second probability, a comprehensive judgment result is obtained that the personnel are located in the danger zone below the boom.

[0012] In some optional embodiments, the step of inputting the original image of the crane construction site into the segmentation network model to segment the personnel and crane boom in the original image and obtain personnel and crane boom mask data includes:

[0013] Acquire video data captured by video equipment at the crane construction site, and extract frames from the video data at a preset frequency to obtain the original image;

[0014] The extracted original image is input into the segmentation network model to segment the personnel and crane boom in the image; the segmentation network model is trained using historical image data of the crane construction site as samples, the historical image data includes historical images and corresponding label data, the label data includes personnel and / or crane boom that have been marked in the image;

[0015] Create a mask image with the same size as the extracted original image. Based on the segmented personnel and crane boom, add corresponding personnel and crane boom identifiers to the corresponding pixels in the mask image to obtain the personnel and crane boom mask image.

[0016] In some optional embodiments, the step of inputting the original image into the depth estimation network to obtain the depth data of the original image includes:

[0017] The original image is input into the depth estimation network. Based on the distance between the target object corresponding to each pixel in the original image and the video device, the depth value representing the distance between each pixel and the video device is determined, resulting in the original image depth map including the depth value corresponding to each pixel.

[0018] In some optional embodiments, combining the personnel boom mask data and the depth data of the original image to obtain the personnel boom depth data includes:

[0019] Set the pixel corresponding to the personnel crane in the personnel crane mask image to 1 and set the other pixels to 0 to obtain the processed personnel crane mask image. Multiply the processed personnel crane mask image with the corresponding pixel values ​​in the original image depth map to obtain the personnel crane depth map.

[0020] In some optional embodiments, the step of determining the degree of overlap between the personnel and the boom in the horizontal, vertical, and depth directions based on the personnel boom depth data, and determining the first probability that the personnel are located in the danger zone below the boom based on the degree of overlap, includes:

[0021] Based on the personnel boom depth map, determine the boom area and personnel area in the image;

[0022] Determine whether the personnel are located below the boom based on the upper and lower boundaries of the vertical projection of the boom area and the personnel area;

[0023] Determine whether personnel are located within the area of ​​the boom area based on the left and right boundaries of the vertical projection of the boom area and the personnel area;

[0024] If a person is located below the boom and within the boom area, based on the person's boom depth data, determine the relationship between the depth difference between the average depth of the person's area and the average depth of the boom area and the set difference threshold range, and determine the first probability that the person is located in the danger zone below the boom.

[0025] In some optional embodiments, the location encoding of the personnel boom depth data to obtain location encoding information, and the input of the location encoding information and pre-built large language model prompts into the pre-built large language model, to obtain a second probability that the personnel are located in the danger zone below the boom, including:

[0026] Adjust the image size of the personnel crane depth map based on the number of lexical units in the large language model;

[0027] The adjusted personnel boom depth map is position-encoded to obtain position-encoded information in text format. The position-encoded information includes the position coordinates of each pixel in the image, the corresponding depth information, and the personnel boom identification information.

[0028] The pre-designed large language model prompt words and text-formatted location encoding information are input into the large language model, and the second probability of the presence of personnel in the danger zone below the boom is output. The prompt words include a content description of the location encoding information and a description of the output result requirements.

[0029] In some optional embodiments, obtaining the comprehensive judgment result that the personnel are located in the danger zone below the crane boom based on the first probability and the second probability includes:

[0030] The first probability and the second probability are weighted and summed to obtain the comprehensive probability that a person is located in the danger zone below the crane boom. If the comprehensive probability is greater than the preset probability threshold, it is determined that there is a person in the danger zone below the crane boom; otherwise, it is considered that there is no person in the danger zone below the crane boom.

[0031] This invention also provides a personnel identification device under a crane boom, comprising:

[0032] The information segmentation module is used to input the original image of the crane construction site into the segmentation network model, segment the personnel and crane boom in the original image, and obtain personnel and crane boom mask data;

[0033] The depth estimation module is used to input the original image into the depth estimation network to obtain the depth data of the original image;

[0034] The information combination module is used to combine the personnel boom mask data and the depth data of the original image to obtain the personnel boom depth data;

[0035] The projection algorithm module determines the degree of overlap between the personnel and the boom in three directions: horizontal, vertical, and depth, based on the personnel boom depth data, and determines the first probability that the personnel are located in the danger zone below the boom based on the degree of overlap.

[0036] The LLM algorithm module is used to encode the personnel boom depth data to obtain position encoding information. The position encoding information and the pre-built large language model prompt words are input into the pre-built large language model to obtain the second probability that the personnel are located in the danger zone below the boom.

[0037] The comprehensive judgment module is used to obtain a comprehensive judgment result based on the first probability and the second probability that the personnel are located in the danger zone below the boom.

[0038] This invention also provides a personnel identification system under a crane boom, comprising: an image acquisition device and a recognition device;

[0039] Image acquisition equipment is used to acquire raw images of a designated area under the crane boom and send them to a recognition device.

[0040] The identification device is equipped with the aforementioned personnel identification device under the crane boom, which is used to determine whether there are personnel under the crane boom based on the original image of a designated area under the crane boom acquired by the image acquisition device.

[0041] This invention also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned method for identifying personnel under a crane boom.

[0042] This invention also provides an identification device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for identifying personnel under a crane boom.

[0043] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0044] The identification method provided in this invention segmentes useful information such as personnel and crane boom from the original image of a crane construction site using a segmentation network model, obtains depth data from the original image using a depth estimation network, and combines the analysis results of both to obtain depth data of personnel and crane boom. Based on the depth data of personnel and crane boom, the overlap between personnel and crane boom in three directions (horizontal, vertical, and depth) is determined, and a first probability is determined based on the overlap. Then, a large language model is used to obtain a second probability of personnel being located in the dangerous area below the crane boom based on the depth data of personnel and crane boom. The two probabilities are combined to comprehensively identify whether there are personnel in the dangerous area below the crane boom. Through the innovative integration of deep learning and depth estimation algorithms, intelligent identification and early warning of personnel intrusion into dangerous areas in hoisting operation scenarios are achieved, improving the accuracy of personnel identification under the crane boom. This method does not require complex equipment configuration, has low deployment cost and technical threshold, and features high algorithm stability, strong real-time judgment process, and high accuracy.

[0045] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart of the personnel identification method under the crane boom in an embodiment of the present invention;

[0049] Figure 2 This is a block diagram illustrating the implementation principle of the personnel identification method under the crane boom in this embodiment of the invention.

[0050] Figure 3 This is a schematic diagram of the personnel identification device under the crane boom in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the personnel identification system under the crane boom in an embodiment of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] To address the problem in existing technologies that cannot accurately determine whether a person is actually under the boom, the inventors of this application have conducted in-depth research into the reasons for this inaccurate determination. The main reason is that deep learning-based target detection algorithms detect the two-dimensional position information of the person and the boom in the image. However, in practical applications, the overlap of the two-dimensional positions of the person and the boom in the image does not necessarily mean that the person is under the boom; the person may be on the side of the boom. Therefore, if depth information can be combined for further judgment, a more accurate judgment result should be obtained.

[0054] While cameras with depth information (such as RGB-D cameras) can provide spatial awareness, their high cost, demanding deployment conditions, and lack of convenience make them unsuitable for complex construction scenarios and stringent safety requirements. Therefore, the applicability of such equipment in oil and gas production and construction scenarios is limited. Thus, the inventors of this application considered using depth estimation algorithms to recover three-dimensional information of a scene from two-dimensional images, such as the depth and distance of objects. In crane operation scenarios, depth estimation algorithms can be used to estimate the distance between personnel and the crane boom: by calculating the depth difference between personnel and the boom, it can be determined whether personnel are in a dangerous area and a timely warning signal can be issued.

[0055] This invention uses a monocular depth estimation algorithm based on RGB images. This method can estimate depth information from monocular RGB images, which retains the low cost and high convenience of RGB data, and makes up for the lack of spatial perception ability in traditional target detection algorithms. It provides a more efficient and practical technical means for judging people standing under the boom in oil and gas production and construction scenarios.

[0056] Depth estimation algorithms can extract depth information from a scene using monocular RGB images or depth sensor data, providing crucial 3D perception capabilities for understanding the spatial relationship between the crane boom and workers. However, these algorithms only output depth maps or point cloud data and cannot directly generate a logical judgment on whether someone is standing under the crane boom. To transform depth information into a concrete safety assessment, a dedicated post-processing algorithm needs to be designed. This algorithm combines depth data with target detection results, comprehensively analyzes the spatial relationship between the crane boom and workers, and ultimately obtains the accurate spatial positional relationship between them, leading to a precise judgment on whether someone is standing under the crane boom.

[0057] The present invention provides a method for identifying personnel under crane booms, which combines segmentation networks and depth estimation algorithms, and further combines projection algorithms and large language model algorithms. This method can accurately determine whether a person is standing under the boom, and can solve the following key technical problems in crane safety monitoring in oil and gas production and construction scenarios:

[0058] To address the problem that existing target detection algorithms based on RGB images lack spatial awareness and cannot accurately determine whether personnel are in the danger zone directly below the crane boom, a method is proposed that integrates monocular depth estimation and image segmentation algorithms to achieve accurate analysis of the three-dimensional spatial relationship between the crane boom and personnel.

[0059] To address the issues of high deployment costs and poor environmental adaptability of existing depth sensors (such as RGB-D cameras) in construction scenarios, a depth estimation solution based on monocular RGB images is proposed. This solution achieves spatial perception capabilities comparable to those of depth sensors while maintaining the advantages of low cost and high convenience.

[0060] To address the disconnect between the output of depth estimation algorithms and safety judgment logic, a dedicated spatial relationship analysis algorithm is designed. By establishing a three-dimensional safety space model of the crane boom projection area, depth information and image segmentation results are fused and analyzed at multiple levels to achieve a deterministic judgment of whether "a person is standing under the crane boom".

[0061] To address the issues of numerous visual disturbances and severe occlusion in complex construction environments, a safety judgment mechanism based on dynamic depth thresholds is constructed. By adaptively adjusting the depth difference tolerance, the robustness of the system's judgment under interference conditions such as occlusion and shadows is improved.

[0062] Through the above technological innovations, a safety monitoring system that can be deployed on conventional monitoring equipment, does not rely on dedicated depth sensors, and can accurately determine the intrusion of personnel under the boom is finally realized, effectively solving the problem of blind spot collision warning for cranes at oil and gas construction sites.

[0063] This invention provides a method for identifying personnel under a crane boom, the process of which is as follows: Figure 1 As shown, see the implementation principle block diagram. Figure 2 As shown, the method includes the following steps:

[0064] Step S101: Input the original image of the crane construction site into the segmentation network model, segment the personnel and crane boom in the original image, and obtain personnel and crane boom mask data.

[0065] Step S102: Input the original image into the depth estimation network to obtain the depth data of the original image.

[0066] Step S103: Combine the personnel boom mask data and the depth data of the original image to obtain the personnel boom depth data. The personnel boom depth data can be a personnel data mask image.

[0067] Step S104: Based on the personnel boom depth data, determine the degree of overlap between the personnel and the boom in the horizontal, vertical and depth directions, and determine the first probability that the personnel are located in the danger zone below the boom based on the degree of overlap.

[0068] Step S105: Perform position encoding on the personnel boom depth data to obtain position encoding information. Input the position encoding information and the pre-built large language model prompt words into the pre-built large language model to obtain the second probability that the personnel are located in the danger zone below the boom.

[0069] Step S106: Based on the first probability and the second probability, obtain a comprehensive judgment result that the personnel are located in the danger zone below the boom.

[0070] The method provided in this embodiment of the invention acquires RGB images from on-site video equipment and inputs them into a segmentation network model and a depth estimation model to obtain mask images of personnel and crane booms, as well as a depth image of the same size as the original image. By superimposing the two images, a depth mask image of personnel and crane booms is finally obtained, which is then input into the algorithm module. Based on the projection algorithm results and the LLM algorithm output results obtained respectively, joint weight allocation is performed to obtain the final judgment result of whether a person is standing under the crane boom.

[0071] The method provided in this invention segmentes useful information such as personnel and crane booms in an image using a segmentation network model, obtains depth information of the image using a depth estimation network, and combines the analysis results of both to obtain depth information of personnel and crane booms. Based on the depth information of personnel and crane booms, the overlap between personnel and crane booms in three directions—horizontal, vertical, and depth—is determined. A first probability is determined based on the overlap, indicating that the personnel are located in the danger zone below the crane boom. A second probability is then obtained using a large language model. The two probabilities are combined to comprehensively identify whether personnel are present in the danger zone below the crane boom. Through the innovative fusion of deep learning and depth estimation algorithms, intelligent identification and early warning of personnel intrusion into danger zones during lifting operations are achieved, improving the accuracy of personnel identification under crane booms. This method does not require complex equipment configuration, has low deployment costs and technical barriers, and features high algorithm stability, strong real-time judgment process, and high accuracy.

[0072] Step S101 above identifies personnel and the crane boom based on the original image of the crane construction site and obtains mask information. The acquisition process may include:

[0073] 1) Obtain video data captured by video equipment at the crane construction site, and extract frames from the video data at a preset frequency to obtain the original image.

[0074] See Figure 2 In S1, the video capture device can be a fixed or mobile camera deployed at the construction site. The video capture device captures video data of the construction site, and frames are extracted from the video data at a preset frequency to obtain image data. Alternatively, the video capture device can capture images at a set frequency. The acquired images are used as input data for subsequent modeling.

[0075] 2) Input the extracted original image into the segmentation network model to segment the people and crane in the image.

[0076] The segmentation network model can be trained using sample data. Specifically, it uses historical image data from a crane construction site as samples. This historical image data includes historical images and corresponding label data, which includes labeled personnel and / or the crane boom in the images. The historical image data from the mobile phone can be manually or automatically labeled to identify personnel and the crane boom; different labels can be used for personnel and the crane boom.

[0077] 3) Create a mask image with the same size as the extracted original image. Based on the segmented personnel and crane, add corresponding personnel and crane identifiers to the corresponding pixels in the mask image to obtain the personnel and crane mask image.

[0078] See Figure 2 S2 in the diagram refers to the segmentation and recognition of personnel and crane boom, as well as the generation of mask information, achieved through the segmentation network branches. The segmentation network model can be a segmentation network U-net. The original image is input into the segmentation network U-net, and the high-performance object segmentation capability of the U-net network is used to segment the main content of interest, namely personnel and crane boom.

[0079] Based on the original image size, create a mask image of the same size. According to the positions of the segmented personnel and crane boom in the original image, add personnel and crane boom identifiers to the corresponding positions in the created mask image as useful information markers. Add useless information markers to other positions, thus obtaining a personnel and crane boom mask image. For example, add identifier 1 to the personnel position, identifier 2 to the crane boom position, and identifier 0 to other positions. The specific identifiers can be set as needed and are not limited to these numbers.

[0080] In step S102 above, inputting the original image into the depth estimation network to obtain the depth data of the original image includes: inputting the original image into the depth estimation network, determining the depth value representing the distance between each pixel and the video device based on the distance between the target object corresponding to each pixel in the original image and the video device, and obtaining the original image depth map including the depth value corresponding to each pixel.

[0081] See Figure 2 The depth estimation network branch of S3 inputs the acquired original image into the DepthAnything network to obtain the depth information of the entire image. For example, a depth image of the same size can be created based on the original image. The distance between the content of each pixel in the original image and the video device can be quantized. For instance, a value between 0 and 1 can be used to represent distance information, with the closer to the video device as a reference, the smaller the value. The minimum distance corresponds to 0, and the maximum distance corresponds to 1.

[0082] The monocular depth estimation algorithm based on RGB images is adopted. It estimates depth information from a single RGB image through a deep learning model, which not only reduces hardware costs but also retains the ability to accurately judge the spatial relationship between people and cranes in complex scenes.

[0083] In step S103 above, the personnel boom mask data and the depth data of the original image are combined to obtain personnel boom depth data, including: setting the corresponding pixel of the personnel boom in the personnel boom mask image to 1 and setting other pixels to 0 to obtain the processed personnel boom mask image; multiplying the processed personnel boom mask image with the corresponding pixel values ​​in the depth image of the original image to obtain the personnel boom depth image.

[0084] This step involves combining the results; see [link / reference]. Figure 2 S4 in the process: Combine the personnel and crane mask data obtained by the segmentation network branch with the depth data of the entire image obtained by the depth estimation network branch. That is, after setting the useful information in the personnel and crane mask map to 1 and the useless information to 0, multiply the personnel and crane mask map with the depth data of the entire image (depth estimation map), and finally leave only the depth data of the person and the crane to obtain the personnel and crane depth map.

[0085] In step S104 above, based on the personnel boom depth data, the overlap between the personnel and the boom in the horizontal, vertical, and depth directions is determined. Based on the overlap, the first probability that the personnel are located in the danger zone below the boom is determined, including:

[0086] 1) Based on the personnel boom depth map, determine the boom area and personnel area in the image.

[0087] In the personnel and boom depth map, the maximum bounding box of the segmented personnel area and boom area is determined. For example, the coordinates of the maximum bounding box of the boom segment are: The coordinates of the maximum bounding box of the personnel segment are X1 and x2 represent the horizontal coordinate range of the boom, i.e., the left and right boundaries; y1 and y2 represent the vertical coordinate range of the boom, i.e., the upper and lower boundaries; px1 and px2 represent the horizontal coordinate range of the personnel, i.e., the left and right boundaries; py1 and py2 represent the vertical coordinate range of the boom, i.e., the personnel boundaries.

[0088] 2) Determine whether the personnel are located below the boom based on the upper and lower boundaries of the vertical projection of the boom area and the personnel area.

[0089] Perform vertical overlap detection on the boom area and personnel area, determining whether the bounding box of the personnel overlaps with the vertical projection area of ​​the boom. It is necessary to determine whether the top of the personnel is below the bottom of the boom according to preset criteria. For example, the preset criteria are:

[0090] .

[0091] 3) Determine whether personnel are located within the area of ​​the boom area based on the left and right boundaries of the vertical projection of the boom area and the personnel area;

[0092] The system performs a horizontal overlap check on the boom area and personnel area to determine if personnel are within the horizontal coverage area of ​​the boom. This requires determining if the left and right boundaries of the personnel area intersect with the boom area based on preset criteria. For example, the preset criteria might be:

[0093] .

[0094] 4) If a person is located below the boom and within the boom area, based on the person's boom depth data, determine the relationship between the depth difference between the average depth of the person's area and the average depth of the boom area and the set difference threshold range, and determine the first probability that the person is located in the danger zone below the boom.

[0095] If personnel are within the danger zone below the boom, further depth verification is required to ensure that the personnel and the boom are at similar depth planes, eliminating visual "false overlap." For example, the average depth of the boom area can be extracted. and the average depth of the personnel area Determine whether the depth difference between the two is within a preset difference threshold. Within the range of ), that is, satisfying the following formula:

[0096] .

[0097] Based on the relationship between the depth difference and the preset difference threshold, and the magnitude of the difference between the depth difference and the difference threshold, the first probability that a person is located in the danger zone below the crane boom can be obtained. The first probability can be a value from 0 to 1. 1 indicates that the person is 100% under the crane boom, or that the person is definitely under the crane boom, and 0 indicates that the person is definitely under the crane boom.

[0098] Optionally, this step can output the probability that a person is located in the danger zone below the boom, or it can output the judgment result of whether a person is below the boom, i.e., yes or no. In step S106, this will be quantized as 1 and then quantized as 0.

[0099] See Figure 2 The projection algorithm branch in the S5 algorithm module mainly relies on the judgment of the three-dimensional information of the boom and personnel. It mainly includes the processing of information in three aspects: vertical, horizontal and depth. It judges whether there is overlap in the vertical, horizontal and depth directions, so as to accurately judge the positional relationship between the personnel and the boom and ensure the correctness of the judgment logic.

[0100] Step S105 above performs position encoding on the personnel boom depth data to obtain position encoding information. This position encoding information, along with pre-built large language model prompts, is then input into the pre-built large language model to obtain the second probability that the personnel are located in the danger zone below the boom. See also... Figure 2 S6, the Large Language Model (LLM) algorithm branch in the algorithm module, leverages the powerful understanding capabilities of the large language model to perform logical judgments and obtain the second probability. The process includes:

[0101] 1) Adjust the image size of the personnel crane depth map according to the number of lexical units in the large language model.

[0102] Large language models have a limited number of tokens, and when inputting images into the model, each pixel's data needs to be encoded. The encoded pixel data needs to carry its own information and positional information; that is, the encoded information includes the pixel's positional information (e.g., coordinates), depth data, and identifier information. If the image is input at the original size of the personnel and crane depth map, the model has too many tokens to process. Therefore, the image needs to be resized, generally by reducing the image size to fit the token requirement of the large language model. Since the core problem is to obtain the positional relationship between the personnel and the crane, resizing the image does not affect the positional relationship; therefore, resizing the image is an operation that solves the above problem. The size of the image resizing is determined based on the token requirement of the large language model. For example, considering both image size and token count, the image can be uniformly converted to a 10*10 pixel size.

[0103] 2) Perform position encoding (transformer) on the adjusted personnel boom depth map to obtain position encoding information in text format. The position encoding information includes the position coordinates of each pixel in the image, the corresponding depth information, and personnel boom identification information.

[0104] This step involves model encoding of the personnel and crane arm mask image, and position encoding of the resized personnel and crane arm mask image. For ease of model understanding, the position encoding is set as follows:

[0105]

[0106] Where i represents a pixel in the image, x is the position of pixel i along the x-axis (horizontal direction), and y is the position of pixel i along the y-axis (vertical direction). This represents the depth information corresponding to a pixel. The personnel and boom identification information can be set to 0 and 1 to distinguish between personnel and boom, where 0 represents personnel and 1 represents boom.

[0107] To save on token count and improve model speed, optionally, only the data of the pixels containing information, such as personnel and crane boom, can be entered into the model, while other useless information does not need to be entered.

[0108] 3) Input the pre-designed large language model prompts and text-formatted location encoding information into the large language model, and output the second probability that there are personnel in the danger zone below the crane boom. The prompts include a description of the location encoding information and a description of the required output results.

[0109] In this step, a prompt can be designed first. The location-encoded data in text format of the above image and the prompt are input into the large language model. The large language model outputs the second probability that the person is in the danger zone under the crane boom, that is, the probability that a person is standing under the crane boom.

[0110] The prompts are designed based on the needs of the large language model. For example, they can include descriptions of the content in the location information and the required output. That is, what information is included in the input location information, and what kind of output is required, etc. The second probability can also be a value from 0 to 1. 1 means that the person is 100% under the crane boom or that the person is definitely under the crane boom, and 0 means that the person is definitely under the crane boom.

[0111] In step S106 above, a comprehensive judgment result is obtained based on the first probability and the second probability, including: weighting and summing the first probability and the second probability to obtain the comprehensive probability that a person is located in the dangerous area below the crane boom; if the comprehensive probability is greater than a preset probability threshold, it is determined that there is a person in the dangerous area below the crane boom; otherwise, it is considered that there is no person in the dangerous area below the crane boom.

[0112] See Figure 2 In step S7, the joint weight allocation yields the judgment result. Here, the results of the projection algorithm and the LLM algorithm are combined to perform a comprehensive weight allocation, thus realizing the final judgment result. For example, the final judgment result can be represented as follows:

[0113] .

[0114] Here, α and β are the weight values ​​of the projection algorithm and the large language model algorithm, respectively, which can be set as needed, for example, coefficients of 0.3 and 0.7 respectively. and These are the results obtained from the projection algorithm and the large language model algorithm, respectively, namely the first probability and the second probability.

[0115] By employing parallel projection algorithm branches and LLM algorithm branches, and combining spatial features of depth maps with logical reasoning, accurate identification of human behavior under the boom is achieved, solving the misjudgment problem caused by the lack of depth perception in traditional methods.

[0116] Based on the same inventive concept, embodiments of the present invention also provide a personnel identification device under a crane boom, such as... Figure 3 As shown, it includes:

[0117] The information segmentation module 11 is used to input the original image of the crane construction site into the segmentation network model, segment the personnel and crane boom in the original image, and obtain personnel and crane boom mask data.

[0118] The depth estimation module 12 is used to input the original image into the depth estimation network to obtain the depth data of the original image.

[0119] The information combination module 13 is used to combine the personnel boom mask data and the depth data of the original image to obtain the personnel boom depth data.

[0120] The projection algorithm module 14 is used to determine the degree of overlap between the personnel and the boom in the horizontal, vertical and depth directions based on the personnel boom depth data, and to determine the first probability that the personnel are located in the danger zone below the boom based on the degree of overlap.

[0121] LLM algorithm module 15 is used to encode the personnel boom depth data to obtain position encoding information. The position encoding information and the pre-built large language model prompt words are input into the pre-built large language model to obtain the second probability that the personnel are located in the danger zone below the boom.

[0122] The comprehensive judgment module 16 is used to obtain a comprehensive judgment result of the personnel being located in the dangerous area below the boom based on the first probability and the second probability.

[0123] This invention also provides a personnel identification system under a crane boom, such as... Figure 4 As shown, it includes: an image acquisition device 10 and a recognition device 20.

[0124] Image acquisition device 10 is used to acquire raw images of a designated area under the crane boom and send them to the recognition device;

[0125] The identification device 20 is equipped with the aforementioned personnel identification device under the crane boom, which is used to determine whether there are personnel under the crane boom based on the original image of the designated area under the crane boom acquired by the image acquisition device.

[0126] This invention also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned method for identifying personnel under a crane boom.

[0127] This invention also provides an identification device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for identifying personnel under a crane boom.

[0128] Regarding the apparatus and system in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0129] The methods, apparatus, and systems provided in the embodiments of the present invention utilize deep learning technology to achieve end-to-end judgment and early warning of the presence of personnel under the crane boom. This can solve various safety problems existing in the use of cranes during oil and gas production, improve the safety level of oil and gas production sites, reduce the probability of dangerous behaviors, and ensure the safety of construction personnel.

[0130] Specifically, this invention designs an end-to-end algorithm framework and a composite algorithm. It requires only a single on-site image to perform real-time analysis of hoisting operations at oil and gas production sites, determining whether personnel are present under the crane boom. This method extracts image features using a deep learning model and combines it with a depth estimation algorithm to accurately identify the location of personnel and their relative distance to the crane boom. Then, through two designed algorithms and information such as the model's depth map, it achieves a logical judgment on whether "a person is standing under the crane boom," thereby enabling precise early warning of dangerous behavior. Compared with traditional target detection methods, this invention has the following significant advantages:

[0131] 1) Simple deployment: Only a monocular camera needs to be deployed in a suitable location on the construction site. No complicated equipment configuration is required, which reduces deployment costs and technical barriers.

[0132] 2) High stability: Through end-to-end training of deep learning models and proprietary algorithms, it can effectively cope with complex environmental interferences at oil and gas production sites, such as changes in light and obstructions, thereby improving the robustness of recognition.

[0133] 3) Strong real-time performance: The algorithm is lightweight and can achieve real-time processing while ensuring high accuracy, meeting the needs of construction sites for rapid response.

[0134] 4) Wide range of applications: This method is not only applicable to oil and gas production scenarios, but can also be extended to other fields involving crane operations such as construction and logistics, and has broad application value.

[0135] This invention significantly improves the safety monitoring capabilities of crane operations in oil and gas production and construction scenarios by integrating monocular depth estimation and image segmentation techniques with a dedicated spatial relationship analysis algorithm. Compared with existing methods for identifying personnel under crane booms in safety monitoring, it has the following advantages and positive effects:

[0136] 1) It has high-precision three-dimensional spatial perception capabilities, which can greatly reduce the misjudgment rate.

[0137] Traditional RGB target detection only provides two-dimensional position information and cannot accurately determine whether a person is actually in the danger zone under the crane boom, resulting in a false alarm rate as high as 95%. This invention uses monocular depth estimation combined with three-dimensional modeling of the crane boom's projection area to accurately calculate the spatial distance between the person and the crane boom, reducing the false alarm rate to below 30% and significantly improving the reliability of early warning.

[0138] 2) It has the advantages of low cost and high adaptability, and is suitable for complex construction environments.

[0139] Existing explosion-proof RGB-D depth sensors are expensive (typically exceeding 50,000 yuan per unit) and sensitive to lighting, dust, and other environmental factors, making deployment conditions demanding. This invention requires only a standard surveillance camera (costing around 10,000 yuan) and, through an optimized monocular depth estimation algorithm, maintains over 80% depth estimation accuracy even under harsh conditions such as low light and dust interference, significantly reducing system deployment costs.

[0140] 3) It can realize real-time dynamic security judgment, which greatly improves the response speed.

[0141] Traditional manual monitoring or simple sensor alarms suffer from latency, with an average time of 10-15 seconds from the occurrence of danger to the warning. This invention employs a lightweight depth estimation model (inference speed ≥25 FPS) combined with a dynamic depth threshold adjustment mechanism, which can complete the detection of personnel intrusion under the crane boom within 200ms, meeting the requirements of real-time safety monitoring.

[0142] 4) It is highly robust and adaptable to complex occlusion and interference scenarios.

[0143] Traditional methods suffer from a surge in false positive rates (>40%) under conditions such as crane arm occlusion and shadow interference. This invention, through multi-scale depth feature fusion and an adaptive depth difference tolerance mechanism, maintains a detection accuracy of over 80% even with a 50% occlusion rate, significantly improving stability in complex environments.

[0144] 5) It enables automated safety decision-making, reducing reliance on manual operation.

[0145] Traditional monitoring relies on manual observation, which is prone to missed detections due to fatigue or blind spots (the human missed detection rate is about 15%~20%). This invention achieves fully automatic hazard assessment through an automated "crane-person" spatial relationship analysis algorithm, reducing the missed detection rate to <2%, greatly reducing the safety risks caused by human factors.

[0146] This invention achieves accurate detection of personnel intrusion into dangerous areas during crane operations, while maintaining low cost, high precision, and strong robustness. Compared with existing technologies, the false alarm rate is reduced by more than 50%, the response speed is increased by 3 times, and the reliance on expensive depth sensors is significantly reduced. It is suitable for construction environments with high safety requirements, such as oil and gas production.

[0147] The human behavior recognition scenario addressed by this invention can be widely applied to high-risk work areas such as oil and gas construction sites, drilling platforms, and tank farms. Furthermore, the algorithm structure and core technology have strong scalability, and can be applied in the future to other high-risk work scenarios involving hoisting operations, such as building construction, equipment maintenance, and warehousing and logistics.

[0148] Unless otherwise specifically stated, terms such as processing, calculation, operation, determination, display, etc., may refer to the actions and / or processes of one or more processing or computing systems or similar devices that represent the manipulation and conversion of data representing physical (e.g., electronic) quantities within the registers or memory of the processing system into other data similarly representing physical quantities within the memory, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0149] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0150] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0151] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0152] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0153] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0154] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. A method of identifying a person under a crane jib, characterized by, The method comprises the following steps: inputting the original image of the crane construction site into a segmentation network model to segment out the personnel and the boom in the original image and obtain personnel-boom mask data; inputting the original image into a depth estimation network to obtain depth data of the original image; combining the personnel-boom mask data and the depth data of the original image to obtain personnel-boom depth data; judging the overlap degree of the personnel and the boom in horizontal, vertical and depth directions according to the personnel-boom depth data, and determining a first probability of the personnel being in a dangerous area under the boom according to the overlap degree, comprising: determining a boom area and a personnel area in the image according to the personnel-boom depth map; judging whether the personnel is under the boom according to the upper and lower boundaries of the vertical projection of the boom area and the personnel area; determining whether the personnel is within the range of the boom area according to the left and right boundaries of the vertical projection of the boom area and the personnel area; if the personnel is under the boom and within the range of the boom area, determining the first probability of the personnel being in the dangerous area under the boom according to the relationship between the depth difference between the depth average of the personnel area and the depth average of the boom area and a set difference threshold range. position encoding is performed on the personnel-boom depth data to obtain position encoding information, the position encoding information and a pre-constructed large language model prompt word are input into the pre-constructed large language model to obtain a second probability of the personnel being in the dangerous area under the boom, and the position encoding information includes the position coordinates, the corresponding depth information and the personnel-boom identification information of each pixel point in the image. According to the first probability and the second probability, a comprehensive judgment result of the personnel being in the dangerous area under the boom is obtained.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining video data captured by a video device at a crane construction site, and extracting original images from the video data at a preset frequency; inputting the extracted original images into a segmentation network model to segment out the personnel and the boom in the images; the segmentation network model is trained using historical image data of the crane construction site as samples, the historical image data includes historical images and corresponding label data, and the label data includes personnel and / or booms marked in the images; a mask image with the same size as the extracted original images is established, personnel marks and boom identifiers are added to the pixel points at the corresponding positions in the mask image according to the segmented personnel and booms, and a personnel-boom mask image is obtained.

3. The method of claim 1, wherein, The method comprises the following steps: inputting the original image into a depth estimation network, determining a depth value representing the distance between each pixel point and the video device according to the distance between the target object corresponding to each pixel point and the video device in the original image, and obtaining an original image depth map including the depth values corresponding to each pixel point.

4. The method of claim 1, wherein, The method comprises the following steps: The corresponding pixel points of the personnel boom in the personnel boom mask image are set to 1, and other pixel points are set to 0, to obtain a processed personnel boom mask image. The processed personnel boom mask image and the values of the corresponding pixel points in the original image depth image are multiplied to obtain a personnel boom depth image.

5. The method of claim 1, wherein, The personnel boom depth data is positionally encoded to obtain positionally encoded information. The positionally encoded information and a pre-constructed large language model prompt word are input into the pre-constructed large language model to obtain a second probability that the personnel is in the dangerous area under the boom, including: According to the number of word units of the large language model, the image size of the personnel boom depth image is adjusted; The personnel boom depth image after adjustment is positionally encoded to obtain positionally encoded information in a text format; The pre-designed large language model prompt word and the positionally encoded information in the text format are input into the large language model to output a second probability that the personnel is in the dangerous area under the boom. The prompt word includes a content description of the positionally encoded information and an output result requirement description.

6. The method of claim 1, wherein, The first probability and the second probability are used to obtain a comprehensive judgment result that the personnel is in the dangerous area under the boom, including: The first probability and the second probability are weighted and summed to obtain a comprehensive probability that the personnel is in the dangerous area under the boom. If the comprehensive probability is greater than a preset probability threshold, it is determined that the personnel is in the dangerous area under the boom. Otherwise, it is considered that the personnel is not in the dangerous area under the boom.

7. A personnel recognition device under a crane jib, characterized by including: An information segmentation module is configured to input an original image of a crane construction site into a segmentation network model to segment personnel and a boom in the original image to obtain personnel boom mask data. A depth estimation module is configured to input the original image into a depth estimation network to obtain depth data of the original image. An information combination module is configured to combine the personnel boom mask data and the depth data of the original image to obtain personnel boom depth data. A projection algorithm module is configured to determine a first probability that the personnel is in a dangerous area under the boom according to the personnel boom depth data, the overlap of the personnel and the boom in horizontal, vertical and depth directions, including: determining a boom area and a personnel area in the image according to the personnel boom depth image; determining whether the personnel is under the boom according to upper and lower boundaries of vertical projections of the boom area and the personnel area; determining whether the personnel is in the range of the boom area according to left and right boundaries of the vertical projections of the boom area and the personnel area; if the personnel is under the boom and in the range of the boom area, determining the first probability that the personnel is in the dangerous area under the boom according to a relationship between a depth difference between a depth mean value of the personnel area and a depth mean value of the boom area and a set difference threshold range according to the personnel boom depth data. An LLM algorithm module is configured to positionally encode the personnel boom depth data to obtain positionally encoded information. The positionally encoded information and a pre-constructed large language model prompt word are input into the pre-constructed large language model to obtain a second probability that the personnel is in the dangerous area under the boom. The positionally encoded information includes position coordinates, corresponding depth information and personnel boom identification information of each pixel point in the image. The comprehensive judgment module is configured to obtain a comprehensive judgment result of whether the personnel is located in the dangerous area under the boom according to the first probability and the second probability.

8. A personnel identification system under a crane jib, characterized by The method comprises the following steps: An image acquisition device and an identification device are provided. The image acquisition device is configured to acquire an original image of a designated area under the crane boom and send the original image to the identification device. The identification device is provided with the personnel identification device under the crane boom according to claim 7, and is configured to determine whether there is personnel under the crane boom based on the original image of the designated area under the crane boom acquired by the image acquisition device.

9. A computer storage medium, characterized in that The computer storage medium stores computer executable instructions, and the computer executable instructions are executed by the processor to implement the personnel identification method under the crane boom according to any one of claims 1-6.

10. An identification device, characterized in that The method comprises the following steps: A memory, a processor and a computer program stored in the memory and executable on the processor are provided, and the processor executes the program to implement the personnel identification method under the crane boom according to any one of claims 1-6.

Images