Crane outrigger dynamic electronic fence construction method and device and electronic equipment

Abstract

The application provides a crane outrigger dynamic electronic fence construction method and device and electronic equipment, and relates to the technical field of crane safety monitoring. The method comprises the following steps: acquiring historical key point information in the continuous previous N frames of history images of a target crane outrigger and a current frame image; performing key point extraction on the current frame image to obtain real-time key point information of a plurality of key points of the current frame image; if the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger, calculating the missing information of the key points of the current frame image according to the historical key point information by using a Kalman algorithm, and obtaining effective point information of the preset position points by comprehensively combining the missing information of the key points and the real-time key point information; and calculating a minimum convex hull region by using a convex hull algorithm according to the effective point information, and taking the minimum convex hull region as an electronic fence. The application can accurately construct an electronic fence when the key points of the crane outrigger are blocked.

Description

Technical Field

[0001] This invention relates to the field of crane safety monitoring technology, and in particular to a method, device and electronic equipment for constructing a dynamic electronic fence for crane outriggers. Background Technology

[0002] With the continuous development of construction, large-scale hoisting operations, and other fields, cranes are being used more and more widely in various construction scenarios. To ensure the safety of personnel and vehicles around the crane during construction and hoisting processes, it is necessary to conduct safety monitoring of the area formed by the crane's outriggers.

[0003] In existing technologies, although fixed-area monitoring methods are used, when monitoring the area formed by the crane outriggers, the crane outriggers are often affected by mud and water covering, obstructions and other interference in complex working environments. This results in incomplete detection of key points of the crane outriggers, affecting the determination of the safety monitoring area and thus affecting the safety monitoring of the crane. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for constructing a dynamic electronic fence for crane outriggers, in order to solve the problem that when crane outriggers are obstructed during crane safety monitoring, the detection of key points of the crane outriggers is incomplete, affecting the determination of the safety monitoring area.

[0005] In a first aspect, embodiments of the present invention provide a method for constructing a dynamic electronic fence for crane outriggers, comprising: Obtain historical key point information and the current frame image from the previous N consecutive historical images of the target crane's outriggers; The key points of the current frame image are extracted to obtain real-time key point information of several key points in the current frame image; among them, the key points are the preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points. If the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger, the Kalman algorithm is used to calculate the missing key point information of the current frame image based on the historical key point information, and the effective point information of the preset position point is obtained by combining the missing key point information and the real-time key point information of several key points of the current frame image. The minimum convex hull region is calculated using the convex hull algorithm based on the valid point information of the preset location points, and the minimum convex hull region is used as the electronic fence.

[0006] In one possible implementation, key points are extracted from the current frame image to obtain real-time key point information of several key points in the current frame image, including: The current frame image is input into the HRNet-W48 model to obtain the feature map; The feature map is input into the CenterNet model to obtain real-time key point information of several key points of the leg; Before using the Kalman algorithm to calculate the missing keypoint information of the current frame image based on historical keypoint information, the following steps are also included: Compare and analyze real-time key point information with historical key point information; If a key point corresponding to a certain label in the previous N consecutive historical images does not exist in the current frame image, then a target key point with missing key point information is identified.

[0007] In one possible implementation, the Kalman algorithm is used to calculate the missing keypoint information of the current frame image based on historical keypoint information, including: Based on the label of the target key point, the coordinate position of the key point corresponding to the label in different frames is obtained from the historical key point information, and an initial state vector is generated based on the timestamp of each frame image. Using the Kalman metric state transition matrix and initial state vector, the state vector of the target key point is calculated, and the missing key point information of the target key point is determined based on the state vector of the target key point. By combining missing keypoint information and real-time keypoint information of several keypoints in the current frame image, valid point information of preset location points is obtained, including: The real-time key point information and key point missing information of the current frame image are merged to obtain the valid point information of the preset position point.

[0008] In one possible implementation, based on the target keypoint's label, the coordinates of the keypoint corresponding to that label in different frames are obtained from historical keypoint information, and an initial state vector is generated based on the timestamps of each frame image, including: Calculate the average velocity in the X-axis direction and the average velocity in the Y-axis direction of the key point corresponding to the label based on the coordinate position; An initial state vector is generated based on the coordinates of the key point corresponding to the label in the last frame image, the average velocity in the X-axis direction, and the average velocity in the Y-axis direction.

[0009] In one possible implementation, the minimum convex hull region is calculated using a convex hull algorithm based on the valid point information of preset position points, including: The reference point is determined based on all valid point information; the reference point is the valid point with the smallest ordinate. If the ordinates are the same, the valid point with the smallest abscissa is selected as the reference point. Construct a straight line from all valid points and the reference point, and calculate the polar angle between the line and the horizontal axis; Sort all valid points in ascending order of polar angle; if the polar angles are the same, retain the valid points that are closer to the reference point. Starting from the baseline, all valid points are traversed sequentially according to the sorting results, and valid points are filtered according to preset conditions; Connect the selected valid points in order to obtain the minimum convex hull region.

[0010] In one possible implementation, valid points are filtered based on preset conditions, including: Construct a vector from the current valid point and the first valid point after the current valid point, construct a vector from the first valid point after the current valid point and the second valid point after the current valid point, and calculate the cross product of the two vectors. When the cross product is positive, the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point are considered to form a counterclockwise left turn, and the current valid point is retained. If the cross product is negative or zero, determine whether the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point form a right turn or are collinear. Delete the first valid point after the current valid point and backtrack to the current valid point; traverse all valid points.

[0011] In one possible implementation, after using the minimum convex hull region as the electronic fence, the following is also included: Calculate the movement distance of each key point based on the key point information corresponding to the last frame of the image in the real-time key point information and historical key point information. When at least one moving distance exceeds the update threshold, the minimum convex hull region is recalculated based on the real-time key point information of the current frame image and the convex hull algorithm, and then updated. The updated minimum convex hull region is used as the updated electronic fence.

[0012] In one possible implementation, if the number of key points corresponding to the real-time key point information of the current frame image is equal to the number of preset position points of the target crane outrigger, the real-time key point information of several key points of the current frame image is used as the valid point information of the preset position points.

[0013] Secondly, embodiments of the present invention provide a dynamic electronic fence construction device for crane outriggers, comprising: The data acquisition module is used to acquire historical key point information and the current frame image from the previous N consecutive historical images of the target crane outrigger. The information extraction module is used to extract key points from the current frame image to obtain real-time key point information of several key points in the current frame image; wherein, the key points are preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points; The effective point information determination module is used to calculate the missing key point information of the current frame image based on historical key point information when the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger. The module then uses the Kalman algorithm to calculate the missing key point information of the current frame image based on historical key point information and combines the missing key point information with the real-time key point information of several key points of the current frame image to obtain the effective point information of the preset position points. The fence generation module is used to calculate the minimum convex hull region based on the valid point information of preset location points using the convex hull algorithm, and then use the minimum convex hull region as the electronic fence.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0015] In this embodiment of the invention, by acquiring historical key point information from the previous N consecutive historical images of the target crane outrigger and the current frame image, and extracting key points from the current frame image, the state information of the crane outrigger can be obtained in real time. If the number of key points corresponding to the real-time key point information of the current frame image is less than the preset number of position points of the target crane outrigger, the Kalman algorithm is used to predict the missing key point information. Thus, even in complex operating environments such as those with obstructions, the key points of the outrigger can still be accurately identified, solving the safety hazard problem caused by incomplete key point detection in traditional methods and enhancing the robustness of the system.

[0016] Furthermore, this invention integrates key point missing information and real-time key point information of several key points in the current frame image to obtain effective point information of preset position points. Based on the effective point information of preset position points, the minimum convex hull region is calculated using the convex hull algorithm to construct an electronic fence that can cover the actual deployed area of ​​the crane outriggers. Compared with traditional fixed fences or ground markings, it avoids the problems of area failure or insufficient protection. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a scenario for the crane outrigger dynamic electronic fence construction method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of the dynamic electronic fence construction method for crane outriggers provided in this embodiment of the invention. Figure 3This is a structural schematic diagram of the dynamic electronic fence construction device for crane outriggers provided in an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] Figure 1 This is a schematic diagram of a scenario illustrating the dynamic electronic fence construction method for crane outriggers provided in this embodiment of the invention. This application requires testing... Figure 1 The red key points are numbered 1, 2, 3, and 4. Key point 3 is obscured by a crane. This invention aims to complete key point 3, and then construct an electronic fence based on key points 1, 2, 3, and 4. Figure 1 The area marked by the green lines indicates when a person or vehicle intrudes into this area. Figure 1 The red dot outside the green box (representing a person) triggers an alarm to protect people or vehicles.

[0020] See Figure 2 The document illustrates a flowchart of the implementation of the dynamic electronic fence construction method for crane outriggers provided in an embodiment of the present invention, which is described in detail below: Step 201: Obtain historical key point information and the current frame image from the previous N consecutive historical images of the target crane outrigger.

[0021] In this embodiment, the consecutive previous N frames of historical images refer to the N frames of images that the system has acquired and processed before the current moment. These images contain the status information of the crane outriggers at different points in time. Historical key point information refers to the key point information extracted from these historical images, including the key point labels and coordinates. This information reflects the past state of the crane outriggers. The current frame image refers to the image that the system is currently processing, i.e., the latest image data. The current frame image is the basis for the system's real-time monitoring.

[0022] Step 202: Extract key points from the current frame image to obtain real-time key point information of several key points in the current frame image; wherein, the key points are preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points.

[0023] The historical key point information in the previous N consecutive historical images and the real-time key point information in the current frame image contain the same data type, including labels, coordinates, and timestamps. At the current moment, the historical key point information in the previous N consecutive historical images is historical information. At the next moment, the real-time key point information of the current frame image will be added to the historical key point information, i.e., the historical information. That is to say, as time goes by, new key point information will be added to the historical information, and the historical key point information of the first frame will be deleted, so that the historical key point information retains the historical key point information in the previous N consecutive historical images.

[0024] In this embodiment, image processing and computer vision techniques, such as deep learning models, are used to identify and extract key points of the crane outriggers from the current frame image. These key points are predefined, important location points used to describe the outrigger state, such as the outrigger endpoints and joints. The extracted key point information includes the key point's label and coordinates. The number of extracted key points may be less than or equal to the preset number of location points because occlusion may occur in the actual scene, causing some key points to be undetectable. By extracting key points from the current frame image, the system can obtain the current position and state of the crane outriggers in real time. These key points are important feature points of the outrigger structure and can reflect information such as the outrigger's deployment and positional changes.

[0025] Step 203: If the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger, the Kalman algorithm is used to calculate the key point missing information of the current frame image based on the historical key point information, and the effective point information of the preset position points is obtained by combining the key point missing information and the real-time key point information of several key points of the current frame image.

[0026] Specifically, in real-world scenarios, due to occlusion, the number of keypoints extracted from the current frame image may be less than the number of keypoints at preset locations. When the number of keypoints is insufficient, the system uses the Kalman algorithm to predict the location of missing keypoints in the current frame image based on historical keypoint information. The missing keypoint information predicted by the Kalman algorithm is then combined with the actual keypoint information extracted from the current frame image to obtain complete and valid keypoint information at the preset locations.

[0027] In this embodiment, in complex working environments, key points of the crane outriggers may be obscured or undetectable due to other interference. The Kalman algorithm is used to predict the locations of missing key points, thereby compensating for these missing points. This compensation mechanism ensures that even with partial data loss, the system can still fully acquire the key point information of the outriggers.

[0028] Step 204: Calculate the minimum convex hull region using the convex hull algorithm based on the valid point information of the preset location points, and use the minimum convex hull region as the electronic fence.

[0029] In this embodiment, the smallest convex polygon region calculated using the convex hull algorithm is defined as the electronic fence. This region contains all key points and represents the smallest possible boundary of the convex polygon. The calculated smallest convex hull region is used to monitor and protect the working area of ​​the crane outriggers in real time. The smallest convex hull region calculated using the convex hull algorithm accurately defines the working boundary of the outriggers, ensuring that the electronic fence covers all key points without being too large or too small.

[0030] In this embodiment of the invention, by acquiring historical key point information from the previous N consecutive historical images of the target crane outrigger and the current frame image, and extracting key points from the current frame image, the state information of the crane outrigger can be obtained in real time. If the number of key points corresponding to the real-time key point information of the current frame image is less than the preset number of position points of the target crane outrigger, the Kalman algorithm is used to predict the missing key point information. Thus, even in complex operating environments such as those with obstructions, the key points of the outrigger can still be accurately identified, solving the safety hazard problem caused by incomplete key point detection in traditional methods and enhancing the robustness of the system.

[0031] Furthermore, this invention integrates key point missing information and real-time key point information of several key points in the current frame image to obtain effective point information of preset position points. Based on the effective point information of preset position points, the minimum convex hull region is calculated using the convex hull algorithm to construct an electronic fence that can cover the actual deployed area of ​​the crane outriggers. Compared with traditional fixed fences or ground markings, it avoids the problems of area failure or insufficient protection.

[0032] In one possible implementation, key points are extracted from the current frame image to obtain real-time key point information of several key points in the current frame image, including: The current frame image is input into the HRNet-W48 model to obtain the feature map; The feature map is input into the CenterNet model to obtain real-time key point information of several key points of the leg; Before using the Kalman algorithm to calculate the missing keypoint information of the current frame image based on historical keypoint information, the following steps are also included: Compare and analyze real-time key point information with historical key point information; If a key point corresponding to a certain label in the previous N consecutive historical images does not exist in the current frame image, then a target key point with missing key point information is identified.

[0033] Specifically, the formula for key point location is:

[0034] In the formula, For the coordinates of the key points, These are the heatmap values ​​for the CenterNet model.

[0035] Real-time keypoint information is extracted from the current frame image using the HRNet-W48 and CenterNet models. The extracted real-time keypoint information is compared with historical keypoint information to determine if any keypoints are missing. If a keypoint corresponding to a certain label in the previous N consecutive historical frames is not present in the current frame image, then that keypoint is determined to be a missing keypoint.

[0036] In this embodiment, through comparative analysis, the system can identify missing key points in the current frame image, thereby ensuring the integrity of key point information. This is the basis for subsequent prediction of the location of missing key points using the Kalman algorithm. In complex working environments, key points may be occluded or lost. Through comparative analysis, the system can promptly detect these situations and take corresponding compensatory measures, which improves the system's robustness in dynamic and complex environments. By extracting key points in real time and performing comparative analysis, the system can dynamically adjust the range of the electronic fence to ensure that it always covers the actual working area of ​​the outriggers. This real-time and dynamic nature is crucial to ensuring the safety of crane operations.

[0037] In one possible implementation, the Kalman algorithm is used to calculate the missing keypoint information of the current frame image based on historical keypoint information, including: Based on the label of the target key point, the coordinate position of the key point corresponding to the label in different frames is obtained from the historical key point information, and an initial state vector is generated based on the timestamp of each frame image. Using the Kalman metric state transition matrix and initial state vector, the state vector of the target key point is calculated, and the missing key point information of the target key point is determined based on the state vector of the target key point. By combining missing keypoint information and real-time keypoint information of several keypoints in the current frame image, valid point information of preset location points is obtained, including: The real-time key point information and key point missing information of the current frame image are merged to obtain the valid point information of the preset position point.

[0038] Specifically, the Kalman measurement state transition matrix is:

[0039] In the formula, Δt It is the time interval between the current frame and the previous frame.

[0040] The formula for calculating the state vector of the target key point is:

[0041] In the formula, It is the state vector of the target key point. It is the initial state vector.

[0042] In this embodiment, the Kalman algorithm predicts the location of missing keypoints, ensuring that the system can acquire complete support leg keypoint information even when keypoints are occluded or lost. This compensation mechanism improves the system's robustness, ensuring the accuracy and integrity of the electronic fence. The Kalman algorithm utilizes historical data for prediction, effectively addressing interference in complex environments and enhancing the system's robustness in dynamic and complex environments. By combining historical and current data, the system can more accurately estimate the location of keypoints, reducing the possibility of misjudgments and missed detections.

[0043] In one possible implementation, based on the target keypoint's label, the coordinates of the keypoint corresponding to that label in different frames are obtained from historical keypoint information, and an initial state vector is generated based on the timestamps of each frame image, including: Calculate the average velocity in the X-axis direction and the average velocity in the Y-axis direction of the key point corresponding to the label based on the coordinate position; An initial state vector is generated based on the coordinates of the key point corresponding to the label in the last frame image, the average velocity in the X-axis direction, and the average velocity in the Y-axis direction.

[0044] Specifically, based on the target keypoint's label, the coordinate positions of the keypoint corresponding to that label in different frames are extracted from historical keypoint information. x 1 ,y 1) , ( x 2 ,y 2) ,…, ( x N ,y N )}.

[0045] Assume the timestamp of each frame is { t 1 ,t 2 ,…,t N}, calculate time interval Δt i =t i+1 t i .

[0046] Calculate the average velocity of the target key points in the X and Y axis directions. and :

[0047]

[0048] In the formula, N It's the frame rate. k It is the timestamp of the current frame.

[0049] Use the coordinates of the target key points in the last frame ( , ) and the calculated average velocity ( and Generate the initial state vector:

[0050] In this embodiment, the initial state vector provides the initial position and motion state of the target keypoints, which is the basis for the Kalman filter's prediction. By calculating the average velocity using historical data, the system can more accurately estimate the motion trend of the keypoints, thereby improving prediction accuracy. The average velocity reflects the motion trend of the keypoints, and combined with the coordinates of the last frame, the initial state vector can more accurately describe the current state of the keypoints. This initial state estimation method based on historical data enables the Kalman filter to more effectively predict the location of missing keypoints.

[0051] In one possible implementation, when multiple target key points are detected to be missing, the coordinate information of the corresponding key point in each frame image is determined from the historical key point information based on the label of the target key point, and a state vector is generated based on the coordinate information and the timestamp.

[0052] In this embodiment, when multiple missing target keypoints are detected, the processing steps are the same as when only one target keypoint is detected, except that Kalman prediction and other steps are performed separately for each missing keypoint. An initial state vector is generated using historical data, allowing the prediction of the current positions of these missing keypoints. This compensation mechanism ensures that the system can acquire complete support leg keypoint information even when keypoints are occluded or lost.

[0053] In one possible implementation, the minimum convex hull region is calculated using a convex hull algorithm based on the valid point information of preset position points, including: The reference point is determined based on all valid point information; the reference point is the valid point with the smallest ordinate. If the ordinates are the same, the valid point with the smallest abscissa is selected as the reference point. Construct a straight line from all valid points and the reference point, and calculate the polar angle between the line and the horizontal axis; Sort all valid points in ascending order of polar angle; if the polar angles are the same, retain the valid points that are closer to the reference point. Starting from the baseline, all valid points are traversed sequentially according to the sorting results, and valid points are filtered according to preset conditions; Connect the selected valid points in order to obtain the minimum convex hull region.

[0054] Specifically, the minimum convex hull region is:

[0055] in, It is the minimum convex hull of the point set P. It is the set of all valid points. Points on the boundary of the convex hull.

[0056] In this embodiment, the minimum convex hull region is used to define the boundary of the electronic fence, ensuring that the electronic fence accurately covers the working area of ​​the crane outriggers. Through the convex hull algorithm, the system can dynamically adjust the range of the electronic fence to adapt to the movement and changes of the outriggers. The minimum convex hull region is the smallest and most precise convex polygon, effectively preventing personnel or objects from accidentally entering the hazardous area.

[0057] In one possible implementation, valid points are filtered based on preset conditions, including: Construct a vector from the current valid point and the first valid point after the current valid point, construct a vector from the first valid point after the current valid point and the second valid point after the current valid point, and calculate the cross product of the two vectors. When the cross product is positive, the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point are considered to form a counterclockwise left turn, and the current valid point is retained. If the cross product is negative or zero, determine whether the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point form a right turn or are collinear. Delete the first valid point after the current valid point and backtrack to the current valid point; traverse all valid points.

[0058] In this embodiment, by filtering valid points, it is ensured that the final polygon is the minimum convex hull. The minimum convex hull is a minimal convex polygon that contains all points, effectively covering all key points. This is the core step of the convex hull algorithm, ensuring the correctness and minimization of the convex hull. By accurately filtering valid points, the system can construct the electronic fence more precisely, ensuring that it accurately covers the working area of ​​the crane outriggers, which helps improve the overall performance and safety of the system.

[0059] In one possible implementation, after using the minimum convex hull region as the electronic fence, the following is also included: Calculate the movement distance of each key point based on the key point information corresponding to the last frame of the image in the real-time key point information and historical key point information. When at least one moving distance exceeds the update threshold, the minimum convex hull region is recalculated based on the real-time key point information of the current frame image and the convex hull algorithm, and then updated. The updated minimum convex hull region is used as the updated electronic fence.

[0060] Specifically, the formula for calculating the distance traveled is:

[0061] In the formula, ( , (), , ) are the coordinates of the key points in the current frame image and the coordinates of the key points in the last frame image in the historical key point information, respectively.

[0062] In this embodiment, the movement distance of each keypoint is calculated based on the real-time keypoint information of the current frame image and the keypoint information corresponding to the last frame image in the historical keypoint information. If the movement distance of at least one keypoint exceeds a preset update threshold, the minimum convex hull region is recalculated. The recalculated minimum convex hull region is used as the updated electronic fence.

[0063] In one possible implementation, if the number of key points corresponding to the real-time key point information of the current frame image is equal to the number of preset position points of the target crane outrigger, the real-time key point information of several key points of the current frame image is used as the valid point information of the preset position points.

[0064] In this embodiment, when the number of keypoints is exactly equal, it indicates that the keypoint information in the current frame image is complete, without missing or occluded points. Directly using this keypoint information simplifies the processing flow and avoids unnecessary calculations and predictions.

[0065] One possible implementation also includes: Acquire real-time images of the target crane's outriggers, input these images into the YOLOv8m model, and obtain the intrusion target detection results for the electronic fence area; the intrusion targets include people and vehicles. An alarm is triggered when an intrusion target is detected.

[0066] In this embodiment, through real-time image detection, the system can monitor the dynamic situation within the electronic fence area in real time and promptly detect intrusion targets. This real-time monitoring mechanism effectively prevents personnel or vehicles from accidentally entering dangerous areas. When an intrusion target is detected, the system can immediately trigger an alarm, alerting on-site personnel and operators to take measures. This timely alarm mechanism can effectively reduce accident risks and improve the safety of the work site. The YOLOv8m model can accurately identify and locate intrusion targets, providing intelligent early warning functions. This intelligent early warning mechanism can help on-site personnel take timely measures to avoid potential safety accidents. The alarm system can be linked with the crane's control system to achieve emergency shutdown or warning response. This linkage mechanism can further improve the system's safety and ensure timely measures can be taken in emergency situations.

[0067] For example, the number of key points of the outrigger is 4. Assume that the system stores the key point information in the previous N consecutive frames of images, and each frame of image contains the label and coordinates of each key point.

[0068] The key points detected in the current frame image are: {(k1,(x current 1 ,y current 1 )),(k2,(x current 2 ,y current 2 )),(k3,(x current 3 ,y current 3 ))}。 Labeled as The keypoint was not detected in the current frame image. The coordinates of the keypoint labeled k4 were extracted from historical keypoint information: {(x1 4 ,y1 4 ),(x2 4 ,y2 4 ),…,(x N 4 ,y N 4 )}.

[0069] Assume the timestamp of each frame is {t1, t2, ..., t} N}. Calculate the time interval Δt between every two frames. i =t i+1 t i .

[0070] Calculate the average velocity of keypoint k4 along the X and Y axes in historical data. Obtain the coordinates (x, y) of keypoint k4 in the last frame.N 4 ,y N 4 ).

[0071] The initial state vector is generated based on the coordinates and average velocity of the last frame. The Kalman measurement state transition matrix is ​​then multiplied with the initial state vector to obtain the state vector of the key point labeled k4.

[0072] Determine the coordinate position of the key point labeled k4 based on the coordinate position in the state vector of the key point.

[0073] Based on the coordinates of the key point labeled k4 and the existing coordinates of k1, k2, and k3, complete valid point information is obtained.

[0074] The minimum convex hull region is calculated based on the valid point information, and the electronic fence region is obtained based on the minimum convex hull region.

[0075] In one possible implementation, the number of key points on the outriggers can be multiple, and the process of constructing the electronic fence is the same as when the number of key points on the outriggers is 4.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0078] Figure 3 A schematic diagram of the structure of the dynamic electronic fence construction device for crane outriggers provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 3 As shown, the crane outrigger dynamic electronic fence construction device 3 includes: Data acquisition module 31 is used to acquire historical key point information and current frame image from the previous N consecutive historical images of the target crane outrigger; The information extraction module 32 is used to extract key points from the current frame image to obtain real-time key point information of several key points in the current frame image; wherein, the key points are preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points; The effective point information determination module 33 is used to calculate the missing key point information of the current frame image based on historical key point information when the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger. Then, it uses the Kalman algorithm to calculate the missing key point information of the current frame image based on historical key point information, and combines the missing key point information and the real-time key point information of several key points of the current frame image to obtain the effective point information of the preset position points. The fence generation module 34 is used to calculate the minimum convex hull region based on the valid point information of the preset location points using the convex hull algorithm, and use the minimum convex hull region as the electronic fence.

[0079] In one possible implementation, the information extraction module 32 can be used for: The current frame image is input into the HRNet-W48 model to obtain the feature map; The feature map is input into the CenterNet model to obtain real-time key point information of several key points of the leg; Compare and analyze real-time key point information with historical key point information; If a key point corresponding to a certain label in the previous N consecutive historical images does not exist in the current frame image, then a target key point with missing key point information is identified.

[0080] In one possible implementation, the information extraction module 32 can be used for: Based on the label of the target key point, the coordinate position of the key point corresponding to the label in different frames is obtained from the historical key point information, and an initial state vector is generated based on the timestamp of each frame image. Using the Kalman metric state transition matrix and initial state vector, the state vector of the target key point is calculated, and the missing key point information of the target key point is determined based on the state vector of the target key point. The real-time key point information and key point missing information of the current frame image are merged to obtain the valid point information of the preset position point.

[0081] In this embodiment of the invention, by acquiring historical key point information from the previous N consecutive historical images of the target crane outrigger and the current frame image, and extracting key points from the current frame image, the state information of the crane outrigger can be obtained in real time. If the number of key points corresponding to the real-time key point information of the current frame image is less than the preset number of position points of the target crane outrigger, the Kalman algorithm is used to predict the missing key point information. Thus, even in complex operating environments such as those with obstructions, the key points of the outrigger can still be accurately identified, solving the safety hazard problem caused by incomplete key point detection in traditional methods and enhancing the robustness of the system.

[0082] Furthermore, this invention integrates key point missing information and real-time key point information of several key points in the current frame image to obtain effective point information of preset position points. Based on the effective point information of preset position points, the minimum convex hull region is calculated using the convex hull algorithm to construct an electronic fence that can cover the actual deployed area of ​​the crane outriggers. Compared with traditional fixed fences or ground markings, it avoids the problems of area failure or insufficient protection.

[0083] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the above method embodiments.

[0084] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0085] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for constructing a dynamic electronic fence for crane outriggers, characterized in that, include: Obtain historical key point information and the current frame image from the previous N consecutive historical images of the target crane's outriggers; The current frame image is subjected to key point extraction to obtain real-time key point information of several key points in the current frame image; wherein, the key points are preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points; If the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger, the Kalman algorithm is used to calculate the missing key point information of the current frame image based on the historical key point information, and the missing key point information and the real-time key point information of several key points of the current frame image are combined to obtain the effective point information of the preset position points. The minimum convex hull region is calculated using the convex hull algorithm based on the valid point information of the preset location points, and the minimum convex hull region is used as the electronic fence.

2. The method for constructing a dynamic electronic fence for crane outriggers according to claim 1, characterized in that, The step of extracting key points from the current frame image to obtain real-time key point information of several key points in the current frame image includes: The current frame image is input into the HRNet-W48 model to obtain the feature map; The feature map is input into the CenterNet model to obtain real-time key point information of several key points of the leg; Before using the Kalman algorithm to calculate the missing keypoint information of the current frame image based on historical keypoint information, the following steps are also included: Compare and analyze real-time key point information with historical key point information; If a key point corresponding to a certain label in the previous N consecutive historical images does not exist in the current frame image, then a target key point with missing key point information is identified.

3. The method for constructing a dynamic electronic fence for crane outriggers according to claim 2, characterized in that, The step of using the Kalman algorithm to calculate the missing keypoint information of the current frame image based on historical keypoint information includes: Based on the label of the target key point, the coordinate position of the key point corresponding to the label in different frames is obtained from the historical key point information, and an initial state vector is generated based on the timestamp of each frame image. Using the Kalman metric state transition matrix and the initial state vector, the state vector of the target key point is calculated, and the key point missing information of the target key point is determined based on the state vector of the target key point; The effective point information of the preset location point is obtained by combining the missing key point information and the real-time key point information of several key points in the current frame image, including: The real-time key point information of several key points in the current frame image and the missing key point information are merged to obtain the effective point information of the preset position point.

4. The method for constructing a dynamic electronic fence for crane outriggers according to claim 3, characterized in that, The step of obtaining the coordinate positions of the key points corresponding to the target key points in different frames from the historical key point information based on the target key point's label, and generating an initial state vector based on the timestamps of each frame image, includes: Calculate the average velocity in the X-axis direction and the average velocity in the Y-axis direction of the key point corresponding to the label based on the coordinate position; The initial state vector is generated based on the coordinate position of the key point corresponding to the label in the last frame image, the average velocity in the X-axis direction, and the average velocity in the Y-axis direction.

5. The method for constructing a dynamic electronic fence for crane outriggers according to claim 1, characterized in that, The step of calculating the minimum convex hull region using the convex hull algorithm based on the effective point information of the preset position points includes: The reference point is determined based on all valid point information; wherein, the reference point is the valid point with the smallest ordinate. If the ordinates are the same, the valid point with the smallest abscissa is selected as the reference point. Construct a straight line from all valid points and the reference point, and calculate the polar angle between the straight line and the horizontal axis; Sort all valid points in ascending order of polar angle; if the polar angles are the same, retain the valid points that are closer to the reference point. Starting from the reference point, all valid points are traversed sequentially according to the sorting results, and valid points are filtered according to preset conditions; The selected valid points are connected in sequence to obtain the minimum convex hull region.

6. The method for constructing a dynamic electronic fence for crane outriggers according to claim 5, characterized in that, The step of filtering valid points according to preset conditions includes: Construct a vector from the current valid point and the first valid point after the current valid point, construct a vector from the first valid point after the current valid point and the second valid point after the current valid point, and calculate the cross product of the two vectors. When the cross product is positive, it is determined that the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point constitute a counterclockwise left turn, and the current valid point is retained; If the cross product is negative or zero, determine whether the current valid point, the first valid point after the current valid point, and the second valid point after the current valid point form a right turn or are collinear. Delete the first valid point after the current valid point and backtrack to the current valid point; traverse all valid points.

7. The method for constructing a dynamic electronic fence for crane outriggers according to claim 1, characterized in that, After using the minimum convex hull region as the electronic fence, the following is also included: Calculate the movement distance of each key point based on the key point information corresponding to the last frame of the image in the real-time key point information and historical key point information. When at least one of the moving distances is higher than the update threshold, the minimum convex hull region is recalculated based on the real-time key point information of the current frame image and the convex hull algorithm, so as to update the minimum convex hull region. The updated minimum convex hull region is used as the updated electronic fence.

8. The method for constructing a dynamic electronic fence for crane outriggers according to claim 1, characterized in that, If the number of key points corresponding to the real-time key point information of the current frame image is equal to the number of preset position points of the target crane outrigger, then the real-time key point information of several key points of the current frame image shall be used as the valid point information of the preset position points.

9. A dynamic electronic fence construction device for crane outriggers, characterized in that, include: The data acquisition module is used to acquire historical key point information and the current frame image from the previous N consecutive historical images of the target crane outrigger. The information extraction module is used to extract key points from the current frame image to obtain real-time key point information of several key points in the current frame image; wherein, the key points are preset position points of the target crane outriggers; the key point information includes labels and coordinates; the number of key points is less than or equal to the number of preset position points; The effective point information determination module is used to calculate the missing key point information of the current frame image based on historical key point information when the number of key points corresponding to the real-time key point information of the current frame image is less than the number of preset position points of the target crane outrigger. The module then uses the Kalman algorithm to calculate the missing key point information of the current frame image based on historical key point information and combines the missing key point information with the real-time key point information of several key points of the current frame image to obtain the effective point information of the preset position points. The fence generation module is used to calculate the minimum convex hull region based on the effective point information of the preset location points using the convex hull algorithm, and to use the minimum convex hull region as the electronic fence.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

Images