Method for dynamically detecting falling area below crane load based on computer vision

Abstract

The invention relates to the technical field of computer vision, in particular to a method for dynamically detecting a falling area below a crane load based on computer vision, which comprises the following steps of: acquiring a construction site scene image in real time, and inputting the image into a preset construction site personnel detection tracking model and a crane load angular point detection tracking model; the model outputs the predicted construction site personnel position of each frame of image and the corresponding crane load angular point position; establishing a geometric mapping relation between the construction site scene image and the construction site scene, calculating a vanishing point in the construction site scene image, and estimating a falling area below the crane load based on the geometric mapping relation, the vanishing point and the crane load angular point position; and judging whether a worker is in a falling area or not in real time and automatically triggering an alarm based on the position of the worker on the construction site and the estimated falling area below the crane load. According to the invention, real-time dynamic detection and early warning can be carried out on the falling area below the crane load, so that the safety management level is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a computer vision-based method for dynamically detecting a falling area under a crane load. Background Art

[0002] In modern construction and industrial production, cranes, as important lifting equipment, are widely used for material handling and structural installation. However, crane operations are accompanied by major safety risks, among which "being hit by the crane load" is one of the common crane-related injuries. To ensure the safety of the construction site, it is necessary to effectively detect the falling area under the crane load so as to promptly detect people entering the dangerous area and issue an alarm in advance, thereby reducing the probability of safety accidents. In addition, accurate identification of the falling area can also assist crane operators in optimizing the lifting path and operating range, reducing safety risks caused by misjudgment. At the same time, the long-term accumulated falling area data can also be used to analyze unsafe behaviors and potential risks in operations, providing strong support for the safety management of the construction site and the formulation of subsequent safety measures.

[0003] The existing methods for detecting the falling area under the crane load mainly include three methods. The first is manual and camera-assisted detection. This method relies on cameras to shoot the working area, and the situation under the crane load is monitored manually, and video information is used to assist in judging whether the workers have entered the dangerous area. The second is the sensor detection system, which builds a safety detection system based on various sensors and reflects the on-site conditions by collecting sensor data in real time. The third is the application of computer vision technology, which uses image processing and visual analysis technology to process on-site images to identify the falling area and the location of personnel, providing data support for safety warnings.

[0004] However, although the above detection methods each have certain application advantages, they still have obvious shortcomings in the actual operation process. First, the method of manual and camera-assisted detection relies on the camera to collect images and then make judgments manually. Due to the complex operation of the crane and the large interference of the on-site environment, it is difficult for humans to fully grasp the real-time situation under the load, and it is difficult to judge the position of the workers. Secondly, the sensor detection system collects data in real time by deploying various sensors. Although it can reflect some on-site conditions, the system is expensive, the equipment is easy to damage, and the anti-interference ability is poor. Finally, computer vision technology is mostly based on the assumption that the falling area is static and fixed in existing applications, which is difficult to adapt to the dynamic changes in the area in actual operations, thus affecting the accuracy of monitoring and early warning. In summary, how to achieve real-time dynamic detection and early warning of the falling area under the crane load, thereby effectively improving the level of safety management has become a key technical problem that needs to be solved urgently. Summary of the invention

[0005] This application provides a computer vision-based dynamic detection method for the falling area under the crane load, which can perform real-time dynamic detection and early warning of the falling area under the crane load, thereby effectively improving the safety management level. This application provides the following technical solutions: In a first aspect, the present application provides a method for dynamic detection of a falling area under a crane load based on computer vision, the method comprising: Real-time acquisition of construction site scene images, wherein the construction site scene images are composed of continuous frame images; The construction site scene image is input into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image; Establishing a geometric mapping relationship between the construction site scene image and the construction site scene and calculating the vanishing point in the construction site scene image, estimating the falling area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point; Based on the location of construction site personnel and the estimated falling area under the crane load, it is determined in real time whether there are workers in the falling area. If there are workers in the falling area, an alarm is automatically triggered.

[0006] In a specific implementation scheme, the construction site scene image is input into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position of each frame image and its corresponding crane load corner point position, including: Collect multiple construction site scene images in advance, and manually mark the positions of construction site personnel and the positions of crane load corners in each construction site scene image; Each construction site scene image is taken as input, and the labeled positions of the construction site personnel and the labeled positions of the crane load corner points corresponding to each construction site scene image are taken as outputs respectively. The preset neural network model is trained to obtain the construction site personnel detection and tracking model and the crane load corner point detection and tracking model.

[0007] In a specific implementation scheme, after the model outputs the predicted construction personnel position of each frame image and its corresponding crane load corner position, the following further comprises: For the construction site personnel position predicted by each frame image in the construction site scene image, the displacement vector between the construction site personnel position predicted by the previous frame image and the previous frame image is calculated; the displacement vector The calculation formula is as follows: ; in, The first The location of construction personnel predicted by frame images, The first The position of the construction personnel predicted by the previous frame image; The average displacement vector between adjacent frames is calculated based on a sliding window of a preset size; the average displacement vector The calculation formula is as follows: ; in, For the The average displacement vector of the frame image, is the size of the sliding window, is the frame index in the sliding window, For the Displacement vector of frame image; The deviation between the displacement vector of each frame image and the average displacement vector is as follows: ; in, For the The deviation between the displacement vector of the frame image and the average displacement vector is used to determine whether a frame with a sudden jump occurs based on the deviation between the displacement vector of each frame image and the average displacement vector, and position correction is performed on the frame with a sudden jump.

[0008] In a specific possible implementation scheme, the determining whether there is a frame with an abrupt jump, and performing position correction on the frame determined to have an abrupt jump comprises: The deviation Compare with the preset threshold, if it exceeds the preset threshold, it is judged that the frame has a sudden jump; for the frame judged to have a sudden jump, the position is corrected, and the reasonable position of the current frame is predicted using the position information of the previous frame and the average displacement vector of the sliding window as follows: ; in, Indicates the occurrence of abrupt jumps The current frame position is predicted based on the previous frame position and motion trend; the abrupt jump will occur Predicted position of the frame With reasonable location Perform the fusion as follows: ; in, The first one with abrupt jump is finally adopted after correction. The position of the frame, is the adaptive weight, adaptive weight The calculation formula is as follows: ; in, is the preset threshold value to which the deviation is compared after calculation, is the sensitivity parameter.

[0009] In a specific possible implementation scheme, establishing a geometric mapping relationship between the construction site scene image and the construction site scene and calculating the vanishing point in the construction site scene image includes: A grid matching the actual size of the construction site scene is established on the construction floor plan of the construction site scene, regular grid points are set on the actual construction floor plan, and these grid points are matched with corresponding points in the camera view; The user selects at least 4 pairs of clearly corresponding points in the construction floor plan and the construction site scene image, and establishes a geometric mapping relationship between the construction site scene image and the actual construction site scene by solving the homography matrix; The edge detection technology is used to extract the straight line segment data in the construction site scene image, and then the RANSAC algorithm is used to fit these straight line segments to find the vanishing point and its coordinates where all parallel lines in the image converge under perspective projection.

[0010] In a specific implementation scheme, estimating the falling area under the crane load based on the geometric mapping relationship, the vanishing point, and the position of the crane load corner point includes: The load corner coordinates predicted by the model , , , The homography matrix obtained by calibration is projected onto the actual physical plane to obtain the projection coordinates of each load corner point on the physical plane, as follows: ; in, Indicates the load corner Projected coordinates on the physical plane; The side lengths of the four corner points are calculated using the projection coordinates of each load corner point on the physical plane and recorded as , , , , the long side dimension of the load projection is obtained by taking the average value of the opposite sides and short side size ,as follows: ; ; Calculate the sum of two sets of opposite sides of the load and compare them to determine the similarity coefficient , the perspective similarity principle is used to determine the corner points of the falling area under the load. In the image, the load corner points, vanishing points and falling area corner points satisfy the proportional relationship, as shown below: , ; in, are the pre-computed vanishing point coordinates, To estimate the coordinates of the corner points of the falling area on the actual physical plane, solve this proportional relationship and get The calculation formula is as follows: , ; After calculating the coordinates of the corner points on the actual physical plane, the falling area is defined by , , , The quadrilateral is mapped to the pre-constructed floor grid, and the grid area matching the load size is searched in the construction floor plan. The vertical distance from each corner point in the grid area to the vertical line formed by the load corner point and the vanishing point is calculated, and the grid area with the smallest total vertical distance is selected as the final fall area.

[0011] In a specific embodiment, the calculation loads the sum of two groups of opposite sides and compares them to determine the similarity coefficient. include: if If it is larger, take: , ; Otherwise, take: , ; in, and It is the long and short dimension of the actual physical size of the load.

[0012] In the second aspect, the present application provides a computer vision-based dynamic detection system for the falling area under the crane load, which adopts the following technical solutions: A computer vision-based dynamic detection system for a falling area under a crane load, comprising: An image acquisition module, used for acquiring a construction site scene image in real time, wherein the construction site scene image is composed of continuous frame images; A position prediction module, used for inputting the construction site scene image into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image; A fall area estimation module is used to establish a geometric mapping relationship between the construction site scene image and the construction site scene and calculate the vanishing point in the construction site scene image, and estimate the fall area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point; The real-time warning module is used to determine in real time whether there are workers in the falling area based on the location of the construction site personnel and the estimated falling area under the crane load. If there are workers in the falling area, an alarm is automatically triggered.

[0013] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a computer vision-based dynamic detection method for a falling area under a crane load as described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, wherein a program is stored in the storage medium, and when the program is executed by a processor, it is used to implement a computer vision-based dynamic detection method for a falling area under a crane load as described in the first aspect.

[0015] In summary, the beneficial effects of this application include at least: (1) By collecting construction site scene images in real time and performing dynamic monitoring, the limitations of traditional static area and fixed size algorithms are overcome. Unlike the static area assumption, this application can dynamically adjust and estimate the fall area as the crane operates and the construction site personnel move in real time, ensuring that the monitoring results are more in line with the actual construction site environment. It can also adapt to the complexity of the construction site environment, the dynamic changes of personnel, and the movement trajectory of the crane load, significantly improving the flexibility and accuracy of safety monitoring and reducing monitoring blind spots and misjudgment problems caused by environmental changes.

[0016] (2) A deep learning detection algorithm is used to accurately locate the positions of construction site personnel and the corner points of the crane load. Through the construction site personnel detection and tracking model and the crane load corner point detection and tracking model, accurate analysis of each frame of the image is achieved. Compared with the traditional manual judgment method, the deep learning algorithm not only greatly improves the detection accuracy of the worker and load positions, but also combines the tracking algorithm to dynamically track the construction site personnel and loads, ensuring more accurate real-time judgment and dynamic detection of the falling area. This combination of precise detection and dynamic tracking reduces human errors and significantly improves the overall monitoring efficiency of the system.

[0017] (3) It has strong integration capabilities and can be seamlessly connected with other safety monitoring systems (such as unsafe behavior detection systems) to form a more comprehensive safety management system. By integrating multiple monitoring technologies, this system can not only detect in real time whether workers have entered the fall zone, but also monitor other unsafe behaviors simultaneously, greatly improving the efficiency of observing and recording safe behaviors. The integration of this system optimizes the safety management process, allowing managers to grasp the safety status of the site in real time and take quick action, thereby improving the safety, management efficiency and accident prevention capabilities of the entire construction site.

[0018] The camera collects the scene images of the construction site in real time, and combines the construction site personnel detection and tracking model and the crane load corner point detection and tracking model to continuously monitor the real-time position of the construction site personnel and the movement state of the crane load. After establishing the geometric mapping relationship between the image and the actual construction site scene, the falling area under the crane load is accurately estimated according to the calculation results of the crane load corner point and vanishing point. Then, through the principles of geometric mapping and perspective projection, the load corner point is matched with the actual physical plane coordinate system, and the coordinates of the quadrilateral corner points of the falling area under the load are further calculated. Combined with the position coordinates of the construction site personnel and the estimated falling area, it is judged in real time whether the workers have entered the danger zone, and an alarm is automatically triggered when a worker is in the falling area to notify the crane operator and site management personnel. In this way, the safety dynamic monitoring and early warning of construction site personnel can be realized, and potential dangers can be prevented in time. It effectively solves the limitations of traditional manual monitoring and sensor monitoring methods, and overcomes the problems of environmental complexity and high equipment cost. By combining computer vision and geometric derivation, real-time monitoring of the dynamically changing construction site environment is realized, which not only improves the monitoring accuracy of the falling area, but also responds to the position changes of workers in time to ensure the safety of the operation.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for dynamically detecting a falling area under a crane load based on computer vision in an embodiment of the present application.

[0021] Figure 2 It is a schematic diagram of the overall process of a method for dynamic detection of a falling area under a crane load based on computer vision in an embodiment of the present application.

[0022] Figure 3 It is a structural block diagram of a dynamic detection system for the falling area under a crane load based on computer vision in an embodiment of the present application.

[0023] Figure 4 It is a block diagram of an electronic device for dynamic detection of a falling area under a crane load based on computer vision in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0025] Optionally, the present application takes the computer vision-based dynamic detection method of the falling area under the crane load provided in each embodiment as an example of being used in an electronic device, where the electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0026] Reference Figure 1 , is a flowchart of a method for dynamic detection of a falling area under a crane load based on computer vision provided by an embodiment of the present application, the method comprising at least the following steps: Step S101: collect construction site scene images in real time, where the construction site scene images are composed of continuous frame images.

[0027] In step S101, a construction site scene image is collected in real time by a camera device, wherein the construction site scene image is composed of continuous frame images to form a real-time video stream, ensuring that each frame image can reflect the latest status of the crane operation area. The construction site scene in this application refers to the environment of the entire construction site of the crane operation area, including the layout of the construction ground, surrounding buildings and structures, mechanical equipment, and areas where construction site personnel may move.

[0028] It should be noted that the camera equipment needs to be set up in a location with a wide field of view, no obstructions and covering the entire key area. For example, it can be installed on an elevated platform, a fixed monitoring tower or a fixed location near a crane operating platform. This can not only capture a wide panoramic image, but also ensure that the image contains information about all areas that need to be monitored.

[0029] Step S102: input the construction site scene image into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image.

[0030] In implementation, a proper amount of construction site scene images containing multiple scenes are collected in advance, usually requiring about a thousand images, and then the positions of the construction site personnel and the positions of the crane load corner points in each construction site scene image are manually annotated. Next, each construction site scene image is used as input, and the annotated positions of the construction site personnel and the annotated positions of the crane load corner points corresponding to each construction site scene image are used as outputs, respectively, to train the preset neural network model, thereby training to obtain a construction site personnel detection and tracking model and a crane load corner point detection and tracking model. Each frame of the construction site scene image collected in step S101 is input into the pre-trained construction site personnel detection and tracking model and the crane load corner point detection and tracking model, respectively, and the two models output the predicted construction site personnel position and the corresponding crane load corner point position of each frame of the image.

[0031] In addition, as a preferred embodiment, after obtaining the position of the construction personnel and the corresponding crane load corner position of each frame image predicted by the model output, in practical applications, relying solely on the position prediction results of each frame image by the deep learning model will often cause abrupt changes in the position of the construction personnel or the crane load corner position due to image noise, occlusion or detection error, thereby destroying the smoothness of the target motion in continuous frames. This not only affects the accuracy of subsequent geometric calculations and safety warnings, but may also cause misjudgments. Therefore, the present application designs a real-time tracking algorithm based on sliding window average velocity detection and adaptive weight fusion correction, which detects the continuity of the predicted position in continuous frames and corrects the abnormal frames to ensure that the target trajectory is smooth and continuous. It should be noted that since the correction process of the construction personnel position and the corresponding crane load corner position is exactly the same, the following will only take the construction personnel position as an example for explanation, and the specific steps are as follows: Step S1021: For the construction site personnel position predicted by each frame image in the construction site scene image, calculate the displacement vector between the construction site personnel position predicted by the previous frame image and the previous frame image.

[0032] Specifically, the displacement vector The calculation formula is as follows: ; in, The first The predicted positions of construction personnel based on the frame images (except the first frame), The first The position of the construction personnel predicted by the previous frame of the frame image. In computer vision and motion tracking, the displacement vector usually represents the movement of the target between two frames, that is, its position from the previous frame Go to the current frame position , the change is the displacement vector .

[0033] Step S1022: Calculate the average displacement vector between adjacent frames according to a sliding window of a preset size.

[0034] Specifically, the average displacement vector The calculation formula is as follows: ; in, For the The average displacement vector of the frame image, is the size of the sliding window, usually set to 3 to 5 frames to balance the smoothing effect and real-time performance. is the frame index in the sliding window, For the Displacement vector of frame image. During the construction site personnel tracking process, due to factors such as detection model errors, lighting changes, occlusion, and jitter, the detection results of a single frame may be abnormal, resulting in abrupt jumps in the target position. In order to reduce the impact of single frame errors, the average displacement vector of the sliding window is introduced to smooth the target's motion trend, making the detected personnel position more stable.

[0035] Step S1023: Based on the deviation between the displacement vector of each frame image and the average displacement vector, determine whether there is a frame with abrupt jump, and perform position correction on the frame determined to have abrupt jump.

[0036] Specifically, first calculate the deviation between the displacement vector of each frame image and the average displacement vector as follows: ; in, For the The deviation between the displacement vector of the frame image and the average displacement vector is calculated. Compare it with the preset threshold. If it exceeds the preset threshold, it is judged that the frame has a sudden jump. At this time, the position of the frame judged to have a sudden jump is corrected. First, the position information of the previous frame and the average displacement vector of the sliding window are used to predict the reasonable position of the current frame as follows: ; in, Indicates the occurrence of abrupt jumps The current frame position is predicted based on the previous frame position and motion trend. Then, in order to smooth the detection results of abnormal frames, an adaptive weighting formula is designed to smooth the detection results of abnormal frames. Predicted position of the frame With reasonable location Perform the fusion as follows: ; in, The first one with abrupt jump is finally adopted after correction. The position of the frame, is the adaptive weight, which determines the fusion ratio of the current frame detection result and the predicted position. The calculation is based on the deviation between the current frame displacement and the average displacement. The formula is as follows: ; in, is the preset threshold value to which the deviation is compared after calculation, is the sensitivity parameter, which is used to control the steepness of the sigmoid function. This formula achieves a smooth and continuous weight conversion through the sigmoid function. When it is smaller, it means that the current frame is consistent with the historical motion. Close to 1, the final position mainly depends on the detection result of the current frame. When the threshold is exceeded, It will quickly approach 0, making the correction result mainly depend on the predicted position, thereby automatically suppressing abnormal jumps.

[0037] In implementation, the average displacement is calculated using historical frame data to capture the overall trend of target movement, thereby effectively smoothing abnormal changes in a single frame caused by noise or error. Abnormal frame correction can avoid the accumulation of detection errors in consecutive frames, ensuring that subsequent geometric calculations and safety warning modules obtain continuous and reasonable target trajectories. The continuity and stability of the target position are crucial for safety warnings, and the corrected smooth trajectory can significantly reduce the risk of false alarms or missed alarms. Since the correction principles for the site personnel position and the crane load corner position are the same, this method is universal and can simultaneously improve the accuracy and continuity of both target tracking.

[0038] Step S103: establishing a geometric mapping relationship between the construction site scene image and the construction site scene and calculating the vanishing point in the construction site scene image, and estimating the falling area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point.

[0039] In implementation, it is first necessary to match the detection results in the construction site scene image (including the position of the crane load corners) with the actual construction site scene, and establish a grid that matches the actual size of the construction site scene on the construction floor plan. Set regular grid points on the actual construction floor plan, and match these grid points with the corresponding points in the camera view. The user selects at least 4 pairs of clearly corresponding points in the construction floor plan and the construction site scene image, and establishes a geometric mapping relationship between the construction site scene image and the actual construction site scene by solving the homography matrix. In this way, all subsequent target positions detected in the image can be converted into actual physical coordinates through this mapping relationship, thereby providing a stable benchmark for area size matching and geometric calculations. This step only needs to be re-executed when the camera installation position or viewing angle changes to ensure that the mapping relationship is always accurate.

[0040] Next, in order to capture the perspective effect in the construction site scene and provide a key reference for the subsequent estimation of the falling area, this application also needs to calculate the coordinates of the vanishing point in the construction site scene image. The straight line segment data in the construction site scene image is extracted using edge detection technology, and then the RANSAC algorithm is used to fit these straight line segments to find the vanishing point and its coordinates where all parallel lines in the image converge under perspective projection. The vanishing point not only reflects the perspective center of the construction site scene image, but also plays a key role in the subsequent construction of the vertical line vector and scale calculation under the crane load. Since the construction site scene environment is relatively stable, the vanishing point can generally be used as a fixed parameter after being determined in the preprocessing stage, without the need to recalculate each frame.

[0041] In step S103, after the preprocessing is completed, the falling area under the crane load is estimated based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point. , , , The homography matrix obtained by calibration is projected onto the actual physical plane to obtain the projection coordinates of each load corner point on the physical plane, as follows: ; in, Indicates the load corner Projection coordinates on the physical plane. The homography matrix reflects the correspondence between the camera view and the actual scene.

[0042] Then, the projection coordinates of each load corner point on the physical plane are used to calculate the side lengths of the four corner points, recorded as , , , ,For example for and The distance between the two sides is calculated by analogy. The long side dimension of the load projection is obtained by taking the average value of the opposite sides. and short side size , as shown below: ; ; The sum of the two sets of opposite sides of the load is then calculated and compared to determine the similarity coefficient It should be noted that crane loads are usually rectangular, or approximately rectangular, and their four sides can be divided into two groups of opposite sides: one group is and , and the other group is and , by calculating the sum of two sets of opposite sides, that is and , determine which group of sides has a larger sum. The larger group corresponds to the long side size of the load on the physical plane, while the smaller group corresponds to the short side size of the load on the physical plane. Therefore, if If it is larger, take: , ; Otherwise, take: , ; in, and is the long side size and short side size of the actual physical size of the load, similarity coefficient It reflects the proportional relationship between the load projection size in the image and the actual load size.

[0043] Finally, the perspective similarity principle is used to determine the corner points of the falling area under the load. In the image, the load corner points, vanishing points, and falling area corner points satisfy the proportional relationship, as shown below: , ; in, are the pre-computed vanishing point coordinates, To estimate the coordinates of the corner points of the falling area on the actual physical plane, solve this proportional relationship and get The calculation formula is as follows: , ; After calculating the coordinates of the corner points on the actual physical plane, the falling area is defined by , , , The quadrilateral is mapped to the pre-constructed floor grid, and the grid area matching the load size is searched in the construction floor plan. The vertical distance from each corner point in the grid area to the vertical line formed by the load corner point and the vanishing point is calculated, and the grid area with the smallest total vertical distance is selected as the final fall area.

[0044] Step S104: Based on the location of the construction site personnel and the estimated falling area under the crane load, it is determined in real time whether there are workers in the falling area. If there are workers in the falling area, an alarm is automatically triggered.

[0045] In step S104, the position of the construction personnel in each frame of the construction site scene image has been detected by the construction site personnel detection and tracking model, and the position coordinates of each worker in the construction site scene are output. The coordinate data is converted into the position in the physical coordinate system of the construction site scene after geometric mapping processing. Then, combined with the estimated falling area under the load, it is determined whether the real-time position of the construction site personnel is within the falling area to determine whether the worker has entered the danger zone. The judgment method is the point inside the polygon judgment algorithm in the prior art, which will not be described in detail here. Once it is determined that a worker is in the falling area, an alarm is automatically triggered immediately, and the crane operator and on-site management personnel are notified at the same time.

[0046] It should be noted that, since the positions of workers and loads are constantly changing, real-time personnel position detection and fall zone judgment will be performed on each frame of the image. As the construction site environment and personnel change, the relationship between the worker's position and the fall zone will be constantly re-judged. If a worker leaves the fall zone, the alarm will be automatically released; if a new worker enters the fall zone, the alarm will be immediately re-triggered. Through this dynamic monitoring and multiple detection mechanism, it is possible to respond to the movement of construction site personnel in real time, ensuring that an alarm can be issued as soon as a worker enters a dangerous area.

[0047] In summary, combined with Figure 2, this application uses a camera to collect site scene images in real time, combines the site personnel detection and tracking model and the crane load corner point detection and tracking model, and continuously monitors the real-time position of the site personnel and the motion state of the crane load. After establishing the geometric mapping relationship between the image and the actual site scene, the falling area under the crane load is accurately estimated according to the calculation results of the crane load corner point and the vanishing point. Then, through the principles of geometric mapping and perspective projection, the load corner point is corresponded to the actual physical plane coordinate system, and the coordinates of the quadrilateral corner points of the falling area under the load are further calculated. Combined with the position coordinates of the site personnel and the estimated falling area, it is judged in real time whether the workers enter the dangerous area, and an alarm is automatically triggered when a worker is in the falling area to notify the crane operator and the on-site manager. Thereby, the safe dynamic monitoring and early warning of the site personnel can be realized, and potential dangers can be prevented in time. It effectively solves the limitations of traditional manual monitoring and sensor monitoring methods, and overcomes the problems of environmental complexity and high equipment cost. By combining computer vision and geometric derivation, real-time monitoring of the dynamically changing site environment is realized, which not only improves the monitoring accuracy of the falling area, but also responds to the position changes of workers in time to ensure the safety of the operation.

[0048] Figure 3 This is a structural block diagram of a computer vision-based dynamic detection system for a falling area under a crane load provided by an embodiment of the present application. The system includes at least the following modules: An image acquisition module is used to acquire construction site scene images in real time. The construction site scene images are composed of continuous frame images. A position prediction module is used to input the construction site scene image into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image; A fall area estimation module is used to establish a geometric mapping relationship between the construction site scene image and the construction site scene and calculate the vanishing point in the construction site scene image, and estimate the fall area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point; The real-time warning module is used to determine in real time whether there are workers in the falling area based on the location of the construction site personnel and the estimated falling area under the crane load. If there are workers in the falling area, an alarm is automatically triggered.

[0049] For relevant details, refer to the above method embodiment.

[0050] Figure 4 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0051] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0052] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the computer vision-based dynamic detection method for the falling area under the crane load provided in the method embodiment of the present application.

[0053] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0054] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0055] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the computer vision-based dynamic detection method of the falling area under the crane load of the above method embodiment.

[0056] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the dynamic detection method of the fall area under the crane load based on computer vision of the above-mentioned method embodiment.

[0057] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A computer vision-based dynamic detection method for the falling area under a crane load, characterized in that: The method comprises: Real-time acquisition of construction site scene images, wherein the construction site scene images are composed of continuous frame images; The construction site scene image is input into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image; Establishing a geometric mapping relationship between the construction site scene image and the construction site scene and calculating the vanishing point in the construction site scene image, estimating the falling area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point; Based on the location of construction site personnel and the estimated falling area under the crane load, it is determined in real time whether there are workers in the falling area. If there are workers in the falling area, an alarm is automatically triggered.

2. The method for dynamic detection of the falling area under the crane load based on computer vision according to claim 1 is characterized in that: The construction site scene image is input into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position of each frame image and its corresponding crane load corner point position, including: Collect multiple construction site scene images in advance, and manually mark the positions of construction site personnel and the positions of crane load corners in each construction site scene image; Each construction site scene image is taken as input, and the labeled positions of the construction site personnel and the labeled positions of the crane load corner points corresponding to each construction site scene image are taken as outputs respectively. The preset neural network model is trained to obtain the construction site personnel detection and tracking model and the crane load corner point detection and tracking model.

3. The method for dynamic detection of the falling area under the crane load based on computer vision according to claim 1 is characterized in that: The model outputs the predicted construction personnel position of each frame image and its corresponding crane load corner position, and further includes: For the construction site personnel position predicted by each frame image in the construction site scene image, the displacement vector between the construction site personnel position predicted by the previous frame image and the previous frame image is calculated; the displacement vector The calculation formula is as follows: ； in, The first The location of construction personnel predicted by frame images, The first The position of the construction personnel predicted by the previous frame image; The average displacement vector between adjacent frames is calculated based on a sliding window of a preset size; the average displacement vector The calculation formula is as follows: ； in, For the The average displacement vector of the frame image, is the size of the sliding window, is the frame index in the sliding window, For the Displacement vector of frame image; The deviation between the displacement vector of each frame image and the average displacement vector is as follows: ； in, For the The deviation between the displacement vector of the frame image and the average displacement vector is used to determine whether a frame with a sudden jump occurs based on the deviation between the displacement vector of each frame image and the average displacement vector, and position correction is performed on the frame with a sudden jump.

4. The computer vision-based dynamic detection method for the falling area under the crane load according to claim 3 is characterized in that: The determining whether there is a frame with a sudden jump, and performing position correction on the frame determined to have a sudden jump comprises: The deviation Compare with the preset threshold, if it exceeds the preset threshold, it is judged that the frame has a sudden jump; for the frame judged to have a sudden jump, the position is corrected, and the reasonable position of the current frame is predicted using the position information of the previous frame and the average displacement vector of the sliding window as follows: ； in, Indicates the occurrence of abrupt jumps The current frame position is predicted based on the previous frame position and motion trend; the abrupt jump will occur Predicted position of the frame With reasonable location Perform the fusion as follows: ； in, The first one with abrupt jump is finally adopted after correction. The position of the frame, is the adaptive weight, adaptive weight The calculation formula is as follows: ； in, is the preset threshold value to which the deviation is compared after calculation, is the sensitivity parameter.

5. The computer vision-based dynamic detection method for the falling area under the crane load according to claim 1 is characterized in that: The step of establishing a geometric mapping relationship between the construction site scene image and the construction site scene and calculating the vanishing point in the construction site scene image comprises: A grid matching the actual size of the construction site scene is established on the construction floor plan of the construction site scene, regular grid points are set on the actual construction floor plan, and these grid points are matched with corresponding points in the camera view; The user selects at least 4 pairs of clearly corresponding points in the construction floor plan and the construction site scene image, and establishes a geometric mapping relationship between the construction site scene image and the actual construction site scene by solving the homography matrix; The edge detection technology is used to extract the straight line segment data in the construction site scene image, and then the RANSAC algorithm is used to fit these straight line segments. The point where all parallel lines in the image converge under perspective projection is found to be the vanishing point.

6. The computer vision-based dynamic detection method for the falling area under the crane load according to claim 5 is characterized in that: The method of estimating the falling area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point includes: The load corner coordinates predicted by the model , , , The homography matrix obtained by calibration is projected onto the actual physical plane to obtain the projection coordinates of each load corner point on the physical plane, as follows: ； in, Indicates the load corner Projected coordinates on the physical plane; The side lengths of the four corner points are calculated using the projection coordinates of each load corner point on the physical plane and recorded as , , , , the long side dimension of the load projection is obtained by taking the average value of the opposite sides and short side size ,as follows: ； ； Calculate the sum of two sets of opposite sides of the load and compare them to determine the similarity coefficient , the perspective similarity principle is used to determine the corner points of the falling area under the load. In the image, the load corner points, vanishing points and falling area corner points satisfy the proportional relationship, as shown below: ， ； in, are the pre-computed vanishing point coordinates, To estimate the coordinates of the corner points of the falling area on the actual physical plane, solve this proportional relationship and get The calculation formula is as follows: ， ； After calculating the coordinates of the corner points on the actual physical plane, the falling area is defined by , , , The quadrilateral is mapped to the pre-constructed floor grid, and the grid area matching the load size is searched in the construction floor plan. The vertical distance from each corner point in the grid area to the vertical line formed by the load corner point and the vanishing point is calculated, and the grid area with the smallest total vertical distance is selected as the final fall area.

7. The computer vision-based dynamic detection method for the falling area under the crane load according to claim 6 is characterized in that: The calculation load is the sum of two groups of opposite sides and compared to determine the similarity coefficient include: if If it is larger, take: ， ； Otherwise, take: ， ； in, and It is the long and short dimension of the actual physical size of the load.

8. A computer vision-based dynamic detection system for the falling area under a crane load, characterized in that: include: An image acquisition module, used for acquiring a construction site scene image in real time, wherein the construction site scene image is composed of continuous frame images; A position prediction module, used for inputting the construction site scene image into a preset construction site personnel detection and tracking model and a crane load corner point detection and tracking model, and the model outputs the predicted construction site personnel position and the corresponding crane load corner point position of each frame image; A fall area estimation module is used to establish a geometric mapping relationship between the construction site scene image and the construction site scene and calculate the vanishing point in the construction site scene image, and estimate the fall area under the crane load based on the geometric mapping relationship, the vanishing point and the position of the crane load corner point; The real-time warning module is used to determine in real time whether there are workers in the falling area based on the location of the construction site personnel and the estimated falling area under the crane load. If there are workers in the falling area, an alarm is automatically triggered.

9. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a computer vision-based dynamic detection method for a falling area under a crane load as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, which, when executed by a processor, is used to implement a computer vision-based dynamic detection method for a falling area under a crane load as described in any one of claims 1 to 7.

Images