Traveling crane hoisting monitoring method and system based on multi-normal-form visual fusion and medium

Abstract

The invention provides a crane hoisting monitoring method and system based on multi-normal-form visual fusion and a medium, and relates to the technical field of computer vision. Acquiring an image of the target acquisition area through an image acquisition device, and obtaining a real-time image acquisition video stream; preprocessing the real-time image acquisition video stream to obtain real-time image frame data; performing real-time image frame data identification based on the visual detection model to generate detection information; inputting the detection information into a visual tracking module, and tracking the identification object to obtain tracking information; acquiring a fixed dangerous area or a dynamic dangerous area based on the detection information and the tracking information; and identifying a dangerous condition in the video stream according to the dynamic dangerous area or the fixed dangerous area, obtaining dangerous condition accumulated data and triggering an alarm. The technical problems that in the prior art, the monitoring means is single, and dynamic dangers are difficult to accurately recognize are solved, and the technical effect of accurately recognizing potential dangers of hoisting operation in real time is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, system and medium for monitoring crane hoisting based on multi-paradigm vision fusion. Background Art

[0002] With the rapid development of industrial automation technology, the efficiency of loading and unloading operations has been significantly improved, but it has also brought more complex safety challenges. Problems such as accidental unhooking of heavy objects, collisions caused by movement or swinging, and obstructed operating vision all pose a serious threat to the safety of operators. Traditional monitoring technology mainly relies on manual monitoring and simple sensor alarms, such as closed-circuit television monitoring systems. However, manual monitoring is prone to blind spots due to human negligence, and simple sensor alarms can only respond to specific environmental changes, and cannot monitor and predict complex and changing operating environments in real time.

[0003] In recent years, with the advancement of computer vision technology, especially the application of deep learning technology, new possibilities have been brought to the field of industrial engineering safety. By combining cameras, sensors and advanced image processing algorithms, real-time monitoring of engineering scenes can be achieved. However, the variability of engineering sites, the dynamic complexity of loading and unloading equipment, and the high accuracy requirements of monitoring systems have posed huge challenges to existing technologies. Existing technologies often use a single traditional visual algorithm or modern deep learning method, relying on simple motion detection or fixed warning lines to trigger alarms, which is difficult to adapt to different working environments and complex lifting operation scenarios. Summary of the invention

[0004] The present application provides a method, system and medium for monitoring crane and hoisting based on multi-paradigm visual fusion, which solves the technical problem that the prior art cannot accurately identify and predict dynamic hazards in complex lifting operation scenarios due to the single monitoring means. The multi-paradigm visual fusion technology is used to monitor the crane and hoisting operation process in real time, intelligently identify dynamic and static hazards in the operation process and trigger alarms, thereby achieving the technical effect of improving the accuracy and reliability of monitoring and identifying dangerous conditions of crane and hoisting and ensuring the safety of the crane and hoisting operation process.

[0005] In view of the above problems, on the one hand, the present application provides a vehicle hoisting monitoring method based on multi-paradigm visual fusion, including: performing image acquisition of a target acquisition area based on an image acquisition device to obtain a real-time image acquisition video stream, wherein the target acquisition area includes a hoisting operation area; preprocessing the real-time image acquisition video stream to obtain real-time image frame data; identifying the real-time image frame data based on a visual detection model to generate detection information; inputting the detection information into a visual tracking module to track and identify the object to obtain tracking information; obtaining a fixed danger area or a dynamic danger area based on the detection information and the tracking information; identifying dangerous conditions in the real-time image acquisition video stream according to the dynamic danger area or the fixed danger area to obtain accumulated data of dangerous conditions; and triggering an alarm according to the accumulated data of dangerous conditions.

[0006] On the other hand, the present application also provides a vehicle hoisting monitoring system based on multi-paradigm visual fusion, including: an image acquisition unit, the image acquisition unit is used to perform image acquisition of a target acquisition area based on an image acquisition device, and obtain a real-time image acquisition video stream, wherein the target acquisition area includes a lifting operation area; a preprocessing unit, the preprocessing unit is used to preprocess the real-time image acquisition video stream and obtain real-time image frame data; a detection information generation unit, the detection information generation unit is used to perform real-time image frame data recognition based on a visual detection model to generate detection information; a tracking and identification unit, the tracking and identification unit is used to input the detection information into a visual tracking module, track and identify the object to obtain tracking information; a dangerous area acquisition unit, the dangerous area acquisition unit is used to obtain a fixed dangerous area or a dynamic dangerous area based on the detection information and the tracking information; a dangerous condition recognition unit, the dangerous condition recognition unit is used to perform dangerous condition recognition in the real-time image acquisition video stream according to the dynamic dangerous area or the fixed dangerous area, and obtain accumulated dangerous condition data; an alarm triggering unit, the alarm triggering unit is used to trigger an alarm according to the accumulated dangerous condition data.

[0007] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in any one of the above-mentioned methods when executed by a processor.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages: Based on the image acquisition device, the image acquisition of the target acquisition area is performed to obtain a real-time image acquisition video stream, and the target acquisition area includes the lifting operation area, so as to realize continuous real-time monitoring of the operation area and provide original visual information for subsequent processing. The real-time image acquisition video stream is preprocessed to obtain real-time image frame data, thereby improving the availability and quality of the image data. The real-time image frame data is recognized based on the visual detection model, and detection information is generated to intelligently identify the key elements in the monitoring scene. The detection information is input into the visual tracking module, and the recognized object is tracked to obtain the tracking information, so as to obtain the real-time state of the recognized object in the scene, and provide a basis for predicting its future position and activity. Based on the detection information and the tracking information, a fixed dangerous area or a dynamic dangerous area is obtained; according to the dynamic dangerous area or the fixed dangerous area, the dangerous condition in the real-time image acquisition video stream is identified to obtain the accumulated data of the dangerous condition; quantitative data support is provided for safety management, which is helpful to identify accident patterns and trends and provide a basis for preventive measures. According to the accumulated data of the dangerous condition, an alarm is triggered, which improves the response speed to sudden dangers, helps to prevent accidents and ensure the safety of workers.

[0009] As mentioned above, this application monitors the lifting operation area in real time, divides the fixed danger area and the dynamic danger area, and uses the visual detection model and the visual tracking model to perform intelligent identification and tracking as well as data analysis, accurately predicting dangerous conditions and issuing warnings. This improves the accuracy and reliability of monitoring and identifying dangerous conditions for crane lifting, ensures the safe operation of crane lifting, and significantly improves the safety management level of crane lifting operations.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a flow chart of a crane hoisting monitoring method based on multi-paradigm visual fusion provided in an embodiment of the present application; Figure 2 A schematic diagram of the process of drawing a dangerous area in the crane hoisting monitoring method based on multi-paradigm visual fusion provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a crane hoisting monitoring system based on multi-paradigm visual fusion provided in an embodiment of the present application; Explanation of the reference numerals: image acquisition unit 10 , preprocessing unit 20 , detection information generation unit 30 , tracking and identification unit 40 , dangerous area acquisition unit 50 , dangerous condition identification unit 60 , alarm triggering unit 70 . DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a crane and hoisting monitoring method, system and medium based on multi-paradigm visual fusion, thereby solving the technical problem that the prior art cannot accurately identify and predict dynamic hazards in complex lifting operation scenarios due to the single monitoring means. The multi-paradigm visual fusion technology is used to monitor the crane and hoisting operation process in real time, intelligently identify dynamic and static hazards in the operation process and trigger alarms, thereby achieving the technical effect of improving the accuracy and reliability of monitoring and identifying dangerous conditions of crane and hoisting and ensuring the safety of the crane and hoisting operation process.

[0013] Example 1: Figure 1 As shown, the embodiment of the present application provides a crane hoisting monitoring method based on multi-paradigm visual fusion, including: Step S1: performing image acquisition of a target acquisition area based on an image acquisition device to obtain a real-time image acquisition video stream, wherein the target acquisition area includes a lifting operation area.

[0014] Specifically, the image acquisition device refers to a device for capturing static or dynamic images, such as a camera, a camcorder, etc., which can convert real-world scenes into electronic signals to form a digital or analog video stream. The target acquisition area refers to the specific physical space range monitored by the image acquisition device, and in this embodiment, it specifically refers to the lifting operation area. The lifting operation area refers to the area where objects are lifted and moved during lifting operations. This is a high-risk operation area and special attention should be paid to safety. The real-time image acquisition video stream refers to a sequence of images continuously captured and transmitted by the image acquisition device, forming a real-time video stream that can be used to observe and analyze activities in the monitored area.

[0015] The image acquisition equipment installed around the lifting operation area can monitor the lifting operation area in all directions in real time, capture dynamic images of the operation area, and convert these images into digital signals to form a continuous real-time video stream. In order to ensure the effectiveness of monitoring, the deployment of image acquisition equipment needs to take into account factors such as coverage, viewing angle, and light conditions, so that all activities in the lifting operation area can be clearly captured.

[0016] Step S1 uses image acquisition equipment to establish a basic framework for real-time monitoring of the lifting operation area, providing the necessary data source for subsequent image processing and safety analysis.

[0017] Step S2: pre-processing the real-time image acquisition video stream to obtain real-time image frame data.

[0018] Specifically, preprocessing refers to the preliminary processing steps performed on an image before the main analysis is performed in computer vision and image processing. These steps include denoising, enhancement, scaling, cropping, etc., with the goal of improving image quality and making it more suitable for subsequent analysis and processing. Real-time image frame data refers to the information of a single image frame extracted from a real-time image acquisition video stream, which usually contains all the pixel values ​​of the image and other relevant information.

[0019] First, a specific video codec, such as H.264 or H.265, is used to decode the real-time image acquisition video stream obtained in step S1 and convert it from a compressed format back to the original image data. Then the video stream is preprocessed by denoising, enhancement, and standardization to improve the image quality. Denoising refers to removing noise from an image, which can be achieved through a variety of algorithms, such as mean filtering, median filtering, or a noise reduction model based on deep learning. Enhancement is to adjust the brightness, contrast, saturation, etc. of an image to enhance certain features of the image and make it easier to analyze. Standardization is to standardize the size and color space of the image for subsequent processing. For example, the image is scaled to a uniform resolution and the color space is converted to grayscale or RGB. Finally, independent image frames are extracted from the processed video stream by reading the video stream frame by frame or setting a specific time interval. The above-mentioned process of preprocessing the real-time image acquisition video stream can be completed by various existing image processing software and libraries, such as OpenCV, FFmpeg, etc. These tools provide a wealth of functions and interfaces for processing tasks such as decoding, denoising, enhancement, and frame extraction of video streams. Through the preprocessing steps, the real-time image frame data is prepared for further visual analysis and hazardous situation identification.

[0020] Step S3: performing real-time image frame data recognition based on a visual detection model to generate detection information.

[0021] Furthermore, step S3 of the embodiment of the present application also includes: Step S31: Collect historical operation video stream data.

[0022] Step S32: performing frame extraction processing on the historical operation video stream data to obtain extracted frame image data, performing data identification on the extracted frame image data to obtain a data identification result.

[0023] Step S33: constructing an image database based on the data identification result and the extracted frame image data.

[0024] Step S34: training the visual detection model based on the image database.

[0025] Preferably, the detection information includes: three target categories of hooks, cargoes and workers and corresponding positions and pixel area parameters.

[0026] Specifically, a visual inspection model is a specially designed algorithmic model that can analyze image data and identify specific objects, features or patterns therein. This model is usually based on deep learning techniques, such as convolutional neural networks (CNNs). After being trained with a large amount of image data, it can automatically learn and extract key information from images. Detection information refers to the results obtained by the visual inspection model after analyzing real-time image frame data, including three target categories of hooks, cargo and workers and their corresponding positions and pixel area parameters. Among them, the pixel area parameter refers to the number of pixels occupied by the target object in the image, which is used to quantify the size of the target.

[0027] Historical operation video stream data refers to the video data recorded in the past during the lifting operation. These data reflect the actual situation of the operation area and are used to train and verify the visual detection model. According to the historical operation video stream data, a visual detection model is pre-trained. The specific training process is as follows: First, the video data recorded during the past lifting operation is collected through the image acquisition device. These data are the unprocessed initial video streams and contain rich scene information. Then, video processing software such as FFmpeg is used to extract frames from these initial video streams to obtain extracted frame image data. Frame extraction is the process of selecting image frames at specific time points from the video stream. These frames represent the key moments or representative scenes of the video. These extracted frame image data are manually cleaned and labeled, that is, the positions of the hooks, goods and workers are marked in the image, and a category label is assigned to each identified object to obtain the data identification result. The identified extracted frame image data is organized into a native database. In order to further enhance the data diversity and the generalization ability of the model, data enhancement technology is used to process the data in the native database to ensure the balance of various target samples. Data enhancement is to locally crop a large number of sample maps of goods with different appearances and collection angles from data outside the native database, and paste the sample maps at random positions, sizes, and angles into random images in the native database to generate an image database. The image database is a collection of labeled image data used to train and test visual detection models to improve the accuracy and generalization of the model. This database contains not only the image itself, but also the identification information associated with it, such as object category and location. Model training is performed using deep learning frameworks such as TensorFlow or PyTorch, and related libraries and tools such as Keras, OpenCV, etc. Based on recall rate and mAP@0.5 as evaluation indicators, by adjusting the number of training samples in the image database, GPU graphics card configuration and number, batch size, initial learning rate, SGD momentum, target box loss weight, and category loss weight, the multi-card parallel tuning iterative training is achieved to generate a visual detection model. Through this series of steps, a visual inspection model specifically for the lifting operation area is trained. This model can identify and report the location and pixel area of ​​hooks, cargo and workers, providing strong technical support for operation safety.

[0028] Load the trained visual detection model. Prepare the preprocessed real-time image frame data in the format required by the model, including adjusting the image size, normalizing pixel values, etc. Input the prepared image data into the visual detection model for inference analysis, identify the objects in the image and their locations, and give one or more detection boxes, as well as the category label and pixel area parameters corresponding to each box. Post-process the detection information output by the model, which may include filtering out detection results with low confidence, merging overlapping detection boxes, performing non-maximum suppression (NMS) on the detection results, etc., to obtain the final detection information.

[0029] Step S3 can extract accurate and effective detection information from real-time image frame data by training the visual detection model, providing a basis for further safety analysis and decision-making.

[0030] Step S4: input the detection information into the visual tracking module to track and identify the object to obtain tracking information.

[0031] Furthermore, step S4 of the embodiment of the present application also includes: Step S41: Construct a visual tracking module, which includes two types of trackers , .

[0032] Step S42: The tracker The tracker is constructed using the second-order matching method. The Top-K matching method is used for construction.

[0033] Step S43: The tracker Perform worker target tracking and output worker target tracking information. The tracker Carry out hook target and cargo target tracking, and output hook target tracking information and cargo target tracking information.

[0034] Specifically, the visual tracking module is a specially designed system or algorithm for tracking moving objects in an image sequence. It can receive detection information and further analyze it to determine the motion trajectory of the object in consecutive frames. Tracking information refers to the results generated by the visual tracking module after analyzing the detection information, which usually includes the object's motion direction, speed, path, and other relevant dynamic information. The second-order matching method is a tracking algorithm that may consider more than one observation result when matching an object, that is, it not only relies on the information of the current frame, but also may combine the information of the previous frame or more frames for more stable object matching. The Top-K matching method is a selection strategy, where K represents a number, indicating that among all possible matches, the best top K matches are selected as the final result. This method is often used to deal with many-to-many matching problems and can improve the robustness of tracking. Artificial targets refer to people in the video screen, that is, workers. Hook targets refer to hooks used in lifting operations. Cargo targets refer to cargo being lifted or transported.

[0035] Build a visual tracking module, which contains two types of trackers , , Used to track workers, Used to track hooks and cargo. The second-order matching method is used, and the calculation formula of the second-order matching method is: ← ; , , ← ; Among them, the superscript Represents low confidence detection information, subscript represents the tth frame, subscript and Indicates whether the information matches. Represents the tracking trajectory, represents the detection information of all targets in one frame, T represents the tracking information of all targets in one frame, Represents the detection information of a single target, Represents the IOU distance cost matrix calculation function, Stands for Hungarian algorithm for information distribution.

[0036] Tracker The Top-K matching method is used to track the existence of only unique hooks and cargo; a removal condition judgment method is designed to handle unmatched detection information or tracking information. The removal condition judgment method first creates a motion state equation for all tracking trajectories And initialize the state equation of motion and the gain matrix , the specific formula is: ; ; , ; ; in, Respectively represent the initial detection information center coordinates, aspect ratio, height, Respectively represent the change values ​​of the detection information center coordinates, aspect ratio, and height. The initial value is 0. is an 8×8 diagonal matrix representing the initial state covariance, is a 4×8 matrix representing the observation matrix, represents the state transfer matrix, is an all-zero matrix, Representatives rely on The noise matrix of Match to detection information , update the filter parameters , and , the specific update formula is: ; in, represents the measurement noise covariance; when tracking the trajectory No matching detection information found , the Kalman filter is used to estimate the motion of the target in the tracking lost state. The motion estimation determines that the target disappears when the motion estimation value reaches the boundary of the field of view or is 0 for a long time; the motion information is calculated based on the tracking information. The specific calculation formula is: ; in, , Represents the motion information of the current frame direction, The magnitude of the velocity in the direction, , Respectively represent the output of the tracking information of the current frame direction, The magnitude of the velocity in the direction, , Respectively represent the output of the tracking information of the previous frame direction, The magnitude of the velocity in the direction, Represents the smoothing coefficient.

[0037] The detection information obtained from the visual detection model is passed to the visual tracking module. The visual tracking module uses this detection information to match objects in adjacent frames and predict their possible positions in subsequent frames. By analyzing the position of objects in consecutive frames, the visual tracking module can calculate the motion trajectory of the object, including information such as speed, acceleration, and direction of movement, and integrate the analyzed motion trajectory and other dynamic information into tracking information, which can be used for further data analysis or real-time monitoring.

[0038] Step S4, by constructing a visual tracking module, can identify key objects in the work area and track their movement status in real time, providing more comprehensive information support for work monitoring and safety management.

[0039] Step S5: Acquire a fixed danger area or a dynamic danger area based on the detection information and the tracking information.

[0040] Furthermore, if Figure 2 As shown, step S5 of the embodiment of the present application also includes: Step 1, determining whether the detection information or the tracking information contains the cargo target, if the cargo target is contained, obtaining the range coefficient and executing step 3, otherwise executing step 2; Step 2, determining whether the detection information or the tracking information contains a hook target, and if the hook target is contained, obtaining a range coefficient and executing step 3; Step 3, judging whether the target state of the hook is in a low posture and whether the swing amplitude is greater than a threshold value, if the target state of the hook is in the low posture and the swing amplitude of the hook is less than the threshold value, not drawing the fixed danger area and executing step 4, otherwise, drawing the fixed danger area based on the range coefficient and executing step 4; Step 4: determine the moving state of the vehicle. If the vehicle is in a moving state, draw a dynamic danger zone based on the range coefficient; otherwise, do not draw the dynamic danger zone.

[0041] Preferably, step S5 of the embodiment of the present application further includes: Obtaining the pixel area of ​​the hook based on the detection information of the hook target or the tracking information of the hook target , set the base plane height and the pixel area where the hook reaches the reference plane , according to the formula Calculate the hook height , set the height threshold , when the hook height Less than or equal to the height threshold , the hook is in the low posture.

[0042] Specifically, a fixed danger zone refers to a potential danger zone formed by the static position of the cargo or hook during the lifting operation. This area is fixed and determined by the static position of the cargo or hook, and does not change with the movement of the vehicle. Fixed danger zones are drawn by calculating the range coefficient. Dynamic danger zones refer to potential danger zones that change with changes in the working environment and the motion state of the object. These areas may be caused by the movement of the hook, cargo, or workers, and require real-time monitoring to prevent accidents. The detection information from the visual detection model and the tracking information from the visual tracking module are integrated. First, a logical judgment is used to determine whether the dynamic danger zone needs to be drawn. If necessary, the length of the dynamic danger zone is calculated by the visual optical flow module to complete the dynamic danger zone drawing.

[0043] First, check the detection information and tracking information to determine whether the cargo target or hook target is included. If the cargo target is included, calculate the range coefficient and then execute step 3, where the range coefficient ; If the cargo target is not included, execute step two to determine whether the monitoring information contains the hook target. When the hook target is detected, calculate the range coefficient and execute step three. This process can take corresponding safety measures for different work objects. Then further analyze the status of the hook target. If the hook is in a low posture, that is, the hook height is less than or equal to the height threshold, and the swing amplitude is less than the threshold, the hook is considered to be relatively stable and the fixed danger area is not drawn. On the contrary, if the swing amplitude of the hook is greater than the threshold, or the hook is not in a low posture, a fixed danger area is drawn according to the calculated range coefficient to warn of potential risks. After drawing the fixed danger area, execute step four.

[0044] Step 4: First, determine the moving state of the vehicle. If the vehicle is in a moving state, a dynamic danger zone is drawn based on the fixed danger zone. If the vehicle is not in a moving state, no dynamic danger zone is drawn. This dynamic danger zone takes into account the situation that when the vehicle moves, the fixed danger zone will also move with it, so it needs to be updated and marked in real time to ensure the safety of personnel. The moving state of the vehicle can be determined by various sensors configured on the vehicle, such as speed sensors, gyroscopes, accelerometers, etc. These sensors can monitor the movement state of the vehicle in real time. When the speed, angular velocity or acceleration of the vehicle exceeds the set threshold, the system can determine that the vehicle is in a moving state. It is also possible to capture visual images through image acquisition devices, and analyze the position and motion trajectory of objects in the picture through image processing technology. If the position of an object changes in continuous images, it can be determined that the object is in a moving state.

[0045] Then a visual optical flow module is constructed to calculate the length of the dynamic danger zone. The visual optical flow module is a technical module used to estimate the motion of objects in image sequences. It infers the motion trajectory of objects by analyzing the movement of pixels between adjacent frames. The visual optical flow module first initializes the optical flow parameters of the first frame in the first frame of the video sequence, where the optical flow parameters refer to various parameters used in optical flow calculation, including but not limited to pixel gradients, time gradients, etc. Pixel gradient refers to the direction and rate of change of pixel intensity in an image, which is usually used to detect edges and textures. Time gradient refers to the change in pixel intensity at the same position between adjacent frames in a video sequence. Then the module selects a fixed area in the video frame without moving targets as the reference area for optical flow calculation. Such an area usually contains multiple pixels, which can be used as a stable reference point to calculate the optical flow, obtain the pixel gradients and time gradients of each pixel around each pixel, and finally use the least squares fitting method to generate the moving speed of each pixel coordinate. The specific calculation formula is: ; Calculate the moving speed of each pixel coordinate The average vector of the current frame is used to generate the motion inertia , according to the motion inertia of the current frame , risk factor Calculate the length of the dynamic danger zone Among them, the risk factor It is a predefined parameter that can be customized by safety experts or engineers based on factors such as the type and size of the vehicle and the working environment.

[0046] Finally, according to the calculated dynamic danger zone length and a pre-set range factor , a dynamic danger zone is drawn in the direction of the vehicle's movement. This zone moves with the movement of the vehicle to mark potential danger areas in real time.

[0047] Step S5 can automatically draw corresponding dangerous areas for different work objects, which is convenient for subsequent identification of dangerous conditions and taking corresponding safety measures.

[0048] This embodiment provides a preferred method for determining whether the hook is in a low posture, that is, obtaining the pixel area of ​​the hook based on the detection information or tracking information of the hook target. , set the base plane height and the pixel area where the hook reaches the reference plane , according to the formula Calculate the hook height , set the height threshold , when the hook height Less than or equal to the height threshold , the hook is in the low posture.

[0049] Step S5 determines the target type and hook status, and can intelligently determine and update the dynamic danger zone according to the real-time working environment and object status, thereby effectively preventing accidents and ensuring working safety.

[0050] Step S6: performing real-time image acquisition according to the dynamic dangerous area or the fixed dangerous area to identify dangerous conditions in the video stream and update the accumulated dangerous list to obtain accumulated dangerous condition data.

[0051] Furthermore, step S6 of the embodiment of the present application also includes: Step A, perform worker target identification based on the visual detection model and the visual tracking module. If the detection information or the tracking information contains the worker target, execute step B; otherwise, update the hazard accumulation list, insert a safe status element at the head of the hazard accumulation list, remove a tail element, and execute step E.

[0052] Step B, determine whether the dynamic dangerous area exists. If so, perform intrusion judgment on the dynamic dangerous area based on the target position of the worker in the detection information or the tracking information, and execute step C, otherwise execute step D.

[0053] Step C, determine whether the invasion of the worker's target pixel area is greater than the danger threshold. If the invasion is greater than the danger threshold, update the danger accumulation list, insert a danger state element at the head of the danger accumulation list, and remove a tail element. Otherwise, update the danger accumulation list, insert a safety state element at the head of the danger accumulation list, remove a tail element, and execute step E.

[0054] Step D, based on the target position of the worker in the detection information or the tracking information, perform intrusion judgment on the fixed dangerous area, and judge whether the intrusion of the target pixel area of ​​the worker is greater than the dangerous threshold. When the intrusion is greater than the dangerous threshold, update the danger accumulation list, insert a dangerous state element at the head of the danger accumulation list, and remove a tail element. Otherwise, update the danger accumulation list, insert a safe state element at the head of the danger accumulation list, remove a tail element, and execute step E.

[0055] Step E: Count the number of dangerous state elements in the dangerous accumulation list to obtain the accumulated dangerous state data.

[0056] Specifically, dangerous condition recognition refers to automatically detecting and identifying dangerous conditions in the video by analyzing the real-time image acquisition video stream. This includes objects entering dangerous areas, abnormal movement patterns, signs of equipment failure, etc. Dangerous condition cumulative data refers to the data set of all identified dangerous conditions that are recorded. This data may include information such as the type of dangerous event, time of occurrence, duration, frequency, etc.

[0057] Receive and process the real-time image acquisition video stream captured by the image acquisition device in real time. Analyze and detect whether any object, including hooks, cargo or workers, has entered the dynamic danger zone or fixed danger zone in each frame of the image. Automatically detect the occurrence of dangerous conditions by comparing the position of the object in the image with the known danger zone boundary. Whenever a dangerous condition is detected, record the relevant data, including but not limited to the type of dangerous event, time of occurrence, duration, objects involved, etc. These data are then accumulated to form a historical record of the dangerous condition. The specific dangerous condition identification process is as follows: First, use the visual detection model to analyze the real-time image acquisition video stream and identify the worker target in the image. If the worker target is detected, proceed to step B; if not, continue to monitor the next frame of the image, and update the hazard accumulation list, insert a safe state element at the head of the hazard accumulation list, and remove a tail element. When the worker target is detected, further determine whether there is a dynamic danger zone. If so, a dynamic danger zone intrusion judgment is performed based on the worker's location information; if there is no dynamic danger zone, a fixed danger zone intrusion judgment is performed. Dynamic danger zone intrusion judgment refers to determining whether the worker has entered the dynamic danger zone, which is a judgment process based on real-time data and specific algorithms. Fixed danger zone intrusion judgment refers to determining whether the worker has entered the fixed danger zone, which is a judgment process based on preset areas and real-time data.

[0058] In the case of invasion of dynamic danger zone or fixed danger zone, it is judged whether the invasion of the target pixel area of ​​the worker is greater than the preset danger threshold. If the invasion degree exceeds the danger threshold, a danger state element is inserted at the head of the danger accumulation list and a tail element is removed; if the invasion degree does not reach the danger threshold, a safe state element is inserted and a tail element is also removed. Among them, the danger threshold is a preset value used to judge whether the invasion degree of the target pixel area of ​​the worker has reached the dangerous level. According to the actual operation requirements, the analysis of historical cases is pre-defined by experts or technicians. The danger accumulation list is a data structure used to store and manage detected danger state elements, which can dynamically add and delete elements. Dangerous state elements refer to records in the danger accumulation list that represent the occurrence of a dangerous condition. Safe state elements refer to records in the danger accumulation list that represent a safe state, that is, records in which no dangerous condition has occurred.

[0059] The number of dangerous state elements in the hazard accumulation list is counted as the cumulative data of dangerous conditions. These data can be used to analyze the safety performance of workers during the operation and evaluate the safety situation at the operation site.

[0060] Step S6 monitors and records the dangerous behaviors of workers during the operation in real time through dangerous condition identification, obtains accumulated data on dangerous conditions, realizes real-time monitoring of the safety conditions of the operation site, promptly discovers and records dangerous conditions, and ensures the safety of the driving operation process.

[0061] Step S7: triggering an alarm based on the accumulated data of the dangerous situation.

[0062] Specifically, the accumulated data of dangerous conditions collected in step S6 are compared with the preset alarm threshold. This threshold may be set based on historical data, industry standards or expert advice, and is used to determine when to activate the alarm. If the analysis results show that the accumulated data of dangerous conditions has reached or exceeded the alarm threshold, the system will determine that an alarm needs to be triggered. And activate the preset alarm mechanism, including lighting up indicator lights, sounding audible alarms, sending text messages or emails to relevant personnel, etc. The operation process is continuously monitored according to steps S1-S6, and the alarm is stopped when the accumulated data of dangerous conditions is less than the alarm threshold.

[0063] Exemplarily, the preset alarm threshold is that the number of dangerous state elements in the danger accumulation list exceeds half. The acquired dangerous state cumulative data is compared with the preset alarm threshold to determine whether the number of dangerous state elements in the danger accumulation list exceeds half. If so, the alarm is activated, and the surrounding personnel are notified by flashing red lights and beeps. The relevant information at the alarm time is recorded and a text message is sent to the administrator to determine whether the number of safe state elements in the danger accumulation list exceeds half. If so, the alarm is cancelled after a delay of N frames.

[0064] Step S7 can respond to potential safety threats in real time by analyzing the accumulated data of dangerous conditions, trigger alarms in time, and remind operators to take necessary preventive measures, thereby improving the overall safety of the work site.

[0065] In addition, after the alarm is triggered, the system can also record detailed information about the alarm, including the triggering time, cause, duration, etc., for subsequent analysis and reporting. At the same time, it can provide operators with some guiding information or recommended action plans to help them respond to dangerous situations quickly and effectively.

[0066] In summary, the crane hoisting monitoring method based on multi-paradigm visual fusion provided in the embodiment of the present application has the following technical effects: The image acquisition device is used to acquire images of the lifting operation area, and a real-time image acquisition video stream is obtained, so that continuous real-time monitoring of the operation area is realized, the continuity and comprehensiveness of the monitoring are ensured, and the original visual information is provided for subsequent data processing and analysis. The real-time image acquisition video stream is preprocessed to acquire real-time image frame data, thereby improving the availability and quality of the image data. According to the historical operation video stream data, a deep learning model is used to train a visual detection model, and the real-time image frame data is recognized based on the visual detection model to generate detection information and intelligently identify the key elements in the monitoring scene. The second-order matching method and the Top-K matching method are used to construct a visual tracking module, and the worker target and the hook and cargo target are tracked and identified respectively to obtain tracking information, accurately identify the real-time state of the object in the scene, and provide a basis for predicting its future position and activity. Based on the detection information and the tracking information, a fixed dangerous area or a dynamic dangerous area is obtained; according to the dynamic dangerous area or the fixed dangerous area, the dangerous condition in the real-time image acquisition video stream is identified, and the accumulated data of the dangerous condition is obtained, so as to realize real-time monitoring of the safety condition of the operation site, timely discover and record the dangerous condition, provide quantitative data support for safety management, and ensure the safety of the driving operation process. Based on the accumulated data of the dangerous conditions, an alarm is triggered to respond to potential safety threats in real time, prevent accidents from happening, and ensure the safety of the staff.

[0067] As described above, the embodiments of the present application have realized comprehensive and continuous real-time monitoring of the lifting operation area, divided the fixed danger area and the dynamic danger area by using the range coefficient and the visual optical flow model, and intelligently identified and tracked the targets in the area through the visual detection model and the visual tracking model, accurately predicted the dangerous conditions and issued early warnings. The accuracy and reliability of the monitoring and identification of the dangerous conditions of the crane lifting are improved, thereby significantly improving the safety management level of the crane lifting operation and ensuring the safety of the crane lifting operation process.

[0068] Example 2: Figure 3As shown, the embodiment of the present application provides a crane hoisting monitoring system based on multi-paradigm visual fusion, including: The image acquisition unit 10 is used to acquire images of a target acquisition area based on an image acquisition device to obtain a real-time image acquisition video stream, and the target acquisition area includes a lifting operation area.

[0069] The preprocessing unit 20 is used to preprocess the real-time image acquisition video stream to obtain real-time image frame data.

[0070] The detection information generating unit 30 is used to perform the real-time image frame data recognition based on the visual detection model to generate detection information.

[0071] The tracking and identifying unit 40 is used to input the detection information into the visual tracking module, track and identify the object to obtain tracking information.

[0072] The dangerous area acquisition unit 50 is used to acquire a dynamic dangerous area based on the detection information and the tracking information.

[0073] The dangerous situation identification unit 60 is used to identify dangerous situations in the real-time image acquisition video stream according to the dynamic dangerous area or the fixed dangerous area, and obtain accumulated data of dangerous situations.

[0074] The alarm triggering unit 70 is used to trigger an alarm according to the accumulated data of the dangerous situation.

[0075] Furthermore, the detection information generating unit 30 in the embodiment of the present application is further configured to perform the following steps: Collect historical operation video stream data.

[0076] Perform frame extraction processing on the historical operation video stream data to obtain frame-extracted image data, perform data identification on the frame-extracted image data, and obtain a data identification result.

[0077] An image database is constructed based on the data identification result and the extracted frame image data.

[0078] The visual detection model is trained based on the image database.

[0079] Preferably, the detection information includes: three target categories of hooks, cargoes and workers and corresponding positions and pixel area parameters.

[0080] Furthermore, the tracking and identification unit 40 of the embodiment of the present application is also used to perform the following steps: Construct a visual tracking module, which includes two types of trackers , .

[0081] The tracker The tracker is constructed using the second-order matching method. The Top-K matching method is used for construction.

[0082] The tracker Perform worker target tracking and output worker target tracking information. The tracker Carry out hook target and cargo target tracking, and output hook target tracking information and cargo target tracking information.

[0083] Furthermore, the dangerous area acquisition unit 50 in the embodiment of the present application is further configured to perform the following steps: Step 1, determining whether the detection information or the tracking information contains the cargo target, if the cargo target is contained, obtaining the range coefficient and executing step 3, otherwise executing step 2; Step 2, determining whether the detection information or the tracking information contains a hook target, and if the hook target is contained, obtaining a range coefficient and executing step 3; Step 3, judging whether the target state of the hook is in a low posture and whether the swing amplitude is greater than a threshold value, if the target state of the hook is in the low posture and the swing amplitude of the hook is less than the threshold value, not drawing the fixed danger area and executing step 4, otherwise, drawing the fixed danger area based on the range coefficient and executing step 4; Step 4: determine the moving state of the vehicle. If the vehicle is in a moving state, draw a dynamic danger zone based on the range coefficient; otherwise, do not draw the dynamic danger zone.

[0084] Preferably, the dangerous area acquisition unit 50 in the embodiment of the present application is further configured to perform the following steps: Obtaining the pixel area of ​​the hook based on the detection information of the hook target or the tracking information of the hook target , set the base plane height and the pixel area where the hook reaches the reference plane , according to the formula Calculate the hook height , set the height threshold , when the hook height Less than or equal to the height threshold , the hook is in the low posture.

[0085] Furthermore, the dangerous situation identification unit 60 of the embodiment of the present application is also used to perform the following steps: Step A, perform worker target identification based on the visual detection model and the visual tracking module. If the detection information or tracking information contains the worker target, execute step B; otherwise, update the hazard accumulation list, insert a safe status element at the head of the hazard accumulation list, remove a tail element, and execute step E.

[0086] Step B, determine whether the dynamic dangerous area exists. If so, perform intrusion judgment on the dynamic dangerous area based on the target position of the worker in the detection information or the tracking information, and execute step C, otherwise execute step D.

[0087] Step C, determine whether the invasion of the worker's target pixel area is greater than the danger threshold. If the invasion is greater than the danger threshold, update the danger accumulation list, insert a danger state element at the head of the danger accumulation list, and remove a tail element. Otherwise, update the danger accumulation list, insert a safety state element at the head of the danger accumulation list, remove a tail element, and execute step E.

[0088] Step D, based on the target position of the worker in the detection information or the tracking information, perform intrusion judgment on the fixed dangerous area, and judge whether the intrusion of the target pixel area of ​​the worker is greater than the dangerous threshold. When the intrusion is greater than the dangerous threshold, update the danger accumulation list, insert a dangerous state element at the head of the danger accumulation list, and remove a tail element. Otherwise, update the danger accumulation list, insert a safe state element at the head of the danger accumulation list, remove a tail element, and execute step E.

[0089] Step E: Count the number of dangerous state elements in the dangerous accumulation list to obtain the accumulated dangerous state data.

[0090] Through the above-mentioned detailed description of the method for monitoring the crane and hoisting based on multi-paradigm visual fusion, those skilled in the art can clearly understand the crane and hoisting monitoring system based on multi-paradigm visual fusion in this embodiment. For the system disclosed in Example 2, since it corresponds to the method disclosed in Example 1 and has corresponding functional modules and beneficial effects, the relevant parts can be referred to the description of the method part.

[0091] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned embodiment of the crane lifting monitoring method based on multi-paradigm visual fusion are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0092] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A crane hoisting monitoring method based on multi-paradigm visual fusion is characterized by: include: Based on the image acquisition device, image acquisition is performed in the target acquisition area to obtain a real-time image acquisition video stream, wherein the target acquisition area includes a lifting operation area; Preprocessing the real-time image acquisition video stream to obtain real-time image frame data; Performing real-time image frame data recognition based on a visual detection model to generate detection information; Inputting the detection information into a visual tracking module to track and identify the object to obtain tracking information; Acquire a fixed danger area or a dynamic danger area based on the detection information and the tracking information; Identify dangerous conditions in a real-time image acquisition video stream according to the dynamic dangerous area or the fixed dangerous area, and obtain accumulated data of dangerous conditions; According to the accumulated data of the dangerous condition, an alarm is triggered.

2. The method for monitoring crane hoisting based on multi-paradigm visual fusion according to claim 1, characterized in that: Before the real-time image frame data recognition is performed based on the visual detection model and the detection information is generated, the method further includes: Collect historical operation video stream data; Perform frame extraction processing on the historical operation video stream data to obtain frame-extracted image data, perform data identification on the frame-extracted image data, and obtain a data identification result; Building an image database based on the data identification result and the frame-sampling image data; The visual detection model is trained based on the image database.

3. The method for monitoring crane hoisting based on multi-paradigm visual fusion according to claim 1, characterized in that: The detection information includes three target categories: hooks, cargo and workers, and corresponding positions and pixel area parameters.

4. The method for monitoring crane hoisting based on multi-paradigm visual fusion according to claim 1, characterized in that: Before inputting the detection information into the visual tracking module and tracking and identifying the object to obtain the tracking information, the method further includes: Construct a visual tracking module, which includes two types of trackers , ; The tracker The tracker is constructed using the second-order matching method. The Top-K matching method is used for construction; The tracker Perform worker target tracking and output worker target tracking information. The tracker Carry out hook target and cargo target tracking, and output hook target tracking information and cargo target tracking information.

5. The method for monitoring crane hoisting based on multi-paradigm visual fusion according to claim 1, characterized in that: Acquiring a fixed danger zone or a dynamic danger zone based on the detection information and the tracking information, the method further includes: Step 1, determining whether the detection information or the tracking information contains the cargo target, if the cargo target is contained, obtaining the range coefficient and executing step 3, otherwise executing step 2; Step 2, determining whether the detection information or the tracking information contains a hook target, and if the hook target is contained, obtaining a range coefficient and executing step 3; Step 3, judging whether the target state of the hook is in a low posture and whether the swing amplitude is greater than a threshold value, if the target state of the hook is in the low posture and the swing amplitude of the hook is less than the threshold value, not drawing the fixed danger area and executing step 4, otherwise, drawing the fixed danger area based on the range coefficient and executing step 4; Step 4: determine the moving state of the vehicle. If the vehicle is in a moving state, draw a dynamic danger zone based on the range coefficient; otherwise, do not draw the dynamic danger zone.

6. The method for monitoring crane hoisting based on multi-paradigm visual fusion according to claim 5, characterized in that: The method further comprises: Obtaining the pixel area of ​​the hook based on the detection information of the hook target or the tracking information of the hook target , set the base plane height and the pixel area where the hook reaches the reference plane , according to the formula Calculate the hook height , set the height threshold , when the hook height Less than or equal to the height threshold , the hook is in the low posture.

7. The method according to claim 1, characterized in that According to the dynamic dangerous area or the fixed dangerous area, the dangerous condition in the real-time image acquisition video stream is identified and the dangerous accumulation list is updated to obtain the accumulated data of the dangerous condition, including: Step A, identifying worker targets based on the visual detection model and the visual tracking module, if the detection information or the tracking information contains worker targets, executing step B, otherwise updating the hazard accumulation list, inserting a safe state element at the head of the hazard accumulation list, removing a tail element, and executing step E; Step B, determining whether the dynamic dangerous area exists, if so, performing intrusion determination on the dynamic dangerous area based on the detection information or the worker target position in the tracking information, and executing step C, otherwise executing step D; Step C, determining whether the worker target pixel area invasion is greater than the danger threshold, if the invasion is greater than the danger threshold, updating the danger accumulation list, inserting a danger state element at the head of the danger accumulation list, and removing a tail element, otherwise updating the danger accumulation list, inserting a safety state element at the head of the danger accumulation list, and removing a tail element, and executing step E; Step D, based on the worker target position in the detection information or the tracking information, perform the fixed danger zone intrusion judgment, and judge whether the worker target pixel area intrusion is greater than the danger threshold. If the intrusion is greater than the danger threshold, update the danger accumulation list, insert a danger state element at the head of the danger accumulation list, and remove a tail element. Otherwise, update the danger accumulation list, insert a safety state element at the head of the danger accumulation list, and remove a tail element, and execute step E. Step E: Count the number of dangerous state elements in the dangerous accumulation list to obtain the accumulated dangerous state data.

8. The crane hoisting monitoring system based on multi-paradigm visual fusion is characterized by: The method for monitoring crane hoisting based on multi-paradigm visual fusion according to any one of claims 1 to 7 comprises: An image acquisition unit, the image acquisition unit is used to acquire images of a target acquisition area based on an image acquisition device to obtain a real-time image acquisition video stream, the target acquisition area including a lifting operation area; A preprocessing unit, the preprocessing unit is used to preprocess the real-time image acquisition video stream to obtain real-time image frame data; A detection information generating unit, the detection information generating unit is used to perform the real-time image frame data recognition based on a visual detection model to generate detection information; A tracking and identifying unit, the tracking and identifying unit is used to input the detection information into a visual tracking module, track and identify the object to obtain tracking information; A dangerous area acquisition unit, the dangerous area acquisition unit is used to acquire a fixed dangerous area or a dynamic dangerous area based on the detection information and the tracking information; A dangerous condition identification unit, the dangerous condition identification unit is used to identify dangerous conditions in the real-time image acquisition video stream according to the dynamic dangerous area or the fixed dangerous area, and obtain accumulated dangerous condition data; An alarm triggering unit is used to trigger an alarm according to the accumulated data of the dangerous situation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method for monitoring crane hoisting based on multi-paradigm visual fusion as described in any one of claims 1 to 7 are implemented.

Images