Graded safety protection method and device for bridge crane based on computer vision

Abstract

The embodiment of the invention provides a graded safety protection method and device for a bridge crane based on computer vision, and the method comprises the steps: arranging an intelligent vision camera on a moving trolley, and collecting image data of an area around a lifting hook; image preprocessing is carried out by using dynamic background modeling and sliding median filtering, and high-precision personnel detection is realized by combining a deep neural network configured by an adaptive residual block. A double-layer protection mechanism with the lifting hook as the center is innovatively designed, a dangerous area is divided into a peripheral early warning area and an inner side braking area, and sound-light alarm and emergency braking response are triggered respectively. According to the invention, the limitation of traditional fixed protection is broken through, intelligent graded safety protection is realized, and the operation safety is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a hierarchical safety protection method and device for bridge cranes based on computer vision. Background Art

[0002] Personnel safety protection during bridge crane operation has always been an important issue in the field of industrial safety. Traditional safety protection solutions mainly rely on mechanical guardrails or simple infrared detection, which have obvious limitations in complex operating environments.

[0003] From the perspective of protection methods, existing systems generally use fixed protection devices or single sensor detection, which cannot achieve dynamic monitoring of the work area. Mechanical protection devices can easily affect work efficiency, while simple sensor detection is easily affected by environmental interference and has problems of false alarms and missed alarms. Especially in areas where people frequently move around, traditional solutions are difficult to accurately identify potential dangers.

[0004] From the perspective of detection technology, the existing monitoring system has low accuracy in identifying human activities. In complex situations such as lighting changes and occlusion, it is difficult for the system to reliably detect and track people in the work area. At the same time, the fixed threshold alarm method cannot make differentiated responses based on the actual degree of danger.

[0005] From the perspective of system response, traditional protection solutions often adopt a single warning or braking method and lack a hierarchical protection mechanism. This simple protection strategy may affect normal operation efficiency and fail to meet the protection needs of different danger levels.

[0006] Therefore, how to build an intelligent hierarchical safety protection system to improve the accuracy of personnel detection and the flexibility of protection response is a technical problem that needs to be solved urgently. This is not only related to the life safety of operators, but also an important guarantee for improving work efficiency. Summary of the invention

[0007] In response to the problems in the prior art, the present application provides a computer vision-based bridge crane hierarchical safety protection method and device, which can effectively build an intelligent hierarchical safety protection system and improve the accuracy of personnel detection and the flexibility of protection response.

[0008] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a hierarchical safety protection method for a bridge crane based on computer vision, comprising: An intelligent visual camera is arranged on the mobile trolley of the bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection, wherein the personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer, wherein the feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on a training set, and selects the residual block combination with the optimal objective function value as the final network structure, wherein each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer, and the output layer outputs the personnel position coordinates through a fully connected layer; Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center. The warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0009] Furthermore, the intelligent visual camera is configured on the mobile trolley of the bridge crane to collect image data streams of the area around the hook, and the intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view, including: The smart vision camera is fixed to the bottom of the mobile trolley by a metal bracket, the metal bracket adopts a shock-proof and vibration-reducing structure, the lens axis of the smart vision camera is perpendicular to the ground, the focal plane of the smart vision camera is set parallel to the ground, and the optical axis of the smart vision camera coincides with the center line of the hook suspension; The intelligent vision camera has a built-in image acquisition processor, which continuously scans the area around the hook at a preset sampling frequency to generate a real-time image data stream. The field of view of the intelligent vision camera covers the area directly below the hook and the surrounding area. The intelligent vision camera uses an autofocus function to maintain image clarity.

[0010] Furthermore, the image data stream is sampled by a data acquisition unit, noise filtering and dynamic background modeling are performed on the image data stream, outliers are eliminated by a sliding median filter, and the pre-processed image data is written into a distributed database, including: The data acquisition unit performs a double buffer sampling operation on the image data stream, passes the sampled image data into a Gaussian blur filter to eliminate random noise, starts an adaptive threshold segmentation algorithm to remove background interference information, and calculates and updates scene background data through a mixed Gaussian model; A sliding median filter processing window is constructed to scan and filter the image data, the processing window size parameters are calculated and adjusted according to the local features of the image, and the image data after the filtering process is converted into a binary format and written into a designated storage partition of a distributed database.

[0011] Further, the pre-processed image data is read from the distributed database, and the image data is input into a personnel activity recognition model for personnel detection, wherein the personnel activity recognition model includes an input layer, a feature extraction layer and an output layer, including: The distributed database receives an image data reading instruction, reads pre-processed image data from a specified storage partition in a timestamp order, parses the read image data into a tensor format, and normalizes the image data and adjusts the image size through a data pre-processing module; The normalized image data is passed into the input layer of the personnel activity recognition model, the input layer maps the image tensor data to the feature space, the feature extraction layer performs multi-scale analysis on the image features and extracts the key point features of the personnel, and the output layer generates the personnel detection results based on the extracted feature information.

[0012] Furthermore, the feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on the training set, and selects the residual block combination with the best objective function value as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer, including: Constructing a residual block search space and setting search parameters, inputting the training set into the adaptive search algorithm to perform iterative optimization of residual block combinations, calculating the performance index of each group of residual block configurations according to the objective function, and writing the residual block combination configuration with the best performance index into the network structure parameter table. The residual block adopts a tandem method to construct a reduced-dimensional convolution layer, a standard convolution layer, and an increased-dimensional convolution layer; The residual block combination configuration is loaded into the feature extraction layer, the dimensionality reduction convolution layer is used to compress the number of feature channels, the standard convolution layer is used to extract spatial feature information, the dimensionality increase convolution layer is used to restore the feature dimension, and the fully connected layer is used to perform nonlinear transformation on the extracted features and calculate the personnel position coordinate values.

[0013] Furthermore, based on the personnel position coordinates, the warning area and the braking area are divided with the hook position as the center, the warning area is an annular area around the hook, and the braking area is a circular area inside the hook, including: A two-dimensional plane coordinate system is established with the hook position coordinates, and the boundary of the monitoring area is determined according to the coordinate system, and the monitoring area is divided into a circular braking area and a ring-shaped early warning area according to an inner and outer two-layer structure, and the boundary coordinates of the braking area and the early warning area are written into the area configuration table; The boundary coordinate data in the area configuration table is read, the distance difference between the personnel position coordinates and the hook position coordinates is calculated, the area type where the personnel is located is determined according to the distance difference, and the area type determination result is written into the status register.

[0014] Further, when a person is detected to enter the warning area, the sound and light alarm device is triggered to send a warning signal, and when a person is detected to enter the braking area, a braking instruction is generated, and the braking instruction is sent to the execution unit of the crane controller through the industrial bus network, and the execution unit includes a motor driver and a brake, including: Read the area type judgment result in the status register, send a trigger signal to the sound and light alarm device when the judgment result is a warning area, activate the speaker and flash light of the sound and light alarm device, write the alarm state into the system log, and generate a braking control instruction and write it into the control instruction buffer area when the judgment result is a braking area; The brake control instruction is transmitted to the crane controller through the industrial bus network. The crane controller parses the brake control instruction and sends a control signal to the execution unit. After receiving the control signal, the motor driver cuts off the power supply to the motor, and the brake simultaneously starts the brake mechanism to lock the moving parts.

[0015] In a second aspect, the present application provides a hierarchical safety protection device for a bridge crane based on computer vision, comprising: A visual acquisition module is used to configure an intelligent visual camera on a mobile trolley of a bridge crane to collect image data streams around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. A personnel identification module is used to read the preprocessed image data from the distributed database, and input the image data into a personnel activity identification model for personnel detection. The personnel activity identification model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on a training set, and selects the residual block combination with the optimal objective function value as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. The safety alarm module is used to divide the warning area and the braking area into the hook position center based on the personnel position coordinates, the warning area is the outer annular area around the hook, and the braking area is the inner circular area around the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0016] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the computer vision-based graded safety protection method for bridge cranes are implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the computer vision-based bridge crane graded safety protection method.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the computer vision-based bridge crane graded safety protection method.

[0019] It can be seen from the above technical solution that the present application provides a hierarchical safety protection method and device for bridge cranes based on computer vision, by configuring an intelligent visual camera on a mobile trolley to collect image data of the area around the hook. Dynamic background modeling and sliding median filtering are used for image preprocessing, and a deep neural network with adaptive residual block configuration is used to achieve high-precision personnel detection. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is one of the flow charts of the hierarchical safety protection method for bridge cranes based on computer vision in the embodiment of the present application; Figure 2 This is a second flow chart of a hierarchical safety protection method for bridge cranes based on computer vision in an embodiment of the present application; Figure 3 The third flowchart of the computer vision-based bridge crane hierarchical safety protection method in the embodiment of the present application; Figure 4 This is a fourth flow chart of the hierarchical safety protection method for bridge cranes based on computer vision in an embodiment of the present application; Figure 5 This is a fifth flow chart of the hierarchical safety protection method for bridge cranes based on computer vision in an embodiment of the present application; Figure 6 This is a sixth flow chart of the hierarchical safety protection method for bridge cranes based on computer vision in an embodiment of the present application; Figure 7 This is the seventh flow chart of the hierarchical safety protection method for bridge cranes based on computer vision in the embodiment of the present application; Figure 8 It is a structural diagram of a hierarchical safety protection device for a bridge crane based on computer vision in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0022] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0025] Taking into account the problems existing in the prior art, the present application provides a hierarchical safety protection method and device for bridge cranes based on computer vision. An intelligent visual camera is configured on a mobile trolley to collect image data of the area around the hook. Dynamic background modeling and sliding median filtering are used for image preprocessing, and a deep neural network with adaptive residual block configuration is used to achieve high-precision personnel detection. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety.

[0026] In order to effectively build an intelligent hierarchical safety protection system and improve the accuracy of personnel detection and the flexibility of protection response, this application provides an embodiment of a hierarchical safety protection method for bridge cranes based on computer vision, see Figure 1 The computer vision-based bridge crane hierarchical safety protection method specifically includes the following contents: Step S101: an intelligent visual camera is configured on a mobile trolley of a bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Optionally, this embodiment elaborates on the hardware configuration and image data preprocessing scheme of the bridge crane visual monitoring system, focusing on solving the technical problems of image acquisition stability and data processing real-time in industrial environments. Through reasonable camera installation methods and efficient data processing algorithms, accurate monitoring of the area around the hook is achieved.

[0027] The intelligent vision camera uses an industrial-grade high-speed camera, which is fixed to the bottom of the mobile trolley through a customized metal bracket. The bracket uses shock-absorbing rubber pads and elastic connectors to effectively reduce the impact of mechanical vibration during crane operation on the imaging quality. The camera lens axis is strictly kept vertical to the ground to ensure that the image will not produce perspective distortion, which is convenient for subsequent coordinate mapping calculations.

[0028] In practical applications, the installation height of the camera is determined according to the operating height of the crane, generally 2-3 meters above the hook. The selection of the field of view angle requires a balance between coverage and resolution. This embodiment uses a 120-degree wide-angle lens, which can cover a 12-meter diameter range centered on the hook to meet security monitoring requirements.

[0029] The data acquisition unit adopts a dual buffer structure. The main buffer is responsible for real-time data acquisition, and the secondary buffer is responsible for data processing. The ping-pong switching mechanism ensures the continuity of the data flow. The sampling frequency is set to 30 frames per second, which ensures real-time performance while avoiding excessive storage pressure.

[0030] Image preprocessing uses a multi-level filtering strategy. First, a Gaussian filter is used to eliminate random noise, and the filter kernel size is dynamically adjusted according to the image resolution. Then the background modeling module is started, and an improved mixed Gaussian model is used to handle background changes under complex working conditions. The model contains multiple Gaussian distribution components and can adapt to dynamic interference such as lighting changes and equipment movement.

[0031] The sliding median filter uses an adaptive window mechanism to eliminate outliers. The window size is automatically adjusted according to the local image features. A smaller window is used in the edge area to maintain details, and a larger window is used in the flat area to enhance the filtering effect. This dynamic adjustment strategy not only ensures the filtering effect, but also avoids information loss caused by over-smoothing.

[0032] The preprocessed image data is stored in a distributed database in binary format. The database adopts a partition storage strategy and organizes data blocks by timestamp to facilitate parallel processing and historical data backtracking in subsequent modules. The data writing process adopts an asynchronous method to avoid I / O operations affecting the real-time processing process.

[0033] This embodiment has been verified in complex industrial environments such as steel mills. Even under adverse conditions such as dust, vibration, and strong light, stable and clear image data can still be obtained. Through an efficient preprocessing process, the accuracy and real-time performance of subsequent personnel detection are significantly improved, providing a reliable data foundation for the safety protection system.

[0034] In practical applications, the processing flow of this embodiment can adapt to different weather conditions and operating environments. For example, when the light changes dramatically, dynamic background modeling can quickly update the scene model; when the device vibrates greatly, the shock-absorbing structure and image stabilization algorithm can effectively maintain the image quality. This robustness and adaptability are the key to ensuring the reliable operation of the security monitoring system.

[0035] Step S102: Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection. The personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, and iteratively calculates the objective function values ​​of different residual block combinations on the training set. The residual block combination with the optimal objective function value is selected as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. Optionally, this embodiment focuses on the design and implementation of a personnel activity recognition model, and proposes a solution based on an adaptive residual network to address the technical difficulties in accuracy and real-time personnel detection in a bridge crane working environment.

[0036] The model first reads the preprocessed image data from the distributed database. Considering the complexity of the industrial site, data reading adopts a multi-threaded asynchronous method. The main thread is responsible for model reasoning, and the independent reading thread maintains the data buffer pool to ensure the continuity of data supply. After the image data is read, it is normalized, the pixel value is scaled to the [-1,1] interval, and uniformly adjusted to the size specifications required by the model input.

[0037] The human activity recognition model adopts an innovative adaptive residual architecture. The input layer receives the preprocessed image tensor and performs feature mapping through the initial convolution layer. The feature extraction layer is the core part of the model, which contains multiple residual blocks connected in series. Each residual block consists of a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. This design reduces the computational complexity while maintaining the feature extraction capability.

[0038] A key innovation of this embodiment is the use of an adaptive search algorithm to optimize the residual block configuration. The search space contains different numbers of residual blocks, channels, and connections. The search algorithm iteratively evaluates the performance of different configurations on the training set, and the objective function comprehensively considers the detection accuracy and computational overhead. This adaptive optimization method enables the model to automatically adjust the structure according to the specific application scenario and find the best balance between accuracy and efficiency.

[0039] In practical applications, the design of the residual block takes into account the characteristics of the industrial environment. The dimensionality reduction convolution layer uses a 1×1 convolution kernel to compress the feature channel and reduce the amount of calculation; the standard convolution layer uses a 3×3 convolution kernel to extract spatial features, which can effectively identify human targets in different postures; the dimensionality increase convolution layer restores the feature dimension to ensure information integrity. This multi-scale feature extraction mechanism improves the model's adaptability to complex scenarios.

[0040] The output layer of the model adopts a fully connected structure to convert the feature map into the coordinates of the personnel position. Considering the characteristics of the crane operation area, the output coordinates are expressed in polar coordinates relative to the center of the hook, including two components: distance and azimuth. This representation method facilitates the subsequent determination of the safe area.

[0041] In actual application scenarios such as steel mills, this embodiment demonstrates excellent environmental adaptability. Even under complex conditions such as uneven lighting and occlusion interference, the model can still accurately detect the position of personnel in the work area. In particular, for workers wearing protective equipment such as safety helmets and reflective clothing, the detection effect remains stable.

[0042] To improve the practicality of the model, this embodiment also implements an online learning mechanism. The system continuously collects new sample data during operation and regularly fine-tunes the model so that the detection performance gradually improves over time. This continuous optimization feature enables the system to adapt to the specific conditions of different factories and provide more reliable technical support for safety protection.

[0043] Through the above technical solution, this embodiment solves the problem of insufficient accuracy and poor environmental adaptability of traditional visual inspection systems in industrial environments, and provides reliable technical guarantee for the safe operation of bridge cranes. The adaptive characteristics of the model and the efficient computing architecture ensure the real-time nature of the inspection process and meet the actual needs of industrial sites.

[0044] Step S103: Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center, wherein the warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0045] Optionally, this embodiment elaborates on the implementation scheme of intelligent safety protection based on personnel position coordinates, and effectively solves the technical problem of insufficient safety protection during traditional crane operations through multi-level warning and automatic braking mechanisms.

[0046] Based on the personnel position coordinates obtained in the previous step, this embodiment establishes a double-layer safety protection mechanism centered on the hook. The warning area is set as an outer annular area, whose inner diameter coincides with the boundary of the braking area, and the outer diameter is determined according to the operating characteristics of the crane. In actual applications, the scope of the warning area is usually set as a 4-6 meter annular area around the hook. This range can provide sufficient warning time without excessively interfering with normal operations.

[0047] The braking area is set as a circular area around the hook, usually 3-4 meters in diameter. This range is determined by taking into account practical factors such as the swing amplitude of the hook and the braking distance of the crane. The area division adopts the polar coordinate system, which is convenient for quickly calculating the position relationship between personnel and area boundaries, and improves the system response speed.

[0048] When a person is detected entering the warning area, the system immediately activates the sound and light alarm device. The sound and light alarm adopts a multi-level strategy: initially, a yellow flashing warning light is combined with an intermittent alarm sound. As the distance between the person and the hook decreases, the warning intensity increases step by step. The alarm is installed at the four corners of the crane to ensure that the sound and light signals can cover the operating area in all directions.

[0049] A major innovation of this embodiment is the adaptive adjustment mechanism of the warning signal. In a noisy working environment, the system automatically increases the volume of the alarm; in dim light, the brightness and flashing frequency of the warning light are increased. This intelligent adjustment ensures the effectiveness of the warning signal under various working conditions.

[0050] When a person enters the braking area, the system immediately generates a braking command. The command generation module predicts the possible risk of collision based on the person's movement direction and speed, and generates a corresponding braking strategy. In special cases, such as during the lifting of heavy objects, the system will consider the load status and adopt a graded braking method to avoid secondary safety hazards caused by sudden braking.

[0051] The braking command is transmitted to the crane controller via the industrial bus network. This embodiment uses the real-time Ethernet protocol to ensure the timeliness and reliability of command transmission. The network architecture adopts a redundant design, with the main and backup channels working simultaneously. Even if a single channel fails, it will not affect the system function.

[0052] After the actuator receives the braking command, the motor driver first performs a speed reduction operation while the brake is ready to engage. This progressive braking method can effectively reduce impact and protect equipment and hoisted objects. The brake adopts a dual-coil design with fast response and reliable braking performance.

[0053] In actual application scenarios such as steel mills, this embodiment shows excellent protection effects. Even in complex industrial environments, the system can accurately identify the location of personnel, issue warning signals in a timely manner, and perform reliable braking operations when necessary. Especially in scenarios where multiple people work together, the system can track multiple targets at the same time and provide independent safety protection for each person.

[0054] Through the above technical solution, this embodiment realizes intelligent safety protection during crane operation and effectively prevents the occurrence of human-machine collision accidents. The system's multi-level warning mechanism and reliable braking control provide strong guarantee for safe production at industrial sites.

[0055] From the above description, it can be seen that the computer vision-based hierarchical safety protection method for bridge cranes provided in the embodiment of the present application can collect image data of the area around the hook by configuring an intelligent visual camera on a mobile cart. Dynamic background modeling and sliding median filtering are used for image preprocessing, and high-precision personnel detection is achieved in combination with a deep neural network configured with adaptive residual blocks. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety.

[0056] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 2 , and can also include the following: Step S201: fix the smart vision camera to the bottom of the mobile trolley by a metal bracket, the metal bracket adopts a shock-proof and vibration-reducing structure, the lens axis of the smart vision camera is perpendicular to the ground, the focal plane of the smart vision camera is set parallel to the ground, and the optical axis of the smart vision camera coincides with the center line of the hook suspension; Step S202: The smart vision camera has a built-in image acquisition processor, which continuously scans the area around the hook at a preset sampling frequency to generate a real-time image data stream. The field of view of the smart vision camera covers the area directly below the hook and the surrounding area. The smart vision camera uses an autofocus function to maintain image clarity.

[0057] Optionally, this embodiment elaborates on the installation configuration and image acquisition scheme of the intelligent vision camera, and proposes a solution based on precision mechanical structure and intelligent imaging control to address technical problems such as poor image acquisition stability and monitoring blind spots in industrial environments.

[0058] The design of the metal bracket adopts a multi-layer vibration reduction structure, including an upper fixing plate, an intermediate vibration reduction layer and a lower mounting plate. The upper fixing plate is fastened to the bottom of the mobile trolley by high-strength bolts, and the connection is reinforced with thickened steel plates to ensure the overall structural strength. The intermediate vibration reduction layer consists of rubber vibration reduction pads and spring shock absorbers. The rubber vibration reduction pads use specially formulated industrial-grade damping rubber, which has excellent vibration reduction performance and durability, and can effectively absorb high-frequency vibrations during crane operation. The spring shock absorber mainly responds to low-frequency and large-scale vibrations, and adopts a parallel design of multiple sets of springs, which has good mechanical properties.

[0059] Precise adjustment of the camera installation position is a key innovation of this embodiment. The lower mounting plate is designed with a fine-tuning mechanism, including precision adjustment screws in three directions, which can accurately adjust the camera posture after installation. To ensure that the lens axis is strictly perpendicular to the ground, a high-precision level and laser plumb are used for calibration during installation. The adjustment of the focal plane parallelism uses a special angle compensation device to compensate for slight tilts of the mounting surface.

[0060] In practical applications, the installation height of the camera is an important parameter. In this embodiment, the camera is installed 2-3 meters above the hook according to the operating height and field of view requirements of the crane. This height can ensure sufficient monitoring range and prevent the camera from being blocked by the hoisted objects. To ensure that the optical axis coincides with the center line of the hook suspension, a laser positioner is used for calibration during installation, and fine-tuning space is reserved on the bracket.

[0061] The intelligent vision camera uses an industrial-grade high-speed image processor with powerful real-time processing capabilities. The image acquisition processor is based on the FPGA architecture and adopts a pipeline parallel processing mechanism, which can simultaneously perform tasks such as image acquisition, preprocessing and data transmission. The sampling frequency is set to 30 frames per second, which ensures real-time monitoring while avoiding data redundancy.

[0062] Another innovation of this embodiment is the intelligent exposure control system. In strong light and dark light environments, the system can automatically adjust the exposure parameters to ensure the visibility of image details. For backlight conditions, HDR technology is used for dynamic range compensation, which effectively solves the problem of uneven illumination at industrial sites.

[0063] The design of the field of view fully considers the actual application needs. The camera uses a 120-degree wide-angle lens, combined with a large-size image sensor, which can cover a large area centered on the hook. In order to avoid wide-angle distortion affecting the measurement accuracy, the system has a built-in distortion correction algorithm to perform geometric correction on the image in real time.

[0064] The autofocus system uses phase detection technology and advanced focus algorithms to track focus quickly and accurately. The system continuously monitors image clarity and immediately initiates focus adjustment when blur is detected. To cope with the rapid movement of the crane, the focus system uses a predictive tracking mechanism to adjust the focus in advance based on the movement trend.

[0065] Through the above technical solutions, this embodiment realizes stable and reliable image acquisition in industrial environments, providing a high-quality data basis for subsequent personnel detection. The system shows excellent environmental adaptability in complex working conditions such as steel plants, and can maintain image clarity and stability even under adverse conditions such as vibration, dust, and strong light.

[0066] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 3 , and can also include the following: Step S301: the data acquisition unit performs a double buffer sampling operation on the image data stream, passes the sampled image data into a Gaussian blur filter to eliminate random noise, starts an adaptive threshold segmentation algorithm to remove background interference information, and calculates and updates the scene background data through a mixed Gaussian model; Step S302: construct a sliding median filter processing window to scan and filter the image data, calculate and adjust the processing window size parameters according to the local features of the image, convert the filtered image data into a binary format and write it into a designated storage partition of the distributed database.

[0067] Optionally, this embodiment describes an image data preprocessing solution for industrial scenes, which effectively solves technical problems such as unstable image quality and complex background interference in industrial scenes through multi-stage filtering and adaptive processing mechanisms.

[0068] This embodiment first adopts a double buffer mechanism for data sampling, and sets a front buffer and a back buffer. When the front buffer receives new image data, the back buffer synchronously processes the previous frame data, and the two buffers work alternately to ensure the continuity of data processing. The buffer size is dynamically adjusted according to the image resolution, and is usually set to 1.5 times the original data size to reserve sufficient processing space.

[0069] The Gaussian blur filter is designed with an adaptive kernel function. The kernel size and standard deviation of the filter are automatically adjusted according to the image noise level. In areas with more dust in industrial sites, the system will increase the filter kernel size to enhance the noise reduction effect; in areas with rich image details, the kernel size will be reduced to retain edge information. The filtering process uses a fast convolution algorithm to improve processing efficiency.

[0070] Background segmentation uses an improved adaptive threshold algorithm to divide the image into multiple sub-regions and calculate local thresholds separately. The algorithm takes into account the grayscale distribution characteristics and spatial correlation of the image, and can effectively deal with the uneven lighting conditions at industrial sites. In the threshold calculation process, a weight factor is introduced to balance the contribution of each sub-region, thereby improving the accuracy of segmentation.

[0071] The innovation of the mixed Gaussian model lies in its adaptive learning mechanism. The model initially uses 3-5 Gaussian components, and the number of components can be dynamically increased or decreased as the scene changes. The model parameters are updated through online learning, and the learning rate is adaptively adjusted according to the speed of scene changes. In view of the movement of industrial field equipment and other situations, the model sets a fast update mechanism to ensure that the background model can reflect scene changes in a timely manner.

[0072] The sliding median filter uses a variable size processing window, and the window size is dynamically adjusted according to the local features of the image. In the edge area of ​​the image, a smaller window is used to retain details; in the flat area, a larger window is used to enhance the filtering effect. The calculation of the window size takes into account the gradient information and grayscale variance of the local area to ensure that the filtering strength matches the image features.

[0073] A major feature of this embodiment is the adaptive extraction mechanism of image features. By calculating parameters such as texture features, edge strength and grayscale distribution of local areas, the system can identify key areas in the image and adjust processing parameters accordingly. This adaptive mechanism is particularly suitable for industrial sites with large lighting changes and complex backgrounds.

[0074] The data storage adopts a distributed database architecture, and the processed binary image data is partitioned and stored according to timestamps and spatial locations. The database adopts a multi-copy mechanism to ensure data reliability and fast access. The design of storage partitions takes into account the temporal characteristics of the data, which facilitates subsequent trajectory analysis and behavior recognition.

[0075] Through the above technical solution, this embodiment significantly improves the quality and availability of image data in industrial scenarios. In complex industrial environments such as steel mills, this solution can effectively suppress various types of interference and provide stable and reliable image data, laying a solid foundation for subsequent personnel detection and trajectory analysis. The system has good environmental adaptability and can maintain stable processing performance even in harsh industrial environments.

[0076] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 4 , and can also include the following: Step S401: the distributed database receives an image data reading instruction, reads pre-processed image data from a specified storage partition in a timestamp order, parses the read image data into a tensor format, and normalizes the image data and adjusts the image size through a data pre-processing module; Step S402: The normalized image data is passed into the input layer of the personnel activity recognition model, the input layer maps the image tensor data to the feature space, the feature extraction layer performs multi-scale analysis on the image features and extracts personnel key point features, and the output layer generates personnel detection results based on the extracted feature information.

[0077] Optionally, this embodiment describes a deep learning-based personnel activity recognition solution in an industrial scenario. In view of technical difficulties such as complex industrial environments and changeable personnel postures, a complete solution from data reading to feature recognition is proposed.

[0078] The distributed database uses a time-series index mechanism to ensure that image data is efficiently read in chronological order. Data reading adopts batch processing mode, and the amount of data read in each batch is dynamically adjusted according to the real-time processing capability. When the processor load is detected to be too high, the system will automatically reduce the batch size to ensure real-time performance. Zero-copy technology is used in the data parsing process to convert binary data to tensor format directly in memory, avoiding additional data copying overhead.

[0079] Image normalization is a key step in model input. This embodiment adopts an adaptive normalization strategy, which first calculates the statistical features of the image, including parameters such as the mean and variance, and then dynamically adjusts the normalization parameters based on these features. In industrial scenes with large changes in lighting, this adaptive normalization method can effectively balance the light and dark contrast of the image and improve the stability of feature extraction.

[0080] Image resizing uses an intelligent scaling algorithm to maintain the image aspect ratio while ensuring that key areas are not cropped or deformed. The algorithm dynamically calculates the most appropriate scaling parameters based on the size ratio of the person in the image. For people who are far away from the camera, the system will appropriately increase the sampling density of the area to ensure that the feature information of the distant person is not lost.

[0081] The personnel activity recognition model of this embodiment adopts a multi-branch network structure. The input layer uses an improved spatial pyramid pooling module, which can process input images of different scales. This module enhances the model's adaptability to people at different distances and in different postures through multi-scale feature fusion. In industrial field applications, this structure significantly improves the ability to recognize people in partially occluded situations.

[0082] The feature extraction layer is designed with a cascaded attention mechanism, which includes two branches: spatial attention and channel attention. The spatial attention mechanism focuses on areas where people may appear, while the channel attention mechanism strengthens the response to key features of the human body. This dual attention mechanism can effectively cope with the interference of complex backgrounds in industrial sites and improve the accuracy of feature extraction.

[0083] The key point feature extraction of personnel adopts an improved posture estimation algorithm. The algorithm first locates the main joints of the human body, and then reconstructs the human body posture through the skeleton model. Considering that personnel in industrial scenes often carry tools or wear protective equipment, the algorithm specially adds key point templates specific to industrial scenes to improve the robustness of recognition.

[0084] The output layer uses a multi-task learning framework to simultaneously output the person detection box, key point location, and activity type prediction. The confidence score of the detection result takes into account multiple aspects, including the completeness of the person's outline, the rationality of the posture, and the temporal coherence. The system will perform temporal smoothing on the detection results of consecutive frames to reduce the jitter of the detection results.

[0085] Through this set of technical solutions, this embodiment achieves stable and reliable identification of personnel activities in industrial environments. Even in complex situations such as uneven lighting, partial occlusion, and dense crowds, it can still accurately detect and track on-site personnel, providing strong technical support for safety management. Especially in crane operation areas, the system can promptly detect safety hazards such as not wearing safety helmets and illegal standing positions, effectively preventing the occurrence of safety accidents.

[0086] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 5 , and can also include the following: Step S501: construct a residual block search space and set search parameters, input the training set into the adaptive search algorithm to perform iterative optimization of residual block combinations, calculate the performance index of each group of residual block configurations according to the objective function, write the residual block combination configuration with the best performance index into the network structure parameter table, and construct the reduced-dimensional convolution layer, the standard convolution layer and the increased-dimensional convolution layer in series with the residual block; Step S502: Load the residual block combination configuration into the feature extraction layer, compress the number of feature channels through the dimensionality reduction convolution layer, extract spatial feature information through the standard convolution layer, restore the feature dimension through the dimensionality increase convolution layer, and perform nonlinear transformation on the extracted features and calculate the personnel position coordinate values ​​through the fully connected layer.

[0087] Optionally, this embodiment proposes a residual block optimization scheme based on neural network architecture search, and designs an adaptive network structure optimization mechanism to address the issues of feature extraction efficiency and accuracy of personnel detection models in industrial scenarios.

[0088] This embodiment first constructs a comprehensive residual block search space, which includes different adjustable parameters such as convolution kernel size, number of channels, activation function, etc. The design of the search space fully considers the characteristics of industrial scenarios. For example, in the selection of convolution kernel size, it includes options suitable for detecting human targets of different scales. The setting of search parameters adopts a hierarchical strategy, first determining the main structural parameters, and then refining the specific connection mode and operation type.

[0089] The adaptive search algorithm uses an improved reinforcement learning method to model the combination selection of residual blocks as a sequential decision problem. The algorithm efficiently searches for the optimal configuration in the search space through an exploration-exploitation balance mechanism. In order to speed up the search process, the algorithm adopts an early stopping strategy to promptly terminate the evaluation of configurations with significantly poor performance and concentrate computing resources on configurations with more potential.

[0090] The performance evaluation of the residual block adopts a multi-objective optimization method, taking into account the accuracy, computational efficiency, and memory usage of the model. An adaptive weight mechanism is designed for the objective function to dynamically adjust the importance of each indicator according to the specific industrial application scenario requirements. For example, in scenarios with high real-time monitoring requirements, the weight of computational efficiency will be appropriately increased.

[0091] This embodiment innovatively designs a three-stage residual block structure. The dimensionality reduction convolution layer compresses the feature channel through a 1×1 convolution operation, effectively reducing the computational complexity. In the complex background of the industrial site, this dimensionality reduction operation can highlight important features and suppress the interference of irrelevant information. For example, for people wearing work clothes, the dimensionality reduction operation can effectively extract the human body contour features and reduce the influence of detail features such as clothing texture.

[0092] The standard convolution layer adopts a separable convolution design to extract features in the spatial dimension and channel dimension respectively. This design significantly reduces the number of parameters while maintaining good feature extraction capabilities. When extracting spatial features, a multi-scale receptive field mechanism is used to capture local detail features and global position information of people at the same time.

[0093] The design of the dimension-raising convolutional layer takes into account the restoration and enhancement of feature information. By gradually expanding the feature channels, important feature information can be fully preserved. In the process of restoring the feature dimension, the residual connection mechanism is used to effectively prevent the gradient vanishing problem in the deep network.

[0094] The fully connected layer adopts an attention mechanism-enhanced design, which can adaptively focus on important features. When calculating the position coordinates of people, a confidence assessment mechanism is introduced to quantitatively assess the reliability of the detection results. When the detection confidence is low, the system triggers a multi-frame verification mechanism to improve positioning accuracy through timing information.

[0095] Through this set of technical solutions, this embodiment significantly improves the performance of personnel detection in industrial scenarios. In complex lighting and crowded scenes at industrial sites, the solution demonstrates excellent detection accuracy and operating efficiency. Especially in complex industrial scenes such as steel workshops, the system can accurately identify and locate personnel at different distances and in different postures, providing reliable technical support for industrial safety monitoring.

[0096] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 6 , and can also include the following: Step S601: establishing a two-dimensional plane coordinate system with the hook position coordinates, determining the monitoring area boundary according to the coordinate system, dividing the monitoring area into a circular braking area and a ring-shaped warning area according to an inner and outer two-layer structure, and writing the boundary coordinates of the braking area and the warning area into the area configuration table; Step S602: read the boundary coordinate data in the area configuration table, calculate the distance difference between the personnel position coordinates and the hook position coordinates, determine the area type where the personnel is located according to the distance difference, and write the area type determination result into the status register.

[0097] Optionally, this embodiment proposes a hook operation monitoring solution based on dynamic safety zones, which effectively solves the potential safety hazard of personnel accidentally entering dangerous areas during industrial lifting operations by establishing a multi-level safety protection mechanism centered on the hook.

[0098] This embodiment first establishes a two-dimensional plane coordinate system with the real-time position of the hook as the origin, and adopts a dynamic following mechanism to ensure that the coordinate system is updated in real time as the hook moves. The establishment of the coordinate system takes into account the characteristics of the crane's operating range. The X-axis is set along the crane's running direction, and the Y-axis is perpendicular to the running direction. This setting facilitates subsequent area division and distance calculation.

[0099] The determination of the monitoring area boundary fully considers the working characteristics and safety requirements of the hook. The boundary range is dynamically calculated by analyzing the maximum swing amplitude, braking distance and other parameters of the hook. When operating with heavy loads, the system will automatically expand the monitoring area to cope with the large inertia of the hook; when operating with light loads or no loads, the monitoring range will be appropriately reduced to improve the flexibility of the system.

[0100] This embodiment innovatively proposes a double-layer safety area structure. The diameter of the inner circular braking area is usually set to 1.5 times the maximum swing of the hook. This area is a dangerous area where people are absolutely prohibited from entering. Once a person is detected entering this area, the system will immediately trigger an emergency braking signal. The width of the outer annular warning area can be flexibly configured according to site conditions, and is usually set to 0.5 to 1 times the radius of the braking area as a safety buffer zone.

[0101] The area configuration table uses a segmented storage structure. The boundary coordinates of each area are stored in polar coordinates, including radius and angle information. The configuration table supports real-time updates and can dynamically adjust the area range according to the operating conditions. For example, when strong winds are detected, the system will automatically expand the warning area range and issue a warning in advance.

[0102] The distance difference calculation uses an improved Euclidean distance algorithm, which takes into account the dynamic characteristics of the origin of the coordinate system. The algorithm calculates the relative position between the person and the hook by vector decomposition, not only calculating the absolute distance, but also analyzing the relative movement trend. When a person moves toward the hook, the system will predict the possible collision risk in advance.

[0103] The area type judgment adopts a multi-level judgment mechanism. First, a rough classification is performed based on the distance threshold, and then a detailed judgment is made based on the speed and direction of the person's movement. The system pays special attention to the person's stay time and movement trajectory in the warning area. If it is found that a person stays in the warning area for too long or moves towards the braking area, a warning signal will be issued in time.

[0104] The status register adopts a bit field structure design to efficiently store the results of area type judgment. The register not only records the current status, but also retains historical status information, which is convenient for analyzing the activity patterns of personnel. When an abnormal situation is detected, the system will automatically trigger data recording to provide a basis for subsequent safety analysis.

[0105] Through this set of technical solutions, this embodiment achieves accurate personnel safety protection in industrial lifting operation scenarios. This solution can effectively prevent personnel from mistakenly entering dangerous areas and promptly detect and stop unsafe behaviors. Especially in complex scenarios where multiple cranes work together, the system significantly improves operation safety through dynamic area division and real-time monitoring, providing reliable technical support for industrial site safety management.

[0106] In one embodiment of the computer vision-based bridge crane hierarchical safety protection method of the present application, see Figure 7 , and can also include the following: Step S701: reading the area type judgment result in the status register, sending a trigger signal to the sound and light alarm device when the judgment result is a warning area, activating the speaker and flash light of the sound and light alarm device, writing the alarm state into the system log, and generating a braking control instruction and writing it into the control instruction buffer area when the judgment result is a braking area; Step S702: The brake control instruction is transmitted to the crane controller through the industrial bus network. The crane controller parses the brake control instruction and sends a control signal to the execution unit. After receiving the control signal, the motor driver cuts off the power supply to the motor, and the brake simultaneously starts the brake mechanism to lock the moving parts.

[0107] Optionally, this embodiment proposes a crane intelligent braking control scheme based on multi-level safety response, which realizes active safety protection during industrial lifting operations through graded warning and emergency braking mechanisms.

[0108] The status register uses a real-time polling mechanism to continuously monitor the results of the area type judgment. When a person is detected entering the warning area, the system immediately starts the warning response process. The triggering of the warning signal adopts a progressive strategy. In the initial stage, the sound and light prompts are mainly low-frequency. As the person continues to approach the dangerous area, the alarm frequency and intensity gradually increase. The layout of the sound and light alarm device takes into account the noise interference of the industrial site, and adopts a multi-point collaborative layout to ensure that the alarm signal can be effectively transmitted.

[0109] The sound and light alarm device adopts an intelligent control strategy. The alarm audio output by the speaker contains a combination of sounds of different frequency bands, which can penetrate industrial noise without causing auditory discomfort to on-site personnel. The flash light uses a high-brightness LED array to transmit different levels of warning information through specific flashing frequencies and color coding. For example, yellow flashing indicates a warning state, and red fast flashing indicates that emergency braking is about to be triggered.

[0110] The alarm status record uses a distributed log system, which not only records the alarm time and type, but also includes the scene information at that time, such as personnel location, movement trajectory and other key data. These records are of great value for subsequent safety analysis and optimization of preventive measures. The system supports real-time log query and statistical analysis, which helps to discover potential safety hazards.

[0111] When a person is detected entering the braking area, the present embodiment activates a multi-level braking control mechanism. First, the braking control command is written into a high-priority cache to ensure that the command can be processed first. The braking command adopts a redundant design, including a main control command and a backup command, to prevent single-point failure from causing braking failure.

[0112] The industrial bus network uses real-time Ethernet protocol and supports millisecond-level instruction transmission. To ensure the reliable transmission of control instructions, CRC check and retransmission mechanism are adopted. The network topology adopts a ring structure. Even if a node fails, the control instruction can still reach the target device through the backup path.

[0113] The crane controller uses a dual-core processor architecture, with one core dedicated to parsing and processing emergency braking commands to ensure minimal response time. After receiving the braking command, the controller first verifies its legitimacy and then selects the most appropriate braking strategy based on the current load status.

[0114] The motor driver uses a fast power-off protection circuit, which can cut off the motor power supply in a very short time after receiving the control signal. The power-off process uses soft cut-off technology to avoid mechanical shock caused by sudden power failure. At the same time, the driver has an energy recovery function, which can convert the energy generated during braking into electrical energy storage.

[0115] The brake design uses a dual braking mechanism, including a normally closed mechanical brake and an auxiliary electromagnetic brake. The mechanical brake uses a spring pressurized method to automatically start braking after power failure; the electromagnetic brake provides adjustable braking force to achieve smooth braking. The brake's locking mechanism adopts a self-locking design, which can maintain the locked state even in the case of power failure.

[0116] Through this set of technical solutions, this embodiment achieves fast and reliable safety protection in industrial lifting operations. The solution can provide timely warnings when potential safety hazards are discovered, quickly shut down equipment in dangerous situations, and effectively prevent safety accidents during lifting operations. Especially during the operation of large lifting equipment, this multi-level safety protection mechanism significantly improves operational safety and provides reliable protection for the safety of personnel at industrial sites.

[0117] In order to effectively build an intelligent hierarchical safety protection system and improve the accuracy of personnel detection and the flexibility of protection response, the present application provides an embodiment of a hierarchical safety protection device for bridge cranes based on computer vision for realizing all or part of the contents of the hierarchical safety protection method for bridge cranes based on computer vision, see Figure 8 The computer vision-based bridge crane graded safety protection device specifically includes the following contents: The visual acquisition module 10 is used to configure an intelligent visual camera on the mobile trolley of the bridge crane to collect image data streams of the area around the hook. The intelligent visual camera adopts a vertical downward installation angle to position the hook at the center of the field of view, samples the image data stream through a data acquisition unit, performs noise filtering and dynamic background modeling on the image data stream, uses a sliding median filter to eliminate outliers, and writes the pre-processed image data into a distributed database; A personnel identification module 20 is used to read the preprocessed image data from the distributed database, and input the image data into a personnel activity identification model for personnel detection. The personnel activity identification model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on a training set, and selects the residual block combination with the optimal objective function value as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. The safety alarm module 30 is used to divide the warning area and the braking area into the center of the hook position based on the personnel position coordinates. The warning area is the outer annular area around the hook, and the braking area is the inner circular area around the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0118] From the above description, it can be seen that the computer vision-based graded safety protection device for bridge cranes provided in the embodiment of the present application can collect image data of the area around the hook by configuring an intelligent visual camera on a mobile cart. Dynamic background modeling and sliding median filtering are used for image preprocessing, and high-precision personnel detection is achieved in combination with a deep neural network configured with adaptive residual blocks. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent graded safety protection, and improves operational safety.

[0119] From the hardware level, in order to effectively build an intelligent hierarchical safety protection system and improve the accuracy of personnel detection and the flexibility of protection response, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the hierarchical safety protection method for bridge cranes based on computer vision, and the electronic device specifically includes the following contents: Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the hierarchical safety protection device for bridge crane based on computer vision and related equipment such as core business system, user terminal and related database; the logic controller can be a desktop computer, tablet computer and mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the hierarchical safety protection method for bridge crane based on computer vision and the embodiment of the hierarchical safety protection device for bridge crane based on computer vision in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0120] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0121] In practical applications, part of the computer vision-based bridge crane hierarchical safety protection method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0122] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0123] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0124] In one embodiment, the function of the bridge crane hierarchical safety protection method based on computer vision can be integrated into the central processor 9100. The central processor 9100 can be configured to perform the following control: Step S101: an intelligent visual camera is configured on a mobile trolley of a bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Step S102: Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection. The personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, and iteratively calculates the objective function values ​​of different residual block combinations on the training set. The residual block combination with the optimal objective function value is selected as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. Step S103: Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center, wherein the warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0125] From the above description, it can be seen that the electronic device provided in the embodiment of the present application collects image data of the area around the hook by configuring an intelligent visual camera on a mobile cart. Dynamic background modeling and sliding median filtering are used for image preprocessing, and a deep neural network with adaptive residual block configuration is used to achieve high-precision personnel detection. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety.

[0126] In another embodiment, the hierarchical safety protection device for bridge cranes based on computer vision can be configured separately from the central processing unit 9100. For example, the hierarchical safety protection device for bridge cranes based on computer vision can be configured as a chip connected to the central processing unit 9100, and the function of the hierarchical safety protection method for bridge cranes based on computer vision can be realized through the control of the central processing unit.

[0127] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0128] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 9600.

[0129] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0130] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0131] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0132] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0133] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0134] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0135] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the hierarchical safety protection method for bridge cranes based on computer vision in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps in the hierarchical safety protection method for bridge cranes based on computer vision in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented: Step S101: an intelligent visual camera is configured on a mobile trolley of a bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Step S102: Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection. The personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, and iteratively calculates the objective function values ​​of different residual block combinations on the training set. The residual block combination with the optimal objective function value is selected as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. Step S103: Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center, wherein the warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0136] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application collects image data of the area around the hook by configuring an intelligent visual camera on a mobile cart. Dynamic background modeling and sliding median filtering are used for image preprocessing, and a deep neural network with adaptive residual block configuration is used to achieve high-precision personnel detection. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety.

[0137] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the hierarchical safety protection method for bridge cranes based on computer vision in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the hierarchical safety protection method for bridge cranes based on computer vision are implemented. For example, the computer program / instruction implements the following steps: Step S101: an intelligent visual camera is configured on a mobile trolley of a bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Step S102: Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection. The personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, and iteratively calculates the objective function values ​​of different residual block combinations on the training set. The residual block combination with the optimal objective function value is selected as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. Step S103: Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center, wherein the warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

[0138] From the above description, it can be seen that the computer program product provided in the embodiment of the present application collects image data of the area around the hook by configuring an intelligent visual camera on a mobile cart. Dynamic background modeling and sliding median filtering are used for image preprocessing, and a deep neural network with adaptive residual block configuration is used to achieve high-precision personnel detection. An innovative double-layer protection mechanism centered on the hook is designed, which divides the dangerous area into an outer warning area and an inner braking area, triggering sound and light alarms and emergency braking responses respectively. The present application breaks through the limitations of traditional fixed protection, realizes intelligent hierarchical safety protection, and improves operational safety.

[0139] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A hierarchical safety protection method for bridge cranes based on computer vision, characterized in that: The method comprises: An intelligent visual camera is arranged on the mobile trolley of the bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. Read the preprocessed image data from the distributed database, and input the image data into a personnel activity recognition model for personnel detection, wherein the personnel activity recognition model includes an input layer, a feature extraction layer, and an output layer, wherein the feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on a training set, and selects the residual block combination with the optimal objective function value as the final network structure, wherein each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer, and the output layer outputs the personnel position coordinates through a fully connected layer; Based on the personnel position coordinates, a warning area and a braking area are divided with the hook position as the center. The warning area is an annular area on the outer periphery of the hook, and the braking area is a circular area on the inner periphery of the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send out a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

2. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: The intelligent visual camera is configured on the mobile trolley of the bridge crane to collect image data streams of the area around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view, including: The smart vision camera is fixed to the bottom of the mobile trolley by a metal bracket, the metal bracket adopts a shock-proof and vibration-reducing structure, the lens axis of the smart vision camera is perpendicular to the ground, the focal plane of the smart vision camera is set parallel to the ground, and the optical axis of the smart vision camera coincides with the center line of the hook suspension; The intelligent vision camera has a built-in image acquisition processor, which continuously scans the area around the hook at a preset sampling frequency to generate a real-time image data stream. The field of view of the intelligent vision camera covers the area directly below the hook and the surrounding area. The intelligent vision camera uses an autofocus function to maintain image clarity.

3. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: The method of sampling the image data stream by a data acquisition unit, performing noise filtering and dynamic background modeling processing on the image data stream, eliminating outliers by using a sliding median filter, and writing the pre-processed image data into a distributed database includes: The data acquisition unit performs a double buffer sampling operation on the image data stream, passes the sampled image data into a Gaussian blur filter to eliminate random noise, starts an adaptive threshold segmentation algorithm to remove background interference information, and calculates and updates scene background data through a mixed Gaussian model; A sliding median filter processing window is constructed to scan and filter the image data, the processing window size parameters are calculated and adjusted according to the local features of the image, and the image data after the filtering process is converted into a binary format and written into a designated storage partition of a distributed database.

4. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: The pre-processed image data is read from the distributed database, and the image data is input into a personnel activity recognition model for personnel detection, wherein the personnel activity recognition model includes an input layer, a feature extraction layer and an output layer, including: The distributed database receives an image data reading instruction, reads pre-processed image data from a specified storage partition in a timestamp order, parses the read image data into a tensor format, and normalizes the image data and adjusts the image size through a data pre-processing module; The normalized image data is passed into the input layer of the personnel activity recognition model, the input layer maps the image tensor data to the feature space, the feature extraction layer performs multi-scale analysis on the image features and extracts the key point features of the personnel, and the output layer generates the personnel detection results based on the extracted feature information.

5. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on the training set, and selects the residual block combination with the best objective function value as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer, including: Constructing a residual block search space and setting search parameters, inputting the training set into the adaptive search algorithm to perform iterative optimization of residual block combinations, calculating the performance index of each group of residual block configurations according to the objective function, and writing the residual block combination configuration with the best performance index into the network structure parameter table. The residual block adopts a tandem method to construct a reduced-dimensional convolution layer, a standard convolution layer, and an increased-dimensional convolution layer; The residual block combination configuration is loaded into the feature extraction layer, the dimensionality reduction convolution layer is used to compress the number of feature channels, the standard convolution layer is used to extract spatial feature information, the dimensionality increase convolution layer is used to restore the feature dimension, and the fully connected layer is used to perform nonlinear transformation on the extracted features and calculate the personnel position coordinate values.

6. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: Based on the personnel position coordinates, the warning area and the braking area are divided with the hook position as the center, the warning area is an annular area around the hook, and the braking area is a circular area inside the hook, including: A two-dimensional plane coordinate system is established with the hook position coordinates, and the boundary of the monitoring area is determined according to the coordinate system, and the monitoring area is divided into a circular braking area and a ring-shaped early warning area according to an inner and outer two-layer structure, and the boundary coordinates of the braking area and the early warning area are written into the area configuration table; The boundary coordinate data in the area configuration table is read, the distance difference between the personnel position coordinates and the hook position coordinates is calculated, the area type where the personnel is located is determined according to the distance difference, and the area type determination result is written into the status register.

7. The computer vision-based bridge crane hierarchical safety protection method according to claim 1 is characterized in that: When a person is detected entering the warning area, the sound and light alarm device is triggered to send a warning signal. When a person is detected entering the braking area, a braking command is generated and the braking command is sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake, including: Read the area type judgment result in the status register, send a trigger signal to the sound and light alarm device when the judgment result is a warning area, activate the speaker and flash light of the sound and light alarm device, write the alarm state into the system log, and generate a braking control instruction and write it into the control instruction buffer area when the judgment result is a braking area; The brake control instruction is transmitted to the crane controller through the industrial bus network. The crane controller parses the brake control instruction and sends a control signal to the execution unit. After receiving the control signal, the motor driver cuts off the power supply to the motor, and the brake simultaneously starts the brake mechanism to lock the moving parts.

8. A hierarchical safety protection device for bridge cranes based on computer vision, characterized in that: The device comprises: A visual acquisition module is used to configure an intelligent visual camera on a mobile trolley of a bridge crane to collect image data streams around the hook. The intelligent visual camera is installed at a vertical downward angle to position the hook at the center of the field of view. The image data stream is sampled by a data acquisition unit, and noise filtering and dynamic background modeling are performed on the image data stream. A sliding median filter is used to eliminate outliers, and the pre-processed image data is written into a distributed database. A personnel identification module is used to read the preprocessed image data from the distributed database, and input the image data into a personnel activity identification model for personnel detection. The personnel activity identification model includes an input layer, a feature extraction layer, and an output layer. The feature extraction layer determines the optimal residual block configuration through an adaptive search algorithm, iteratively calculates the objective function values ​​of different residual block combinations on a training set, and selects the residual block combination with the optimal objective function value as the final network structure. Each residual block includes a dimensionality reduction convolution layer, a standard convolution layer, and a dimensionality increase convolution layer. The output layer outputs the personnel position coordinates through a fully connected layer. The safety alarm module is used to divide the warning area and the braking area into the hook position center based on the personnel position coordinates, the warning area is the outer annular area around the hook, and the braking area is the inner circular area around the hook. When a person is detected entering the warning area, the sound and light alarm device is triggered to send a warning signal. When a person is detected entering the braking area, a braking command is generated and sent to the execution unit of the crane controller through the industrial bus network. The execution unit includes a motor driver and a brake.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the computer vision-based bridge crane hierarchical safety protection method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the computer vision-based bridge crane graded safety protection method described in any one of claims 1 to 7 are implemented.

Images