Method for identifying illegal behaviors of crane scene

Abstract

The invention relates to a method for identifying illegal behaviors in a crane scene, which comprises the following steps of: acquiring at least two adjacent frames of first images acquired by a first camera and a second image acquired by a second camera; generating a first recognition result based on the first image, wherein the first recognition result comprises whether a crane of the crane is in a working state, whether a hook is in an inclined state and whether the hook is in an auxiliary state; generating a second identification result based on the second image, wherein the second identification result comprises whether a person enters the target risk area and whether the person wears a safety helmet; and judging an identification result and generating an alarm corresponding to the preset alarm condition. Through carrying out video snapshot on a crane scene, by virtue of related technical means such as artificial intelligence video behavior analysis and the like, production field safety visual management is realized, the effect of realizing intelligent identification and control on safety violation behaviors is achieved, the production field safety management capability is improved, and accident hidden dangers are eliminated in the bud by utilizing informatization and intelligent technologies.

Description

Technical Field

[0001] This application relates to the technical field of video capture algorithms, and particularly to a method for identifying illegal behaviors in a gantry crane scenario. Background Art

[0002] In the factory and mining areas of domestic manufacturing enterprises, due to the lack of appropriate hardware devices and related technical means, it is often difficult to capture videos of the safety risk areas during the gantry crane operation, and it is impossible to achieve the safety visualization management of the production site.

[0003] In the gantry crane scenario, it is necessary to identify safety violation behaviors during the gantry crane operation, including the state of the gantry crane hook, the state of the personnel entering the area, and the wearing situation of safety helmets. When it is identified that the hook is skewed, there are personnel entering below the gantry crane, and the personnel in the gantry crane working area do not wear safety helmets, an on-site alarm reminder is given in a timely manner, and at the same time, the alarm information is sent to the intelligent terminal management system platform in the control room for the supervision personnel to view.

[0004] Traditional safety monitoring systems mainly use fixed cameras for monitoring, but they cannot achieve intelligent identification and control of safety violation behaviors. Therefore, a method for identifying illegal behaviors in a gantry crane scenario is needed to solve this technical problem. Summary of the Invention

[0005] To solve or partially solve the problems existing in the related technologies, this application provides a method for identifying illegal behaviors in a gantry crane scenario, which identifies safety violation behaviors during the gantry crane operation based on a perimeter intrusion recognition algorithm, a gantry crane skew pulling and slanting recognition algorithm, and a safety helmet recognition algorithm.

[0006] The first aspect of this application provides a method for identifying illegal behaviors in a gantry crane scenario, and the method includes the following steps:

[0007] Obtain at least two adjacent first images collected by a first camera and a second image collected by a second camera. The first camera is set directly above the gantry crane hook, and the image acquisition area covers the gantry crane hook and the perimeter area. The image acquisition area of the second camera covers the target risk area of the gantry crane;

[0008] Based on the first images, generate a first recognition result, which includes at least one of whether the gantry crane is in a working state, whether the hook is in a skewed state, and whether the hook is in an auxiliary state. In the auxiliary state, the hook height is lower than the threshold height and there are personnel entering the perimeter area;

[0009] Based on the second image, generate a second recognition result, which includes whether there are personnel entering the target risk area and whether the personnel wear safety helmets;

[0010] Determine whether the first recognition result and / or the second recognition result meet at least one preset alarm condition. If so, generate an alarm corresponding to the preset alarm condition.

[0011] Among them, the preset alarm conditions include:

[0012] When the first recognition result is that the overhead crane is in the working state, the hook is in the skewed state, and the hook is not in the auxiliary state, trigger the alarm for abnormal skewing of the hook.

[0013] Among them, the preset alarm conditions include:

[0014] When the first recognition result is that the hook is not in the auxiliary state and there is a person entering the perimeter area, trigger the alarm for abnormal intrusion into the perimeter.

[0015] Among them, the preset alarm conditions include:

[0016] When the second recognition result is that there is a person entering the target risk area and the person is not wearing a safety helmet, trigger the safety helmet alarm.

[0017] Among them, whether the overhead crane of the crane is in the working state is judged by calculating the difference value between adjacent frames of the first image and the slope of the straight line fitted by pixel points.

[0018] Among them, judging whether the overhead crane is in the working state by calculating the difference value between adjacent frames includes:

[0019] Based on the 4 regions of the four-sided frame, perform frame difference to judge the similarity of the picture;

[0020] When the similarity of the 4-region picture exceeds the set threshold compared with the previous frame, determine that this frame of picture is a moving state picture;

[0021] Continuously judge N frames. When the difference values of all N frames exceed the threshold, determine that the overhead crane is in the working state.

[0022] Among them, judging whether the overhead crane is in the working state by the slope of the straight line fitted by pixel points includes:

[0023] Continuously record the pixel point size of the hook in the video stream data images of 8 frames, and obtain the pixel point fitting straight line by the least squares method;

[0024] Obtain the slope of the pixel point fitting straight line, and compare the slope with the preset slope threshold;

[0025] When the absolute value of the slope is greater than the preset slope threshold, determine that the overhead crane is in the working state.

[0026] Among them, whether the hook is in the skewed state is judged based on the position deviation angle of the hook. When the position deviation angle is greater than the hook skewing angle threshold, it is judged that the hook is in the skewed state. The calculation of the position deviation angle includes:

[0027] Calibrate the first image, and calculate the focal length of the camera based on the calibration data;

[0028] Based on the actual hook width, actual pixel width, and camera focal length parameters, the actual distance arc line position is calculated;

[0029] A calibration data template is established based on the proportional relationship, and the position deviation angle of the hook is calculated based on the inverse trigonometric function.

[0030] The proportional relationship is established as follows: the ratio of the pixel distance of the target from the center point in the horizontal direction to the pixel width of the picture is approximately equal to the ratio of the actual distance of the target from the center point in the horizontal direction to the actual width of the picture.

[0031] Among them, whether there is a person entering the target risk area is judged based on human posture detection. If there is someone, a rectangular cutout area is drawn and whether the person is wearing a helmet is judged based on the cutout detection. The rectangular cutout area is: with the nose as the center point, the distance between the two ears as the reference width, 1.5 times expanded horizontally, 2 times expanded vertically, and 0.2 times translated upward to draw the rectangular cutout area.

[0032] The technical solution provided by this application may have the following beneficial effects:

[0033] The present application provides a method for identifying illegal behaviors in crane scenes. By capturing videos of crane scenes and using related technical means such as artificial intelligence video behavior analysis, it is possible to achieve visual management of production site safety, achieve the effect of intelligent identification and control of safety violations, improve production site safety management capabilities, and use information and intelligent technologies to eliminate potential accidents in the bud.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0036] Figure 1 It is a flowchart of a method for identifying illegal behaviors in a crane scene shown in an embodiment of the present application;

[0037] Figure 2 It is a schematic diagram of the installation of the first camera of the method for identifying illegal behaviors in a overhead crane scene shown in an embodiment of the present application;

[0038] Figure 3It is a schematic diagram of the installation of the second camera in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application;

[0039] Figure 4 It is a schematic diagram of the hook skew recognition algorithm in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application;

[0040] Figure 5 It is a schematic diagram of the gantry crane motion state recognition algorithm in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application;

[0041] Figure 6 It is an auxiliary state schematic diagram in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application;

[0042] Figure 7 It is a schematic diagram of the perimeter intrusion recognition algorithm in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application;

[0043] Figure 8 It is a schematic diagram of the safety helmet recognition algorithm in the method for identifying illegal behaviors in the gantry crane scenario shown in the embodiments of the present application. Detailed implementation manners

[0044] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0045] It should be understood that although terms such as "first", "second", and "third" may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality" means two or more unless otherwise specifically defined.

[0046] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present application.

[0047] Unless otherwise clearly specified and defined, the terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0048] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings.

[0049] Embodiment 1

[0050] As Figure 1 shown in the method for identifying violations in the gantry crane scenario, which includes the following steps:

[0051] S1. Obtain at least two adjacent first images collected by the first camera and the second image collected by the second camera.

[0052] In this embodiment, first, obtain at least two consecutive frames of images containing a specific scene collected by the first camera as the first images, and obtain at least two consecutive frames of images containing a specific scene collected by the second camera as the second images.

[0053] In this embodiment, the first camera is set below the gantry of the crane and above the hook. The image acquisition angle of view of the first camera covers the entire crane hook and the perimeter area around the hook. The image collected by the first camera is used as the first image, and the first image contains the imaging scene of the crane hook.

[0054] It should be noted that the perimeter area is the standing area directly below the hook and within the rotation radius of the boom. During the rotation of the boom, any object within its rotation radius may be affected. Therefore, it is necessary to ensure that no personnel are allowed to enter the perimeter area under non-assisted conditions to prevent personal injury or property damage caused by improper operation or accidents.

[0055] Exemplarily, referring to Figure 2The set position of the first camera shown on the crane. At this field of view angle, the crane hook is located at the middle position of the first image frame, and the surrounding image area is the perimeter area.

[0056] In this embodiment, the second camera is set at a position where the entire hoisting process of the crane can be seen, and the image acquisition area needs to cover the entire target risk area of the crane. The image collected by the second camera is used as the second image, and the second image needs to include the complete working area of the hoisting scene including the crane.

[0057] The target risk area is characterized as the complete working area of the hoisting scene including the crane. There are safety risks in entering this area. Whether the hoisting operation is in progress or not, wearing a safety helmet is required when moving in this area.

[0058] Exemplarily, referring to Figure 3 The schematic diagram of the set position of the second camera shown. The second camera can be set on the side of the crane working area, at a position where the entire hoisting process of the crane can be seen and covering the entire target risk area of the crane.

[0059] S2. Generate a first recognition result based on the first image.

[0060] In this embodiment, after the first image is collected, image processing is performed on the image content in the first image to generate a first recognition result.

[0061] The first recognition result includes one or more recognition results, which include at least one of whether the hoisting of the crane is in a working state, whether the hook is in a skewed state, and whether the hook is in an auxiliary state.

[0062] In this embodiment, the main function of the first image is to judge various states of the hook and the hoisting. It should be noted that the working state of the hoisting includes horizontal movement (X, Y) and vertical movement (Z) of the hook. When any one of the horizontal movement and vertical movement is judged to occur, it can be judged that the hoisting is in a working state.

[0063] During the operation of the crane, when the hook is in a skewed state, the structure of the hook itself is unevenly stressed, which may lead to a decrease in its load-carrying capacity, making it unable to bear the original design load and increasing the risk of lifting accidents. The skewed hook may cause the center of gravity of the lifted object to shift, making the lifted object unstable and prone to dangerous situations such as swaying or slipping, resulting in casualties and equipment damage. The skewed state of the hook will affect the balance and stability during the lifting process. Once a lifting accident occurs, it may cause the lifted object to slip or tilt, resulting in casualties or injuries. Therefore, ensuring that the hook is in the correct position and posture is an important measure for the safe operation of the crane. Timely warning and adjustment when the hook is skewed can effectively reduce safety risks and ensure the safety of personnel and equipment.

[0064] It should be noted that the auxiliary state is specifically as follows: when the overhead crane is in the working state and the hook lowers an object, there will be a staff member holding it by hand and pulling the hook to a specific position before the hook continues to slowly lower; when the hook hangs a heavy object, the staff member will first pull the hook above the hanging rope of the heavy object, and then the hook will slowly lift it. As Figure 6 shown, set the hook height when there is personnel assistance. When the hook height is lower than this height and at the same time it is detected that there is a person within the perimeter area of the hook, it is considered that this is the staff assistance state at this time. In the auxiliary state, the skewing of the hook is autonomously generated and does not pose a safety hazard, so no warning is generated.

[0065] S3. Generate a second recognition result based on the second image. The second recognition result includes whether there is a person entering the target risk area and whether the person is wearing a safety helmet.

[0066] In this embodiment, after the second image is collected, the image content in the second image is processed to generate a second recognition result.

[0067] The second recognition result contains one or more recognition results, which include at least one of whether there is a person entering the target risk area and whether the person is wearing a safety helmet.

[0068] S4. Determine whether the first recognition result and / or the second recognition result meet at least one preset warning condition. If so, generate a warning corresponding to the preset warning condition.

[0069] It should be noted that multiple preset warning conditions are set based on the warning requirements, and the preset warning conditions are associated with the recognition results and warnings. When the preset warning conditions are met, the corresponding warnings are triggered.

[0070] When the first recognition result or the second recognition result meets a preset alarm condition, an alarm corresponding to the preset alarm condition is generated; when the first recognition result or the second recognition result meets multiple preset alarm conditions, alarms corresponding to the multiple preset alarm conditions are generated. When both the first recognition result and the second recognition result meet the preset alarm condition, alarms corresponding to the multiple preset alarm conditions are generated.

[0071] Embodiment 2

[0072] In this embodiment, based on the first embodiment, in step S2, the horizontal motion state of the overhead crane is judged by calculating the difference value between adjacent frames, and the vertical (Z) motion state of the hook is judged by detecting the size of the target pixel points of the hook.

[0073] The judgment of the horizontal motion (X, Y) state is as Figure 5 shown, and includes the following steps:

[0074] S211. Make a frame difference judgment on the similarity of the picture based on 4 regions of the four-sided frame;

[0075] S212. When the similarity of the pictures in the 4 regions exceeds the set threshold compared with the previous frame, determine that the picture of this frame is a motion state picture;

[0076] S213. Continuously judge N frames. When the difference values of the N frames all exceed the threshold, determine that the overhead crane is in the working state.

[0077] To filter out larger moving targets in the picture, the frame difference of 4 regions of the four-sided frame is used to judge the similarity of the picture. Only when the similarity of the picture is found to exceed the set threshold in all 4 regions compared with the previous frame, it is determined that the picture of this frame is a motion state picture. Continuously judge N frames. When the difference values of the N frames all exceed the threshold, it is determined that the overhead crane is moving (in the x, y directions). The threshold here is set through a configuration file and can be modified by the user according to the actual situation.

[0078] Judgment of the vertical (Z) motion state of the hook:

[0079] The motion state of the hook is judged by detecting the size of the target pixel points of the hook. When the hook is lowered, the pixel point size gradually becomes smaller. When the hook is raised, the pixel point size gradually increases. To prevent the influence of the size change of the frame caused by the jitter problem detected by the target detection algorithm, the size of 8 consecutive frames is recorded. The best fitting straight line equation is found by the least squares method. When the absolute value of the slope of the fitting straight line exceeds the set threshold, it is determined to be in motion, and the positive or negative of the slope determines whether it is rising or falling.

[0080] Embodiment 3

[0081] In this embodiment, based on the first embodiment, in step S2, it is determined whether the hook is skewed by setting the maximum allowable skewed angle of the hook. Based on AI video frame analysis, the position of the hook is detected through the yolov8 object detection algorithm, and then, with the center point of the target rectangular frame as the reference, the deviation angle of this center point in the vertical direction is calculated. If this angle exceeds the threshold set by the user, it is considered that the hook is skewed; otherwise, it is determined that the hook is not skewed. To prevent the jitter of the detection target, which is a rectangular frame, from affecting the result, the user can set that the hook is skewed for 3 consecutive frames before finally determining that it is skewed.

[0082] The calculation of the position deviation angle includes the following steps:

[0083] S221. Calibrate the first image and calculate the camera focal length based on the calibration data;

[0084] S222. Calculate the position of the actual distance radian line based on the actual width of the hook, the actual pixel width, and the camera focal length parameters;

[0085] S223. Establish a calibration data template based on the proportional relationship and calculate the position deviation angle of the hook using the inverse trigonometric function.

[0086] The calculation of the hook tilt angle is mainly obtained by using the principle of visual calibration technology and trigonometric functions. First, before calculation, in order to improve accuracy, calibration technology is used to calibrate the data as a reference image. When calibrating, a reference target with a size similar to the actual circumscribed rectangular frame of the hook is selected for calibration. The main calibration parameters are the width of the reference target, the pixel width in the synchronous camera frame, and the actual distance from the camera (stand directly below the camera to calibrate the data during calibration). Then, the camera focal length is calculated using the following formula.

[0087] The distance from the target to the camera = (actual width of the target × camera focal length) ÷ pixel width of the target;

[0088] After obtaining the camera focal length parameters, substituting the actual width of the target object and the actual pixel width can determine the position of the actual distance radian line.

[0089] After obtaining the radian line, next, calculate the horizontal distance. The ratio of the pixel distance of the target from the horizontal center point to the pixel width of the frame is approximately equal to the ratio of the actual distance of the target from the horizontal center point to the actual width of the frame. And this ratio coefficient changes with the height. At this time, a calibration data template is established according to this proportional relationship, and the resolution of the minimum radian line in the template is set to 0.2m. Finally, using the inverse trigonometric function, the tilt angle can be obtained.

[0090] The detection of personnel intrusion in the perimeter area is different from traditional human detection algorithms. Due to the camera angle, the human detection here is the state of a person seen from a top-down perspective (including the head (wearing a safety helmet or not), shoulders, and a human body with a certain inclination angle). Therefore, a specific dataset is required for training.

[0091] Example 4

[0092] In this embodiment, based on the first embodiment, in step S3, the safety helmet wearing detection includes the following steps:

[0093] S311. Perform human pose detection based on the second image to determine whether a person has entered the target risk area;

[0094] S312. If someone enters, expand a rectangular box centered on the head key point, and crop the image to obtain the detection image;

[0095] S313. Perform safety helmet detection on the detection image to determine whether a safety helmet is detected.

[0096] In the detection of safety helmet wearing, considering the complex factory environment, the environmental noise is reduced. An algorithm is designed through the idea of local magnification by cropping. First, detect the personnel in the picture, confirm the approximate position of the head according to the pose key points, then crop this part, and then detect the safety helmet on the cropped picture. On the one hand, it can reduce the environmental noise, and on the other hand, it can also filter out the situation where a safety helmet is brought close to the head but not worn. The rectangular cropping area is: with the nose as the center point, the distance between the two ears as the reference width, expand 1.5 times horizontally, expand 2 times vertically, and translate 0.2 times upward to draw the rectangular cropping area.

[0097] Example 5

[0098] In this embodiment, based on the first embodiment, step S4 specifically includes three types of alarms, including abnormal hook skew alarm, abnormal perimeter intrusion alarm, and safety helmet alarm.

[0099] Specifically, when the first recognition result is that the overhead crane is in the working state, the hook is in the skewed state, and the hook is not in the auxiliary state, an abnormal hook skew alarm is triggered.

[0100] Such as Figure 4 The abnormal hook skew alarm judgment shown includes the following steps:

[0101] S411. Based on the first image, determine whether the overhead crane is in the working state. If it is in the working state, determine whether the hook is skewed;

[0102] S412. If the hook is skewed, determine whether it is in the auxiliary state. If it is not in the auxiliary state, determine that the hook is abnormally skewed.

[0103] When the first recognition result is that the hook is not in the auxiliary state and there is a person entering the perimeter area, a perimeter abnormal intrusion alarm is triggered.

[0104] As Figure 7 shown in the perimeter area personnel intrusion alarm judgment, includes the following steps:

[0105] S421. Based on the first image, judge whether the overhead crane is in the working state. If it is in the working state, judge whether there is a person entering the perimeter area;

[0106] S422. If there is a person entering, judge whether it is in the auxiliary state. If it is not in the auxiliary state, judge the perimeter area personnel intrusion.

[0107] When the second recognition result is that there is a person entering the target risk area and the person is not wearing a safety helmet, a safety helmet alarm is triggered.

[0108] As Figure 8 shown in the safety helmet alarm judgment, includes the following steps:

[0109] S431. Based on the second image, judge whether there is a person entering the target risk area;

[0110] S432. If there is a person entering, judge whether the person is wearing a safety helmet;

[0111] S433. If no safety helmet is detected, judge that the safety helmet wearing is abnormal.

[0112] Embodiment Six

[0113] As Figure 1 shown in the recognition method of overhead crane scene violation behavior, includes the following steps:

[0114] S1. Obtain at least two adjacent first images collected by the first camera and the second image collected by the second camera.

[0115] Installation position of the first camera: directly below the overhead crane, directly above the hook, vertically shooting with the hook in the center of the picture, and the image acquisition area covering the crane hook and the perimeter area; Association algorithm: hook skew detection algorithm, perimeter area personnel detection algorithm.

[0116] Installation position of the second camera: from the side wall that can see the entire working area of the overhead crane to a height, and the image acquisition area covering the target risk area of the crane; Association algorithm: safety helmet detection algorithm.

[0117] S2. Based on the first image, generate the first recognition result.

[0118] The first recognition result includes at least one of whether the overhead crane of the crane is in the working state, whether the hook is in the skewed state, and whether the hook is in the auxiliary state.

[0119] The working state of the overhead crane is (horizontal movement (X, Y), vertical movement of the hook (Z)). For the horizontal movement of the overhead crane, it is determined whether there is movement by calculating the difference value between adjacent frames.

[0120] The judgment of the horizontal movement (X, Y) state is as Figure 5 shown. To filter out large moving targets in the picture, frame differences are made in 4 regions of the four-sided frame to judge the similarity of the picture. Only when the similarity of the picture is found to exceed the set threshold in all 4 regions compared with the previous frame, this frame of picture is determined to be a moving state picture. Continuously judge N frames. When the difference values of all N frames exceed the threshold, it is determined that the overhead crane is moving (in the x and y directions). The threshold here is set through a configuration file and can be modified by the user according to the actual situation.

[0121] The judgment of the vertical (Z) movement state of the hook:

[0122] The movement state of the hook is judged by detecting the size of the target pixel points of the hook. When the hook is lowered, the pixel point size gradually becomes smaller. When the hook is raised, the pixel point size gradually increases. To prevent the change in the size of the frame caused by the jitter problem detected by the target detection algorithm, the size is continuously recorded for 8 frames. The best fitting straight line equation is found by the least squares method. When the absolute value of the slope of the fitting straight line exceeds the set threshold, it is determined to be moving, and the positive or negative of the slope determines whether it is rising or falling.

[0123] Whether the hook is in the skewed state is judged based on the hook skewness detection algorithm as Figure 4 shown. The skewness detection of the overhead crane is judged by setting the maximum allowable skewed angle of the hook to determine whether the hook is skewed. For AI video picture analysis, first, the position of the hook is detected by the yolov8 target detection algorithm, and then, based on the center point of the target rectangle frame, the deviation angle of this center point in the vertical direction is calculated. If this angle exceeds the threshold set by the user, it is considered that the hook is skewed; otherwise, it is determined that the hook is not skewed. To prevent the influence of the jitter of the detected target being a rectangle frame, the user can set that it is skewed continuously for 3 frames before finally determining the skewness.

[0124] Angle calculation method:

[0125] The calculation of the hook tilt angle is mainly obtained by using visual calibration technology and the principle of trigonometric functions. First, before the calculation, in order to improve the accuracy, the calibration technology is used to calibrate the data as a reference image. When calibrating, a reference target with a size similar to the actual circumscribed rectangle of the hook is selected for calibration. The main calibration parameters include the width of the reference target, the pixel width in the synchronous camera image, and the actual distance from the camera (stand directly below the camera to calibrate the data when calibrating). Then, the camera focal length is calculated using the following formula.

[0126] The distance from the target object to the camera = (actual width of the target object × camera focal length) ÷ pixel width of the target object;

[0127] After obtaining the camera focal length parameters, substituting the actual width of the target object and the actual pixel width can determine the position of the actual distance radian line.

[0128] After obtaining the radian line, next, calculate the horizontal distance. The ratio of the pixel distance of the target from the horizontal center point to the pixel width of the image is approximately equal to the ratio of the actual distance of the target from the horizontal center point to the actual width of the image. And this ratio coefficient changes with the height. At this time, a calibration data template is established according to this proportional relationship, and the resolution of the minimum radian line in the template is set to 0.2m. Finally, using the inverse trigonometric function, the tilt angle can be obtained.

[0129] In the auxiliary state, the hook height is lower than the threshold height and there is a person entering the perimeter area. The specific auxiliary state is as follows: when the overhead crane is in the working state, when the hook lowers an object, there will be a staff member holding it by hand and pulling the hook to a specific position before the hook continues to lower slowly; when the hook is hanging a heavy object, the staff member will first pull the hook above the hanging rope of the heavy object and then the hook will slowly lift it. As Figure 6 shown, set the hook height during personnel assistance. When the hook height is lower than this height and at the same time a person is detected within the perimeter area of the hook, it is considered that this is the staff assistance state at this time.

[0130] The algorithm for monitoring the intrusion of personnel in the perimeter area is as Figure 7 shown. The working state detection of the overhead crane and the personnel assistance state detection are the same as the working state detection and personnel assistance state detection algorithms in the above-mentioned entire process of detecting the hook skew of the overhead crane. The only difference is the detection of personnel below the overhead crane. However, different from the traditional humanoid detection algorithm, due to the camera angle, the humanoid detection here is the state of a person seen from the top-down perspective (including the head (wearing a safety helmet or not), shoulders, and a human body with a certain tilt angle), so a specific dataset is required for training.

[0131] S3. Generate a second recognition result based on the second image. The second recognition result includes whether there is a person entering the target risk area and whether the person is wearing a safety helmet.

[0132] The safety helmet detection algorithm is as follows Figure 8 As shown, in the detection of whether a safety helmet is worn, considering the complex factory environment, environmental noise is reduced. An algorithm is designed based on the idea of cropping and locally magnifying. First, the personnel in the picture are detected, and the approximate position of the head is confirmed according to the pose key points. Then, this part is cropped, and the safety helmet is detected on the cropped picture. On the one hand, it can reduce environmental noise, and on the other hand, it can also filter out the situation where the safety helmet is brought close to the head but not worn. The rectangular cropping area is: taking the nose as the center point, the distance between the two ears as the reference width, expanding 1.5 times horizontally, expanding 2 times vertically, and translating 0.2 times upward to draw the rectangular cropping area.

[0133] S4. Determine whether the first recognition result and / or the second recognition result meet at least one preset warning condition. If so, generate a warning corresponding to the preset warning condition.

[0134] When the first recognition result is that the overhead crane is in the working state, the hook is in the skew state, and the hook is not in the auxiliary state, trigger the warning of abnormal hook skew.

[0135] When the first recognition result is that the hook is not in the auxiliary state and there is a person entering the perimeter area, trigger the warning of abnormal perimeter intrusion.

[0136] When the second recognition result is that there is a person entering the target risk area and the person is not wearing a safety helmet, trigger the safety helmet warning.

[0137] Finally, it should also be noted that in this article, relationships such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms including, containing or any other variant are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0138] The unit described as a separate component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying illegal behaviors in a crane scene, characterized in that: The method comprises the following steps: Acquire at least two adjacent first image frames captured by a first camera and a second image captured by a second camera, wherein the first camera is arranged directly above a crane hook and an image acquisition area covers the crane hook and a perimeter area, and the image acquisition area of ​​the second camera covers a target risk area of ​​the crane; Based on the first image, a first recognition result is generated, wherein the first recognition result includes at least one of whether the crane's overhead crane is in a working state, whether the hook is in a skewed state, and whether the hook is in an auxiliary state, wherein the height of the hook is lower than a threshold height and a person enters the enclosure area; Based on the second image, generating a second recognition result, the second recognition result including whether a person has entered the target risk area and whether the person is wearing a safety helmet; Determine whether the first recognition result and / or the second recognition result meets at least one preset alarm condition, and if so, generate an alarm corresponding to the preset alarm condition.

2. The method for identifying illegal behaviors in a crane scene according to claim 1, characterized in that: The preset alarm conditions include: When the first recognition result is that the traveling crane is in a working state, the hook is in a skewed state, and the hook is not in an auxiliary state, an abnormal skew alarm of the hook is triggered.

3. The method for identifying illegal behaviors in a crane scene according to claim 1, characterized in that: The preset alarm conditions include: When the first recognition result is that the hook is not in the auxiliary state and a person has entered the enclosure area, an abnormal enclosure intrusion alarm is triggered.

4. The method for identifying illegal behaviors in a crane scene according to claim 1, characterized in that: The preset alarm conditions include: When the second recognition result is that a person has entered the target risk area and the person is not wearing a safety helmet, a safety helmet alarm is triggered.

5. The method for identifying illegal behaviors in a crane scene according to claim 1 is characterized in that: Whether the crane's overhead crane is in working condition is determined by calculating the difference between adjacent frames of the first image and the slope of the pixel point fitting line.

6. The method for identifying illegal behaviors in a crane scene according to claim 5 is characterized in that: The step of determining whether the crane is in a working state by calculating the difference value between adjacent frames includes: Based on the four areas of the four borders, the frame difference is used to determine the similarity of the pictures; When the similarity between the four regions and the previous frame exceeds the set threshold, the current frame is determined to be a motion state frame; N frames are judged continuously, and when the difference values ​​of the N frames all exceed a threshold, it is determined that the overhead crane is in a working state.

7. The method for identifying illegal behaviors in a crane scene according to claim 5 is characterized in that: The step of judging whether the crane is in working state by fitting the slope of the straight line of the pixel points comprises: The pixel size of the hook in the video stream data image of 8 frames is continuously recorded, and the pixel fitting straight line is obtained by the least square method; Obtaining the slope of the pixel point fitting line, and comparing the slope with a preset slope threshold; When the absolute value of the slope is greater than the preset slope threshold, it is determined that the overhead crane is in a working state.

8. The method for identifying illegal behaviors in a crane scene according to claim 1 is characterized in that: Whether the hook is in a skewed state is determined based on the position deviation angle of the hook. When the position deviation angle is greater than a hook skew angle threshold, the hook is determined to be in a skewed state. The calculation of the position deviation angle includes: Calibrate the first image, and calculate the focal length of the camera based on the calibration data; Based on the actual hook width, actual pixel width, and camera focal length parameters, the actual distance arc line position is calculated; A calibration data template is established based on the proportional relationship, and the position deviation angle of the hook is calculated based on the inverse trigonometric function.

9. The method for identifying illegal behaviors in a crane scene according to claim 8 is characterized in that: The proportional relationship is established as the ratio of the pixel distance of the target from the center point in the horizontal direction to the pixel width of the picture is approximately equal to the ratio of the actual distance of the target from the center point in the horizontal direction to the actual width of the picture.

10. The method for identifying illegal behaviors in a crane scene according to claim 1 is characterized in that: Whether there is a person entering the target risk area is judged based on human posture detection. If there is someone, a rectangular cutout area is drawn and based on the cutout detection, it is judged whether the person is wearing a safety helmet. The rectangular cutout area is: with the nose as the center point, the distance between the two ears as the reference width, the horizontal expansion is 1.5 times, the vertical expansion is 2 times, and the upward translation is 0.2 times. The rectangular cutout area is drawn.

Images