Hoisting target unsafe state identification method, device and equipment and storage medium

Abstract

The application relates to the technical field of construction operation monitoring, and provides a hoisting target unsafe state identification method, system, device and storage medium, the method comprises the following steps: hoisting object images are captured in real time, a target detection algorithm is used to detect the hoisting object images, and a hoisting target area is obtained; based on two adjacent hoisting target areas, a dense optical flow algorithm is used to extract optical flow features of the hoisting target area; the optical flow features of the hoisting target area are subjected to dimension reduction processing, and the dimension-reduced optical flow features are obtained; based on the distance between two adjacent optical flow points and the dimension-reduced optical flow features of two adjacent frames, feature association and target tracking are carried out through a Hungarian algorithm combining distance and cosine similarity; whether the hoisting target is in an unsafe state is identified according to the angle change of target points of the same tracking target between different frames. The application realizes high-precision real-time tracking of the hoisting target, and can discover motion abnormalities in a timely manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of construction operation monitoring, and in particular to a hoisting target unsafe state identification method, device, equipment and storage medium. BACKGROUND

[0002] The static load test is an important link of construction quality detection. In the hoisting process of the static load test, the hoisted object is extremely prone to irregular swinging in the air movement due to the influence of information such as operation method and hoisting route, which seriously affects the safety of hoisting operation. Through dynamic tracking and monitoring of the hoisted object, potential safety hazards can be found and handled in time to avoid safety accidents. However, there are the following differences between the hoisted target of the static load test platform and other dynamic target tracking. First, the motion law of the hoisted object has a certain complexity. In the traditional video image target tracking process, a generative tracking method is often used, such as a template matching method, which finds the most matched region in the current frame by using an exhaustive method, and updates the target template using the information of the current frame, and matches through probability mapping, such as dynamic vehicle tracking. However, the above method requires that the feature information of the dynamic motion target is relatively stable, and the motion is only the motion of the object in the two-dimensional plane and presents a certain regularity. However, when hoisting in the air, the object not only has left-right and up-down motion, but also has three-dimensional irregular motion such as rotation, and the surface feature information of the object is extremely prone to large difference changes at different times. The tracking effect of the traditional method is poor, and the target cannot be tracked in real time and effectively.

[0003] Secondly, in video target tracking, deep learning research and application have become more and more common in recent years, such as DeepSort which adds target motion and appearance feature information on the basis of SORT, and trains on a large-scale pedestrian dataset through a convolutional neural network (CNN), thereby improving accuracy and robustness. However, most of the above tracking algorithms are based on the continuity assumption of the target state, that is, the motion and appearance change of the target in the continuous frame is not large. However, in the case of low frame rate or transmission bandwidth limitation, the motion displacement and shape change of the target are large, which leads to a significant decline in tracking effect. In the complex background of building construction, when the hoisted target is from the ground to the high altitude, the target scale presents a certain abnormality in different stages due to the imaging law of near large and far small of machine vision, and the above tracking algorithm effect is also not good.

[0004] Optical flow feature is an important representation of the intensity change of light on the surface of an object, and is a representation of a dynamic object. Compared with single-surface feature or two-dimensional motion rule information of an object, optical flow feature information can better reflect the complex feature rule of the object. According to the sampling condition of optical flow feature information, optical flow feature method includes sparse optical flow feature and dense optical flow feature. Sparse optical flow feature is collected for specific video feature points, has small data volume and fast calculation speed, but has high requirement for stability of the feature points, while dense optical flow feature is not limited to features, is sampled for all optical flow feature changes of an image, is suitable for application scenarios with large dynamic change, but has large calculation amount and difficult to guarantee stability. SUMMARY

[0005] In view of the above technical problems, the purpose of the present application is to provide a hoisting target unsafe state recognition method, system, device and storage medium, aiming to solve at least one of the above technical problems.

[0006] In a first aspect, the embodiments of the present application provide a hoisting target unsafe state recognition method, which comprises:

[0007] capturing a hoisting object image in real time and performing hoisting target detection on the hoisting object image by using a target detection algorithm to obtain a hoisting target region;

[0008] based on the adjacent two frames of the hoisting target region, extracting optical flow features of the hoisting target region by using a dense optical flow algorithm;

[0009] performing dimension reduction processing on the optical flow features of the hoisting target region to obtain reduced dimension optical flow features;

[0010] based on the distance between the optical flow points of the adjacent two frames and the reduced dimension optical flow features of the adjacent two frames, performing feature association and target tracking by using a Hungarian algorithm combining distance and cosine similarity;

[0011] recognizing whether the hoisting target is in an unsafe state according to the angle change of the target points of the same tracking target between different frames.

[0012] Further, the step of extracting the optical flow features of the hoisting target region based on the adjacent two frames of the hoisting target region by using the dense optical flow algorithm comprises:

[0013] based on the adjacent two frames of the hoisting target region and the brightness constancy assumption and Taylor expansion, establishing an optical flow feature equation, and solving the optical flow information of the hoisting target region by using a least square method; wherein the optical flow information includes a velocity vector of the optical flow along the X axis and a velocity vector of the optical flow along the Y axis;

[0014] extracting the optical flow features of the hoisting target region based on the optical flow information of the hoisting target region; wherein the optical flow features include optical flow size and optical flow direction.

[0015] Further, the light flow feature of the hoisting target region is a first light flow feature matrix, and the step of performing dimension reduction on the light flow feature of the hoisting target region to obtain a reduced light flow feature includes:

[0016] Converting each light flow direction in the first light flow feature matrix into a sine component and a cosine component to generate a second light flow feature matrix;

[0017] Standardizing each light flow feature in the second light flow feature matrix to a form with a mean value of 0 and a standardization of 1 to obtain a standardized light flow feature matrix;

[0018] The covariance of the feature matrix describes the correlation between the features, and the feature decomposition is performed using the covariance to obtain eigenvalues and corresponding eigenvectors;

[0019] The eigenvectors are arranged in descending order of eigenvalues, and the first p eigenvectors are selected to form a projection matrix;

[0020] The standardized light flow feature matrix is projected onto the projection matrix to obtain a reduced light flow feature matrix.

[0021] Further, the step of identifying whether the hoisting target is in an unsafe state according to the angle change of the target point of the same tracking target between different frames includes:

[0022] Based on the same target determined by the tracking algorithm, the target center point, the upper and lower four edge end points are selected as the target point, and the polar coordinate system is constructed based on the relative reference object in the environment;

[0023] Determine the angle change of one or more target points in the polar coordinate system;

[0024] Identify whether the hoisting target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

[0025] Further, the hoisting image is captured by a fixed camera.

[0026] Further, in the Hungarian algorithm combined with distance and cosine similarity, the matching parameter is calculated by the following formula:

[0027] ;

[0028] Where S is the matching parameter, is the distance between the light flow points of two adjacent frames, , and B are the reduced light flow features of the previous and subsequent two frames, respectively.

[0029] ​In a second aspect, the embodiments of the present application provide a hoisting target unsafe state recognition device, the device comprising:

[0030] a capturing module configured to capture hoisting object images in real time and perform hoisting target detection on the hoisting object images using a target detection algorithm to obtain hoisting target regions;

[0031] an extraction module configured to extract optical flow features of the hoisting target regions based on the adjacent two hoisting target regions using a dense optical flow algorithm;

[0032] a dimension reduction module configured to perform dimension reduction processing on the optical flow features of the hoisting target regions to obtain reduced dimension optical flow features;

[0033] a feature correlation and target tracking module configured to perform feature correlation and target tracking based on distances between optical flow points of adjacent two frames and the reduced dimension optical flow features of adjacent two frames by using a Hungarian algorithm combining distance and cosine similarity;

[0034] a recognition module configured to recognize whether the hoisting target is in an unsafe state according to angle changes of target points of the same tracking target between different frames.

[0035] In a third aspect, the embodiments of the present application provide a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the hoisting target unsafe state recognition method according to any one of the above embodiments when executing the computer program.

[0036] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing a computer program, and the computer program implements the steps of the hoisting target unsafe state recognition method according to any one of the above embodiments when executed by a processor.

[0037] The embodiments of the present application have the following technical effects:

[0038] (1) In the construction scene, the hoisting operation may be affected by the environment, the object may present multi-angle motion changes, the same target before and after may present different features on different frame images, the traditional feature matching only considers the similarity of a single feature, and cannot accurately complete the management between the same target in different frames. To solve this problem, the present application uses a dense optical flow algorithm to quantify the displacement and rotational motion of target points, and adapts to the three-dimensional irregular swing characteristics of the hoisting object. By fusing distance and cosine similarity, the optical flow feature similarity and spatial distance are combined in the Hungarian algorithm to improve the robustness of feature correlation between front and rear frames under complex motion.

[0039] (2) Compared with sparse optical flow, dense optical flow is more comprehensive in extracting feature information of the target and has higher tracking accuracy, but it will greatly increase the calculation amount and complexity of the algorithm. To solve this problem, the embodiment of the present application adopts dense optical flow to capture the global optical flow features of the hoisting target, locks the hoisting object contour area through a target detection algorithm, and extracts only the optical flow of the target area to reduce redundant calculation. At the same time, the dimension reduction technology of optical flow features is introduced to balance the accuracy and real-time performance of the algorithm.

[0040] (3) In view of the problems of low frame rate and scale change in the movement process of the hoisting target in the construction scene, the embodiment of the present application adopts a dense optical flow algorithm, that is, through the spatial consistency assumption of optical flow and short-time neighborhood motion modeling, the tracking drift caused by the "near large and far small" imaging and the transmission bandwidth limitation of the target is solved.

[0041] In summary, the embodiment of the present application realizes the irregular three-dimensional motion state of the dynamic tracking of the hoisting target of the crane by the method of sparse dense optical flow, combines the Hungarian algorithm of distance and cosine similarity, solves the tracking failure problem caused by feature instability of the target rotation, scale change and low frame rate scene in the traditional tracking algorithm, and solves the problem of large calculation amount of dense optical flow, effectively improves the real-time tracking performance of the dynamic object, realizes high-precision real-time tracking of the hoisting target, can timely discover motion abnormalities, thereby optimizing the hoisting operation parameters and reducing the collision risk, thereby ensuring construction safety, improving efficiency and ensuring compliance. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flow diagram of the unsafe state recognition method of the hoisting target provided by the embodiment of the present application;

[0044] Figure 2 is a structural schematic diagram of the unsafe state recognition device of the hoisting target provided by the embodiment of the present application;

[0045] Figure 3 is a structural schematic block diagram of the computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0047] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising" and the like, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components and / or groups, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or wireless coupling. The terminology used herein also covers all ranges from and including one or more of the associated listed items as well as all combinations thereof.

[0048] It is to be understood that the terms so used are intended to encompass common and dictionary definitions of terms as well as the definitions in context of the relevant art. The definition of common and dictionary terms should be understood to encompass usual and ordinary meanings inclusive of implied meanings unless otherwise specifically defined herein.

[0049] As shown in Figure 1 The method provided by the embodiment of the present application comprises the following steps.

[0050] S1, capturing a hoisting object image in real time and performing hoisting object detection on the hoisting object image by using a target detection algorithm to obtain a hoisting target region;

[0051] S2, based on two adjacent hoisting target regions, extracting optical flow features of the hoisting target region by using a dense optical flow algorithm;

[0052] S3, performing dimension reduction processing on the optical flow features of the hoisting target region to obtain reduced dimension optical flow features;

[0053] S4, based on the distance between optical flow points of two adjacent frames and the reduced dimension optical flow features of two adjacent frames, performing feature association and target tracking by using a Hungarian algorithm combining distance and cosine similarity;

[0054] S5, identifying whether the hoisting target is in an unsafe state according to the angle change of the target points of the same tracking target between different frames.

[0055] In step S1, specifically, taking a static load test as an example, a crane control operation cab top fixed camera is used to ensure that the hoisting target is always located in the center of the field of view. Thereafter, a target detection algorithm such as a YOLOv11 model is used to detect the hoisting target, and a deep learning-based bounding box regression algorithm is used to quickly lock the contour area of the hoisting target to obtain the hoisting target area. It should be noted that the purpose of the target detection used in the embodiments of the present application is to reduce redundant calculation when calculating dense optical flow, and only focus on the target area to avoid the inefficiency of full image optical flow analysis. In addition, the deep learning model has strong robustness and can cope with the imaging difference between large near and small far.

[0056] In step S2, the hoisting target area is cropped from the hoisting object image to calculate the optical flow features of the hoisting target area. Step S2 only extracts dense optical flow features for the hoisting target area detected by the target detection algorithm, and discards background interference. The selection of dense optical flow is conducive to the algorithm to capture the three-dimensional motion of the hoisting target, that is, to reflect irregular motions such as rotation and swinging through global optical flow features. In the static load test, the hoisting target area cropping reduces the pixel calculation amount by more than 80%, supporting real-time performance.

[0057] In step S3, dimensionality reduction and denoising of data can be achieved by dimensionality reduction processing of the optical flow features, that is, eliminating the overall offset interference of the optical flow features. This operation can also improve the matching efficiency by reducing the complexity of the subsequent algorithm through the simplified feature data.

[0058] In step S4, it should be noted that feature information correlation is the tracking core of data in the front and rear time sequences, and considering the accuracy and real-time performance of the algorithm, the Hungarian algorithm is used for matching. However, how to calculate the correlation between the features before and after the matching, the traditional method mainly includes Euclidean distance, cosine similarity, etc. In the hoisting process of the static load test platform, due to the motion characteristics of the hoisted object, the target position will not jump, therefore, only considering the distance or only considering the similarity between the optical flow features has certain limitations. In view of the above problems, the distance is added on the basis of the cosine similarity to jointly constitute the matching parameters of the front and rear optical flows. In the Hungarian algorithm, the optical flow feature similarity and the spatial distance (such as position jump constraint) are combined to improve the robustness of the feature correlation between the front and rear frames under complex motion.

[0059] In step S5, based on the angle change, it can be identified whether the hoisting target is in an abnormal rotation, falling or other unsafe state.

[0060] The embodiments of the present application have the following technical effects:

[0061] (1) In the construction scene, the lifting operation may be affected by the environment, the object presents multi-angle motion change, the same target before and after appears different features on different frame images, the traditional feature matching only considers the similarity of a single feature, and cannot accurately complete the management between the same target in different frames. To solve this problem, the present application uses a dense optical flow algorithm to quantify the displacement and rotational motion of the target point, which is suitable for the three-dimensional irregular swinging characteristics of the lifted object. By fusing distance and cosine similarity, the optical flow feature similarity and spatial distance are combined in the Hungarian algorithm to improve the robustness of the feature correlation between the front and rear frames under complex motion.

[0062] (2) Compared with sparse optical flow, dense optical flow can extract more comprehensive feature information of the target and has higher tracking accuracy, but it will greatly increase the calculation amount and complexity of the algorithm. To solve this problem, the embodiment of the present application uses dense optical flow to capture the global optical flow features of the lifting target, locks the lifted object contour area through a target detection algorithm, and only extracts the optical flow of the target area to reduce redundant calculation. At the same time, the dimension reduction technology of optical flow features is introduced to balance the accuracy and real-time performance of the algorithm.

[0063] (3) In view of the low frame rate and scale change problem of the lifting target in the construction scene, the embodiment of the present application uses a dense optical flow algorithm, that is, through the space consistency assumption and short-time neighborhood motion modeling, the tracking drift caused by the "near large and far small" imaging and the transmission bandwidth limitation of the target is solved.

[0064] In summary, the embodiment of the present application realizes dynamic tracking of the irregular three-dimensional motion state of the lifting target of the crane by the method of sparse dense optical flow, and solves the tracking failure problem caused by feature instability of the traditional tracking algorithm under the condition of target rotation, scale change and low frame rate scene. At the same time, the problem of large calculation amount of dense optical flow is solved, the real-time tracking performance of dynamic objects is effectively improved, high-precision real-time tracking of the lifting target is realized, motion abnormalities can be discovered in time, lifting operation parameters can be optimized, and collision risks can be reduced, so as to ensure construction safety, improve efficiency and ensure compliance.

[0065] In an embodiment, the step of extracting the optical flow features of the lifting target region based on the dense optical flow algorithm includes:

[0066] Based on the adjacent two frames of the lifting target region and the brightness constancy assumption and Taylor expansion, the optical flow feature equation is established, and the optical flow information of the lifting target region is solved by the least square method; wherein the optical flow information includes the velocity vector of the optical flow along the X axis and the velocity vector of the optical flow along the Y axis;

[0067] The optical flow features of the lifting target region are extracted based on the optical flow information of the lifting target region; wherein the optical flow features include the optical flow size and the optical flow direction.

[0068] In the embodiment of the present application, the optical flow information of the hoisting target region includes the optical flow information of each pixel point of the hoisting target region, and the optical flow feature of the hoisting target region includes the optical flow feature of each pixel point. Specifically, the optical flow refers to the instantaneous velocity of the pixel movement of a spatial moving object on an observation imaging plane, which reflects the motion information of the object in a scene. In a time-based video image sequence, the brightness change and correlation on the object pixel in the image are calculated to obtain the motion state of the object between adjacent frames by using the matching relationship between the optical flow features of the previous frame image and the next frame image (i.e. adjacent two images). The brightness change of each target point on a two-dimensional image plane is defined as an optical flow vector, and then the displacement of the target point on the previous and next frame images can be reflected by the change trend of the optical flow on the previous and next images, that is, it is assumed that the time interval between frames is very small, and the brightness of the light source remains unchanged. For the hoisting object, the displacement change amount of the object is also relatively small in a very short time, and the light reflection intensity of the hoisting object will not appear a step change, that is, the space consistency is also met on the space represented by the optical flow, and then the brightness of the image can be represented as:

[0069] ;

[0070] wherein I represents the image brightness, (x, y) represents the spatial coordinates of the pixel point on the image plane, x is the horizontal direction coordinate, y is the vertical direction coordinate, which is used to determine the position of the pixel point in the two-dimensional image plane; t represents the time, which is used to identify different frames in the video image sequence.

[0071] According to Taylor expansion, it can be simplified as:

[0072] ;

[0073] Let be the velocity vector of the optical flow along the X-axis and the Y-axis respectively, and then:

[0074] ;

[0075] When the distance between two frames is relatively close , that is, there is no change in the neighborhood, then the least square method can be used to obtain u and v.

[0076] ;

[0077] wherein is a rotation matrix containing , T represents transposition, is a vector.

[0078] After the optical flow information (u, v) is calculated, the optical flow size and the optical flow direction can be calculated according to the following formula:

[0079] ;

[0080] wherein, is the optical flow size, is the optical flow direction.

[0081] It should be noted that when the current frame image is the first frame image, in order to effectively calculate the optical flow information map of the hoisting target region, the original image is translated, and it is assumed that the optical flow change information is the brightness change value between two adjacent pixels of the original image.

[0082] In view of the fact that the hoisting operation in the construction scene may be affected by the environment, the object presents multi-angle motion change, and the same target before and after presents different characteristics on different frame images, the traditional feature matching only considers the similarity of a single feature, and cannot accurately complete the management between the same target in different frames. In order to solve the problem, the embodiment of the application adopts a dense optical flow algorithm to quantize the displacement and rotational motion of the target point. Specifically, an optical flow feature vector model is established based on Taylor expansion and least square method to quantize the displacement and rotational motion of the target point, and adapt to the three-dimensional irregular swing characteristics of the hoisted object.

[0083] In an embodiment, assuming that the optical flow feature of the hoisting target region is a first optical flow feature matrix, the step of performing dimension reduction processing on the optical flow feature of the hoisting target region to obtain a reduced dimension optical flow feature includes:

[0084] Converting each optical flow direction in the first optical flow feature matrix into a sine component and a cosine component to generate a second optical flow feature matrix;

[0085] Standardizing each optical flow feature in the second optical flow feature matrix to a form with a mean value of 0 and a standardization of 1 to obtain a standardized optical flow feature matrix;

[0086] The covariance of the feature matrix is used to describe the correlation between the features, and the covariance is used for feature decomposition to obtain feature values and corresponding feature vectors;

[0087] The feature vectors are arranged in descending order of the feature values, and the first p feature vectors are selected to form a projection matrix;

[0088] The standardized optical flow feature matrix is projected onto the projection matrix to obtain a reduced dimension optical flow feature matrix.

[0089] In the embodiment of the application, specifically, assuming that the hoisting target region is an image block of , then the optical flow feature matrix X of the hoisting target region is:

[0090] ; ​

[0091] Each element This represents the optical flow feature vector (or simply optical flow feature) located at coordinates (i,j). , Specifically defined as:

[0092] ;

[0093] in, This represents the magnitude of the optical flow at coordinates (i,j). This indicates the direction of optical flow at coordinates (i,j).

[0094] Because optical flow information is two-dimensional, including both magnitude and direction, this data possesses not only numerical value but also directionality, thus exhibiting stronger structured characteristics. More complex dependencies and relationships may exist between features. Since the direction of optical flow is an angular value (…),… arrive This can be converted into a two-dimensional vector, specifically, for the optical flow direction. The real and imaginary parts are represented as follows: and That is, each optical flow direction It is converted into cosine and sine components. This preserves the continuity and periodicity of its directional information. Subsequently, each optical flow feature (components of optical flow magnitude and direction) is normalized to a form with a mean of 0 and a normalization to 1. The expanded form of the first optical flow feature matrix X is: .

[0095] Second optical flow feature matrix for:

[0096] ;

[0097] Secondly, the projection matrix is ​​constructed. Specifically, the standardized optical flow features are a two-dimensional matrix, where each row or column represents a pixel, and each element is a feature vector (including the cosine and sine components of the optical flow magnitude and direction). For example, if there are... If there are *k* pixels, and each pixel has *k* features, then the dimension of the feature matrix is... Then, the correlation between features is described using covariance based on the feature matrix, and eigenvalues ​​and corresponding eigenvectors are obtained through eigenvalue decomposition using covariance. The eigenvectors are then sorted in descending order of eigenvalues, and the top p (0, 1) eigenvalues ​​are selected. <p< The projection matrix is ​​formed by 10 eigenvectors (principal components). Its dimensions are The formula for calculating covariance is: Where N is the number of pixels, N= X is the normalized feature matrix.

[0098] Then the normalized feature matrix X is projected onto the projection matrix P to obtain the dimension-reduced optical flow feature matrix X' = PX. The dimension of X' is n x m. .

[0099] It should be noted that the traditional optical flow information is divided into dense optical flow and sparse optical flow. The sparse optical flow is mainly applied to the scene in which the key point feature of the image is fixed and obvious, such as the corner point or other salient feature image, and only the optical flow change of the feature point is obtained. The dense optical flow is to extract the optical flow feature of all pixel points of the feature position. However, this method is more effective for complex dynamic environment, but the calculation amount is large, which easily affects the real-time performance of the system. In view of the above problem, the feature dimension reduction is performed after all the optical flow feature information in the hoisting target region is extracted in the embodiment of the application, so as to realize the dimension reduction and denoising of data, that is, to eliminate the overall offset interference of the optical flow feature and highlight the local dynamic feature. The operation can also improve the matching efficiency, and reduce the complexity of the subsequent algorithm through the simplified feature data. It should be noted that the present application is different from the traditional PCA data dimension reduction, and the optical flow information has a direction in addition to the size. After the above processing, the redundant information can be removed, and the relative change of the direction and size is retained.

[0100] In an embodiment, the matching parameter in the Hungarian algorithm combined with the cosine similarity is calculated by the following formula:

[0101] ;

[0102] Wherein, S is the matching parameter, is the distance between the optical flow points of two adjacent frames, that is, the greater the position change, the lower the similarity of matching; , and B are the dimension-reduced optical flow features of the two frames.

[0103] The formula combines the cosine similarity (feature direction) and the spatial distance (position jump constraint). The operation can realize the spatial distance constraint, avoid the false matching caused by low frame rate or occlusion, and enhance the tracking robustness of the algorithm to dynamic targets, that is, the feature similarity and the motion continuity are considered, and the complex swing scene is adapted.

[0104] In an embodiment, the hoisted object image is captured by a fixed camera.

[0105] The embodiment of the application adopts the fixed camera position and target region dynamic interception strategy, suppresses the complex building background interference, and enhances the concentration of the algorithm on the single target tracking of the hoisted object.

[0106] ​​​In an embodiment, the step of identifying whether the hoisting target is in an unsafe state according to the angle change of the target points of the same tracking target between different frames comprises:

[0107] Based on the same target determined by the tracking algorithm, the target center point, the upper and lower four edge points, and the left and right four edge points are selected as the target points, a polar coordinate system is constructed based on the relative reference object in the environment;

[0108] The angle change of the one or more target points in the polar coordinate system is determined.

[0109] Based on the angle change of the one or more target points in the polar coordinate system, whether the hoisting target is in an unsafe state is identified.

[0110] In the embodiments of the present application, whether the hoisting target is in an unsafe state such as abnormal rotation or falling can be identified based on the angle change.

[0111] In the embodiments of the present application, the polar coordinate system is constructed, and the static load test platform is taken as an example. Specifically, the polar coordinate system is constructed based on the fixed static load test platform in the scene to ensure the stability of the reference system of motion analysis. First, the base of the static load test platform is selected as the polar coordinate origin. This position remains stationary during the hoisting operation and can eliminate the influence of camera angle changes or background interference on motion analysis. Second, the direction of the polar coordinate system is defined. The vertical direction of the static load test platform is defined as the polar axis, and the corresponding angle is 0°. The horizontal right direction is 90°, and the angle increases counterclockwise to form a complete polar coordinate reference system. This direction definition is consistent with the intuitive spatial perception of the crane operator, which facilitates the subsequent explanation and decision of the angle change. Finally, the upper, lower, left, and right four edge points (such as the left upper, right upper, left lower, and right lower vertices) are extracted from the hoisting target region (i.e., the hoisting target bounding box) of the original target detection, and the Cartesian coordinates are converted to polar coordinates. The selection of these four points covers the contour extreme positions of the target and can fully reflect the translation, rotation, and deformation characteristics of the hoisting object. By fixing the origin and direction, the polar coordinate system decomposes the complex motion of the dynamic target into quantitative parameters of radius (displacement from the origin) and angle (rotation state), providing a stable mathematical framework for subsequent motion state discrimination.

[0112] After the polar coordinate system is established, the angle dynamic detection is performed. For the tracked dynamic target, the angle change of the target points in the polar coordinates is detected at different times and different frames based on the determined target in the embodiments of the present application, and the angle detection of the hoisting target is completed.

[0113] In the Cartesian coordinate system, x is the horizontal coordinate and y is the vertical coordinate. In the polar coordinate system, is the polar radius, is the polar angle, and the Cartesian coordinates of the center of the static load test platform base in the original image are ( ), the Cartesian coordinates of the hoisting target edge point are (x, y) ), the polar coordinates (r, θ) are calculated as follows:

[0114] ;

[0115] ;

[0116] If , ; if , then , ensuring that the angle value falls in the correct quadrant.

[0117] For two consecutive frames of images (time detection is ), the angle values of the same edge point in the t-th frame and the t-1-th frame are recorded as and , and the angle change is calculated as:

[0118] ;

[0119] To avoid angle jumps (such as from 359° to 0°), the angle difference needs to be periodically corrected:

[0120] ;

[0121] The angle change directly represents the rotation amplitude of the target, such as = +0.1 represents a clockwise rotation of 18°, so through the anomaly of this parameter, the detection of whether the hoisting target has an abnormal rotation can be realized.

[0122] The angle change rate reflects the rotation or drop speed of the target per unit time:

[0123] ;

[0124] wherein is the angle change rate of the target edge point i, and if is the frame rate reciprocal (such as frame rate 30 fps, then = 1 / 30 seconds), the instantaneous angular velocity can be directly calculated from the inter-frame angle difference.

[0125] If the target drops with horizontal displacement, the angle of the edge point will change rapidly due to the decrease of height ( ), and sudden increase can trigger an early warning, so through whether the change rate of the angle is abnormal, the detection of whether the hoisting target is in rapid drop can be realized.

[0126] ​The above method can determine whether the hoisting target is in an unsafe state based on one target point. Further, the angle changes of multiple target points can be considered comprehensively, and the hoisting target can be more accurately determined to be in an unsafe state or not by comprehensively considering the angle changes of multiple target points.

[0127] As shown in Figure 2 The embodiment of the present application provides a hoisting target unsafe state recognition device, and the device comprises:

[0128] A capturing module 1 is configured to capture hoisting object images in real time and detect hoisting targets in the hoisting object images by using a target detection algorithm to obtain hoisting target regions.

[0129] An extraction module 2 is configured to extract optical flow features of the hoisting target regions by using a dense optical flow algorithm based on adjacent two hoisting target regions.

[0130] A dimension reduction module 3 is configured to perform dimension reduction processing on the optical flow features of the hoisting target regions to obtain reduced optical flow features.

[0131] A feature correlation and target tracking module 4 is configured to perform feature correlation and target tracking by combining a distance between adjacent two optical flow points and a cosine similarity Hungarian algorithm based on the distance and the reduced optical flow features of adjacent two frames.

[0132] A recognition module 5 is configured to recognize whether the hoisting target is in an unsafe state according to angle changes of target points of the same tracking target between different frames.

[0133] In an embodiment, the extraction module 2 comprises:

[0134] An optical flow information solving unit is configured to establish an optical flow feature equation based on adjacent two hoisting target regions and a brightness constancy assumption and Taylor expansion, and solve optical flow information of the hoisting target regions by using a least square method; wherein the optical flow information comprises a velocity vector of optical flow along an X axis and a velocity vector of optical flow along a Y axis.

[0135] An extraction unit is configured to extract optical flow features of the hoisting target regions based on the optical flow information of the hoisting target regions; wherein the optical flow features comprise optical flow size and optical flow direction.

[0136] In an embodiment, assuming that the optical flow features of the hoisting target regions are a first optical flow feature matrix, the dimension reduction module 3 comprises:

[0137] A conversion unit is configured to convert each optical flow direction in the first optical flow feature matrix into a sine component and a cosine component to generate a second optical flow feature matrix.

[0138] ​a standardization unit configured to normalize each optical flow feature in the second optical flow feature matrix to a form with a mean value of 0 and a standardization value of 1, to obtain a normalized optical flow feature matrix;

[0139] a decomposition unit configured to describe the correlation between features based on a covariance of the feature matrix, and perform feature decomposition on the covariance to obtain eigenvalues and corresponding eigenvectors;

[0140] an arrangement unit configured to arrange the eigenvectors in a descending order of the eigenvalues, and select the first p eigenvectors to form a projection matrix;

[0141] a projection unit configured to project the normalized optical flow feature matrix onto the projection matrix to obtain a reduced-dimension optical flow feature matrix.

[0142] In an embodiment, in the Hungarian algorithm combined with the cosine similarity, a matching parameter is calculated by the following formula:

[0143]

[0144] wherein S is the matching parameter, is a distance between optical flow points of two adjacent frames, and B are the reduced-dimension optical flow features of the two frames respectively.

[0145] In an embodiment, the identification module 5 comprises:

[0146] a polar coordinate system construction unit configured to select a target center point and four edge end points (up, down, left and right) of a same target determined by a tracking algorithm as target points, and construct a polar coordinate system based on a relative reference object in an environment;

[0147] a determination unit configured to determine an angle change of one or more target points in the polar coordinate system;

[0148] an identification unit configured to identify whether the hoisting target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

[0149] In an embodiment, the hoisting object image is captured by a fixed camera.

[0150] Reference Figure 3 The embodiments of the present application also provide a computer device, and an internal structure of the computer device can be as follows: Figure 3 ​As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is designed to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores operating devices, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data such as a hoisting target unsafe state identification method. The network interface of the computer device is used to communicate with the external terminal through the network connection. Further, the above-mentioned computer device can also be provided with input devices and display screens, etc. The above-mentioned computer program is executed by the processor to realize a hoisting target unsafe state identification method, which includes the following steps: real-time capturing of hoisting object image and hoisting target detection of hoisting object image by using target detection algorithm to obtain hoisting target area; based on the adjacent two frames of hoisting target area, the optical flow features of the hoisting target area are extracted by using the dense optical flow algorithm; the dimensionality reduction processing is performed on the optical flow features of the hoisting target area to obtain the dimensionality reduction of the optical flow features; based on the distance between the adjacent two frames of optical flow points and the dimensionality reduction of the adjacent two frames of optical flow features, the feature association and target tracking are performed by combining the distance with the cosine similarity Hungarian algorithm; according to the angle change of the target points of the same tracking target between different frames, whether the hoisting target is in an unsafe state is identified. Those skilled in the art can understand, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied.

[0151] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to realize a hoisting target unsafe state identification method, which includes the following steps: real-time capturing of hoisting object image and hoisting target detection of hoisting object image by using target detection algorithm to obtain hoisting target area; based on the adjacent two frames of hoisting target area, the optical flow features of the hoisting target area are extracted by using the dense optical flow algorithm; the dimensionality reduction processing is performed on the optical flow features of the hoisting target area to obtain the dimensionality reduction of the optical flow features; based on the distance between the adjacent two frames of optical flow points and the dimensionality reduction of the adjacent two frames of optical flow features, the feature association and target tracking are performed by combining the distance with the cosine similarity Hungarian algorithm; according to the angle change of the target points of the same tracking target between different frames, whether the hoisting target is in an unsafe state is identified.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application and in examples used herein, unless specifically stated otherwise, can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0153] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, device, article, or method that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, device, article, or method. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article, or method that includes the element.

[0154] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A hoisting target unsafe state identification method, characterized in that, The method comprises: real-time capturing of a hoisting object image and performing hoisting target detection on the hoisting object image by using a target detection algorithm to obtain a hoisting target region; based on the hoisting target regions of two adjacent frames, extracting optical flow features of the hoisting target region by using a dense optical flow algorithm; dimensionality reduction processing of the optical flow features of the hoisting target region to obtain reduced dimension optical flow features; based on the distance between optical flow points of two adjacent frames and the reduced dimension optical flow features of two adjacent frames, performing feature association and target tracking by using a Hungarian algorithm combining distance and cosine similarity; determining whether the hoisting target is in an unsafe state according to the angle change of the target points of the same tracking target between different frames; assuming that the optical flow features of the hoisting target region are a first optical flow feature matrix, the dimensionality reduction processing of the optical flow features of the hoisting target region to obtain reduced dimension optical flow features comprises: converting each optical flow direction in the first optical flow feature matrix into a sine component and a cosine component, generating a second optical flow feature matrix; standardizing each optical flow feature in the second optical flow feature matrix to a form with a mean value of 0 and a standardization of 1 to obtain a standardized optical flow feature matrix; based on the covariance of the feature matrix to describe the correlation between features, and performing feature decomposition by using the covariance to obtain eigenvalues and corresponding eigenvectors; arranging the eigenvectors in descending order of the eigenvalues, and selecting the first p eigenvectors to form a projection matrix; projecting the standardized optical flow feature matrix onto the projection matrix to obtain a reduced dimension optical flow feature matrix; in the Hungarian algorithm combining distance and cosine similarity, the matching parameters are calculated by the following formula: ; wherein S is a matching parameter, is a distance between two adjacent optical flow points, B are the optical flow features of the previous and next frames after dimension reduction, respectively. the step of determining whether the hoisting target is in an unsafe state according to the angle change of the target points of the same tracking target between different frames comprises: based on the same target determined by the tracking algorithm, selecting the target center point, the upper and lower four edge end points as the target points, and constructing a polar coordinate system based on the relative reference object in the environment; determining the angle change of one or more target points in the polar coordinate system; based on the angle change of the one or more target points in the polar coordinate system, determining whether the hoisting target is in an unsafe state.

2. The hoist target unsafe state recognition method according to claim 1, characterized in that, the step of extracting the optical flow features of the hoisting target region based on the hoisting target regions of two adjacent frames and the dense optical flow algorithm comprises: based on the hoisting target regions of two adjacent frames and the brightness constancy assumption and Taylor expansion, establishing an optical flow feature equation, and solving the optical flow information of the hoisting target region by using the least square method; wherein the optical flow information includes the velocity vector of the optical flow along the X axis and the velocity vector of the optical flow along the Y axis; extracting the optical flow features of the hoisting target region based on the optical flow information of the hoisting target region; wherein the optical flow features include the optical flow size and the optical flow direction.

3. The hoist target unsafe state recognition method according to claim 1, characterized by, The hoisting object image is captured by a fixed camera.

4. A hoisting target unsafe state recognition device characterized by comprising: The device is used to perform the hoisting target unsafe state recognition method in any one of claims 1-3, and the device comprises: a capturing module for real-time capturing of a hoisting object image and performing hoisting target detection on the hoisting object image by using a target detection algorithm to obtain a hoisting target region; an extraction module for extracting optical flow features of the hoisting target region based on the hoisting target regions of two adjacent frames by using a dense optical flow algorithm; a dimension reduction module, configured to perform dimension reduction on the optical flow features of the hoisting target region to obtain reduced dimension optical flow features; a feature association and target tracking module, configured to perform feature association and target tracking by combining a Hungarian algorithm of distance and cosine similarity based on distances between optical flow points of adjacent two frames and the reduced dimension optical flow features of the adjacent two frames; an identification module, configured to identify whether the hoisting target is in an unsafe state according to angle changes of target points of a same tracking target between different frames. 5.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-4 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the hoisting target unsafe state identification method in any one of claims 1 to 3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the hoisting target unsafe state identification method in any one of claims 1 to 3.

Images