Hoisting target unsafe state identification method, device and equipment and storage medium

Abstract

The invention relates to the technical field of building operation monitoring, and provides a hoisting target unsafe state identification method, system and device and a storage medium, and the method comprises the steps: capturing a hoisting object image in real time, carrying out the hoisting target detection of the hoisting object image through employing a target detection algorithm, and obtaining a hoisting target region; based on two adjacent frames of hoisting target areas, extracting optical flow features of the hoisting target areas by adopting a dense optical flow algorithm; performing dimension reduction processing on the optical flow features of the hoisting target area to obtain dimension-reduced optical flow features; on the basis of the distance between two adjacent frames of optical flow points and the optical flow features of the two adjacent frames after dimension reduction, feature association and target tracking are carried out through a Hungary algorithm combining the distance and cosine similarity; and identifying whether the hoisting target is in an unsafe state or not according to the angle change of the target point of the same tracking target between different frames. According to the invention, high-precision real-time tracking of the hoisting target is realized, and motion abnormity can be found in time.

Description

Technical Field

[0001] The present application relates to the technical field of construction operation monitoring, and in particular to a method, device, equipment and storage medium for identifying an unsafe state of a hoisted target. Background Art

[0002] Static load testing is a crucial step in building construction quality inspection. During static load testing, the hoisting target is easily affected by factors such as the operating method and the hoisting route. This can lead to irregular swinging of the hoisted object during aerial motion, seriously impacting the safety of the hoisting operation. Dynamic tracking and monitoring of hoisted objects can promptly identify and address potential safety hazards, preventing accidents. However, tracking targets on static load testing platforms differs from tracking other dynamic targets in several ways. First, the motion patterns of hoisted objects are complex. Traditional video image target tracking often employs generative tracking methods, such as template matching. This method uses an exhaustive search to find the region in the current frame that best matches the previous frame. Information from the current frame is then used to update the target template, and matching is performed using probabilistic mapping, as is the case with dynamic vehicle tracking. However, these methods require that the feature information of the dynamically moving target be relatively stable, and that the motion exhibits a certain regularity only in a two-dimensional plane. During aerial hoisting, however, objects not only experience left-right, up-and-down motion, but also three-dimensional irregular motion, such as rotation. Consequently, the surface feature information of the object can easily vary significantly from moment to moment. Traditional methods have poor tracking effects and cannot ensure real-time and effective tracking of targets.

[0003] Secondly, in video object tracking, deep learning research and applications have become increasingly numerous in recent years. For example, DeepSort, based on SORT, adds target motion and appearance feature information and uses a convolutional neural network (CNN) trained on a large-scale pedestrian dataset to improve accuracy and robustness. However, most of these discriminative tracking algorithms are based on the assumption of target state continuity, assuming that the target's motion and appearance do not change much in consecutive frames. However, at low frame rates or with limited transmission bandwidth, the target's motion displacement and shape vary significantly, resulting in a significant decrease in tracking effectiveness. Furthermore, in the complex background of building construction environments, when hoisted targets are lifted from the ground to high altitudes, the target scale exhibits certain variability at different stages due to the imaging principle of machine vision that objects appear larger near and smaller far away. These tracking algorithms also perform poorly.

[0004] Optical flow features are an important representation of the intensity changes of light on the surface of an object and are a representation of dynamic objects. Compared with the single surface features of an object or the information on the two-dimensional motion laws, optical flow feature information can better reflect the complex characteristic laws of an object. Based on the sampling of optical flow feature information, the optical flow feature method includes sparse optical flow features and dense optical flow features. Sparse optical flow features are collected for specific video feature points. The data volume is small and the calculation speed is fast, but the stability requirements of the feature points are high. Dense optical flow features are not limited to features and are based on sampling of all optical flow feature changes in the image. They are suitable for application scenarios with large dynamic changes, but the algorithm has a large amount of calculation and stability is difficult to guarantee. Summary of the Invention

[0005] In response to the above technical problems, the purpose of this application is to provide a method, system, device and storage medium for identifying unsafe states of hoisting targets, aiming to solve at least one of the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying an unsafe state of a hoisted object, the method comprising:

[0007] Capture the hoisted object image in real time and use the target detection algorithm to detect the hoisting target in the hoisted object image to obtain the hoisting target area;

[0008] Based on the target area of ​​two adjacent frames, the dense optical flow algorithm is used to extract the optical flow features of the target area.

[0009] Performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction;

[0010] Based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of two adjacent frames, feature association and target tracking are performed through the Hungarian algorithm combining distance and cosine similarity;

[0011] It is possible to identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

[0012] Furthermore, the step of extracting optical flow features of the hoisting target area using a dense optical flow algorithm based on the hoisting target area of ​​two adjacent frames includes:

[0013] Based on the target area of ​​the hoisting in two adjacent frames and the brightness constancy assumption and Taylor expansion, an optical flow characteristic equation is established, and the optical flow information of the hoisting target area is solved by the least squares method; wherein the optical flow information includes the velocity vector of the optical flow along the X axis and the velocity vector of the optical flow along the Y axis;

[0014] The optical flow features of the hoisting target area are extracted based on the optical flow information of the hoisting target area; wherein the optical flow features include optical flow size and optical flow direction.

[0015] Furthermore, assuming that the optical flow features of the hoisting target area are a first optical flow feature matrix, the step of performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain the optical flow features after dimensionality reduction includes:

[0016] Each optical flow direction in the first optical flow feature matrix Convert to sine components and cosine components to generate the second optical flow feature matrix;

[0017] Normalizing each optical flow feature in the second optical flow feature matrix to have a mean of 0 and a normalization of 1, to obtain a standardized optical flow feature matrix;

[0018] The covariance of the feature matrix is ​​used to describe the correlation between features, and the covariance is used to perform feature decomposition to obtain the eigenvalues ​​and corresponding eigenvectors;

[0019] Arrange the eigenvectors in descending order of eigenvalue, and select the first p eigenvectors to form the projection matrix;

[0020] The normalized optical flow feature matrix is ​​projected onto the projection matrix to obtain the reduced-dimensional optical flow feature matrix.

[0021] Furthermore, the step of identifying whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames includes:

[0022] Based on the same target determined by the tracking algorithm, the target center point and the four edge endpoints of the upper, lower, left and right are selected as the target points, and the polar coordinate system is constructed based on the relative reference objects in the environment;

[0023] Determine the angular change of one or more target points in a polar coordinate system;

[0024] It is determined whether the hoisted target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

[0025] Furthermore, the image of the hanging object is captured by a fixed camera.

[0026] Furthermore, in the Hungarian algorithm combining distance and cosine similarity, the matching parameter is calculated by the following formula:

[0027] ;

[0028] Among them, S is the matching parameter, is the distance between the optical flow points of two adjacent frames, , B are the optical flow features of the two frames after dimensionality reduction.

[0029] In a second aspect, an embodiment of the present application provides a device for identifying an unsafe state of a hoisted object, the device comprising:

[0030] The capture module is used to capture the image of the hoisted object in real time and use the target detection algorithm to detect the hoisting target on the image of the hoisted object to obtain the hoisting target area;

[0031] An extraction module is used to extract the optical flow features of the hoisting target area based on the hoisting target area of ​​two adjacent frames using a dense optical flow algorithm;

[0032] A dimensionality reduction module is used to perform dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction;

[0033] The feature association and target tracking module is used to perform feature association and target tracking based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of the two adjacent frames, using the Hungarian algorithm that combines distance and cosine similarity;

[0034] The recognition module is used to identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

[0035] In a third aspect, an embodiment of the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for identifying an unsafe state of a hoisting target as described in any one of the above items are implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for identifying an unsafe state of a hoisted target as described in any one of the above items.

[0037] The embodiments of the present application have the following technical effects:

[0038] (1) In construction scenarios, hoisting operations may be affected by the environment, and objects may exhibit multi-angle motion changes. The same target in front and back may show different features in different frame images. Traditional feature matching only considers the similarity of a single feature and cannot accurately manage the same target between different time frames. To address this problem, the present invention uses a dense optical flow algorithm to quantify the displacement and rotation of the target point to adapt to the three-dimensional irregular swing characteristics of the hoisted object. By fusing distance and cosine similarity, the Hungarian algorithm combines optical flow feature similarity with spatial distance to improve the robustness of the correlation between front and back frame features under complex motion.

[0039] (2) Compared with sparse optical flow, dense optical flow extracts more comprehensive target feature information and has higher tracking accuracy, but it also greatly increases the computational complexity and complexity of the algorithm. To address this issue, the embodiment of the present application uses dense optical flow to capture the global optical flow characteristics of the hoisted target. The target detection algorithm locks the outline area of ​​the hoisted object and only extracts the optical flow of the target area to reduce redundant calculations. At the same time, the optical flow feature dimensionality reduction technology is introduced to balance the algorithm accuracy and real-time performance.

[0040] (3) In order to solve the problems of low frame rate and scale change during the movement of hoisted targets in construction operation scenes, the embodiment of the present application adopts a dense optical flow algorithm, that is, through the optical flow spatial consistency assumption and short-time neighborhood motion modeling, to solve the tracking drift of the target caused by the imaging of "near large and far small" and transmission bandwidth limitations.

[0041] In summary, the embodiments of the present application use a dense optical flow sparsification method combined with the Hungarian algorithm of distance and cosine similarity to achieve dynamic tracking of the irregular three-dimensional motion state of the crane hoisting target, solve the tracking failure problem of traditional tracking algorithms caused by target rotation, scale change and feature instability in low frame rate scenarios, and at the same time solve the problem of large amount of dense optical flow calculation, effectively improve the real-time tracking performance of dynamic objects, and achieve high-precision real-time tracking of hoisting targets. It can timely detect motion anomalies, thereby optimizing hoisting operation parameters and reducing collision risks, thereby ensuring construction safety, improving efficiency and ensuring compliance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 1 is a flow chart of a method for identifying an unsafe state of a hoisted target provided in an embodiment of the present application;

[0044] Figure 2 Schematic diagram of the structure of the device for identifying an unsafe state of a hoisted target provided in an embodiment of the present application;

[0045] Figure 3 It is a schematic block diagram of the structure of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0048] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as such, will not be interpreted in an idealized or overly formal sense.

[0049] like Figure 1 As shown, an embodiment of the present application provides a method for identifying an unsafe state of a hoisted target, the method comprising:

[0050] S1. Capture the image of the hoisted object in real time and use the target detection algorithm to detect the hoisting target on the image of the hoisted object to obtain the hoisting target area;

[0051] S2. Based on the target area of ​​two adjacent frames, the dense optical flow algorithm is used to extract the optical flow features of the target area;

[0052] S3, performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction;

[0053] S4, based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of the two adjacent frames, feature association and target tracking are performed through the Hungarian algorithm combining distance and cosine similarity;

[0054] S5. Identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

[0055] In step S1, specifically, taking the static load test as an example, a camera is fixed on the top of the crane control cab to ensure that the hoisted target is always in the center of the field of view. Thereafter, a target detection algorithm, such as the YOLOv11 model, is used to detect the hoisted target, and the bounding box regression algorithm based on deep learning quickly locks the contour area of ​​the hoisted target to obtain the hoisted target area. It should be noted that the purpose of using target detection in the embodiment of the present invention is to reduce redundant calculations during dense optical flow calculations, focus only on the target area, and avoid the inefficiency of full-image optical flow analysis. In addition, the deep learning model used is highly robust and can cope with the imaging differences of hoisted objects that are large near and small far away.

[0056] In step S2, the target area is cropped from the load image and its optical flow features are calculated. Step S2 extracts dense optical flow features only for the target area detected by the target detection algorithm, discarding background interference. The choice of dense optical flow helps the algorithm capture the three-dimensional motion of the target, reflecting irregular movements such as rotation and swing through global optical flow features. In static load tests, cropping the target area reduces pixel computation by over 80%, supporting real-time performance.

[0057] In step S3, dimensionality reduction and denoising are achieved by performing dimensionality reduction on the optical flow features, eliminating the overall offset interference of the optical flow features. This operation can also improve matching efficiency and reduce the complexity of subsequent algorithms by simplifying the feature data.

[0058] In step S4, it should be noted that the association of feature information is the core of tracking data in the previous and next time series. Considering the accuracy and real-time performance of the algorithm, the Hungarian algorithm is specifically used for matching. However, the basis of matching is how to calculate the correlation between the previous and next features. Traditional methods mainly include Euclidean distance, cosine similarity, etc. During the hoisting process of the static load test platform, due to the motion characteristics of the hoisted object itself, the target position will not jump. Therefore, considering only the distance or only the similarity between the optical flow feature information has certain limitations. Based on the above problems, the algorithm adds distance on the basis of cosine similarity to jointly constitute the matching parameters of the previous and next optical flows. In the Hungarian algorithm, the combination of optical flow feature similarity and spatial distance (such as position jump constraints) can improve the robustness of the association of previous and next frame features under complex motion.

[0059] In step S5, based on the angle change, it can be identified whether the hoisted object is in an unsafe state such as abnormal rotation or falling.

[0060] The embodiments of the present application have the following technical effects:

[0061] (1) In construction scenarios, hoisting operations may be affected by the environment, and objects may exhibit multi-angle motion changes. The same target in front and back may show different features in different frame images. Traditional feature matching only considers the similarity of a single feature and cannot accurately manage the same target between different time frames. To address this problem, the present invention uses a dense optical flow algorithm to quantify the displacement and rotation of the target point to adapt to the three-dimensional irregular swing characteristics of the hoisted object. By fusing distance and cosine similarity, the Hungarian algorithm combines optical flow feature similarity with spatial distance to improve the robustness of the correlation between front and back frame features under complex motion.

[0062] (2) Compared with sparse optical flow, dense optical flow extracts more comprehensive target feature information and has higher tracking accuracy, but it also greatly increases the computational complexity and complexity of the algorithm. To address this issue, the embodiment of the present application uses dense optical flow to capture the global optical flow characteristics of the hoisted target. The target detection algorithm locks the outline area of ​​the hoisted object and only extracts the optical flow of the target area to reduce redundant calculations. At the same time, the optical flow feature dimensionality reduction technology is introduced to balance the algorithm accuracy and real-time performance.

[0063] (3) In order to solve the problems of low frame rate and scale change during the movement of hoisted targets in construction operation scenes, the embodiment of the present application adopts a dense optical flow algorithm, that is, through the optical flow spatial consistency assumption and short-time neighborhood motion modeling, to solve the tracking drift of the target caused by the imaging of "near large and far small" and transmission bandwidth limitations.

[0064] In summary, the embodiments of the present application use a method of dense optical flow sparsification combined with the Hungarian algorithm of distance and cosine similarity to achieve dynamic tracking of the irregular three-dimensional motion state of the crane hoisting target, solve the tracking failure problem of traditional tracking algorithms caused by target rotation, scale change and feature instability in low frame rate scenarios, and at the same time solve the problem of large amount of dense optical flow calculation, effectively improve the real-time tracking performance of dynamic objects, and achieve high-precision real-time tracking of hoisting targets. It can timely detect motion anomalies, optimize hoisting operation parameters, and reduce collision risks, thereby ensuring construction safety, improving efficiency and ensuring compliance.

[0065] In one embodiment, the step of extracting optical flow features of the hoisting target area using a dense optical flow algorithm based on the hoisting target area in two adjacent frames includes:

[0066] Based on the target area of ​​the hoisting in two adjacent frames and the brightness constancy assumption and Taylor expansion, an optical flow characteristic equation is established, and the optical flow information of the hoisting target area is solved by the least squares method; wherein the optical flow information includes the velocity vector of the optical flow along the X axis and the velocity vector of the optical flow along the Y axis;

[0067] The optical flow features of the hoisting target area are extracted based on the optical flow information of the hoisting target area; wherein the optical flow features include optical flow size and optical flow direction.

[0068] In an embodiment of the present application, the optical flow information of the hoisting target area includes the optical flow information of each pixel in the hoisting target area, and the optical flow features of the hoisting target area include the optical flow features of each pixel. Specifically, optical flow refers to the instantaneous speed of the pixel motion of a spatial moving object on the observation imaging plane, which reflects the motion information of the object in the scene. In a time-based video image sequence, the brightness changes and correlations on the pixels of the object in the image are calculated using the matching relationship between the optical flow features of the previous frame image and the next frame image (i.e., two adjacent images) to obtain the object motion state between adjacent frames. The brightness change of each target point on the two-dimensional image plane is defined as an optical flow vector, and the displacement of the target point on the previous and next frame images can be reflected by the change trend of the optical flow of the previous and next images, that is, assuming that the time interval between frames is very small, and assuming that the brightness of the light source remains consistent. For hoisted objects, the displacement change of the object is relatively small in a very short period of time, and the light reflection intensity of the hoisted object will not show a step change, that is, the spatial consistency is also satisfied in the space represented by the optical flow, and the brightness of the image can be expressed as:

[0069] ;

[0070] Where I represents the image brightness, (x, y) represents the spatial coordinates of the pixel point on the image plane, x is the horizontal coordinate, and y is the vertical coordinate, which is used to determine the position of the pixel point in the two-dimensional image plane; t represents time and is used to identify different frames in a video image sequence.

[0071] According to Taylor expansion, it can be simplified to:

[0072] ;

[0073] set up are the velocity vectors of the optical flow along the X-axis and Y-axis respectively, and we get:

[0074] ;

[0075] When the distance between two frames is close to each other , that is, it will not change in the neighborhood, then u and v can be obtained by using the least squares method.

[0076] ;

[0077] in, To include The rotation matrix, T represents the transpose, for vector.

[0078] After calculating the optical flow information (u, v), the optical flow size and direction can be calculated according to the following formula:

[0079] ;

[0080] in, is the optical flow size, is the direction of optical flow.

[0081] It should be noted that when the current frame image is the first frame image, in order to effectively calculate the optical flow information map of the lifting target area, the original image is translated, and it is assumed that the optical flow change information is the brightness change value between two adjacent pixels of the original image.

[0082] Hoisting operations in construction scenarios can be affected by the environment, with objects exhibiting multi-angle motion changes. The same target in front and behind may appear with different features in different frames. Traditional feature matching only considers the similarity of a single feature and cannot accurately manage the same target across different frames. To address this issue, an embodiment of the present invention employs a dense optical flow algorithm to quantify the displacement and rotational motion of target points. Specifically, an optical flow feature vector model is established based on Taylor expansion and the least squares method to quantify the displacement and rotational motion of target points, adapting to the three-dimensional irregular swing characteristics of hoisted objects.

[0083] In one embodiment, assuming that the optical flow features of the hoisting target area are a first optical flow feature matrix, the step of performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain the optical flow features after dimensionality reduction includes:

[0084] Each optical flow direction in the first optical flow feature matrix Convert to sine components and cosine components to generate the second optical flow feature matrix;

[0085] Normalizing each optical flow feature in the second optical flow feature matrix to have a mean of 0 and a normalization of 1, to obtain a standardized optical flow feature matrix;

[0086] The covariance of the feature matrix is ​​used to describe the correlation between features, and the covariance is used to perform feature decomposition to obtain the eigenvalues ​​and corresponding eigenvectors;

[0087] Arrange the eigenvectors in descending order of eigenvalue, and select the first p eigenvectors to form the projection matrix;

[0088] The normalized optical flow feature matrix is ​​projected onto the projection matrix to obtain the reduced-dimensional optical flow feature matrix.

[0089] In the embodiment of the present application, specifically, the hoisting target area is The image block, then its optical flow feature matrix X is:

[0090] ;

[0091] Among them, each element Represents the optical flow feature vector (abbreviated as optical flow feature) at coordinate (i, j), , , specifically defined as:

[0092] ;

[0093] in, Indicates the optical flow size of coordinate (i, j), Represents the optical flow direction of coordinate (i, j).

[0094] Since optical flow information is two-dimensional, including the size and direction of optical flow, this data has not only numerical value but also direction, so it has stronger structured characteristics, and there may be more complex dependencies and dependencies between features. Since the direction of optical flow is an angle value ( arrive ), which can be converted into a two-dimensional vector. Specifically, for the optical flow direction , and express its real and imaginary parts as and , that is, each optical flow direction Converted into cosine components and sine components. This can maintain the continuity and periodicity of its directional information. Subsequently, each optical flow feature (the components of the optical flow magnitude and direction) is normalized to a mean of 0 and a standardization of 1. Among them, the expansion form of the first optical flow feature matrix X is: .

[0095] The second optical flow feature matrix for:

[0096] ;

[0097] Secondly, construct the projection matrix. Specifically, the standardized optical flow feature is a two-dimensional matrix, where each row or column represents a pixel point and each element is a feature vector (including the cosine component and sine component of the optical flow magnitude and direction). For example, if there is pixels, each pixel has k features, then the dimension of the feature matrix is Then, the covariance based on the feature matrix is ​​used to describe the correlation between features, and the covariance is used to perform feature decomposition to obtain the eigenvalues ​​and corresponding eigenvectors. The eigenvectors are sorted from large to small according to the eigenvalues, and the first p (0 <p< ) eigenvectors (principal components) form the projection matrix , whose dimensions are The formula for calculating covariance is: ; Where N is the number of pixels, N= , X is the standardized feature matrix.

[0098] Then the standardized feature matrix Projection to the projection matrix On the above, we get the optical flow feature matrix after dimensionality reduction , , whose dimensions are .

[0099] It should be noted that traditional optical flow information is divided into dense optical flow and sparse optical flow. Sparse optical flow is mainly used in scenes where the key features of the image are fixed and obvious, such as diagonal points or other significant feature images, and only the optical flow changes of the feature points are obtained. Dense optical flow, on the other hand, extracts optical flow features from all pixel points at the feature position. However, this method is more effective for complex dynamic environments, but it has a large amount of calculation and can easily affect the real-time performance of the system. In response to the above problems, the embodiment of the present application extracts all the optical flow feature information within the lifting target area and then performs feature dimensionality reduction to achieve data dimensionality reduction and denoising, that is, eliminates the overall offset interference of the optical flow features and highlights local dynamic features. This operation can also improve matching efficiency and reduce the complexity of subsequent algorithms through simplified feature data. It should be noted that the present application is different from traditional PCA data dimensionality reduction. In addition to size, the optical flow information also has direction. After the above processing, redundant information can be removed and the relative changes in direction and size can be retained.

[0100] In one embodiment, in the Hungarian algorithm combining distance and cosine similarity, the matching parameter is calculated using the following formula:

[0101] ;

[0102] Among them, S is the matching parameter, is the distance between the optical flow points of two adjacent frames. That is, the greater the position change before and after, the lower the matching similarity. , B are the optical flow features of the two frames after dimensionality reduction.

[0103] This formula combines cosine similarity (feature direction) with spatial distance (position jump constraint). This operation implements spatial distance constraint and avoids mismatches caused by low frame rate or occlusion. This enhances the algorithm's tracking robustness for dynamic targets, balancing feature similarity and motion continuity, and adapting to complex swinging scenarios.

[0104] In one embodiment, the image of the suspended object is captured by a fixed camera.

[0105] The embodiment of the present invention adopts a fixed camera position and a dynamic interception strategy for the target area to suppress the interference of complex building background and enhance the algorithm's focus on tracking a single target of the suspended object.

[0106] In one embodiment, the step of identifying whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames includes:

[0107] Based on the same target determined by the tracking algorithm, the target center point and the four edge endpoints of the upper, lower, left and right are selected as the target points, and the polar coordinate system is constructed based on the relative reference objects in the environment;

[0108] Determine the angular change of one or more target points in a polar coordinate system;

[0109] It is determined whether the hoisted target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

[0110] In the embodiment of the present application, it is possible to identify whether the hoisted target is in an unsafe state such as abnormal rotation or falling based on the angle change.

[0111] In the embodiments of this application, a polar coordinate system is constructed using a static load test platform as an example. Specifically, the polar coordinate system is based on the static load test platform, which is fixed in the scene, to ensure a stable reference system for motion analysis. First, the base of the static load test platform is selected as the polar coordinate origin. This location remains stationary during the lifting operation, eliminating the impact of camera perspective changes or background interference on motion analysis. Second, the polar coordinate system's orientation is defined: the vertical direction of the static load test platform is used as the polar axis, corresponding to an angle of 0°; the horizontal right direction is 90°, and the angle increases counterclockwise, forming a complete polar coordinate reference system. This orientation aligns with the crane operator's intuitive spatial perception and facilitates subsequent interpretation and decision-making of angle changes. Finally, four edge points (e.g., the upper, lower, left, and right vertices) are extracted from the original target detection hoisting target area (i.e., the hoisting target bounding box) and their Cartesian coordinates are converted to polar coordinates. These four points are selected to cover the extreme positions of the target's contour, fully reflecting the translation, rotation, and deformation characteristics of the hoisted object. By fixing the origin and direction, the polar coordinate system decomposes the complex motion of dynamic targets into quantitative parameters of radius (displacement from the origin) and angle (rotation state), providing a stable mathematical framework for subsequent motion state judgment.

[0112] After the polar coordinate system is established, dynamic angle detection is performed. For the dynamic target being tracked, the embodiment of the present application can complete the angle detection of the hoisted target by detecting the angle changes of the target point in the polar coordinates at different times and frames based on the tracked target.

[0113] Assume that in the Cartesian coordinate system, x is the horizontal coordinate and y is the vertical coordinate. In the polar coordinate system, is the polar diameter, is the polar angle, and the Cartesian coordinate of the center of the static load test platform base in the original image is ( ), the Cartesian coordinates of a certain edge point of the lifting target are ( ), whose polar coordinates ( ) is calculated as follows:

[0114] ;

[0115] ;

[0116] like , ;like ,but , make sure the angle value falls in the correct quadrant.

[0117] For two consecutive frames of images (time detection is ), record the angle value of the same edge point in the t-th frame and the t-1-th frame and , calculate the angle change:

[0118] ;

[0119] To avoid angle jumps (e.g., from 359° to 0°), the angle difference needs to be corrected periodically:

[0120] ;

[0121] The angle change directly represents the rotation amplitude of the target, such as =+0.1 It means 18° clockwise rotation. Therefore, the abnormality of this parameter can be used to detect whether the hoisted target has abnormal rotation.

[0122] The angle change rate reflects the rotation or falling speed of the target per unit time:

[0123] ;

[0124] in, is the angle change rate of a target edge point i, if is the inverse of the frame rate (e.g., if the frame rate is 30fps, = 1 / 30 second), the instantaneous angular velocity can be directly calculated from the angle difference between frames.

[0125] If the target falls with horizontal displacement, the angle of the edge point will decrease due to the height ( decrease) and changes rapidly, A sudden increase can trigger an early warning. Therefore, by checking whether the rate of change of the angle is abnormal, it is possible to detect whether the hoisted target is falling rapidly.

[0126] The above method can determine whether the hoisted object is in an unsafe state based on a single target point. Furthermore, the angle changes of multiple target points can be comprehensively considered. By comprehensively considering the angle changes of multiple target points, it is possible to more accurately determine whether the hoisted object is in an unsafe state.

[0127] like Figure 2 As shown, an embodiment of the present application provides a device for identifying an unsafe state of a hoisted target, the device comprising:

[0128] Capture module 1 is used to capture the image of the hanging object in real time and use the target detection algorithm to detect the hanging target of the hanging object image to obtain the hanging target area;

[0129] Extraction module 2, for extracting optical flow features of the hoisting target area based on the hoisting target area of ​​two adjacent frames using a dense optical flow algorithm;

[0130] Dimensionality reduction module 3, used for performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction;

[0131] Feature association and target tracking module 4, used to perform feature association and target tracking based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of the two adjacent frames by using the Hungarian algorithm combining distance and cosine similarity;

[0132] The identification module 5 is used to identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

[0133] In one embodiment, the extraction module 2 includes:

[0134] An optical flow information solving unit is configured to establish an optical flow characteristic equation based on the target hoisting area of ​​two adjacent frames and the brightness constancy assumption and Taylor expansion, and solve the optical flow information of the target hoisting area by the least squares method; wherein the optical flow information includes the velocity vector of the optical flow along the X-axis and the velocity vector of the optical flow along the Y-axis;

[0135] The extraction unit is used to extract the optical flow features of the hoisting target area based on the optical flow information of the hoisting target area; wherein the optical flow features include the optical flow size and the optical flow direction.

[0136] In one embodiment, assuming that the optical flow feature of the hoisting target area is a first optical flow feature matrix, the dimensionality reduction module 3 includes:

[0137] A conversion unit, configured to convert each optical flow direction in the first optical flow feature matrix Convert to sine components and cosine components to generate the second optical flow feature matrix;

[0138] a normalizing unit, configured to normalize each optical flow feature in the second optical flow feature matrix to have a mean of 0 and a normalization of 1, thereby obtaining a normalized optical flow feature matrix;

[0139] The decomposition unit is used to describe the correlation between features based on the covariance of the feature matrix, and use the covariance to perform feature decomposition to obtain eigenvalues ​​and corresponding eigenvectors;

[0140] The arrangement unit is used to arrange the eigenvectors in descending order of eigenvalues ​​and select the first p eigenvectors to form the projection matrix;

[0141] The projection unit is used to project the normalized optical flow feature matrix onto the projection matrix to obtain a dimensionally reduced optical flow feature matrix.

[0142] In one embodiment, in the Hungarian algorithm combining distance and cosine similarity, the matching parameter is calculated using the following formula:

[0143] ;

[0144] Among them, S is the matching parameter, is the distance between the optical flow points of two adjacent frames, , B are the optical flow features of the two frames after dimensionality reduction.

[0145] In one embodiment, the identification module 5 includes:

[0146] The polar coordinate system construction unit is used to select the target center point, the four edge endpoints of the upper, lower, left and right edges of the same target determined by the tracking algorithm as the target point, and use the relative reference objects in the environment as the basis to construct the polar coordinate system;

[0147] a determination unit, configured to determine an angular change of one or more target points in a polar coordinate system;

[0148] The identification unit is used to identify whether the hoisted target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

[0149] In one embodiment, the image of the suspended object is captured by a fixed camera.

[0150] Reference Figure 3 The embodiment of the present application also provides a computer device, the internal structure of which can be as follows Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a method for identifying an unsafe state of a hoisted target. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen. When the above computer program is executed by the processor, it realizes a method for identifying an unsafe state of a hoisted target, including the following steps: capturing the hoisted object image in real time and using the target detection algorithm to perform hoisting target detection on the hoisted object image to obtain the hoisting target area; based on the hoisting target area of ​​two adjacent frames, using the dense optical flow algorithm to extract the optical flow features of the hoisting target area; performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain the optical flow features after dimensionality reduction; based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of two adjacent frames, performing feature association and target tracking by combining the Hungarian algorithm of distance and cosine similarity; identifying whether the hoisted target is in an unsafe state according to the angle change of the target point of the same tracking target between different frames. Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0151] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for identifying an unsafe state of a hoisted target is implemented, comprising the following steps: capturing a hoisted object image in real time and performing hoisting target detection on the hoisted object image using a target detection algorithm to obtain a hoisting target area; extracting optical flow features of the hoisting target area based on the hoisting target area of ​​two adjacent frames using a dense optical flow algorithm; performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction; performing feature association and target tracking using a Hungarian algorithm combining distance and cosine similarity based on the distance between optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of two adjacent frames; and identifying whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

[0152] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0153] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0154] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying an unsafe state of a hoisted target, characterized in that: The method comprises: Capture the hoisted object image in real time and use the target detection algorithm to detect the hoisting target in the hoisted object image to obtain the hoisting target area; Based on the target area of ​​two adjacent frames, the dense optical flow algorithm is used to extract the optical flow features of the target area. Performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction; Based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of two adjacent frames, feature association and target tracking are performed through the Hungarian algorithm combining distance and cosine similarity; It is possible to identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

2. The method for identifying an unsafe state of a hoisted target according to claim 1, characterized in that: The step of extracting optical flow features of the hoisting target area using a dense optical flow algorithm based on the hoisting target area of ​​two adjacent frames includes: Based on the target area of ​​the hoisting in two adjacent frames and the brightness constancy assumption and Taylor expansion, an optical flow characteristic equation is established, and the optical flow information of the hoisting target area is solved by the least squares method; wherein the optical flow information includes the velocity vector of the optical flow along the X axis and the velocity vector of the optical flow along the Y axis; The optical flow features of the hoisting target area are extracted based on the optical flow information of the hoisting target area; wherein the optical flow features include optical flow size and optical flow direction.

3. The method for identifying an unsafe state of a hoisted object according to claim 2, characterized in that: Assuming that the optical flow features of the hoisting target area are a first optical flow feature matrix, the step of performing dimensionality reduction processing on the optical flow features of the hoisting target area to obtain the optical flow features after dimensionality reduction includes: Each optical flow direction in the first optical flow feature matrix Convert to sine components and cosine components to generate the second optical flow feature matrix; Normalizing each optical flow feature in the second optical flow feature matrix to have a mean of 0 and a normalization of 1, to obtain a standardized optical flow feature matrix; The covariance of the feature matrix is ​​used to describe the correlation between features, and the covariance is used to perform feature decomposition to obtain the eigenvalues ​​and corresponding eigenvectors; Arrange the eigenvectors in descending order of eigenvalue, and select the first p eigenvectors to form the projection matrix; The normalized optical flow feature matrix is ​​projected onto the projection matrix to obtain the reduced-dimensional optical flow feature matrix.

4. The method for identifying an unsafe state of a hoisted object according to claim 1, characterized in that: In the Hungarian algorithm combining distance and cosine similarity, the matching parameters are calculated using the following formula: ; Among them, S is the matching parameter, is the distance between the optical flow points of two adjacent frames, , B are the optical flow features of the two frames after dimensionality reduction.

5. The method for identifying an unsafe state of a hoisted object according to claim 1, characterized in that: The step of identifying whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames includes: Based on the same target determined by the tracking algorithm, the target center point and the four edge endpoints of the upper, lower, left and right are selected as the target points, and the polar coordinate system is constructed based on the relative reference objects in the environment; Determine the angular change of one or more target points in a polar coordinate system; It is determined whether the hoisted target is in an unsafe state based on the angle change of the one or more target points in the polar coordinate system.

6. The method for identifying an unsafe state of a hoisted object according to claim 1, characterized in that: The hanging object image is captured by a fixed camera.

7. A device for identifying an unsafe state of a hoisted target, characterized in that: The device comprises: The capture module is used to capture the image of the hoisted object in real time and use the target detection algorithm to detect the hoisting target on the image of the hoisted object to obtain the hoisting target area; An extraction module is used to extract the optical flow features of the hoisting target area based on the hoisting target area of ​​two adjacent frames using a dense optical flow algorithm; A dimensionality reduction module is used to perform dimensionality reduction processing on the optical flow features of the hoisting target area to obtain optical flow features after dimensionality reduction; The feature association and target tracking module is used to perform feature association and target tracking based on the distance between the optical flow points of two adjacent frames and the optical flow features after dimensionality reduction of the two adjacent frames, using the Hungarian algorithm that combines distance and cosine similarity; The recognition module is used to identify whether the hoisted target is in an unsafe state based on the angle change of the target point of the same tracking target between different frames.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method for identifying an unsafe state of a hoisted target according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying an unsafe state of a hoisted object according to any one of claims 1 to 6 are implemented.

Images