Hoisting object space trajectory video tracking and identification method based on multi-source data fusion

Abstract

The application discloses a hoisting object space trajectory video tracking and identification method based on multi-source data fusion, relates to the technical field of machine vision monitoring, and comprises the following steps: acquiring preloaded tower crane physical boundary constants, synchronously collecting multi-source data in a tower crane operation process, performing kinematics calculation on the multi-source data based on the physical boundary constants, obtaining a relative vertical displacement amount, three-dimensional absolute space coordinates of a luffing trolley center point, absolute coordinates of a camera optical center in a global three-dimensional coordinate system and a three-dimensional rotation matrix, performing edge detection on a color video frame matrix after time domain alignment, extracting a steel wire candidate rope outline to generate a binary edge pixel matrix, and mapping collinear edges to a polar coordinate space through Hough transformation. The application reduces monocular vision depth scale uncertainty and improves stability and precision under large amplitude lifting and complex swing working conditions.

Description

Technical Field

[0001] This invention relates to the field of machine vision monitoring technology, and in particular to a video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion. Background Technology

[0002] As a core hub equipment for large-scale civil engineering, bridge construction, and port logistics transportation, the safety and intelligence level of tower cranes directly determine the overall implementation efficiency of the project. With the deepening of building industrialization and prefabricated assembly buildings, the spatial complexity of hoisting operations has increased dramatically. The requirement for real-time perception of the spatial status of hoisted objects has jumped from rough boundary collision prevention to precise spatiotemporal trajectory tracking.

[0003] Existing visual target tracking technologies have revealed significant technical limitations and vulnerabilities in extremely complex lifting and construction operation scenarios:

[0004] Existing methods often directly detect the hoisting object or hook outline in the entire video image. This is easily affected by building structure edges, scaffolding, shadows, strong reflections, occlusion, and changes in scale at long distances, leading to loss of target features or false detection of straight wire ropes. It is difficult to reliably obtain image features that can be used for spatial inversion.

[0005] Existing monocular vision tracking results mostly remain at the level of two-dimensional pixel coordinates or image bounding boxes, without fully combining physical constraints such as the spatial position of the luffing trolley, the length of the hoisting rope, the winding ratio of the wire rope, the camera's intrinsic and extrinsic parameters, and the mechanical installation boundaries. This results in problems such as scale uncertainty, unstable depth estimation, and lack of effective fallback for abnormal frames when converting from two-dimensional images to three-dimensional absolute spatial coordinates, making it difficult to meet the engineering requirements for continuous, stable, and quantifiable output of the spatial trajectory of the hoisted object. Summary of the Invention

[0006] This invention provides a video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion, comprising:

[0007] The system acquires the pre-loaded physical boundary constants of the tower crane and simultaneously collects multi-source data during the tower crane operation. Based on the physical boundary constants, it performs kinematic calculations on the multi-source data to obtain the relative vertical displacement, the three-dimensional absolute spatial coordinates of the center point of the luffing trolley, and the absolute coordinates and three-dimensional rotation matrix of the camera optical center in the global three-dimensional coordinate system.

[0008] Edge detection is performed on the temporally aligned color video frame matrix to extract the outline of the candidate steel wire rope and generate a binary edge pixel matrix. Collinear edges are mapped to polar coordinate space through Hough transform, and polar coordinate parameter pairs are extracted based on the steel wire rope winding ratio. An overdetermined linear equation system is constructed and solved to obtain the two-dimensional pixel center point of the virtual convergent micro-region on the imaging target surface.

[0009] Two-dimensional pixel coordinates of the virtual convergent micro-region are extracted, and a three-dimensional global direction ray is constructed using camera inverse mapping. Based on the mechanically estimated length as an elevation scale constraint, a damped numerical iteration mechanism is constructed to output the three-dimensional estimated coordinates of the virtual convergent micro-region.

[0010] Based on the three-dimensional estimated coordinates of the virtual convergence micro-region, the spatial sway direction vector is calculated. Combined with the absolute physical rope length and the lifting device compensation, the final absolute spatial target point coordinates of the hoisted object are obtained through three-dimensional vector synthesis.

[0011] By combining the current moment with the previous sampling period's three-dimensional absolute coordinates of the hoisted object, a filtered and smoothed velocity difference calculation is performed, and the video frame matrix, spatially estimated target coordinates, velocity vector, and end-to-end tracking status flags are encapsulated into a comprehensive tracking status data packet.

[0012] As a preferred embodiment of the video tracking and recognition method for the spatial trajectory of a hoisted object based on multi-source data fusion described in this invention, the step of performing kinematic calculations on the multi-source data based on physical boundary constants to obtain the relative vertical displacement, the three-dimensional absolute spatial coordinates of the center point of the amplitude-changing trolley, and the absolute coordinates and three-dimensional rotation matrix of the camera's optical center in the global three-dimensional coordinate system includes:

[0013] The multi-source data includes clockwise horizontal deflection angle, horizontal radial distance, signed cumulative pulse count, and color video frame matrix;

[0014] The physical boundary constants include the center-to-center distance of the wire ropes between the center of the luffing trolley fixed pulley block and the center of the hook moving pulley block, the mechanical reference radius of the hoisting winch drum, the equivalent drum radius, the transmission ratio of the hoisting mechanism, the wire rope winding ratio of the current hoisting pulley block, the single-turn pulse resolution of the hoisting winch motor encoder, the equivalent axial distance from the center of the hook to the center of the hoisted object, the absolute installation height of the tower crane boom relative to the earth coordinate system, the lateral spacing of the wire rope pulley blocks, the maximum permissible physical swing angle, the minimum permissible effective hoisting rope length, and the permissible effective hoisting rope length.

[0015] A global three-dimensional coordinate system is established with the intersection of the rotation center axis of the target tower crane and the horizontal mounting surface of the tower crane base as the origin;

[0016] Based on the encoder single-turn pulse resolution, drum equivalent radius, hoisting mechanism transmission ratio, and hoisting pulley block ratio, the signed cumulative pulse count is converted into the vertical displacement of the hook relative to the initial zero position.

[0017] Based on the absolute installation height, the horizontal radial distance of the luffing trolley and the clockwise horizontal deflection angle of the boom are mapped to the global three-dimensional coordinate system, and the three-dimensional absolute spatial coordinate vector of the center point of the luffing trolley is calculated.

[0018] Extract the horizontal rotation angle and pitch angle from the camera's pan-tilt angle, and calculate the camera's three-dimensional rotation matrix;

[0019] By constructing a yaw rotation matrix using the clockwise horizontal deflection angle of the crane boom, rotating the fixed offset vector, and superimposing it onto the three-dimensional absolute space coordinate vector, the absolute coordinates of the camera optical center in the global three-dimensional coordinate system are obtained.

[0020] Extract the acquisition timestamps of sensor parameters and the capture timestamps of color video frame matrices from multi-source data, and encapsulate sensor parameters and calculation results with the same acquisition timestamp within the same acquisition period into a dynamic motion feature set.

[0021] As a preferred embodiment of the video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion described in this invention, the method involves mapping collinear edges to polar coordinate space using Hough transform, extracting polar coordinate parameter pairs based on the wire rope winding ratio, constructing and solving an overdetermined linear equation system to obtain the two-dimensional pixel center point of the virtual convergent micro-region on the imaging target surface, including:

[0022] The time-domain aligned color video frame is converted into a single-channel grayscale image. Based on the three-dimensional absolute coordinates of the variable amplitude trolley, the current effective hoisting rope length, and the camera's intrinsic and extrinsic parameters, the theoretical three-dimensional envelope space of the wire rope is projected onto the current frame to generate a dynamic region of interest.

[0023] Perform Canny edge detection within the dynamically active region of interest to generate a binary edge pixel matrix of candidate edge contours for the wire rope.

[0024] Perform a Hough transform on the binary edge pixel matrix to map collinear edges to polar coordinate parameter space, and limit the polar angle search range according to the projection direction based on the wire rope theory.

[0025] An accumulator matrix is ​​constructed in the polar coordinate parameter space and a peak search is performed. Candidate peaks are filtered based on the accumulated value, angle deviation and continuous pixel length, and valid candidate straight lines are determined by combining the wire rope winding ratio.

[0026] The confidence weights are calculated based on the continuous pixel lengths of valid candidate lines, Hough cumulative response, and angle consistency. An overdetermined linear equation system is constructed and solved in a weighted manner to obtain the two-dimensional pixel center point.

[0027] As a preferred embodiment of the video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion described in this invention, the following steps are included: extracting the two-dimensional pixel coordinates of the virtual convergence micro-region, constructing a three-dimensional global direction ray using camera inverse mapping, constructing a damped numerical iteration mechanism based on mechanically estimated length as an elevation scale constraint, and outputting the three-dimensional estimated coordinates of the virtual convergence micro-region.

[0028] Extract the two-dimensional pixel center points of the virtual convergent micro-region, and inversely map them into direction vectors in the local coordinate system of the camera based on the camera intrinsic parameters. Combine them with the camera instantaneous attitude parameters to convert them into global three-dimensional direction rays.

[0029] The visual ray is constructed by taking the absolute coordinates of the camera's optical center in the global three-dimensional coordinate system as the starting point of the ray and constructing a ray along the global three-dimensional direction.

[0030] Construct an initial mechanical constraint elevation surface, determine whether the visual ray and the initial mechanical constraint elevation surface meet the intersection conditions, and obtain the initial three-dimensional intersection coordinates when the intersection conditions are met.

[0031] Calculate the visual inversion equivalent length from the three-dimensional intersection coordinates to the center point of the variable amplitude trolley, and perform damped iterative correction on the mechanical constraint elevation surface based on the scale residual between the visual inversion equivalent length and the mechanically estimated length.

[0032] When the scale residual meets the preset engineering error requirement, the corresponding three-dimensional intersection coordinates are output as the three-dimensional estimated coordinates of the virtual convergent micro-region. When the maximum number of iterations is reached but the preset engineering error requirement is still not met, the three-dimensional intersection coordinates with the smallest scale residual are selected as the low-confidence three-dimensional estimated coordinates.

[0033] As a preferred embodiment of the video tracking and recognition method for the spatial trajectory of a hoisted object based on multi-source data fusion described in this invention, the method involves: calculating the spatial sway direction vector based on the three-dimensional estimated coordinates of a virtual convergence micro-region, combining the absolute physical rope length and the lifting device compensation amount, and obtaining the final absolute spatial target point coordinates of the hoisted object through three-dimensional vector synthesis, including:

[0034] Based on the estimated three-dimensional coordinates of the virtual convergence micro-region in the global three-dimensional coordinate system and the three-dimensional absolute spatial coordinates of the center point of the luffing trolley, the spatial swaying attitude unit direction vector from the center point of the luffing trolley to the virtual convergence micro-region is calculated.

[0035] The current effective hoisting rope length is obtained based on the initial effective hoisting rope length and the vertical displacement of the hook relative to the initial zero position.

[0036] Starting from the center point of the luffing trolley, extend the current effective hoisting rope length along the unit direction of the spatial swaying posture to obtain the three-dimensional absolute spatial coordinates of the hook's geometric center.

[0037] Based on the equivalent axial distance from the center of the hook to the geometric center of the hoisted object, physical offset compensation is performed along the unit direction of the spatial swing posture to obtain the three-dimensional absolute spatial target point coordinates of the hoisted object.

[0038] As a preferred embodiment of the video tracking and recognition method for the spatial trajectory of a hoisted object based on multi-source data fusion described in this invention, the method involves: combining the current moment with the previous moment's three-dimensional absolute coordinates of the hoisted object based on the sampling period to perform filtered and smoothed velocity difference calculation; and encapsulating the video frame matrix, spatially estimated target point coordinates, velocity vector, and end-to-end tracking status flags into a comprehensive tracking status data packet, including:

[0039] Extract the three-dimensional absolute space target coordinates of the hoisted object at the current moment and the three-dimensional absolute space target coordinates of the hoisted object in the previous sampling period, and perform a first-order backward finite difference calculation based on the sampling time interval to obtain the original differential velocity;

[0040] The original differential velocity is smoothed by a first-order low-pass filter to obtain the smoothed three-dimensional running velocity vector of the hoisted object.

[0041] The system encapsulates a two-dimensional video frame matrix containing region of interest or feature calibration information, the three-dimensional absolute spatial target coordinates of the hoisted object at the current moment, the smoothed three-dimensional running speed vector, visual feature status flags, three-dimensional reconstruction confidence flags, direction validity flags, hoisting rope length validity flags, and a unified system timestamp in a structured manner to generate a comprehensive tracking status data packet.

[0042] The beneficial effects of this invention are as follows: This invention synchronously integrates the physical boundary constants of the tower crane, the data of the hoisting encoder, the position of the luffing trolley, the slewing angle of the boom, the camera posture, and the video frames, so that the video image recognition results and the mechanical motion state of the tower crane are established in a unified spatiotemporal correspondence, avoiding the target loss problem caused by simply relying on the appearance features of the hoisted object for tracking.

[0043] This invention constructs a dynamic region of interest based on the three-dimensional absolute coordinates of the luffing trolley, the current effective hoisting rope length, and the camera's intrinsic and extrinsic parameters. It also combines the theoretical projection direction of the wire rope, the wire rope winding ratio, and the confidence level of candidate straight lines to solve the virtual convergence micro-region. This effectively suppresses the interference of complex building backgrounds, shadows, occlusions, and irrelevant lines on wire rope identification, and improves the stability and accuracy under large-amplitude hoisting and complex swing conditions.

[0044] This invention reverse maps the two-dimensional pixel coordinates of a virtual convergent micro-region to a global three-dimensional directional ray, and uses the mechanically estimated length as an elevation scale constraint. The three-dimensional estimated coordinates are obtained through damped numerical iteration, thereby reducing the uncertainty of the monocular vision depth scale. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an architecture diagram of the video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion in Example 1;

[0047] Figure 2 This is a schematic diagram of wire rope identification and virtual convergence micro-region localization in Example 1. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Reference Figure 1 and Figure 2 This embodiment provides a video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion, including the following steps:

[0050] like Figure 1 As shown, S1, obtain the pre-loaded physical boundary constant of the tower crane, and simultaneously collect multi-source data during the tower crane operation. Based on the physical boundary constant, perform kinematic calculations on the multi-source data to obtain the relative vertical displacement, the three-dimensional absolute spatial coordinates of the center point of the luffing trolley, and the absolute coordinates and three-dimensional rotation matrix of the camera optical center in the global three-dimensional coordinate system.

[0051] Specifically, the origin of the global three-dimensional coordinate system is established by taking the intersection of the rotation center axis of the target tower crane and the horizontal mounting surface of the tower crane base as the origin. The positive direction of the X-axis is north in the horizontal plane, the positive direction of the Y-axis is east in the horizontal plane, and the positive direction of the Z-axis is vertically upward. The global three-dimensional coordinate system is used to represent the absolute position of the tower crane luffing trolley, the optical center of the camera, and related points of the hook in three-dimensional space.

[0052] The luffing trolley coordinate system is established with the center point of the luffing trolley as the origin. The X-axis extends radially outward along the boom, the Z-axis is vertically upward, and the Y-axis is located in the horizontal plane and is determined according to the right-hand rectangular coordinate system. The luffing trolley coordinate system is used to represent the fixed installation offset of the camera optical center relative to the center point of the luffing trolley.

[0053] A unified time base for the system is established through a hardware synchronization line. Each sensor adds a hardware timestamp when collecting data, and each hardware timestamp is converted to the unified time base after entering the control terminal.

[0054] During tower crane operation, intelligent sensors are used to collect multi-source data, including the clockwise horizontal deflection angle of the boom relative to the north of the ground, the horizontal radial distance from the center point of the luffing trolley to the axis of the tower crane's slewing center, the signed cumulative pulse count of the hoisting winch encoder, the color video frame matrix of the camera, the horizontal rotation angle of the camera pan-tilt unit relative to the longitudinal axis of the boom, the camera pitch angle, and the corresponding hardware timestamp.

[0055] Specifically, a slewing angle encoder is installed on the upper part of the tower crane's slewing mechanism to collect the clockwise horizontal deflection angle of the boom relative to the north of the ground; a variable amplitude laser rangefinder is installed at the root end of the boom's traveling track and its measurement zero point is aligned with the tower crane's slewing center axis to collect the horizontal radial distance from the center point of the variable amplitude trolley to the tower crane's slewing center axis; a hoisting encoder is installed on the hoisting winch motor shaft end to collect the absolute cumulative pulse value based on the initial zero point; and a high-definition PTZ camera is installed on the lower side of the variable amplitude trolley to collect a color video frame matrix.

[0056] The above-mentioned clockwise horizontal deflection angle is taken as zero degrees north of the earth, and clockwise rotation is the positive direction. It is uniformly converted to radians before participating in trigonometric function calculations.

[0057] If the zero point of the variable amplitude laser rangefinder does not completely coincide with the slewing center axis of the tower crane, compensation is made by a pre-calibrated radial zero point correction to ensure that the horizontal radial distance from the center point of the variable amplitude trolley to the slewing center axis of the tower crane is satisfied. The zero point correction is determined by the installation position and the structural dimensions of the tower crane, based on the real-time travel distance of the variable amplitude trolley measured by the sensor. The pre-calibrated radial zero point correction is the horizontal distance along the boom direction between the zero point of the sensor's distance measurement and the slewing center axis of the tower crane, preferably 0m to 2.00m.

[0058] The above-mentioned horizontal rotation angle is zeroed with the horizontal zero position of the camera mounting coordinate system as the zero position, and clockwise rotation is positive. The pitch angle is zeroed with the horizontal optical axis of the camera as the zero position, and downward tilt is positive. Both are converted to radians before participating in matrix operations.

[0059] Synchronously, a status polling command is sent to the camera's underlying application interface to obtain the horizontal rotation angle and pitch angle relative to the longitudinal axis of the crane boom. The rotation angle is provided by the slewing angle encoder. The camera's intrinsic parameter matrix, distortion coefficients, and the fixed offset vector of the camera's optical center in the variable amplitude trolley coordinate system, which were obtained in advance by the checkerboard calibration method, are also extracted. ,in, , as well as These represent the positional offsets of the camera's optical center relative to the center point of the luffing trolley in the x, y, and z directions of the luffing trolley coordinate system. T represents the transpose operation. The origin of the luffing trolley coordinate system is located at the center point of the luffing trolley. The x-axis extends radially outward along the boom, the z-axis is vertically upward, and the y-axis is determined according to the right-hand coordinate system, i.e., the x-axis, y-axis, and z-axis constitute a right-hand rectangular coordinate system.

[0060] The system extracts the physical boundary constants of the hoisting mechanism, luffing mechanism, camera, and wire rope pulley block from the target tower crane's engineering mechanical design drawings, factory specification database, on-site installation calibration records, and sensor calibration files. These constants include:

[0061] The center-to-center distance between the fixed pulley block of the luffing trolley and the center of the moving pulley block of the hook, the mechanical reference radius of the hoisting winch drum, the equivalent drum radius, the transmission ratio of the hoisting mechanism, and the wire rope winding ratio of the current hoisting pulley block are equal to the total number of wire rope branches supporting the hook, the single-turn pulse resolution of the hoisting winch motor encoder, the equivalent axial distance from the center of the hook to the center of the hoisted object, the absolute installation height of the tower crane boom relative to the earth coordinate system, the lateral spacing of the wire rope pulley blocks, the maximum allowable physical swing angle, the minimum allowable effective hoisting rope length, and the allowable effective hoisting rope length.

[0062] Based on the signed cumulative pulse count, encoder single-turn pulse resolution, drum equivalent radius, hoisting mechanism transmission ratio, and hoisting pulley group ratio, a linear displacement transformation algebraic equation is executed to convert the encoder absolute cumulative pulse value based on the initial zero point into the vertical displacement of the hook relative to the initial zero position in physical units (meters). A positive value of this displacement indicates lowering, and a negative value indicates lifting.

[0063] When the encoder is installed on the motor shaft end, the formula for calculating the relative vertical displacement of the hook is:

[0064] ;

[0065] in, Let t be the vertical displacement of the hook relative to the system's initial zero position at the current time. The pulse reference value at the system initialization moment. This represents the absolute cumulative pulse value at the current time t. The single-turn pulse resolution of the lifting encoder at the end of the lifting winch motor shaft. The equivalent radius of the drum, The current wire rope winding ratio of the lifting pulley block. Pi is a constant. The transmission ratio of the lifting mechanism is 1 when the encoder is installed on the end of the drum shaft.

[0066] Based on the relative vertical displacement of the hook, calculate the current effective hoisting rope length, which is the initial effective rope length from the center of the luffing trolley fixed pulley block to the center of the hook moving pulley block when the system is initialized, plus the vertical displacement of the hook relative to the system initial zero position;

[0067] Furthermore, the mechanically estimated length is defined as the estimated length from the suspension point of the luffing trolley to the center of the hoisted object, which is the current effective hoisting rope length plus the equivalent axial distance from the center of the hook to the center of the hoisted object;

[0068] Using the center of the tower crane base as the origin of the global three-dimensional coordinate system, with the north direction of the horizontal plane as the positive x-axis, the horizontal east direction perpendicular to the x-axis as the positive y-axis, and the vertical upward direction as the positive z-axis, based on the absolute installation height of the tower crane boom relative to the earth's coordinate system, and combining the mapping relationship between polar coordinates and Cartesian coordinates, the three-dimensional absolute spatial coordinate vector of the luffing trolley at the current moment is calculated using the following formula:

[0069] ;

[0070] in, Let be the three-dimensional absolute spatial coordinate vector of the center point of the amplitude-changing trolley at the current time t. The horizontal radial distance from the center point of the luffing trolley at the current time t to the axis of rotation of the tower crane is given. Let t be the clockwise horizontal deflection of the crane boom relative to the north direction of the earth at the current time, and H be the absolute installation height of the tower crane boom relative to the earth's coordinate system. and For cosine and sine functions;

[0071] This formula converts the radial distance and rotation angle in the polar coordinate system into the x-axis and y-axis components in the Cartesian rectangular coordinate system, and uses the fixed installation height of the tower crane boom as the z-axis component, thereby constructing the absolute coordinates of the center point of the luffing trolley in three-dimensional space.

[0072] Based on the fixed offset vector of the camera optical center in the variable amplitude trolley coordinate system, calculate the absolute coordinates of the camera optical center in the global three-dimensional coordinate system;

[0073] Specifically, based on the current boom slewing angle, the horizontal rotation relationship of the luffing trolley coordinate system relative to the global three-dimensional coordinate system is determined. The fixed offset vector of the camera's optical center in the luffing trolley coordinate system is transformed to the global three-dimensional coordinate system and added to the three-dimensional absolute spatial coordinates of the luffing trolley center point to obtain the absolute coordinates of the camera's optical center in the global three-dimensional coordinate system at the current moment. The formula is as follows:

[0074] ;

[0075] in, Let t be the absolute three-dimensional coordinates of the camera's optical center in the global three-dimensional coordinate system at the current time t. To transform vectors in the variable amplitude trolley coordinate system to the global three-dimensional coordinate system, is the fixed offset vector of the camera's optical center in the variable amplitude carriage coordinate system;

[0076] If the camera's optical center is offset relative to the gimbal's rotation center as the gimbal angle changes, then replace the fixed offset vector of the camera's optical center in the variable amplitude carriage coordinate system with the offset vector updated by the gimbal rotation, and then calculate the absolute coordinates of the camera's optical center in the same way as described above.

[0077] After receiving the relative horizontal and pitch angles of the PTZ, the control terminal combines the current boom direction with the three-dimensional rotation matrix synthesis operation to calculate the absolute horizontal orientation angle of the camera in the global coordinate system, and reconstructs the three-dimensional rotation matrix of the camera according to the Euler angle rotation order of yaw and pitch.

[0078] After the camera is installed, align the horizontal zero position of the camera pan / tilt head with the longitudinal direction of the crane arm, and align the vertical zero position of the camera tilt head with the horizontal optical axis of the camera. If there is still a residual deviation in the horizontal zero position after on-site installation, obtain the horizontal zero position offset angle through calibration. When the horizontal zero position of the camera pan / tilt head is completely aligned with the longitudinal direction of the crane arm, the horizontal zero position offset angle is 0. The preferred horizontal zero position offset angle is... to ;

[0079] Based on the boom rotation angle, camera pan-tilt head horizontal rotation angle, camera pitch angle, and horizontal zero-position offset angle, the current absolute horizontal orientation angle of the camera is calculated using the following formula:

[0080] ;

[0081] in, The horizontal zero-offset angle for camera installation. Let be the absolute orientation angle of the camera's optical axis in the horizontal plane at the current time t. Let be the slewing angle of the crane boom at the current time t. Let t be the horizontal rotation angle of the camera pan-tilt unit at the current time.

[0082] The formula for the above-mentioned camera 3D rotation matrix is:

[0083] ;

[0084] in, Let be the 3D rotation matrix of the camera at the current time t. Let t be the pitch angle at the current time.

[0085] The rotation matrix from the camera coordinate system to the global 3D coordinate system is transformed to obtain the rotation relationship between the global 3D coordinate system and the camera coordinate system.

[0086] Based on the absolute coordinates of the camera's optical center in the global 3D coordinate system, the translation relationship from the global 3D coordinate system to the camera coordinate system is calculated. The rotation relationship and the translation relationship together constitute the instantaneous extrinsic parameter matrix of the camera at the current moment.

[0087] Based on the difference in effective hoisting rope length between the current sampling time and the previous sampling time, and the sampling time interval between the two, the current hoisting mechanism's hoisting or descent speed is calculated. When the effective hoisting rope length increases, it indicates that the hook is in the lowering state; when the effective hoisting rope length decreases, it indicates that the hook is in the lifting state.

[0088] Based on the change in the horizontal position of the center point of the luffing trolley at multiple consecutive sampling times (e.g., 5), the motion acceleration of the luffing trolley in the horizontal plane is calculated. The motion acceleration of the luffing trolley is used to indicate whether the luffing trolley is in a stable motion, starting, braking or rapid luffing state.

[0089] Collect timestamp Vertical displacement The following parameters are encapsulated as a dynamic motion feature set: radial distance of the luffing trolley, relative vertical displacement of the hook, effective lifting rope length, mechanically estimated length, three-dimensional absolute coordinates of the luffing trolley center, absolute coordinates of the camera optical center, three-dimensional rotation matrix of the camera, lifting speed, and luffing trolley motion acceleration.

[0090] Capture timestamp The color video frame matrix, camera intrinsic parameter matrix, distortion coefficient, absolute coordinates of camera optical center, camera 3D rotation matrix, rotation matrix from global coordinate system to camera coordinate system, translation vector, and camera extrinsic parameter matrix are encapsulated into a visual state feature set.

[0091] The hardware timestamps of the dynamic motion feature set and the visual state feature set are extracted separately. This is the timestamp for the sensor data acquisition. The capture timestamp of the video frame;

[0092] The control terminal retrieves the dynamic motion feature set sequence from the most recent acquisition cycle that makes... The set of dynamic motion features with the smallest value is considered as synchronous data under the same time domain node and bound together if the absolute value of the minimum time difference does not exceed the pre-set communication delay tolerance, preferably 10 milliseconds. The later timestamp of the successful match is used as the synchronization timestamp. If the communication delay tolerance is exceeded, the current visual frame is discarded and the next frame is waited for until a successful match is obtained, resulting in a time-domain aligned global state data tuple.

[0093] like Figure 2 As shown, S2, edge detection is performed on the temporally aligned color video frame matrix, the outline of the candidate steel wire rope is extracted to generate a binary edge pixel matrix, the collinear edges are mapped to polar coordinate space through Hough transform, and polar coordinate parameter pairs are extracted based on the steel wire rope winding ratio. An overdetermined linear equation system is constructed and solved to obtain the two-dimensional pixel center point of the virtual convergent micro-region on the imaging target surface.

[0094] Specifically, the color video frame matrix is ​​converted into a single-channel grayscale image matrix f(u,v), where u is the horizontal pixel coordinate in the two-dimensional image coordinate system and v is the vertical pixel coordinate. The origin of the coordinate system is located at the upper left corner of the image. In order to reduce the interference of complex building background and lighting interference in the image on the edge extraction of the steel wire rope, the three-dimensional absolute coordinates of the luffing trolley, the current effective lifting rope length, the camera intrinsic parameter matrix, and the camera instantaneous extrinsic parameter matrix are introduced to construct a dynamic region of interest. The theoretical drooping three-dimensional envelope space of the steel wire rope is projected onto the current grayscale image matrix to generate a polygonal mask.

[0095] Among them, constructing a dynamic region of interest refers to introducing the three-dimensional absolute coordinates of the variable amplitude trolley, the current effective lifting rope length, the camera intrinsic parameter matrix, and the camera instantaneous extrinsic parameter matrix, and projecting the theoretical drooping three-dimensional envelope space of the wire rope onto the current grayscale image matrix to generate a local pixel region for subsequent edge detection and line extraction.

[0096] The three-dimensional envelope space of the theoretical droop of the wire rope is constructed with the center point of the luffing trolley as the starting point, the theoretical droop direction of the wire rope as the axis direction, and the current effective lifting rope length as the axial length. Its envelope radius is determined according to the lateral spacing of the wire rope pulley block, the maximum allowable physical swing angle, and the dynamic region of interest compensation. It is used to cover the possible offset range of the wire rope caused by swing, calibration error and structural vibration during actual hoisting.

[0097] The Canny edge detection algorithm is performed on the grayscale image matrix only within the local pixel region covered by the mask. The absolute magnitude of the pixel gradient is obtained by calculating the local first-order spatial partial derivative of the image pixel.

[0098] This step uses discrete differential operators to perform convolution operations on the image matrix, calculates the gradient magnitude and direction of each pixel, and then performs non-maximum suppression and double threshold connectivity judgment to remove false edges and outputs a binary edge pixel map matrix containing the candidate edge contours of the wire rope. In this matrix, the edge pixel value is 1 and the background pixel value is 0.

[0099] Based on the binary edge pixel map matrix, a Hough transform is performed to map discrete collinear edge pixels in the Cartesian coordinate system of the image to continuous mathematical line equations in the polar coordinate parameter space. The mapping equation in the polar coordinate parameter space is:

[0100] ;

[0101] in, In polar coordinate parameter space, the perpendicular polar radius distance from the line to the origin of the image coordinate system is... is the angle between the straight line normal vector and the horizontal axis of the image in the Hough polar coordinate parameter space, and u and v are the horizontal and vertical coordinates of the edge points with a pixel value of 1 in the binary edge pixel map matrix;

[0102] This mapping equation transforms collinear image pixels into parameter space. The family of sine curves in the polar coordinate parameter space is used, and the curves corresponding to collinear points must intersect at the same local extreme point. An accumulator matrix is ​​constructed in the polar coordinate parameter space, and a peak search is performed in the accumulator matrix based on the current wire rope winding ratio of the lifting pulley block. This represents the actual number of steel wire ropes physically present in the field of view, and is strictly extracted from the accumulator. The polar coordinate parameters corresponding to the highest peak value ,in ;

[0103] The specific operations for constructing the accumulator matrix in the polar coordinate parameter space and performing peak search in the accumulator matrix include:

[0104] The parameter space in the image coordinate system is discretized, and a two-dimensional accumulator matrix is ​​established based on the discretized parameter grid.

[0105] Based on the three-dimensional absolute coordinates of the luffing trolley center, the current effective hoisting rope length, the camera intrinsic parameter matrix, and the camera instantaneous extrinsic parameter matrix, the upper and lower endpoints on the theoretical drooping axis of the wire rope are selected. The upper endpoint is the wire rope suspension point near the center of the luffing trolley fixed pulley block or the center of the luffing trolley, and the lower endpoint is the spatial point obtained by extending the current effective hoisting rope length along the theoretical drooping direction of the wire rope.

[0106] The two spatial points are projected onto the current video frame to obtain two two-dimensional pixels. The theoretical projection line of the steel wire rope is determined by these two two-dimensional pixels, and the angle between the normal vector of the projection line and the horizontal axis of the image is taken as the prior value of the Hough polar angle.

[0107] If the Hough polar angle prior value is the direction angle of the theoretically projected straight line of the wire rope in the image plane calculated by combining the instantaneous attitude of the camera, then the corresponding Hough polar angle prior value satisfies: ,in, The polar angle prior value in the Hough transform. The direction angle of the theoretical projection line of the wire rope in the image plane is given by the three-dimensional coordinates of the luffing trolley, the current effective hoisting rope length, the camera intrinsic parameters, and the camera instantaneous extrinsic parameters. The theoretical drooping axis of the wire rope is projected onto the two-dimensional image to obtain a theoretical projection line, which refers to the direction angle of the line relative to the horizontal axis of the image.

[0108] And Normalize to the range [0, π);

[0109] The range of polar angle parameter θ is truncated from the global range [0, π) into a dynamic search interval. Where Δθ is a tolerance threshold set based on the maximum allowable physical swing angle, preferably... ;

[0110] The polar angle parameter is set to a range of [0, π), and divided into multiple angle ranges according to a preset step size (e.g., 1 degree). The polar radius parameter is set to a range of [-π / π]. , ],in The pixel length of the image diagonal is taken to represent the maximum range of values ​​of the polar radius in the positive half-axis direction, and divided into multiple distance value intervals according to a preset step size (e.g., 1 pixel), thereby constructing the corresponding two-dimensional parameter grid.

[0111] Each element of the accumulator matrix corresponds to a combination of polar radius and polar angle parameters, and all matrix elements are initialized to zero.

[0112] Traverse all edge pixels with a value of 1 in the binary edge pixel map matrix. For each edge pixel, calculate its corresponding polar radius parameter one by one within the entire range of discrete polar angle values, and map the polar radius parameter to the corresponding position in the accumulator matrix. Accumulate and count the corresponding matrix elements to record the number of times the parameter combination is supported by the edge pixel.

[0113] After the voting accumulation of all edge pixels is completed, peak search processing is performed on the accumulator matrix;

[0114] Specifically, by setting a neighborhood window to perform local comparisons on the accumulator matrix, the local maxima points with the largest values ​​in their neighborhoods are selected, and non-maximum values ​​are suppressed to reduce the impact of noise peaks on the extraction of candidate lines. The neighborhood window is preferably a 3×3 or 5×5 parameter grid window.

[0115] Thresholding is performed on the selected local maxima points. The thresholding includes at least the cumulative value threshold, the angle deviation threshold, and the continuous pixel length threshold.

[0116] The accumulated value threshold is used to remove weak peaks formed by scattered noise edges. The accumulated value threshold can be determined based on the maximum accumulated value of the current frame accumulator matrix, and is preferably set to 40% of the current maximum accumulated value.

[0117] The angle deviation threshold is used to eliminate candidate peaks that are significantly inconsistent with the theoretical projection direction of the wire rope. The angle deviation threshold is consistent with the Hough polar angle dynamic search range, meaning that the polar angle corresponding to the candidate peak should fall within... Within the range;

[0118] The continuous pixel length threshold is used to remove short background edges or broken noise edges. The continuous pixel length threshold can be set to 10% of the diagonal pixel length of the dynamic region of interest, while setting a minimum pixel length limit, preferably not less than 20 pixels.

[0119] Among the candidate peak values ​​that meet the above threshold conditions, they are sorted from high to low according to their overall confidence level, and further sorted according to the wire rope winding ratio. And the number of redundancy tolerances m, selected to be no more than There are 1 to 3 candidate peak parameters, and the number of valid candidate lines participating in the final solution is denoted as n.

[0120] When n≥2 and not all effective candidate lines are approximately parallel, proceed to the subsequent weighted least squares solution. When n<2, or the effective candidate lines are approximately parallel, causing the weighted equation to be unsolvable, determine that the visual feature extraction of the current frame is in a short-term failure state and issue a missing warning.

[0121] Extracted The polar coordinate parameters are inversely transformed back to the Cartesian coordinate system of the image, constructing a system based on these parameters. The system of overdetermined linear equations consists of several mathematical linear equations. The algebraic form of a single linear equation is:

[0122] ;

[0123] in, Let be the horizontal coefficient of the i-th straight wire rope, and its analytical expression is: , Let be the vertical coefficient of the i-th straight wire rope, and its analytical expression is: , Let be the constant coefficient of the i-th straight wire rope, and its analytical expression is: , and Let Hough polar angle parameters and polar radius parameters be the parameters corresponding to the i-th candidate line;

[0124] In actual physical optical projection, due to the influence of the spatial isosceles trapezoidal geometry of the multi-magnification steel wire rope and lens distortion, the extension lines of the above-mentioned multiple approximately co-located steel wire ropes may not perfectly intersect at a single point on the two-dimensional image target surface. In order to establish the globally unique deterministic intersection point coordinates, namely the virtual convergence micro-region directly below the hook, the confidence weighting mechanism is used to perform optimal analytical solution of the above-mentioned overdetermined linear equation system.

[0125] Construct a linear confidence weight matrix;

[0126] Specifically, the length score is calculated by normalizing the ratio of the continuous pixel length of the i-th candidate line within the dynamic region of interest to the maximum continuous pixel length of the candidate lines in the current frame. The longer the candidate line, the more likely it is to correspond to the real edge of the steel wire rope, and the higher its length score.

[0127] The cumulative response score is obtained by normalizing the ratio of the Hough cumulative value corresponding to the i-th candidate line to the maximum Hough cumulative value in the candidate lines of the current frame. The more edge pixel support a candidate line receives, the higher its cumulative response score.

[0128] Compare the deviation between the Hough polar angle corresponding to the i-th candidate line and the prior Hough polar angle. The smaller the deviation, the higher the angle consistency score; the larger the deviation, the lower the angle consistency score. The angle consistency score preferably adopts an exponential decay method to rapidly reduce the weight of candidate lines that deviate significantly from the theoretical projection direction of the wire rope.

[0129] Multiply the length score, cumulative response score, and angle consistency score to obtain the confidence weight of the i-th candidate line, and normalize the confidence weights of all candidate lines to the range of 0 to 1.

[0130] According to the order in which the candidate lines participate in the solution, the confidence weights of each candidate line are sequentially filled into the main diagonal position of the weight matrix to obtain the line confidence weight matrix. All elements in the weight matrix except the main diagonal are set to 0.

[0131] Transform the system of linear equations into matrix form. The two-dimensional algebraic coordinates of the virtual convergent microregion are obtained under the condition of minimizing the weighted sum of squared residuals. Its closed-form matrix analytic equation is given by:

[0132] ;

[0133] in, To solve for the two-dimensional pixel coordinates of the obtained virtual convergent micro-region in the current video frame, when the coefficient matrix... When the reversibility condition is met and the condition number is less than a preset condition threshold, the two-dimensional algebraic coordinates are used as the valid output of the current frame. These two-dimensional coordinates do not require the outline of the hoisted object to be clearly visible; therefore, when the hoisted object is partially occluded or its outline is incomplete, the preset condition number threshold is preferably... , representing the horizontal and vertical coordinates of the quantitative pixel center point of the virtual convergence micro-region on the camera imaging target surface. Let M be the confidence weight matrix for the straight line, and M be the matrix constructed based on the system of equations for the straight line. The coefficient matrix is ​​given by the following formula:

[0134] ;

[0135] Y is constructed from the constant terms of the system of linear equations. 3D column vector, the formula is:

[0136] ;

[0137] If the number of valid candidate lines is less than 2, or the candidate lines are approximately parallel, resulting in the weighted least squares equations constructed from the system of candidate line equations being unsolvable, or the coefficient matrix... If the reversibility condition is not met, the visual feature extraction is determined to be in a short-term failure state.

[0138] At this point, a default warning is issued, and the kinematic feedforward inference mechanism is triggered: the coordinates of the virtual convergence micro-region that have been successfully solved in the previous frame are retrieved, and combined with the obtained physical displacement increment of the luffing trolley, the change in the current effective lifting rope length, the change in the boom rotation angle, and the change in camera attitude, the alternative coordinates of the virtual convergence micro-region in the current frame are inferred through the camera projection relationship, and the alternative coordinates are used as the low confidence output of the current frame until the subsequent video frames meet the algebraic solution conditions again;

[0139] If the algebraic solution conditions are not met for multiple consecutive frames (e.g., 5 frames), a visual feature extraction failure flag is output. The visual status flag is used to distinguish whether the two-dimensional pixel coordinates of the current virtual convergent micro-region are derived from the normal weighted least squares solution.

[0140] S3. Extract the two-dimensional pixel coordinates of the virtual convergence micro-region, construct a three-dimensional global direction ray using camera inverse mapping, construct a damped numerical iteration mechanism based on the mechanically estimated length as an elevation scale constraint, and output the three-dimensional estimated coordinates of the virtual convergence micro-region.

[0141] Specifically, the pinhole camera imaging geometry model inversely maps the two-dimensional pixel coordinates in the image domain to directional rays in three-dimensional physical space. Based on the inverse of the camera's intrinsic parameter matrix, the two-dimensional pixel coordinates of the virtual convergent micro-region are converted into unit direction vectors in the camera's local coordinate system. Using the camera's three-dimensional rotation matrix, a global ray direction vector is constructed, as shown in the formula:

[0142] ;

[0143] in, Let be the global three-dimensional unit direction vector from the absolute optical center of the camera to the virtual convergent micro-region at the current time t. Let be the 3D rotation matrix of the camera at the current time t. This is the inverse of the camera intrinsic parameter matrix, used to eliminate the scale effect of lens focal length and optical center offset on pixel coordinates. and These represent the horizontal and vertical pixel coordinates in the two-dimensional pixel coordinates of the virtual convergent micro-region, respectively. A homogeneous coordinate column vector derived from the precise quantitative expansion of pixel center points. The Euclidean norm of the vector, i.e. the vector length, forces the direction vector to be normalized to a unit vector with a magnitude of 1.

[0144] Based on the absolute coordinates of the camera's optical center and the global ray direction vector, an algebraic parametric equation for a continuous visual ray in three-dimensional space is constructed. This ray represents a spatial straight line extending outward from the camera's optical center along the back projection direction of the virtual convergent micro-region in the image. The formula is:

[0145] ;

[0146] in, Let λ be the three-dimensional spatial point on the visual ray corresponding to the scale factor λ, where λ is the viewing scale factor to be solved, and the effective solution should satisfy λ>0, indicating that the corresponding spatial point is located in front of the camera imaging direction;

[0147] Extract the absolute coordinates of the center of the luffing trolley and the current estimated mechanical length to construct the mechanical constraint elevation equipotential surface in the initial stage;

[0148] Extract the equivalent elevation offset of the virtual convergent micro-region relative to the center of the moving pulley system. When the virtual convergent micro-region is located at a higher position relative to the center of the moving pulley system, the equivalent elevation offset is positive; when it is located at a lower position, the equivalent elevation offset is negative; when the accurate structural offset cannot be obtained, the equivalent elevation offset can be initialized to 0.

[0149] Using the center height of the luffing trolley as the upper reference, the estimated mechanical length is subtracted downwards, and then the equivalent elevation offset is superimposed to obtain the initial mechanical constraint elevation equipotential surface. This equipotential surface can be understood as: the initial estimated height plane of the virtual convergence micro-region in the vertical direction under the current mechanical encoder reading.

[0150] Before using the elevation equipotential surface to find the initial intersection point, the solvability of the vertical component of the visual ray is first determined. If the absolute value of the vertical component of the global direction ray is less than or equal to the lower limit threshold, it indicates that the current ray is close to horizontal. Using the elevation surface to find the intersection will lead to the scale factor value being unstable. The lower limit threshold is preferably 0.10.

[0151] When the absolute value of the vertical component of the visual ray is greater than the lower threshold, the initial line-of-sight scale factor is calculated. Specifically, the difference between the height of the initial elevation plane and the height of the camera optical center is taken. This difference represents the vertical height difference between the camera optical center and the target elevation plane. Then, the vertical height difference is divided by the unit component of the visual ray in the global Z-axis direction to obtain the spatial scale distance that needs to be traveled along the direction of the visual ray, which is the initial line-of-sight scale factor.

[0152] When the absolute value of the vertical component of the visual ray is greater than the lower threshold and the initial gaze scale factor is greater than 0, the initial intersection point is determined to be in front of the camera's imaging direction. Substituting the initial gaze scale factor back into the visual ray equation, the initial 3D intersection coordinates are obtained, as follows:

[0153] ;

[0154] in, The estimated three-dimensional coordinates of the virtual convergent micro-region obtained from zero iterations. This is the initial line-of-sight scale factor;

[0155] If the absolute value of the vertical component of the visual ray is not greater than the lower threshold, or the initial visual scale factor is not greater than 0, then the initial intersection of the current frame visual ray and the mechanical elevation constraint is determined to be unsatisfactory, and the erroneous three-dimensional coordinates are not forced to be output.

[0156] For the three-dimensional intersection coordinates that satisfy the initial solvability condition, the visual inversion equivalent length from the point to the suspension center of the luffing trolley is calculated. The specific calculation method is as follows: the three-dimensional coordinates of the virtual convergence micro-region obtained by the k-th iteration are subtracted from the three-dimensional coordinates of the luffing trolley center in the X, Y, and Z directions, respectively. The differences in the three directions are squared and summed. The square root of the summation result is then taken to obtain the three-dimensional Euclidean distance between the two points. This distance is defined as the visual inversion equivalent length.

[0157] After calculating the visual inversion equivalent length, it is first determined whether the visual inversion equivalent length is less than the preset minimum distance threshold. If the visual inversion equivalent length is less than the threshold, it is considered that the current iteration point is abnormally close to the suspension center of the variable amplitude trolley. The minimum distance threshold is preferably 0.2m.

[0158] The scale residual between the mechanically estimated length and the visually inverted equivalent length is calculated. When the absolute value of the scale residual is less than or equal to the engineering tolerance error threshold, the current iteration result is determined to meet the engineering error requirements, and the iteration is terminated. The preferred engineering tolerance error threshold is 0.05m.

[0159] When the absolute value of the scale residual is greater than the engineering allowable error threshold, perform depth correction for the elevation equipotential surface:

[0160] Specifically, the approximate cosine value of the visual ray tilt angle corresponding to the current iteration is first calculated. This value is obtained by dividing the vertical height difference between the current 3D intersection point and the center of the luffing trolley by the 3D spatial straight-line distance between the two points. It is used to represent the correction ratio of the mechanical length residual in the vertical direction.

[0161] To avoid the cosine value from exceeding the physical range due to noise or outliers, the approximate value of the visual ray tilt angle cosine is limited to the interval [0,1], and a damping coefficient is introduced to iteratively correct the elevation equipotential surface.

[0162] Specifically, the scale residual is multiplied by the approximate value of the inclination cosine to obtain the correction amount of the residual in the vertical direction. Then, it is multiplied by the damping coefficient to limit the single correction amplitude. Finally, the damping correction amount is subtracted from the current elevation equipotential surface to obtain the elevation equipotential surface used in the next iteration. The damping correction coefficient is preferably taken as 0.5. The damping mechanism is used to avoid iterative oscillations caused by excessive one-time compensation.

[0163] After the next corrected elevation equipotential surface, the intersection scale of the visual ray and the updated elevation surface is recalculated according to the same calculation logic as the initial line-of-sight scale factor. Specifically, the difference between the height of the updated elevation surface and the height of the camera optical center is taken and then divided by the unit component of the visual ray in the global Z-axis direction to obtain the line-of-sight scale factor for the next iteration.

[0164] After each iteration of the line-of-sight scale factor, it is determined whether it is a positive real number. If the line-of-sight scale factor of the iteration is not less than 0, it is determined that the current iteration result has deviated from the effective imaging direction of the camera, and the iteration is terminated.

[0165] If the viewing scale factor of the iteration is greater than 0, substitute it back into the visual ray equation to obtain the coordinates for the next iteration, as shown in the formula:

[0166] ;

[0167] in, The coordinates of the three-dimensional estimated virtual convergent micro-region obtained in the (k+1)th iteration are: Let t be the three-dimensional absolute coordinates of the camera's optical center in the global coordinate system at the current time t. The line-of-sight scale factor is the (k+1)th iteration.

[0168] Set an upper limit for the maximum number of iterations, preferably 3 to 5. If there is an iteration within the maximum number of iterations that satisfies the condition that the absolute value of the scale residual is less than or equal to the engineering tolerance error threshold, then the line-of-sight scale factor corresponding to that iteration is recorded as the final selected scale factor, and the corresponding three-dimensional coordinates are used as the effective three-dimensional estimated coordinates of the current frame.

[0169] If the error requirement is not met even after reaching the preset maximum number of iterations, the calculation result with the smallest absolute value of the scale residual is selected from the iteration sequence, and its corresponding line-of-sight scale factor is used as a temporary substitute scale factor. A low confidence label is then attached to the output.

[0170] If, during the initial intersection or iterative update phase, the absolute value of the unit component of the visual ray in the global Z-axis direction is less than or equal to the engineering tolerance error threshold, the line-of-sight scale factor is less than or equal to 0, or the visual inversion equivalent length is less than the minimum distance threshold, the system calls the three-dimensional coordinates of the virtual convergence micro-region of the previous valid frame. Combining the displacement increment of the luffing trolley, the change in the effective lifting rope length, the change in the slewing angle of the crane arm, and the change in the camera posture, the system deduces the three-dimensional predicted coordinates of the virtual convergence micro-region of the current frame and uses these predicted coordinates as the low-confidence replacement output of the current frame.

[0171] Substituting the final line-of-sight scale factor back into the visual ray equation, we output the estimated three-dimensional coordinates of the virtual convergent micro-region in the global coordinate system, as shown in the formula:

[0172] ;

[0173] in, The estimated three-dimensional coordinates of the virtual convergent micro-region at time t in the global coordinate system. The line-of-sight scale factor for the final iteration;

[0174] When the final line-of-sight scale factor comes from the normal iteration result that satisfies the residual threshold, the three-dimensional estimated coordinates are marked as normal reconstruction output; when the final line-of-sight scale factor comes from the temporary substitution result with the minimum residual, or when the three-dimensional estimated coordinates of the virtual convergent micro-region in the global coordinate system come from the feedforward inference result, the three-dimensional estimated coordinates are marked as low confidence output.

[0175] S4. Calculate the spatial sway direction vector based on the three-dimensional estimated coordinates of the virtual convergence micro-region, and combine the absolute physical rope length and the lifting device compensation amount to obtain the final absolute spatial target point coordinates of the hoisted object through three-dimensional vector synthesis.

[0176] Specifically, based on the three-dimensional absolute coordinates of the virtual convergence micro-region in the global coordinate system, the direction from the center of the luffing trolley to the virtual convergence micro-region is taken as the estimated direction of the current spatial swing direction of the wire rope. The unit direction vector of the spatial swing attitude is calculated using the following formula:

[0177] ;

[0178] in, Let be the unit direction vector of the steel wire rope's spatial swaying posture at the current time t. This vector is obtained by subtracting and normalizing the values ​​of two absolute coordinate points in space. The estimated three-dimensional coordinates of the virtual convergent micro-region at time t in the global coordinate system. The three-dimensional absolute coordinates of the center of the amplitude-changing trolley at the current time t in the global coordinate system;

[0179] Introducing a spatial direction vector fallback protection mechanism: If If the vector magnitude is less than the preset minimum direction length threshold, preferably 0.2m, or a low confidence flag, then the current frame direction vector is not used directly. Instead, the spatial swing direction vector of the previous valid moment is called. If the previous valid direction is unavailable, the direction is temporarily set to the vertical downward direction and a low confidence flag is added.

[0180] Based on the vertical displacement, which only represents the relative increment relative to the initial zero position, and combined with the initial effective hoisting rope length, the current effective hoisting rope length is calculated using the following formula:

[0181] ;

[0182] in, Let t be the effective hoisting rope length at the current time. This is an absolute initial rope length constant, representing the initial spatial length from the suspension center of the amplitude transformer trolley to the reference center of the hook when the encoder is at zero position;

[0183] Introducing a fallback mechanism for anomaly prediction in one-dimensional data: If ,or If the lifting length exceeds the tower crane's permissible range (the minimum and maximum permissible effective lifting rope lengths), the current encoder length data is considered abnormal. The system issues an abnormality warning signal and prioritizes the effective lifting rope length from the previous valid moment. It then performs short-time kinematic prediction based on the current lifting speed to obtain a replacement lifting rope length for the current frame. If the previous valid length is unavailable, a preset safety length is used, or only a low-confidence trajectory result is output. The formula is as follows:

[0184] ;

[0185] in, Replace the hoisting rope length with the current frame. The effective hoisting rope length at a given moment. This refers to the current lifting or lowering speed of the hoisting mechanism. The sampling period;

[0186] If the previous effective hoisting rope length is unavailable, the preset safety length is used as a temporary replacement value, or only the low confidence trajectory result is output without outputting closed-loop control commands to the control master station. The preset safety length can be the median value of the tower crane's allowable hoisting length range.

[0187] Using the center of the luffing trolley as the geometric origin and the current effective hoisting rope length as the extension length, perform a three-dimensional spatial vector addition operation along the unit direction of the spatial swing posture to calculate the true three-dimensional absolute coordinates of the hook. The formula is:

[0188] ;

[0189] in, Let be the three-dimensional absolute spatial coordinate vector of the geometric center of the hook at the current time t. The effective drop length of the hook relative to the suspension center of the luffing trolley at the current moment t is equal to the current effective hoisting rope length.

[0190] When the current encoder length data is valid When the current encoder length data is abnormal, ;

[0191] Based on the equivalent axial distance from the hook center to the geometric center of the hoisted object, the geometric center of the hoisted object can be approximately located on the wire rope's spatial swing direction, pointing from the center of the luffing trolley to the virtual convergence micro-region. Physical offset compensation is performed downwards along the unit direction vector of the swing posture, and the spatial estimated target point coordinates of the hoisted object are calculated using the following formula:

[0192] ;

[0193] in, Let be the three-dimensional absolute space target point coordinate vector of the geometric center of the hoisted object at the current time t. The equivalent axial distance is the distance from the center of the hook to the geometric center of the object being lifted. When only the hook body is tracked, the equivalent axial distance is set to 0. When the geometric center of the object is tracked, the equivalent axial distance is determined by the length of the lifting device, the position of the suspension point of the object, and the position of the geometric center of the object.

[0194] S5. Combine the current moment with the previous sampling period's three-dimensional absolute coordinates of the hoisted object to perform filtered and smoothed velocity difference calculation, and encapsulate the video frame matrix, spatially estimated target coordinates, velocity vector, and end-link tracking status flags into a comprehensive tracking status data packet.

[0195] Specifically, the coordinates of the target point in the spatial estimation of the hoisted object at the current moment and the coordinates of the target point in the final spatial estimation of the hoisted object in the previous sampling period are extracted. Based on the kinematic definition, a first-order backward finite difference numerical calculation is performed to obtain the original differential velocity.

[0196] To reduce the noise amplification caused by visual pixel quantization, spatial reconstruction errors, or small perturbations in a single frame on speed calculation, a first-order low-pass filtering algorithm is introduced to smooth the original differential speed, resulting in a smoothed three-dimensional running speed vector.

[0197] The system encapsulates a two-dimensional visualized video frame matrix containing the region of interest or feature labels, the spatial estimation target point coordinates of the hoisted object at the current moment, the smoothed three-dimensional running speed vector, visual feature status flags, three-dimensional reconstruction confidence flags, direction validity flags, hoisting rope length validity flags, and a unified system timestamp in a structured manner to generate a comprehensive tracking status data packet.

[0198] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion, characterized in that, include: The system acquires the pre-loaded physical boundary constants of the tower crane and simultaneously collects multi-source data during the tower crane operation. Based on the physical boundary constants, it performs kinematic calculations on the multi-source data to obtain the relative vertical displacement, the three-dimensional absolute spatial coordinates of the center point of the luffing trolley, and the absolute coordinates and three-dimensional rotation matrix of the camera optical center in the global three-dimensional coordinate system. Edge detection is performed on the temporally aligned color video frame matrix to extract the outline of the candidate steel wire rope and generate a binary edge pixel matrix. Collinear edges are mapped to polar coordinate space through Hough transform, and polar coordinate parameter pairs are extracted based on the steel wire rope winding ratio. An overdetermined linear equation system is constructed and solved to obtain the two-dimensional pixel center point of the virtual convergent micro-region on the imaging target surface. Two-dimensional pixel coordinates of the virtual convergent micro-region are extracted, and a three-dimensional global direction ray is constructed using camera inverse mapping. Based on the mechanically estimated length as an elevation scale constraint, a damped numerical iteration mechanism is constructed to output the three-dimensional estimated coordinates of the virtual convergent micro-region. Based on the three-dimensional estimated coordinates of the virtual convergence micro-region, the spatial sway direction vector is calculated. Combined with the absolute physical rope length and the lifting device compensation, the final absolute spatial target point coordinates of the hoisted object are obtained through three-dimensional vector synthesis. By combining the current moment with the previous sampling period's three-dimensional absolute coordinates of the hoisted object, a filtered and smoothed velocity difference calculation is performed, and the video frame matrix, spatially estimated target coordinates, velocity vector, and end-to-end tracking status flags are encapsulated into a comprehensive tracking status data packet.

2. The video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion as described in claim 1, characterized in that, The kinematic calculation of multi-source data based on physical boundary constants yields the relative vertical displacement, the three-dimensional absolute spatial coordinates of the center point of the amplitude-changing trolley, and the absolute coordinates and three-dimensional rotation matrix of the camera optical center in the global three-dimensional coordinate system, including: The multi-source data includes clockwise horizontal deflection angle, horizontal radial distance, signed cumulative pulse count, and color video frame matrix; The physical boundary constants include the center-to-center distance of the wire ropes between the center of the luffing trolley fixed pulley block and the center of the hook moving pulley block, the mechanical reference radius of the hoisting winch drum, the equivalent drum radius, the transmission ratio of the hoisting mechanism, the wire rope winding ratio of the current hoisting pulley block, the single-turn pulse resolution of the hoisting winch motor encoder, the equivalent axial distance from the center of the hook to the center of the hoisted object, the absolute installation height of the tower crane boom relative to the earth coordinate system, the lateral spacing of the wire rope pulley blocks, the maximum permissible physical swing angle, the minimum permissible effective hoisting rope length, and the permissible effective hoisting rope length. A global three-dimensional coordinate system is established with the intersection of the rotation center axis of the target tower crane and the horizontal mounting surface of the tower crane base as the origin; Based on the encoder single-turn pulse resolution, drum equivalent radius, hoisting mechanism transmission ratio, and hoisting pulley block ratio, the signed cumulative pulse count is converted into the vertical displacement of the hook relative to the initial zero position. Based on the absolute installation height, the horizontal radial distance of the luffing trolley and the clockwise horizontal deflection angle of the boom are mapped to the global three-dimensional coordinate system, and the three-dimensional absolute spatial coordinate vector of the center point of the luffing trolley is calculated. Extract the horizontal rotation angle and pitch angle from the camera's pan-tilt angle, and calculate the camera's three-dimensional rotation matrix; By constructing a yaw rotation matrix using the clockwise horizontal deflection angle of the crane boom, rotating the fixed offset vector, and superimposing it onto the three-dimensional absolute space coordinate vector, the absolute coordinates of the camera optical center in the global three-dimensional coordinate system are obtained. Extract the acquisition timestamps of sensor parameters and the capture timestamps of color video frame matrices from multi-source data, and encapsulate sensor parameters and calculation results with the same acquisition timestamp within the same acquisition period into a dynamic motion feature set.

3. The video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion as described in claim 2, characterized in that, The process involves mapping collinear edges to polar coordinate space using Hough transform, extracting polar coordinate parameter pairs based on the wire rope winding ratio, constructing and solving an overdetermined linear equation system to obtain the two-dimensional pixel center point of the virtual convergent micro-region on the imaging target surface, including: The time-domain aligned color video frame is converted into a single-channel grayscale image. Based on the three-dimensional absolute coordinates of the variable amplitude trolley, the current effective hoisting rope length, and the camera's intrinsic and extrinsic parameters, the theoretical three-dimensional envelope space of the wire rope is projected onto the current frame to generate a dynamic region of interest. Perform Canny edge detection within the dynamically active region of interest to generate a binary edge pixel matrix of candidate edge contours for the wire rope. Perform a Hough transform on the binary edge pixel matrix to map collinear edges to polar coordinate parameter space, and limit the polar angle search range according to the projection direction based on the wire rope theory. An accumulator matrix is ​​constructed in the polar coordinate parameter space and a peak search is performed. Candidate peaks are filtered based on the accumulated value, angle deviation and continuous pixel length, and valid candidate straight lines are determined by combining the wire rope winding ratio. The confidence weights are calculated based on the continuous pixel lengths of valid candidate lines, Hough cumulative response, and angle consistency. An overdetermined linear equation system is constructed and solved in a weighted manner to obtain the two-dimensional pixel center point.

4. The video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion as described in claim 3, characterized in that, The process involves extracting the two-dimensional pixel coordinates of the virtual convergent micro-region, constructing a three-dimensional global direction ray using camera inverse mapping, and building a damped numerical iteration mechanism based on mechanically estimated length as an elevation scale constraint. The output includes the three-dimensional estimated coordinates of the virtual convergent micro-region. Extract the two-dimensional pixel center points of the virtual convergent micro-region, and inversely map them into direction vectors in the local coordinate system of the camera based on the camera intrinsic parameters. Combine them with the camera instantaneous attitude parameters to convert them into global three-dimensional direction rays. The visual ray is constructed by taking the absolute coordinates of the camera's optical center in the global three-dimensional coordinate system as the starting point of the ray and constructing a ray along the global three-dimensional direction. Construct an initial mechanical constraint elevation surface, determine whether the visual ray and the initial mechanical constraint elevation surface meet the intersection conditions, and obtain the initial three-dimensional intersection coordinates when the intersection conditions are met. Calculate the visual inversion equivalent length from the three-dimensional intersection coordinates to the center point of the variable amplitude trolley, and perform damped iterative correction on the mechanical constraint elevation surface based on the scale residual between the visual inversion equivalent length and the mechanically estimated length. When the scale residual meets the preset engineering error requirement, the corresponding three-dimensional intersection coordinates are output as the three-dimensional estimated coordinates of the virtual convergent micro-region. When the maximum number of iterations is reached but the preset engineering error requirement is still not met, the three-dimensional intersection coordinates with the smallest scale residual are selected as the low-confidence three-dimensional estimated coordinates.

5. The video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion as described in claim 4, characterized in that, The calculation of the spatial sway direction vector based on the three-dimensional estimated coordinates of the virtual convergent micro-region, combined with the absolute physical rope length and the lifting device compensation, and the acquisition of the final absolute spatial target point coordinates of the hoisted object through three-dimensional vector synthesis, includes: Based on the estimated three-dimensional coordinates of the virtual convergence micro-region in the global three-dimensional coordinate system and the three-dimensional absolute spatial coordinates of the center point of the luffing trolley, the spatial swaying attitude unit direction vector from the center point of the luffing trolley to the virtual convergence micro-region is calculated. The current effective hoisting rope length is obtained based on the initial effective hoisting rope length and the vertical displacement of the hook relative to the initial zero position. Starting from the center point of the luffing trolley, extend the current effective hoisting rope length along the unit direction of the spatial swaying posture to obtain the three-dimensional absolute spatial coordinates of the hook's geometric center. Based on the equivalent axial distance from the center of the hook to the geometric center of the hoisted object, physical offset compensation is performed along the unit direction of the spatial swing posture to obtain the three-dimensional absolute spatial target point coordinates of the hoisted object.

6. The video tracking and recognition method for the spatial trajectory of hoisted objects based on multi-source data fusion as described in claim 5, characterized in that, The process involves combining the current moment with the previous sampling period's three-dimensional absolute coordinates of the hoisted object, performing filtered and smoothed velocity difference calculations, and encapsulating the video frame matrix, spatially estimated target coordinates, velocity vector, and end-to-end tracking status flags into a comprehensive tracking status data packet, including: Extract the three-dimensional absolute space target coordinates of the hoisted object at the current moment and the three-dimensional absolute space target coordinates of the hoisted object in the previous sampling period, and perform a first-order backward finite difference calculation based on the sampling time interval to obtain the original differential velocity; The original differential velocity is smoothed by a first-order low-pass filter to obtain the smoothed three-dimensional running velocity vector of the hoisted object. The system encapsulates a two-dimensional video frame matrix containing region of interest or feature calibration information, the three-dimensional absolute spatial target coordinates of the hoisted object at the current moment, the smoothed three-dimensional running speed vector, visual feature status flags, three-dimensional reconstruction confidence flags, direction validity flags, hoisting rope length validity flags, and a unified system timestamp in a structured manner to generate a comprehensive tracking status data packet.

Images