Non-intrusive dynamic sensing retrofit system for aging ship unloading machines in ports
By non-invasively acquiring the operating characteristics of the ship unloader and using a multi-source sensing module, a dynamic spatial compensation vector is constructed, suspended point clouds are filtered out, and a core material point cloud topology is generated. This solves the problems of point cloud distortion and dust interference in old ship unloaders, and achieves stability and accuracy in 3D reconstruction.
Patent Information
- Application Number
- CN202611629228.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-21
- Publication Date
- 2026-11-17
AI Technical Summary
Under conditions where the observation benchmark has nonlinear spatial jitter and is subject to interference from medium occlusion, existing technologies are difficult to effectively reduce point cloud geometric distortion and maintain the accuracy of three-dimensional topology. Especially in the dynamic unloading process of old ship unloaders in ports, point cloud data is prone to layering, ghosting and edge divergence, and dust interference leads to poor filtering effect.
A non-intrusive operation feature acquisition module monitors the status of the unloader, and a multi-source heterogeneous material perception module collects point cloud and image data. A dynamic spatial compensation vector is constructed through an edge vision computing and registration module to filter out suspended point cloud clusters and generate the core material point cloud topology. The 3D reconstruction module reconstructs the 3D topological morphology model of the remaining material in the ship's hold and performs local updates in response to grab events.
It effectively reduces coordinate deviation caused by sensor observation reference jitter, maintains the consistency of point cloud data, improves the geometric integrity and morphological accuracy of 3D reconstruction results, adapts to changes in dust concentration, fills in the missing point cloud in occluded areas, and ensures the continuity and integrity of the 3D topological morphology model.
Smart Images

Figure CN123118978A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image data processing technology, and in particular relates to a non-intrusive dynamic perception and transformation system for unloading processes of old ship unloaders in ports. Background Technology
[0002] In large-scale spatial geometric perception, point cloud sequences are typically acquired using visual and laser sensors. The spatial coordinates of the target are calculated and its geometric contour is restored through rigid registration and surface fitting between point cloud frames. This type of processing usually assumes that the observation coordinate system remains stable when performing spatial mapping. However, in the dynamic operation scenario of bulk cargo lifting equipment, mechanical transmission gaps and the impact generated by the movement of large-inertia grab buckets can cause nonlinear spatial offsets of different frequencies on the sensor mounting base. This results in the acquired point cloud sequence having pose errors in the observation coordinate system from the beginning. Under such disturbances, the static geometric reference on which conventional registration algorithms rely is no longer stable. The coordinate errors are further amplified during the spatial stitching process of multiple point cloud frames, causing the edges of the reconstructed 3D morphological model to diverge and exhibit layering and ghosting.
[0003] The large amount of dust generated during heavy-duty operations will cause significant diffuse reflection and occlusion within the field of view, resulting in numerous scattered suspended noise clusters in the point cloud data. Simply increasing the truncation threshold of the filtering distance or increasing the number of smoothing filters can remove some suspended noise, but it can also easily filter out geometric feature points on the material surface, causing local surface collapse or feature breakage in the reconstructed mesh. If global downsampling filtering is used, it is difficult to effectively distinguish between the real material boundary and high-density suspended particles, which can also easily cause 3D contour distortion. In addition to the objective limitations brought about by the physical hardware installation and operating environment, existing technologies also have limitations in software logic such as data processing and model reconstruction algorithms. Similarly, there is a lack of effective mechanisms to deal with dynamic and changing working conditions. For example, Chinese invention patent application with publication number CN118780707A discloses a method and system for unloading ships by a grab bucket unloader at a bulk cargo terminal. It collects data through joint calibration of multiple sensors, removes noise using conventional voxel filtering, constructs a three-dimensional mesh, and uses an iterative nearest-point algorithm for incremental model updates. However, this scheme is still based on the ideal assumption that the sensor base is statically rigid and fixed in terms of perception and modeling. In the face of high-frequency mechanical vibrations caused by the impact of large-inertia grab buckets in actual operations, the lack of dynamic spatial pose compensation at the data front end leads to geometric distortion of the point cloud data from the source.
[0004] Therefore, the technical problem to be solved by this invention is how to reduce point cloud geometric distortion and maintain the accuracy of dynamic updates of three-dimensional topology under the condition that the observation benchmark has nonlinear spatial jitter and is subject to medium occlusion interference. Summary of the Invention
[0005] To address the problems in the background art, the technical solution of the present invention is as follows: A non-intrusive dynamic perception retrofit system for the unloading process of aging unloading machines in ports, comprising:
[0006] A non-intrusive operation feature acquisition module is used to collect the operation status feature sequence of the ship unloader through electrical isolation monitoring;
[0007] A multi-source heterogeneous matter sensing module is used to collect multimodal spatial point cloud sequence data and visual image stream data of the work area;
[0008] The edge vision computing and registration module is used to construct a dynamic spatial compensation vector based on the positional fluctuation variance of the running state feature sequence. It compensates for the coordinate offset of discrete data points in the multimodal spatial point cloud sequence data through inverse geometric projection calculation to output the basic point cloud skeleton. It extracts the dust concentration distribution gradient from the visual image stream data. When the gradient is greater than 0.7, it steps up the filter density threshold weight to 2.5 times the basic set value. Based on this, it filters out the floating point cloud clusters in the basic point cloud skeleton to generate the core material point cloud topology, and registers it with the visual image stream data to generate multidimensional fusion scene features.
[0009] The 3D reconstruction and dynamic mapping module is used to reconstruct the 3D topological model of the ship's hold material based on the characteristics of the multi-dimensional fusion scene; in response to the event that the grab bucket changes from closed to open and the coordinates fall into the unloading area, it calculates the spatial differential update of the local working area and keeps the non-working area in a data-locked state.
[0010] Preferably, the edge vision computing and registration module is specifically used to: calculate the position fluctuation variance of the spatial position parameters in the running state feature sequence within a time window of 50ms; when the position fluctuation variance is greater than 0.15, convert the position fluctuation variance into a dynamic spatial compensation vector that characterizes the high-frequency spatial offset of the observation coordinate system; for each discrete data point in the multimodal spatial point cloud sequence data, deduct the coordinate offset component corresponding to the dynamic spatial compensation vector according to the inverse geometric projection mapping relationship, and output the basic point cloud skeleton.
[0011] Preferably, when extracting the dust concentration distribution gradient, the edge vision computing and registration module is specifically used to: extract a continuous video frame sequence from the visual image stream data, calculate the local high-frequency transmittance attenuation coefficient between adjacent video frames; calculate the environmental occlusion feature parameter characterizing the dust spatial diffusion gradient based on the spatial discrete derivative of the local high-frequency transmittance attenuation coefficient on the two-dimensional pixel grid of the image imaging plane; and normalize and map the environmental occlusion feature parameter into a dimensionless scalar with a value between 0 and 1, as the dust concentration distribution gradient.
[0012] Preferably, when the edge vision computing and registration module filters out floating point cloud clusters in the basic point cloud skeleton, it specifically performs the following steps: dividing the basic point cloud skeleton into discrete three-dimensional voxel grid units, and counting the number of discrete point clouds contained in each voxel grid unit within a preset spatial neighborhood search radius; calculating the consistency index of the spatial distribution normal vector of the point cloud within each voxel grid unit; when it is determined that the number of point clouds in the voxel grid unit is lower than the filtering density threshold weight, and the consistency index of the spatial distribution normal vector shows a discrete isotropic scattering distribution, the corresponding voxel grid unit is marked as a non-rigid floating scattering cluster and removed; retaining voxel grid units with continuous tangent plane constraints in their normal vector distribution, and aggregating them to generate the core material point cloud topology.
[0013] Preferably, when registering the core object point cloud topology with the visual image stream data, the edge vision computing and registration module is specifically used for: constructing a perspective projection matrix from the core object point cloud topology to the corresponding imaging plane of the visual image stream data according to preset calibration parameters; matching spatial point clouds and optical image frames of the same acquisition period using timestamp hardware alignment signals; projecting the three-dimensional coordinates in the core object point cloud topology onto the optical image frame through the perspective projection matrix, and extracting the optical color parameters and texture gradient parameters at the projected pixels; and generating multi-dimensional fusion scene features that integrate three-dimensional geometric spatial coordinates and two-dimensional optical texture features based on the corresponding three-dimensional spatial points using the optical color parameters and texture gradient parameters.
[0014] Preferably, the 3D reconstruction and dynamic mapping module is further used to complete the topological mesh of the data hole region caused by mechanical structure occlusion. Specifically, it is used to: identify the boundary of the data hole caused by occlusion in the multi-dimensional fusion scene features; extract the spatial first-order differential slope and second-order tangential continuous curvature features of adjacent effective 3D mesh nodes on the data hole boundary; retrieve the historical prior height parameters that coincide with the horizontal coordinates of the data hole region from the 3D topological morphology model stored in historical time series; use the historical prior height parameters as the initial surface reference, and use the spatial first-order differential slope and second-order tangential continuous curvature features as the surface boundary continuity constraint conditions, calculate the 3D spatial coordinates of the virtual depth point cloud inside the hole through the surface tension interpolation algorithm; and stitch the calculated virtual depth point cloud into the 3D topological morphology model to complete the missing topological mesh completion.
[0015] Preferably, when analyzing the event of the grab bucket changing from closed to open and its coordinates falling into the unloading area, the 3D reconstruction and dynamic mapping module is specifically used to: separate the lifting mechanism height encoding data, trolley travel position data, and grab bucket opening and closing status signal from the operating state feature sequence; calculate the current real-time spatial 3D coordinates of the grab bucket based on the lifting mechanism height encoding data and trolley travel position data; monitor the transition edge of the grab bucket opening and closing status signal, and when the signal is detected to change from closed to open and the calculated real-time spatial 3D coordinates fall within the preset unloading space envelope above the hopper, determine that the unloading physical event is established and issue a local update trigger command.
[0016] Preferably, when the 3D reconstruction and dynamic mapping module performs differential updates in the computational space and keeps the non-operational area in a data-locked state, it is specifically used for: responding to a local update trigger command, dividing a local spatial disturbance mesh envelope in the 3D topological model based on real-time spatial 3D coordinates; calculating the differential offset between the data points in the currently collected multi-dimensional fusion scene features that fall within the local spatial disturbance mesh envelope and the corresponding mesh elevation data in the 3D topological model; refreshing the residual material height topological parameters within the local spatial disturbance mesh envelope using the calculated differential offset; and keeping the 3D morphological parameters of the remaining cabin areas outside the local spatial disturbance mesh envelope in a data-locked state to prevent global geometric model reference drift.
[0017] Preferably, the multi-source heterogeneous matter perception module includes a lidar, an industrial camera, and a six-axis inertial measurement unit installed under the unloader's trolley frame. The lidar and industrial camera are used to simultaneously acquire multimodal spatial point cloud sequence data and visual image stream data. The six-axis inertial measurement unit is used to acquire inertial attitude data sequences characterizing the high-frequency angular velocity and acceleration fluctuations of the sensor's observation reference, and synchronously inputs these inertial attitude data sequences into the edge vision calculation and registration module as an attitude reference for calculating the dynamic spatial compensation vector. The non-intrusive operation feature acquisition module includes non-contact displacement sensors configured on the outside of the unloader's moving parts, used to independently acquire physical position pulse sequences characterizing the trolley's walking position and lifting height via unidirectional passive data transmission, serving as the data source for the operation state feature sequence.
[0018] Compared to existing technologies, this invention provides a non-intrusive dynamic sensing and retrofitting system for aging ship unloading machines in ports, offering the following advantages:
[0019] 1. In the non-intrusive dynamic perception transformation of the unloading process of old ship unloaders in ports, by compensating for the spatial position fluctuations generated during the operation of the ship unloader, the coordinate deviation caused by sensor observation reference jitter can be reduced when point cloud data enters the 3D reconstruction process. This allows point clouds collected at different times to maintain better spatial consistency, thereby reducing point cloud layering, contour ghosting and edge divergence caused by mechanical transmission clearance and grab swing. This makes the basic point cloud skeleton more stable and helps maintain the geometric integrity and morphological accuracy of the subsequent 3D reconstruction results.
[0020] 2. Dynamically adjusting the point cloud filtering intensity based on the changes in dust distribution reflected in the visual image allows the point cloud filtering to better adapt to the constantly changing dust concentration at the unloading site. When dust interference is strong, it can more effectively remove stray point clouds formed by suspended particles, while preserving the effective geometric features of the material surface as much as possible. This reduces the loss of entity boundaries and contour distortion that are easily caused by fixed filtering methods, thereby improving the effectiveness of the core material point cloud topology and maintaining the boundary integrity of the 3D reconstruction results under harsh optical conditions.
[0021] 3. For local point cloud missing caused by mechanical structure obstruction or blind spots, the surface change characteristics of the data hole edge and the corresponding height information in the historical 3D topology model can be used to fill in the missing area. This can make the missing area maintain a relatively continuous surface morphology with the surrounding effective point cloud. This can reduce surface collapse and false boundaries caused by local data missing, and make the completed 3D topology model still have good surface continuity and mesh integrity in the obstructed area. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the multi-source heterogeneous matter sensing and feature registration architecture of the present invention;
[0023] Figure 2 This is the local differential update diagram of the ship's hold remaining material driven by the unloading event of the present invention. Detailed Implementation
[0024] The technical solutions in the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application. A non-intrusive dynamic perception and transformation system for unloading processes of old unloading machines in ports includes: a non-intrusive operation feature acquisition module, a multi-source heterogeneous matter perception module, an edge vision computing and registration module, and a three-dimensional reconstruction and dynamic mapping module.
[0025] A non-intrusive operation feature acquisition module is used to collect the operation status feature sequence of the ship unloader through electrical isolation monitoring;
[0026] A multi-source heterogeneous matter sensing module is used to collect multimodal spatial point cloud sequence data and visual image stream data of the work area;
[0027] The edge vision computing and registration module is used to construct a dynamic spatial compensation vector based on the positional fluctuation variance of the running state feature sequence. It compensates for the coordinate offset of discrete data points in the multimodal spatial point cloud sequence data through inverse geometric projection calculation to output the basic point cloud skeleton. It extracts the dust concentration distribution gradient from the visual image stream data. When the gradient is greater than 0.7, it steps up the filter density threshold weight to 2.5 times the basic set value. Based on this, it filters out the floating point cloud clusters in the basic point cloud skeleton to generate the core material point cloud topology, and registers it with the visual image stream data to generate multidimensional fusion scene features.
[0028] The 3D reconstruction and dynamic mapping module is used to reconstruct the 3D topological model of the ship's hold material based on the characteristics of the multi-dimensional fusion scene; in response to the event that the grab bucket changes from closed to open and the coordinates fall into the unloading area, it calculates the spatial differential update of the local working area and keeps the non-working area in a data-locked state.
[0029] Preferably, the edge vision computing and registration module is specifically used to: calculate the position fluctuation variance of the spatial position parameters in the running state feature sequence within a time window of 50ms; when the position fluctuation variance is greater than 0.15, convert the position fluctuation variance into a dynamic spatial compensation vector that characterizes the high-frequency spatial offset of the observation coordinate system; for each discrete data point in the multimodal spatial point cloud sequence data, deduct the coordinate offset component corresponding to the dynamic spatial compensation vector according to the inverse geometric projection mapping relationship, and output the basic point cloud skeleton.
[0030] Preferably, when extracting the dust concentration distribution gradient, the edge vision computing and registration module is specifically used to: extract a continuous video frame sequence from the visual image stream data, calculate the local high-frequency transmittance attenuation coefficient between adjacent video frames; calculate the environmental occlusion feature parameter characterizing the dust spatial diffusion gradient based on the spatial discrete derivative of the local high-frequency transmittance attenuation coefficient on the two-dimensional pixel grid of the image imaging plane; and normalize and map the environmental occlusion feature parameter into a dimensionless scalar with a value between 0 and 1, as the dust concentration distribution gradient.
[0031] Preferably, when the edge vision computing and registration module filters out floating point cloud clusters in the basic point cloud skeleton, it specifically performs the following steps: dividing the basic point cloud skeleton into discrete three-dimensional voxel grid units, and counting the number of discrete point clouds contained in each voxel grid unit within a preset spatial neighborhood search radius; calculating the consistency index of the spatial distribution normal vector of the point cloud within each voxel grid unit; when it is determined that the number of point clouds in the voxel grid unit is lower than the filtering density threshold weight, and the consistency index of the spatial distribution normal vector shows a discrete isotropic scattering distribution, the corresponding voxel grid unit is marked as a non-rigid floating scattering cluster and removed; retaining voxel grid units with continuous tangent plane constraints in their normal vector distribution, and aggregating them to generate the core material point cloud topology.
[0032] Preferably, when registering the core object point cloud topology with the visual image stream data, the edge vision computing and registration module is specifically used for: constructing a perspective projection matrix from the core object point cloud topology to the corresponding imaging plane of the visual image stream data according to preset calibration parameters; matching spatial point clouds and optical image frames of the same acquisition period using timestamp hardware alignment signals; projecting the three-dimensional coordinates in the core object point cloud topology onto the optical image frame through the perspective projection matrix, and extracting the optical color parameters and texture gradient parameters at the projected pixels; and generating multi-dimensional fusion scene features that integrate three-dimensional geometric spatial coordinates and two-dimensional optical texture features based on the corresponding three-dimensional spatial points using the optical color parameters and texture gradient parameters.
[0033] Preferably, the 3D reconstruction and dynamic mapping module is further used to complete the topological mesh of the data hole region caused by mechanical structure occlusion. Specifically, it is used to: identify the boundary of the data hole caused by occlusion in the multi-dimensional fusion scene features; extract the spatial first-order differential slope and second-order tangential continuous curvature features of adjacent effective 3D mesh nodes on the data hole boundary; retrieve the historical prior height parameters that coincide with the horizontal coordinates of the data hole region from the 3D topological morphology model stored in historical time series; use the historical prior height parameters as the initial surface reference, and use the spatial first-order differential slope and second-order tangential continuous curvature features as the surface boundary continuity constraint conditions, calculate the 3D spatial coordinates of the virtual depth point cloud inside the hole through the surface tension interpolation algorithm; and stitch the calculated virtual depth point cloud into the 3D topological morphology model to complete the missing topological mesh completion.
[0034] Preferably, when analyzing the event of the grab bucket changing from closed to open and its coordinates falling into the unloading area, the 3D reconstruction and dynamic mapping module is specifically used to: separate the lifting mechanism height encoding data, trolley travel position data, and grab bucket opening and closing status signal from the operating state feature sequence; calculate the current real-time spatial 3D coordinates of the grab bucket based on the lifting mechanism height encoding data and trolley travel position data; monitor the transition edge of the grab bucket opening and closing status signal, and when the signal is detected to change from closed to open and the calculated real-time spatial 3D coordinates fall within the preset unloading space envelope above the hopper, determine that the unloading physical event is established and issue a local update trigger command.
[0035] Preferably, the multi-source heterogeneous matter perception module includes a lidar, an industrial camera, and a six-axis inertial measurement unit installed under the unloader's trolley frame. The lidar and industrial camera are used to simultaneously acquire multimodal spatial point cloud sequence data and visual image stream data. The six-axis inertial measurement unit is used to acquire inertial attitude data sequences characterizing the high-frequency angular velocity and acceleration fluctuations of the sensor's observation reference, and synchronously inputs these inertial attitude data sequences into the edge vision calculation and registration module as an attitude reference for calculating the dynamic spatial compensation vector. The non-intrusive operation feature acquisition module includes non-contact displacement sensors configured on the outside of the unloader's moving parts, used to independently acquire physical position pulse sequences characterizing the trolley's walking position and lifting height via unidirectional passive data transmission, serving as the data source for the operation state feature sequence.
[0036] Example 1: In the environment of large grab unloading operations at bulk cargo terminals, bridge grab unloaders that have been in service for over 10 years are subjected to the impact loads from the frequent lifting, braking, and opening / closing of large-inertia grabs. Their metal truss structure experiences continuous mechanical oscillations with alternating stress frequencies ranging from 10Hz to 80Hz. The mechanical wear of the drive transmission mechanism and the external disturbances from the sea wind combine to cause the externally mounted object sensing reference located below the trolley frame to exhibit multidimensional nonlinear random jitter in space. This directly disrupts the static premise of the reference coordinate system upon which conventional 3D point cloud reconstruction relies. Without constraints from the underlying mechanism's motion model, the projection orientation of the laser ranging sampling beam in space changes with the angular displacement and displacement jumps of the base. Transient shifts cause high-frequency spatial pose distortions to be superimposed on multiple consecutive point clouds when spliced together along the trolley's travel axis, resulting in divergent ghosting and layering artifacts at the geometric contour edges. At the same time, the high-speed diffuse dust cloud generated by the grab bucket pouring iron ore into the hopper diffuses in the near-field working space, forming strong diffuse reflection and light transmission attenuation in the optical imaging field of view. This creates a large area of false reflection points in a non-rigid suspended state in the spatial point cloud. If the distance truncation threshold is blindly increased or global smoothing filters are applied to filter them out, the fine geometric feature points on the surface of the stockpile will inevitably be reduced simultaneously, leading to local collapse and breakage of the mesh surface. This is a typical working condition that causes distortion of the basic point cloud skeleton of the external sensing system and failure of spatial topology reconstruction.
[0037] Non-contact displacement sensors, configured on the outside of the moving parts of the ship unloader, independently acquire physical position pulse sequences characterizing the trolley's traveling position and lifting height via unidirectional passive data transmission. The transmission link remains read-only, serving as the data source for the operational status characteristic sequence. Throughout the process, they maintain electrical isolation from the underlying programmable logic controller of older equipment. In a multi-source heterogeneous matter sensing module installed below the trolley frame, a lidar, industrial camera, and six-axis inertial measurement unit (IMU) operate collaboratively. The lidar and industrial camera simultaneously acquire multimodal spatial point cloud sequence data and visual image stream data. The IMU simultaneously acquires inertial attitude data sequences characterizing the high-frequency angular velocity and acceleration fluctuations of the sensor's observation reference. These inertial attitude data sequences are input into the edge vision calculation and registration module as the attitude reference for calculating the dynamic spatial compensation vector. The edge vision calculation and registration module selects a 50ms sliding time window and statistically calculates the spatial position parameters in the operational status characteristic sequence within this window. The variance of the position fluctuation relative to the mean trajectory quantitatively characterizes the transient high-frequency discrete divergence caused by mechanical transmission vibration and grab sway. When the detected position fluctuation variance exceeds the preset 0.15 geometric distortion trigger benchmark, the equipment is determined to be in a state of severe non-rigid vibration. The edge vision computing and registration module converts this position fluctuation variance into a dynamic spatial compensation vector characterizing the high-frequency offset of the observed coordinate system, and calls the high-frequency angular velocity and instantaneous acceleration components output by the six-axis inertial measurement unit as the attitude reference benchmark to correct the spatial direction and components of the dynamic spatial compensation vector. Subsequently, for each discrete spatial sampling data point in the multimodal spatial point cloud sequence data, the edge vision computing and registration module establishes an inverse mapping constraint relationship between the spatial coordinates of the data point and the dynamic spatial compensation vector. Based on the inverse geometric projection mapping relationship, the transient coordinate offset component corresponding to the dynamic spatial compensation vector is subtracted from the original three-dimensional coordinate vector of the discrete data point. The specific relationship is expressed as follows: ,in, The original three-dimensional spatial coordinate vectors of the collected discrete data points. To calculate the output dynamic space compensation vector, To eliminate the transient coordinate offset in the output corrected spatial coordinate vector, this inverse geometric projection mechanism offsets and eliminates the high-frequency jitter component of the observation reference at the initial moment of spatial point cloud data intake. It avoids solving the multi-degree-of-freedom kinematic inverse equations of the mechanism, allowing the discrete point cloud sequence to reconverge to the actual geometric reference without electrical circuit interference, outputting a geometrically topologically convergent basic point cloud skeleton. In the specific implementation of eliminating the combined effects of rotational jitter and linear translation, it addresses the distortion at the lever arm end amplified by a small angular displacement at a spatial projection distance of tens of meters for the sensor base. The high-frequency angular velocity measured by the six-axis inertial measurement unit... After updating the attitude rotation matrix, a rotation transformation is first applied to the measurement vector from the lidar transmission center to the spatial discrete sampling points. The local rotational linear displacement induced at each far-end spatial sampling point due to the tilt of the observation reference is calculated. Then, this local rotational linear displacement is superimposed with the base rigid body displacement obtained by integrating the translational acceleration of the trolley. This superposition is used to correct the dynamic spatial compensation vector obtained by the position fluctuation variance transformation, so that it corresponds to the current specific spatial sampling point. This allows the coordinate subtraction operation to simultaneously compensate for the rotation extrapolation error and the translational displacement component, eliminating the point cloud fan-shaped layering and divergence artifacts caused by small angle deflection under large ranging conditions.
[0038] Faced with severe diffuse reflection interference caused by high wind speeds and falling dust in the near-field space, the edge vision computing and registration module extracts continuous video frame sequences from the visual image stream data acquired by the industrial camera, statistically analyzes the local high-frequency transmittance attenuation coefficient between adjacent video frames, and calculates the spatial discrete derivative of this attenuation coefficient on the two-dimensional pixel grid of the image imaging plane. Based on this, it solves for the environmental shading feature parameter characterizing the spatial diffusion gradient of dust, and then normalizes and maps this environmental shading feature parameter to a dimensionless scalar with a value between 0 and 1, as a dust concentration distribution gradient characterizing the current working surface medium turbidity diffusion rate; the edge vision computing and registration module establishes this dust The system dynamically maps the dust concentration distribution gradient to the filter density threshold weight. When the dust concentration distribution gradient is between 0.4 and 0.7, the system synchronously increases the filter density threshold weight according to a linear proportional mapping rule. When the dust concentration distribution gradient exceeds the preset trigger threshold of 0.7, the system applies a nonlinear step increase control to the filter density threshold weight, causing it to jump directly to 2.5 times the base set value, thus forming a strong filtering constraint for local suspended particle clouds. The filter density threshold weight represents the threshold for the number of voxels in the point cloud after adjustment by the dust concentration distribution gradient. The base set value is the point cloud number threshold used when dynamic adjustment is not triggered. Therefore... This threshold can be directly compared with the number of discrete point clouds within a voxel grid unit. In the 3D spatial filtering operation, the edge vision calculation and registration module divides the basic point cloud skeleton into discrete 3D voxel grid units. Within a preset spatial neighborhood search radius, it counts the number of discrete point clouds contained in each voxel grid unit and calculates the spatial distribution normal vector consistency index of the discrete point clouds within the unit. When the number of point clouds within a voxel grid unit is lower than the filter density threshold weight after the step increase mentioned above, and the spatial distribution normal vector consistency index shows an isotropic discrete scattering distribution, the corresponding point cluster is determined to belong to a non-rigid suspended scattering cluster and is removed, retaining only those with continuous normal vector distribution. The voxel grid cells with physical constraints on the tangent plane collect and output the core material point cloud topology representing the physical surface of the physical material pile. Subsequently, the edge vision computing and registration module constructs a perspective projection matrix from the core material point cloud topology to the imaging plane of the visual image stream based on the preset camera calibration parameters. It uses the timestamp hardware alignment signal to match the spatial point cloud and optical image frame in the same acquisition period. The three-dimensional coordinates in the core material point cloud topology are projected onto the optical image frame through the perspective projection matrix. The optical color parameters and texture gradient parameters at the projected pixels are extracted. Based on the corresponding three-dimensional spatial points, a multi-dimensional fused scene feature containing three-dimensional geometric spatial coordinates and two-dimensional optical texture features is output.
[0039] To address the blind spots caused by the grab bucket's mechanical structure or unloading mechanism at specific opening and closing angles, the 3D reconstruction and dynamic mapping module extracts the spatial depth information of the remaining material inside the hold from multi-dimensional fusion scene features, reconstructs a 3D topological model representing the distribution of the remaining material in the hold, and specifically identifies the boundaries of data voids caused by blind spots or mechanical entity obstruction. The module extracts the spatial first-order differential slope and second-order tangential continuous curvature features of adjacent effective 3D mesh nodes on the data void boundary, quantitatively characterizing the local geometric curvature trend of the actual material pile surface around the void. Simultaneously, it retrieves historical prior height parameters that coincide with the horizontal coordinates of the data void region from the historically stored 3D topological model. These retrieved historical prior height parameters serve as the initial value benchmark for iteratively solving partial differential equations, rather than a hard elevation constraint. When the latest measured mesh node on the void boundary experiences an elevation jump due to the current grab or unloading operation, the boundary tangential... The axial slope and curvature constraints will apply dynamic transmission updates to the cavity during the iteration process, so that the elevation of the reconstructed surface evolves synchronously with the actual changes in the boundary. The historical prior height provides a base elevation reference in the boundaryless deformation area, so that the interpolation result matches the real-time dynamic excavation shape. The retrieved historical prior height parameters are used as the initial elevation benchmark for geometric surface reconstruction. The spatial first-order differential slope and second-order tangential continuous curvature features are used as the constraint conditions of the partial differential equation for the continuity of the surface boundary. The 3D reconstruction and dynamic mapping module solves the virtual depth point cloud 3D spatial coordinates of each grid vertex inside the cavity through the surface tension interpolation algorithm, so that the interpolated reconstructed surface smoothly matches the material evolution history before occlusion in terms of physical elevation. The continuous and smooth transition of the tangential geometric slope is maintained at the grid boundary, eliminating the false edges and topological holes caused by the sudden change of the occlusion blind zone, and outputting a 3D topological morphology model of the ship's cargo hold material with complete spatial geometric grid topology and continuous surface.
[0040] During the continuous unloading and material handling phase, the 3D reconstruction and dynamic mapping module separates the hoisting mechanism height encoding data, trolley travel position data, and grab bucket opening and closing status signals in real time from the self-operating state feature sequence. Based on the hoisting mechanism height encoding data and trolley travel position data, it calculates the grab bucket's current real-time 3D spatial coordinates. Specifically, using the spatial coordinate system adopted by the 3D topological model as a unified reference, and based on the known installation geometry of the unloader, the hoisting mechanism height encoding data and trolley travel position data are converted into spatial position components in this coordinate system and combined to obtain the grab bucket's current real-time 3D spatial coordinates, which are then continuously monitored. The method for non-intrusive acquisition of the grab bucket opening and closing status signal involves detecting the transition edge of the signal. Specifically, the non-intrusive acquisition method for the grab bucket opening and closing status signal is as follows: a through-type current induction transformer is installed on the surface of the power supply line of the opening and closing drum drive contactor outside the electrical control cabinet of the ship unloader, and a non-contact Hall effect sensor switch is installed on the side of the opening and closing wire rope drum. When the grab bucket performs an opening or closing action, the current induction transformer detects the transition of the working current of the drive circuit bus and outputs an induced analog quantity. The Hall effect sensor switch synchronously detects the drum rotation pulse. The induced signal is converted into a pure digital high and low level logic sequence by an opto-isolated sampling board, thereby completely decoupling from the internal circuit of the underlying controller. Under these conditions, electrical isolation monitoring and feature separation of the grab bucket's opening and closing status are achieved in a unidirectional manner. When the grab bucket's opening and closing status signal is detected to change from closed to open, and the calculated real-time spatial 3D coordinates fall within the preset unloading space envelope above the hopper, the 3D reconstruction and dynamic mapping module determines that the physical event of bulk material falling has occurred and issues a local update trigger command. In response to the local update trigger command, the 3D reconstruction and dynamic mapping module delineates a local spatial perturbation grid envelope in the 3D topology model based on the real-time spatial 3D coordinates. Only data points in the currently collected multi-dimensional fusion scene features that fall within the local spatial perturbation grid envelope are compared with the 3D coordinates. The spatial differential offset is calculated based on the elevation data of the corresponding grid in the 3D topological model. The calculated differential offset is used to refresh the topological parameters of the remaining material height inside the local spatial disturbance grid envelope. At the same time, the 3D morphological parameters of the remaining non-operational cabin areas outside the local spatial disturbance grid envelope are forcibly kept in a data-locked state. This event-driven differential update mechanism avoids the computational power consumption caused by indiscriminate reconstruction of the entire scene and blocks the geometric elevation accumulation drift caused by the small noise of repeated point cloud registration in non-operational areas. This allows the entire sensing system to output a 3D dynamic distribution model of the remaining material in the cabin through geometric calculation without being connected to the underlying control loop of the equipment.
[0041] Example 2: A test system for verifying the non-intrusive dynamic perception retrofit system for aging ship unloaders in ports was installed on the structural base of a 12-year-old bridge grab unloader at a continuously operating bulk cargo terminal. The trolley's full travel distance is 38 meters, and its rated lifting load is 35 tons. Multi-source heterogeneous matter perception modules are deployed below the trolley frame structure. The multi-line lidar has a ranging range of 0.5 meters to 60 meters, a ranging accuracy better than ±20 mm, and a scanning frequency of 20 Hz. The industrial camera has an imaging resolution of 2048×1536 pixels, a frame rate of 30 frames per second, and a dynamic range of no less than 65 dB. The six-axis inertial measurement unit has a sampling frequency of 200 Hz, an angular velocity range of ±300 degrees per second, and an acceleration range of ±5g. Simultaneously, non-contact sensors are configured on the outside of the trolley's traveling mechanism and the lifting drum. The touch displacement sensor outputs a physical position pulse sequence at a sampling frequency of 50Hz. It maintains photoelectric isolation from the underlying electrical control cabinet throughout the process. The collected data is transmitted unidirectionally to the edge computing processing hardware node via the internal bus. The test process covers the entire dynamic working conditions of iron ore unloading, trolley high-speed acceleration, trolley braking, and grab bucket dumping bulk material onto the ground. The fundamental frequency of the base vibration generated by the mechanical movement is in the range of 10Hz to 80Hz, and the instantaneous sway acceleration peak reaches 0.8g, forming a diffuse dust cloud with optical high-frequency attenuation around the hopper. To test the suppression effectiveness of the sensing system under actual working conditions, Gaussian white noise with a signal-to-noise ratio of 25dB is superimposed on the laser ranging and optical imaging signal transmission link to simulate sensor housing vibration and high-frequency electromagnetic crosstalk caused by strong sea winds in the port.
[0042] The sliding time window length is set to 50ms to balance the sampling response and computational delay of high-frequency mechanical vibration. If the time window is shortened to 20ms, the number of discrete pulse samples included in the window at a 50Hz position sampling rate is too small, making it susceptible to spurious variance jumps induced by single pulse quantization glitches. If the time window is extended to more than 100ms, the peaks and troughs of high-frequency vibrations will be smoothed and filtered in the mean statistics, resulting in a loss of the ability to capture transient displacement jumps. From the analysis of the vibration spectrum response characteristics of the mechanism, the natural main vibration frequencies of the main beam and trolley frame of the old ship unloader under the impact of the grab bucket are concentrated in the 10Hz to 36Hz frequency band. The 50ms time window corresponds to 0.5 to 1.8 complete vibration cycles of the above-mentioned main vibration frequencies. Taking into account the half-wave peak displacement capture and dynamic attitude calculation delay, the position fluctuation variance threshold is set to 0.15, which is based on the boundary measurement results of the equipment's inherent mild vibration during uniform sliding and heavy-load impact oscillation. The trolley runs uniformly on the track and the grab bucket When stationary, the position fluctuation variance remains stable between 0.04 and 0.11. When the grab closes for lifting or a sudden stop causes alternating swaying of the main beam, this variance rapidly increases and exceeds 0.15, serving as the trigger benchmark for activating the inverse geometric projection calculation. This position fluctuation variance is not directly related to the original absolute spatial displacement in meters, but rather it is a dimensionless variance index calculated by converting the pulse count value output by the non-contact displacement sensor into a normalized displacement unit based on 50 millimeters within a 50ms time window. When the position fluctuation variance is 0.32, the corresponding normalized standard deviation is 0.566, and the root mean square amplitude of the jitter converted to the real physical spatial displacement is 28.3 millimeters. The edge vision calculation and registration module converts this position fluctuation variance into a dynamic spatial compensation vector, and uses the inertial attitude data sequence as the attitude reference benchmark. After passing through a passband filter to eliminate some high-frequency quantization white noise, a dynamic spatial compensation vector with a modulus of 26.8 millimeters is obtained.
[0043] For the environmental occlusion feature parameters obtained from visual image stream parsing, they are normalized and mapped to a dust concentration distribution gradient ranging from 0 to 1. When this gradient is in the range of 0.4 to 0.7, the filter density threshold weight is linearly compensated proportionally with the increase of dust concentration. When the dust concentration distribution gradient exceeds the critical point of 0.7, a step control is applied to directly increase the filter density threshold weight to 2.5 times the basic set value. Experiments further show that when the step factor exceeds 3.5 times, effective material pile point clouds at high curvature edges will be removed as well. In the material discharge test, a dust concentration distribution gradient of 0.7 corresponds to a critical condition where the near-field effective optical visibility is reduced to below 4 meters and the high-frequency optical attenuation rate exceeds 68%. Under this medium turbidity, the spatial density of false noise points formed by the backscattering of suspended dust in a single laser detection beam increases by nearly 3 times. After selecting the weight enhancement ratio as 2.5 times, the density removal threshold of the voxel filter covers the upper limit of the density statistics of suspended dust point clusters and maintains a safety isolation margin of more than 15% with the lower limit of the natural aggregation density of the solid point cloud at the repose angle slope of the material pile.
[0044] To verify the coordinate offset cancellation, spatial suspension noise filtering, local occlusion mesh completion, and geometric reference locking in the non-operational area during the initial data intake stage, test sets were set up for the present invention's sample groups 1 to 5 and control sample groups A to G. Sample groups 1 to 5 correspond to moderate disturbance, heavy load and strong vibration, extreme shaking and strong dust, lower limit critical, and upper limit critical conditions, respectively. Control sample group A removed the dynamic spatial compensation vector and inverse geometric projection driven by position fluctuation variance, and only used static sensor calibration extrinsic parameters as input and directly used conventional point cloud rigid registration. Control sample group B removed the adaptive step filter driven by dust concentration distribution gradient, and used fixed distance threshold truncation and global spatial filtering. Control sample group C removed the virtual depth point cloud reconstruction based on surface tension interpolation, and directly used simple neighborhood plane interpolation for occluded data holes. The values were as follows: Control group D removed the local differential space update in response to the grab opening and closing event, and performed indiscriminate global point cloud repeated registration and periodic reconstruction on the entire ship compartment; Control group E set the sliding time window to 20ms and the variance trigger benchmark to 0.08; Control group F only increased the filter density threshold weight to 1.2 times when the dust concentration distribution gradient was greater than 0.7; Control group G increased the filter density threshold weight to 3.8 times when the dust concentration distribution gradient was greater than 0.7. Each test group continuously collected 300 sets of time series data under the same physical working conditions, and quantitatively tested the point cloud contour average geometric deviation, suspension noise removal rate, entity boundary feature fidelity, topological mesh continuity curvature residual, local differential update single cycle time, and cumulative elevation drift in non-operation areas.
[0045] In the dynamic spatial compensation and point cloud distortion correction link, the position fluctuation variance of sample group 1 of this invention is 0.18, the dynamic spatial compensation vector magnitude is 14.2 mm, and the average deviation of the point cloud contour after distortion correction is 11.6 mm; sample group 2 of this invention corresponds to 0.32, 26.8 mm, and 13.4 mm; sample group 3 of this invention corresponds to 0.46, 38.5 mm, and 15.2 mm; sample group 4 of this invention corresponds to 0.15, 11.5 mm, and 10.8 mm; sample group 5 of this invention corresponds to 0.6, 51.2 mm, and 16.8 mm. Under the same position fluctuation variance of 0.32, control sample group A did not apply dynamic spatial compensation, the dynamic spatial compensation vector magnitude was 0 mm, and the average deviation of the point cloud contour increased to 68.4 mm; control samples B, C, The dynamic spatial compensation vector magnitudes of D and E are 27.1 mm, 26.5 mm, 26.8 mm, and 22.4 mm, respectively, corresponding to average deviations of the point cloud contours of 14.1 mm, 13.9 mm, 13.6 mm, and 34.2 mm, respectively. When the position fluctuation variance is 0.46, the dynamic spatial compensation vector magnitudes of control groups F and G are 38.2 mm and 38.6 mm, respectively, corresponding to average deviations of the point cloud contours of 15.6 mm and 17.5 mm, respectively. After canceling the inverse geometric compensation in control group A, the high-frequency tilt disturbance of the observation benchmark is directly incorporated into the temporal point cloud stitching. Control group E uses a 20ms time window, and the discrete jump of position sampling increases the noise of variance calculation, and the compensation result deviates from the present invention's sample group 2 under the same 0.32 working condition.
[0046] In the dust adaptive filtering link, the dust concentration distribution gradients of samples 1 to 5 of this invention are 0.45, 0.62, 0.82, 0.4, and 0.7, respectively, corresponding to filter density threshold weight multiples of 1.25, 1.73, 2.5, 1, and 2, respectively. The suspended noise removal rates are 96.2%, 97.5%, 99.1%, 95.1%, and 98.3%, respectively, and the fidelity of entity boundary features are 98.4%, 97.6%, 96.8%, 98.9%, and 97.1%, respectively. The dust concentration distribution gradient of control sample A is 0.62, the filter density threshold weight multiple is 1.73, the suspended noise removal rate is 91.2%, and the fidelity of entity boundary features is 82.5%. The dust concentration distribution gradient of control sample B is 0.82, the filter density threshold weight multiple is 1, and the suspended noise removal rate drops to 71%. 3%, with a solid boundary feature fidelity of 80.4%; the dust concentration distribution gradient of control groups C, D, and E is 0.62, and the filter density threshold weight factor is 1.73, corresponding to suspended noise removal rates of 97.2%, 97.4%, and 96.8%, and solid boundary feature fidelity of 97.4%, 97.5%, and 95.2%, respectively. Under the condition that the dust concentration distribution gradient is 0.82, control group F uses the 1.2x step condition, with a suspended noise removal rate of 78.4% and a solid boundary feature fidelity of 97.2%; control group G uses the 3.8x step condition, with a suspended noise removal rate of 99.6%, but the solid boundary feature fidelity drops to 73.8%. When the fixed parameter filtering and step factor are too low, the number of residual suspended point clouds increases; when the step factor is too high, the point cloud at the edge of the solid material pile is lost simultaneously.
[0047] In the 3D topology completion and dynamic update link, the topology mesh continuity curvature residuals of samples 1 to 5 of this invention are 0.28×10⁻³mm⁻¹, 0.31×10⁻³mm⁻¹, 0.35×10⁻³mm⁻¹, 0.26×10⁻³mm⁻¹, and 0.34×10⁻³mm⁻¹, respectively. The single-cycle local update times are 48.5ms, 51.2ms, 53.6ms, 46.2ms, and 52.8ms, respectively, and the cumulative global elevation drifts are 2.1 mm, 2.4 mm, 2.8 mm, and 1 mm, respectively. For control group A (8 mm and 2.6 mm), the topological mesh continuity curvature residual was 1.42 × 10⁻³ mm⁻¹, the local update cycle time was 52 ms, and the global elevation cumulative drift was 14.5 mm; control group B had 0.86 × 10⁻³ mm⁻¹, 54.1 ms, and 18.2 mm; control group C had 2.15 × 10⁻³ mm⁻¹, 50.8 ms, and 3.2 mm, where simple interpolation in the neighborhood plane caused obvious geometric kinks at the edges of data holes; control group D had 0.32 × 10⁻³ mm⁻¹. ¹, 385ms and 36.8 mm, its full-scene repeated registration and periodic reconstruction significantly increased the single-cycle time and the cumulative elevation drift in non-operational areas; control sample group E corresponds to 0.48×10⁻³mm⁻¹, 76.5ms and 8.4 mm; control sample group F corresponds to 0.62×10⁻³mm⁻¹, 53.2ms and 15.6 mm; control sample group G corresponds to 0.95×10⁻³mm⁻¹, 55.4ms and 3.1 mm. In the occlusion area processing of the sample groups of this invention, the first-order differential slope of the cavity boundary and the second-order tangential continuous curve The rate feature is combined with historical time series prior height data to participate in surface tension interpolation, and the curvature residual is kept within the range corresponding to the above-mentioned sample group of the present invention. During the dynamic update process, when the grab bucket changes from closed to open and the real-time spatial three-dimensional coordinates fall into the preset unloading space envelope above the hopper, a local update trigger command is generated, and the local spatial disturbance grid envelope is divided in the three-dimensional topological morphology model according to the real-time spatial three-dimensional coordinates. The differential offset is calculated only for the data points within the envelope and the corresponding grid elevation data, while keeping the three-dimensional morphological parameters of the non-operational cabin area outside the envelope in a data-locked state.
[0048] Under the aforementioned multi-condition and control conditions, the dynamic spatial compensation driven by position fluctuation variance, the filter density threshold weight adjustment driven by dust concentration distribution gradient, the surface tension interpolation of data void regions, and the local differential update triggered by the opening and closing events of the grab bucket correspond to the changes in the average geometric deviation of the point cloud contour, the suspension noise removal rate and the fidelity of the entity boundary features, the curvature residual of the topological mesh continuity, and the single-cycle time consumption of local update and the cumulative drift of global elevation, respectively, thus realizing the corresponding mapping between the processing process and the quantization results.
[0049] Example 3: Addressing the phenomenon of spatial nonlinear jitter in the external object sensing reference of a bulk crane due to mechanical vibration, a non-contact displacement sensor outputs a physical position pulse sequence at a sampling frequency of 50Hz to the edge vision calculation and registration module. The system extracts the position fluctuation variance within a selected 50ms sliding time window. When the position fluctuation variance is greater than 0.15, the edge vision calculation and registration module converts the position fluctuation variance into a dynamic spatial compensation vector characterizing the high-frequency spatial offset of the observation coordinate system. Using the high-frequency angular velocity vector and three-axis acceleration vector output by the six-axis inertial measurement unit as the attitude reference, the spatial direction and components of the dynamic spatial compensation vector are corrected. Specifically, using the start of the sliding time window as the reference point, the attitude rotation matrix is updated by applying the Rodrigues rotation formula integral to the angular velocity vector. Simultaneously, a second time integral is applied to the three-axis acceleration vector after deducting the gravity component to obtain the instantaneous translational drift component of the observation reference. This translational drift component is then adaptively weighted and synthesized with the high-frequency position pulse jitter component measured by the non-contact displacement sensor to correct the dynamic spatial compensation vector obtained from the position fluctuation variance conversion. ,in, The transient displacement component is along the direction of travel of the trolley. The transient displacement component is along the direction of travel of the trolley. For the transient displacement offset component in the vertical lifting direction, adaptive weighted synthesis employs a weight allocation logic based on frequency band complementarity: in the low-frequency displacement band below 5Hz, the position pulse increment of the non-contact displacement sensor is assigned a reference weight of 0.85 to provide an absolute position reference and suppress low-frequency cumulative drift caused by inertial double time integration; in the high-frequency impact band from 5Hz to 80Hz, the acceleration time quadratic integral component is assigned a dynamic weight of 0.75 to capture transient vibration response, and the two are weighted and summed according to the product of discrete frequency responses. While maintaining the stability of the absolute geometric origin, high-frequency swaying displacement is tracked. For each discrete sampling point in the multimodal spatial point cloud sequence data, a three-dimensional coordinate inverse subtraction constraint mapping relationship is established, based on... In-situ subtraction of spatial coordinate offset, where, This refers to the discrete three-dimensional coordinate vector measured by the lidar in the original reference coordinate system of the sensor. The corrected spatial coordinate vector is output to the basic point cloud skeleton to offset the high-frequency jitter of the observation reference at the point cloud ingestion front end, and output the basic point cloud skeleton with converged point cloud edges.
[0050] Faced with the interference of large-scale suspended dust in port material handling conditions, the edge vision computing and registration module extracts a continuous video frame sequence from the visual image stream data acquired by industrial cameras, obtains the grayscale difference image between the current video frame and the previous video frame, calculates the local high-frequency transmittance attenuation coefficient of the local high-frequency region on the two-dimensional imaging pixel grid, and uses the discrete Laplacian spatial convolution operator to calculate the two-dimensional spatial discrete derivative of the coefficient in the image imaging plane, thereby obtaining the environmental occlusion feature parameters. These environmental occlusion feature parameters are then mapped to a preset calibration interval and subjected to a minimax normalization transformation to obtain a dimensionless scalar between 0 and 1 as the dust concentration. In extracting the local high-frequency transmittance attenuation coefficient, the distribution gradient is first determined by generating a two-dimensional projection mask of the grab bucket's rigid body structure on the image plane based on the real-time three-dimensional spatial coordinates and perspective projection matrix of the grab bucket. This mask removes the pixel regions of the grab bucket and the material dropping rigid body motion, isolating the differential interference caused by the displacement of the foreground material. For static background areas, the high-frequency texture gradient magnitude is extracted within local image blocks of adjacent video frames. The light intensity transmission attenuation is determined based on the ratio of the high-frequency component amplitude of the corresponding local block in the dust-free reference image to that in the current image. The local high-frequency transmittance attenuation coefficient changes continuously between 0 and 1. The lower the ratio, the stronger the shading and scattering of optical imaging by suspended dust particles.
[0051] When the dust concentration distribution gradient is between 0.4 and 0.7, a linear gain function with a slope of 3.33 is used to increase the filter density threshold weight in the same direction. When the dust concentration distribution gradient exceeds 0.7, a discrete step gain mechanism is triggered, locking the filter density threshold weight to 2.5 times the base value. The edge vision calculation and registration module divides the base point cloud skeleton into cubic 3D voxel grid units with a side length of 0.05 meters. For voxel grid units where the number of internal point clouds is lower than the filter density threshold weight, the three eigenvalues of the covariance matrix of all point cloud coordinates within the unit are calculated. The ratio of the minimum eigenvalue to the sum of the three eigenvalues is used as the spatial distribution normal vector consistency index. When the spatial distribution normal vector consistency index is greater than 0.25, the point clouds inside the voxel grid unit are determined to have an isotropic discrete spatial scattering distribution. The unit is marked as a non-rigid suspended scattering cluster and removed from the base point cloud skeleton. Elimination: When the consistency index of spatial distribution normal vector is less than or equal to 0.25, it is determined that there is a continuous tangent plane geometric constraint and the corresponding point cloud is retained. The core material point cloud topology is generated by aggregation. Before eigenvalue decomposition, the number of sampling points in each voxel grid cell is critically judged: if the number of discrete point clouds contained in the cell is less than 3, since it is geometrically impossible to form a three-dimensional local space plane with a definite normal vector, the covariance matrix cannot calculate the complete spatial distribution feature value, and the voxel is directly judged as an under-constrained isolated floating noise point and eliminated; eigenvalue decomposition and ratio criterion are only started when the number of sampling points in the voxel is greater than or equal to 3. Then, through the perspective projection matrix corresponding to the camera extrinsic parameters, the three-dimensional coordinates in the core material point cloud topology are projected onto the synchronously acquired optical image frame. The optical color parameters and texture gradient parameters at the projected pixels are extracted and multi-dimensional fused scene features are generated based on the corresponding three-dimensional space points.
[0052] To address the blind spots caused by the opening and closing of the grab bucket and the unloading and descent of materials, the 3D reconstruction and dynamic mapping module locates data voids in the core material point cloud topology where the point cloud density suddenly drops to zero. A ring-shaped boundary loop consisting of three layers of effective 3D mesh nodes is extracted along the closed outer contour of the data void region. A second-order tangential continuous curvature feature and a first-order spatial differential slope are established at each boundary mesh node. Simultaneously, historical prior height parameters coinciding with the mesh coordinates inside the void are retrieved from the historically stored 3D topological model. Using these historical prior height parameters as the initial elevation input, the differential equation satisfying the biharmonic tension spline is solved. The surface control network, in which, It is a bitone differential operator. For the Laplace differential operator, The virtual depth point cloud elevation parameters to be solved are: To preset the surface tension adjustment coefficient, a value of 1.85 was taken under the calibration test of the angle of repose of bulk iron ore material. The differential equation uses the second-order tangential continuous curvature characteristics of the boundary ring grid nodes and the first-order spatial differential slope as the Dirichlet and Neumann joint boundary conditions. The virtual depth point cloud three-dimensional spatial coordinates of each discrete grid node inside the data cavity are solved by the finite difference super-relaxation iteration method, so that the reconstructed surface inside the cavity fits the natural angle of repose geometric slope of the outer material pile, and the second-order continuity of the tangential plane normal vector is maintained at the edge of the cavity, thus completing the local missing topological mesh.
[0053] In the mapping between bulk material mechanical properties and geometric continuous reconstruction, the bulk iron ore pile forms a stable natural angle of repose under the combined action of macroscopic gravity and interparticle friction. The biharmonic tension spline model treats the pile surface as a continuous displacement field that satisfies the local energy minimization constraint. The surface tension adjustment coefficient is used to characterize the stiffness suppression damping of the macroscopic reconstruction surface for high-frequency bending deformation. Its value is determined based on the slope of the boundary tangent of the measured natural angle of repose of iron ore from 36 to 38 degrees. When the discrete difference grid step size is 0.05 meters and the coefficient is calibrated to 1.85, the second derivative tension term and the fourth derivative bending term in the partial differential equation of elevation to be solved are balanced at the boundary to suppress the boundary overshoot oscillation generated by pure biharmonic interpolation and constrain the maximum slope of the composite surface inside the cavity to not exceed the bulk material self-locking limit.
[0054] During the dynamic unloading phase, the 3D reconstruction and dynamic mapping module analyzes the lifting mechanism height encoding data formed by the lifting wire rope encoding pulse and the trolley travel position data formed by the trolley track displacement signal from the operating status feature sequence obtained by one-way monitoring. It calculates the current real-time 3D spatial coordinates of the grab bucket and monitors the transition edges of the grab bucket opening / closing status signal in real time. When the grab bucket opening / closing status signal transitions from a high-level closed state to a low-level open state, and the real-time 3D spatial coordinates fall within the preset unloading space envelope above the hopper receiving port, a local update trigger command is generated. Responding to the local update trigger command, the 3D reconstruction and dynamic mapping module divides the local spatial disturbance mesh envelope in the 3D topology model based on the real-time 3D spatial coordinates. Specifically, the geometric center is the projection point of the grab bucket's real-time 3D spatial coordinates onto the corresponding spatial position in the 3D topology model. A circular horizontal geometric influence domain, 1.5 times the maximum opening of the grab bucket, is taken outward as the local spatial disturbance grid envelope. The differential offset is calculated only for the data points falling into this envelope from the currently captured multi-dimensional fusion scene features and the corresponding grid elevation data of the 3D topology model. The differential offset is then used to refresh the residual material height topology parameters within the envelope. A ring-shaped differential fusion transition zone with a width of 0.3 meters is set at the outer edge of the local spatial disturbance grid envelope. The distance weighting coefficient within the transition zone decreases linearly from 1 to 0 as the distance from the geometric center increases, which is used to smoothly refresh the local grid elevation. At the same time, a write lock operation is applied to all cabin grid elevation parameters outside the local spatial disturbance grid envelope and its transition zone to keep them in a data locked state, blocking the geometric drift error introduced by multi-temporal point cloud splicing in non-operational areas and maintaining the continuous update of the 3D topology model.
[0055] In a complete unloading cycle, the material handling location record and unloading event confirmation are saved in two time-series stages. When the grab bucket enters the ship's hold to perform a closing grab action, the system records the transient three-dimensional spatial coordinates at the moment the grab bucket closes and records the corresponding material handling pit in the three-dimensional topology model. The grab bucket then rises and moves along the track to the top of the shore-side hopper. When it is detected that the grab bucket changes from a closed state to an open state within the preset unloading space envelope, this opening event is used as the trigger criterion for the completion of a single grab cycle. The previously recorded material handling coordinates inside the ship's hold are used to maintain the correspondence between the material handling pit and the unloading event in the same grab cycle, and do not replace the real-time three-dimensional spatial coordinates used to divide the local spatial disturbance mesh envelope after the local update is triggered. The local spatial disturbance mesh envelope is still divided in the three-dimensional topology model according to the real-time three-dimensional spatial coordinates at the time of triggering, and is updated according to the elevation difference offset of the corresponding mesh. This preserves the operational event association between the material handling stage and the unloading stage, while ensuring that the division and differential update of the local spatial disturbance mesh envelope use the same real-time three-dimensional spatial coordinate path.
[0056] Example 4: This example combines Figures 1 to 2 This paper provides a detailed explanation of the collaborative working logic and operation process of a non-intrusive dynamic perception and transformation system for the unloading process of aging ship unloaders in ports, such as... Figure 1 As shown, the multi-source heterogeneous matter perception and feature registration architecture of this invention is mainly composed of a non-intrusive operation feature acquisition module, a multi-source heterogeneous matter perception module, a dynamic spatial compensation vector calculation stage, a core matter point cloud topology generation stage, an edge vision calculation and registration module, and a 3D reconstruction and dynamic mapping module. The data flow and collaborative relationship between each module and processing stage are as follows: The non-intrusive operation feature acquisition module collects the operation status feature sequence of the unloader through electrical isolation monitoring. On the one hand, it transmits the operation status feature sequence to the dynamic spatial compensation vector calculation stage, and on the other hand, it synchronously transmits the grab bucket opening and closing status signal and unloading event contained in the operation status feature sequence to the 3D reconstruction and dynamic mapping module; The multi-source heterogeneous matter perception module collects multimodal spatial point cloud sequence data and visual image stream data. It inputs the multimodal spatial point cloud sequence data into the dynamic spatial compensation vector calculation stage, uses the visual image stream data to extract the dust gradient and sends it to the core matter point cloud topology generation stage, and synchronously sends the visual image stream data to the edge vision calculation and registration module; The dynamic spatial compensation vector calculation stage... The quantity calculation stage offsets coordinate offsets based on inverse geometric projection, combines the input operating state feature sequence and multimodal spatial point cloud sequence data, outputs a basic point cloud skeleton, and sends it to the core material point cloud topology generation stage. The core material point cloud topology generation stage extracts the dust gradient from the visual image stream data, increases the filtering weight based on the dust concentration distribution gradient to filter out suspended point clouds, processes the basic point cloud skeleton into the core material point cloud topology, and outputs it to the edge visual computing and registration module. The edge visual computing and registration module receives the core material point cloud topology and combines it with the visual image stream data input from the multi-source heterogeneous material perception module, performs registration of the core material point cloud topology and visual image stream to generate multidimensional fusion features, and then outputs multidimensional fusion scene features to the 3D reconstruction and dynamic mapping module. The 3D reconstruction and dynamic mapping module receives the multidimensional fusion scene features output from the edge visual computing and registration module, and combines it with the operating state feature sequence input from the non-intrusive operating feature acquisition module, which includes grab bucket opening and closing status signals and unloading events, reconstructs the 3D morphological model, and performs local spatial difference updates in response to unloading events.
[0057] like Figure 2As shown, the specific processing flow of the partial differential update operation of the unloading event driven by the present invention is as follows: Starting from the beginning of the process, the unloader's grab bucket is in a closed state, extending deep into the ship's hold to perform a closing grabbing action and record the material handling pit position; based on the lifting mechanism height encoding data and the trolley travel position data, the current real-time spatial three-dimensional coordinates of the grab bucket are calculated; entering the unloading physical event determination stage, the transition edge of the grab bucket opening / closing state signal is monitored, and the spatial relationship between the real-time spatial three-dimensional coordinates and the preset unloading space envelope above the hopper is detected; when the grab bucket opening / closing state signal is detected to transition from a closed state to an open state, and the real-time spatial three-dimensional coordinates fall within the preset unloading space envelope above the hopper, the unloading physical event is determined to be established, and a partial update trigger command is issued; responding to the partial update trigger command, based on the real-time spatial three-dimensional coordinates in the three... The three-dimensional topological model is divided into local spatial perturbation mesh envelopes. Parallel processing is then performed: first, the topological parameters of the remaining material height are refreshed, and the differential offset is calculated with the elevation data to refresh the parameters within the local spatial perturbation mesh envelope; second, in a data-locked state, the three-dimensional morphological parameters of the remaining ship compartment areas outside the local spatial perturbation mesh envelope are written and locked, preventing global geometric model baseline drift. After the two parallel processing steps are completed, the three-dimensional topological model is updated and enters the continuous update phase, maintaining dynamic updates over time and outputting a ship compartment remaining material distribution model with high geometric convergence and high topological fidelity. During the continuous unloading and material handling operation phase, the overall control flow loop returns to a closed state, penetrating deep into the ship compartment to execute closed-loop material grabbing actions and record the material handling pit locations to maintain continuous operation updates.
[0058] Example 5: After the initial on-site modification of the old ship unloader at the port is completed, the multi-source heterogeneous matter perception module and the edge vision calculation and registration module are calibrated under no-load operation. Reflective targets with known three-dimensional physical coordinates are set up at the edge of the hopper. The target point cloud is collected by LiDAR, and the corresponding optical image feature points are obtained by an industrial camera. Based on this, the initial perspective projection matrix of the multimodal spatial point cloud sequence data to the visual image stream data imaging plane is calculated.
[0059] When the trolley slides along the track unloaded, the non-contact displacement sensor synchronously collects the physical position pulse sequence representing the trolley's movement position in a unidirectional manner. The edge vision calculation and registration module calculates the position fluctuation variance of the spatial position parameters relative to the mean trajectory within a sliding time window of 50ms. When the trolley slides uniformly on the track and the grab bucket is stationary, the measured position fluctuation variance baseline is in the range of 0.04 to 0.11, and the geometric distortion trigger benchmark is determined to be 0.15. When the lifting impact and grab bucket sway cause the position fluctuation variance to cross 0.15, dynamic spatial compensation is activated, and the coordinate offset corresponding to the dynamic spatial compensation vector is deducted in the initial stage of multimodal spatial point cloud sequence data intake according to the inverse geometric projection mapping relationship to offset the high-frequency jitter of the observation benchmark.
[0060] When there is no dust and the wind speed is below level 3, the industrial camera continuously collects visual image stream data of the ship's cabin background. The edge vision computing and registration module extracts the high-frequency texture components of the two-dimensional pixel grid in the continuous video frame sequence and records the initial transmittance benchmark. The dust concentration distribution gradient in the initial clean state is calibrated to 0. During the on-site debugging phase, controlled diffusion smoke is generated by a smoke generator. The environmental occlusion feature parameters are obtained based on the local high-frequency transmittance attenuation coefficient between adjacent video frames and its spatial discrete derivative on the two-dimensional pixel grid of the image imaging plane. The dust concentration distribution gradient is then normalized to obtain the dust concentration distribution gradient. When the dust concentration distribution gradient is in the range of 0.40 to 0.70, the filter density threshold weight is synchronously increased according to the linear proportional mapping rule, and the change curve of the spatial distribution normal vector consistency index is recorded. 0.25 is used as the normal vector discrete judgment boundary of suspended scattering particles. When the dust concentration distribution gradient is greater than 0.7, the filter density threshold weight is directly switched to 2.5 times the basic set value to enhance the filtering of non-rigid suspended scattering clusters and reduce the interference of suspended scattering media on the topological reconstruction of the core matter point cloud.
[0061] Example 6: Before each loading or unloading operation, the system performs a power-on self-test to handle minor structural zero-point changes caused by loosening of sensor base fasteners after long-term operation of the external object sensing kit. After the external parameter calibration of the observation coordinate system and the vibration baseline refresh begin, three-dimensional calibration targets with known coordinates are set at fixed geometric reference positions above the hopper and on the trolley's travel track. With the grab bucket drive motor disconnected and the trolley in an unloaded state, it slides along the track at a constant speed of 0.2 m / s for two sets of reciprocating strokes. Non-contact displacement sensors independently collect the physical position pulse sequence representing the trolley's travel position at a sampling frequency of 50 Hz and send it to the edge vision calculation and registration module. The position fluctuation variance is extracted within a sliding time window of 50 ms, and the mean variance of the inherent structural baseline under the unloaded and stable sliding condition is statistically obtained as 0.06. The geometric distortion triggering reference is locked at 0.15.
[0062] After the trolley reaches the preset calibration parking point, the lidar in the multi-source heterogeneous object perception module and the industrial camera synchronously acquire the point cloud coordinates and optical images of the physical calibration target. The edge vision calculation and registration module extracts the optical corner features of the target and the center coordinates of the corresponding reflection point cloud cluster. It calculates the perspective projection matrix parameters from the lidar measurement coordinate system to the industrial camera imaging plane through direct linear transformation and overwrites and updates the underlying configuration register. When the trolley re-attaches the grab bucket and starts the no-load lifting test, the six-axis inertial measurement unit continuously monitors the vibration waveform generated by the impact of the wire rope winch on the sensor base. The vibration main frequency is measured to be concentrated at 12Hz and 36Hz. After the position fluctuation variance exceeds 0.15, the edge vision calculation and registration module converts the position fluctuation variance into a dynamic spatial compensation vector and corrects its spatial direction and components using the inertial attitude data sequence as the attitude reference. During the correction process, a second-order low-pass Butterworth filter weight with a passband cutoff frequency of 80Hz is loaded in the digital filter integration channel to suppress the cumulative divergence interference caused by the second time integration of the displacement offset component by the mechanical high-frequency collision burrs.
[0063] During the adaptive filtering of environmental media and dynamic maintenance of the ship's cabin topology grid, the edge vision computing and registration module performs dynamic zero-point tracking of environmental occlusion feature parameters. Before the dust suppression spray device is activated at the hopper discharge port, 100 consecutive frames of optical video of the ship's cabin operation area under natural sea breeze conditions are recorded. The industrial camera acquires images and extracts the mean of local high-frequency Laplacian convolution response on the two-dimensional pixel grid. The dust-free baseline response value is mapped to the zero baseline of the dust concentration distribution gradient. When the bulk material continues to fall, the system extracts the local high-frequency transmittance attenuation coefficient from adjacent video frames and obtains the environmental occlusion feature parameters based on its spatial discrete derivative on the two-dimensional pixel grid of the image imaging plane. The dust concentration distribution gradient is then calculated. When the dust concentration distribution gradient is in the range of 0.4 to 0.7, the filter density threshold weight is increased in the same direction according to the linear gain function with a slope of 3.33. When the dust concentration distribution gradient crosses the 0.7 trigger point, nonlinear step control is applied to jump the filter density threshold weight to 2.5 times the basic set value.
[0064] For a 3D voxel grid cell with a side length of 0.05 meters, when the number of points in the cell is lower than the filter density threshold weight, the three eigenvalues of the point cloud covariance matrix within the cell are calculated, and the ratio of the minimum eigenvalue to the sum of the eigenvalues is used as the consistency index of the spatial distribution normal vector. When the index is greater than 0.25, the corresponding voxel grid cell is marked as a non-rigid suspended scattering cluster and removed; when the index is less than or equal to 0.25, the point cloud with continuous tangent plane constraints is retained and aggregated to generate the core matter point cloud topology.
[0065] For data void regions formed by material occlusion in the core material point cloud topology, the 3D reconstruction and dynamic mapping module extracts the spatial first-order differential slope and second-order tangential continuous curvature features of adjacent effective 3D mesh nodes at the data void boundary. It also retrieves historical prior height parameters that coincide with the horizontal coordinates of the data void region from the historically stored 3D topology model. Using these historical prior height parameters as the initial surface reference, the 3D spatial coordinates of the virtual depth point cloud are calculated through surface tension interpolation. The surface tension adjustment coefficient has the dimension of negative square meters. Its value of 1.85 is obtained by dividing the tangent of the natural angle of repose of iron ore by the square of the step size of the voxel grid space. In a discrete computing network with a cube grid side length of 0.05 meters, this parameter makes the tension constraint term in the biharmonic equation match the grid discrete scale. When the coefficient value is below 1.2, the surface is prone to generating unnecessary elevation undulations at the boundary. When it is above 2.4, the reconstructed surface will be over-flattened and lose the true natural slope characteristics. The surface tension adjustment coefficient is calibrated to 1.85 to maintain the second-order smoothness of the boundary and the continuity of the slope.
[0066] During the dynamic update process, the opening and closing status signal of the grab bucket is continuously monitored. When the grab bucket opening and closing status signal changes from closed to open, and the real-time three-dimensional coordinates fall into the preset unloading space envelope above the hopper, a local update trigger command is issued. In response to the local update trigger command, the three-dimensional reconstruction and dynamic mapping module divides the local spatial perturbation mesh envelope in the three-dimensional topology model based on the real-time three-dimensional coordinates. Taking the vertical projection position corresponding to the real-time three-dimensional coordinates of the grab bucket as the center, a circular horizontal region with a radius equal to 1.5 times the maximum opening of the grab bucket is taken as the local spatial perturbation mesh envelope. Linear distance weighted difference is implemented within the 0.3-meter annular transition zone outside the envelope. The system refreshes the data in stages and locks the three-dimensional morphological parameters of the grid in the remaining non-operational cabin area outside the envelope, thus preventing registration drift and completing the time-series dynamic update of the three-dimensional topological morphological model. The width of the transition zone is set to 0.3 meters, which is determined by matching the three-dimensional voxel grid space size with a side length of 0.05 meters with the natural sliding collapse radius of the iron ore particles. The 0.3-meter width spans 6 discrete voxel grids, allowing the linear decay weight to transition smoothly on 6 consecutive grid nodes. This is used to absorb the slight disturbance caused by splashing particles at the edge of the falling material, while limiting the influence of the transition zone on the elevation benchmark of the undisturbed area.
[0067] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.
Claims
1. A non-intrusive dynamic sensing and retrofit system for the unloading process of aging ship unloaders in ports, characterized in that, include: A non-intrusive operation feature acquisition module is used to collect the operation status feature sequence of the ship unloader through electrical isolation monitoring; A multi-source heterogeneous matter sensing module is used to collect multimodal spatial point cloud sequence data and visual image stream data of the work area; The edge vision computing and registration module is used to construct a dynamic spatial compensation vector based on the positional fluctuation variance of the running state feature sequence. It compensates for the coordinate offset of discrete data points in the multimodal spatial point cloud sequence data through inverse geometric projection calculation to output the basic point cloud skeleton. It extracts the dust concentration distribution gradient from the visual image stream data. When the gradient is greater than 0.7, it steps up the filter density threshold weight to 2.5 times the basic set value. Based on this, it filters out the floating point cloud clusters in the basic point cloud skeleton to generate the core material point cloud topology, and registers it with the visual image stream data to generate multidimensional fusion scene features. The 3D reconstruction and dynamic mapping module is used to reconstruct the 3D topological model of the ship's hold material based on the characteristics of the multi-dimensional fusion scene; in response to the event that the grab bucket changes from closed to open and the coordinates fall into the unloading area, it calculates the spatial differential update of the local working area and keeps the non-working area in a data-locked state.
2. The non-intrusive dynamic sensing and retrofit system for aging unloading machines in ports, as described in claim 1, is characterized in that... The edge vision computing and registration module is specifically used to: calculate the position fluctuation variance of spatial position parameters in the running state feature sequence within a time window of 50ms; when the position fluctuation variance is greater than 0.15, convert the position fluctuation variance into a dynamic spatial compensation vector that characterizes the high-frequency spatial offset of the observation coordinate system; for each discrete data point in the multimodal spatial point cloud sequence data, deduct the coordinate offset component corresponding to the dynamic spatial compensation vector according to the inverse geometric projection mapping relationship, and output the basic point cloud skeleton.
3. The non-intrusive dynamic sensing and retrofit system for aging unloading machines in ports, as described in claim 1, is characterized in that... When extracting the dust concentration distribution gradient, the edge vision computing and registration module is specifically used to: extract a continuous video frame sequence from the visual image stream data, calculate the local high-frequency transmittance attenuation coefficient between adjacent video frames; and calculate the environmental occlusion feature parameters characterizing the spatial diffusion gradient of dust based on the spatial discrete derivative of the local high-frequency transmittance attenuation coefficient on the two-dimensional pixel grid of the image imaging plane. The environmental shading characteristic parameters are normalized and mapped to dimensionless scalars with values between 0 and 1, which serve as the dust concentration distribution gradient.
4. The non-intrusive dynamic sensing and retrofit system for the unloading process of aging unloading machines in ports, as described in claim 1, is characterized in that... When filtering out floating point cloud clusters in the basic point cloud skeleton, the edge vision computing and registration module specifically performs the following steps: dividing the basic point cloud skeleton into discrete three-dimensional voxel grid units, and counting the number of discrete point clouds contained in each voxel grid unit within a preset spatial neighborhood search radius; calculating the spatial distribution normal vector consistency index of the point clouds within each voxel grid unit; when it is determined that the number of point clouds in the voxel grid unit is lower than the filtering density threshold weight, and the spatial distribution normal vector consistency index shows a discrete isotropic scattering distribution, the corresponding voxel grid unit is marked as a non-rigid floating scattering cluster and removed. Voxel grid cells with continuous tangent plane constraints are retained in the normal vector distribution and aggregated to generate the core matter point cloud topology.
5. The non-intrusive dynamic sensing and retrofit system for aging unloading machines in ports according to claim 1, characterized in that, When registering the core object point cloud topology with the visual image stream data, the edge vision computing and registration module is specifically used to: construct a perspective projection matrix from the core object point cloud topology to the imaging plane corresponding to the visual image stream data according to preset calibration parameters; and match spatial point clouds and optical image frames of the same acquisition period using timestamp hardware alignment signals. The three-dimensional coordinates in the core object point cloud topology are projected onto the optical image frame using a perspective projection matrix, and the optical color parameters and texture gradient parameters at the projected pixels are extracted. By using optical color parameters and texture gradient parameters based on corresponding three-dimensional spatial points, a multi-dimensional fusion scene feature is generated that integrates three-dimensional geometric spatial coordinates and two-dimensional optical texture features.
6. The non-intrusive dynamic sensing and retrofit system for aging unloading machines in ports according to claim 1, characterized in that, The 3D reconstruction and dynamic mapping module is further used to complete the topological mesh of the data hole region caused by mechanical structure occlusion. Specifically, it is used to: identify the boundary of the data hole caused by occlusion in the multi-dimensional fusion scene features; extract the spatial first-order differential slope and second-order tangential continuous curvature features of adjacent effective 3D mesh nodes on the data hole boundary; retrieve the historical prior height parameters that coincide with the horizontal coordinates of the data hole region from the 3D topological morphology model stored in historical time series; use the historical prior height parameters as the initial surface reference, and use the spatial first-order differential slope and second-order tangential continuous curvature features as the surface boundary continuity constraint conditions, calculate the 3D spatial coordinates of the virtual depth point cloud inside the hole through the surface tension interpolation algorithm; and stitch the calculated virtual depth point cloud into the 3D topological morphology model to complete the missing topological mesh.
7. The non-intrusive dynamic sensing and retrofit system for aging unloading machines in ports according to claim 1, characterized in that, When analyzing the event of the grab bucket changing from closed to open and its coordinates falling into the unloading area, the 3D reconstruction and dynamic mapping module is specifically used for: separating the hoisting mechanism height encoding data, trolley travel position data, and grab bucket opening and closing status signals from the operating state feature sequence; calculating the current real-time spatial 3D coordinates of the grab bucket based on the hoisting mechanism height encoding data and trolley travel position data; monitoring the transition edge of the grab bucket opening and closing status signal, and when the signal is detected to change from closed to open and the calculated real-time spatial 3D coordinates fall within the preset unloading space envelope above the hopper, the module determines that the unloading physical event has occurred and issues a local update trigger command.
8. The non-intrusive dynamic sensing and retrofit system for the unloading process of aging unloading machines in ports, as described in claim 7, is characterized in that... When the 3D reconstruction and dynamic mapping module performs differential updates in the computational space and keeps the non-operational areas in a data-locked state, it is specifically used for: responding to local update trigger commands, dividing the local spatial disturbance mesh envelope in the 3D topology model based on real-time spatial 3D coordinates; calculating the differential offset between the data points that fall within the local spatial disturbance mesh envelope in the currently collected multi-dimensional fusion scene features and the corresponding mesh elevation data in the 3D topology model; refreshing the residual material height topology parameters within the local spatial disturbance mesh envelope using the calculated differential offset; and keeping the 3D morphology parameters of the remaining cabin areas outside the local spatial disturbance mesh envelope in a data-locked state to prevent global geometric model reference drift.
9. The non-intrusive dynamic sensing and retrofit system for the unloading process of aging unloading machines in ports, as described in claim 1, is characterized in that... The multi-source heterogeneous matter perception module includes a lidar, an industrial camera, and a six-axis inertial measurement unit (IMU) installed under the unloader's trolley frame. The lidar and industrial camera are used to simultaneously acquire multimodal spatial point cloud sequence data and visual image stream data. The six-axis IMU is used to acquire inertial attitude data sequences characterizing the high-frequency angular velocity and acceleration fluctuations of the sensor observation reference, and synchronously inputs the inertial attitude data sequences into the edge vision calculation and registration module as an attitude reference reference for solving the dynamic spatial compensation vector. The non-intrusive operation feature acquisition module includes non-contact displacement sensors configured on the outside of the unloader's moving parts, which are used to independently acquire physical position pulse sequences characterizing the trolley's walking position and lifting height in a unidirectional passive data transmission manner, serving as the data source for the operation status feature sequence.
Citation Information
Patent Citations
Bulk cargo wharf grab ship unloader ship unloading method and system
CN118780707A
