An unattended screw ship unloader material and ship control method and system

Abstract

The application discloses an unattended screw ship unloader material and ship control method and system, the method comprising: through a plurality of types of sensors deployed on the screw ship unloader, real-time collection of multi-dimensional sensing data including visual images, material weight, material type and spatial pose; spatio-temporal registration and fusion processing of the collected multi-dimensional sensing data to obtain a job scene situation awareness result for comprehensively representing the real-time state of the ship unloading operation scene; based on the job scene situation awareness result, driving the screw ship unloader digital twin model to perform real-time simulation to obtain simulation results including future material accumulation space morphology, equipment operation performance parameters and / or potential risk probability; according to the simulation results of the digital twin model, generating an optimized control instruction based on a preset decision rule to realize adaptive adjustment of the ship unloading operation parameters and intelligent disposal of abnormal working conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of port automatic handling equipment, in particular to a method and system for unmanned material and ship control of a spiral ship unloader. BACKGROUND

[0002] In the field of bulk cargo terminal handling, spiral ship unloaders, as an important continuous unloading equipment, are widely used in the unloading operations of bulk cargoes such as grain, coal and ore. The traditional spiral ship unloading operation highly depends on the visual observation and manual operation experience of operators to control the operating parameters of the unloader, such as the descent depth, horizontal movement speed and rotation speed of the spiral head. This manual-dominated operation mode not only requires high skills and experience of operators, but also has problems of low production efficiency, high energy consumption and inability to maintain a continuously optimal operating state. In addition, in the conditions of poor visibility such as night and bad weather, the safety risks and operation difficulties of manual operation will be further intensified.

[0003] In order to improve the automation level, some unloaders in the industry use programmed control or a single type of sensor (such as laser radar or camera) to assist in the operation. However, these improved schemes still have significant limitations in actual application. Programmed control cannot adapt to the random changes in the material accumulation form in the ship cabin and the dynamic movement of the ship itself, which is easy to cause equipment idling, insufficient excavation or overload, resulting in efficiency loss and equipment loss. The system based on a single sensor has the problems of incomplete perception dimension and weak anti-interference ability, for example, visual sensors are easily affected by light, dust and water vapor, and weighing sensors are disturbed by ship body sway, making it difficult to obtain comprehensive, stable and reliable operation scene information.

[0004] More complex challenges come from the dynamic changes of the operating environment. During the unloading operation, the ship and the unloader have six degrees of freedom relative motion due to factors such as wind and wave, tidal water, and cargo weight changes. This continuous and random relative motion causes the measurement reference of the sensor to change constantly, resulting in serious deviation of the collected data (such as images, weights, positions). If these deviations cannot be compensated in real time and with high precision, any automatic decision based on this data will be based on incorrect information, thus unable to achieve precise and reliable control, and even may cause safety accidents. SUMMARY

[0005] The purpose of the embodiment of the present application is to provide an unattended spiral ship unloader material and ship control method and system, which combines multi-dimensional sensor data fusion and digital twin real-time simulation to build a closed-loop intelligent control system, completely solves the control problem caused by the dynamic movement of the ship and the random distribution of the material, realizes the autonomous intelligent operation of the spiral ship unloading operation from perception, decision-making to execution, and significantly improves the operation efficiency, safety and economy.

[0006] To solve the above technical problems, the first aspect of the embodiment of the present application provides an unattended spiral ship unloader material and ship control method, comprising the following steps:

[0007] Through the deployment of multiple types of sensors on the spiral ship unloader, multi-dimensional sensor data is collected in real time, which includes visual images, material weight, material type and spatial pose;

[0008] The collected multi-dimensional sensor data is subjected to spatio-temporal registration and fusion processing to obtain a job scene situation awareness result for comprehensively representing the real-time state of the unloading operation scene;

[0009] Based on the job scene situation awareness result, a spiral ship unloader digital twin model is driven to perform real-time simulation to obtain a simulation result, which includes the future material accumulation space form, equipment operation performance parameters and / or potential risk probability;

[0010] According to the simulation result of the digital twin model, an optimized control instruction is generated based on a preset decision-making rule to realize the adaptive adjustment of the unloading operation parameters and the intelligent disposal of abnormal working conditions.

[0011] Further, the spatio-temporal registration and fusion processing of the collected multi-dimensional sensor data to obtain the job scene situation awareness result for comprehensively representing the real-time state of the unloading operation scene comprises:

[0012] Using a unified time and space reference, all the multi-dimensional sensor data is subjected to spatio-temporal alignment, a mapping model of image pixel coordinates and world coordinates is established, and the observation deviation caused by the relative motion of the ship and the unloading machine is dynamically compensated to obtain a standard data stream synchronized in time and space;

[0013] Based on the standard data stream, the fusion weight is adaptively allocated according to the confidence level, timeliness and operation process of each data source, the visual material profile, weight information, type information and pose information are weighted and fused and mutually verified to obtain a comprehensive state estimation containing the material geometric distribution, physical properties, category attributes and environmental pose relationship, which is taken as the job scene situation awareness result.

[0014] Further, the dynamic compensation of the observation deviation caused by the relative motion between the ship and the ship unloader includes:

[0015] Real-time acquisition of six-degree-of-freedom relative pose data between the ship and the ship unloader, input of the six-degree-of-freedom relative pose data into a motion prediction model, and calculation of a motion trajectory and an attitude change of the relative motion between the ship and the ship unloader in a future control period;

[0016] Calculation of a pixel-level coordinate offset caused by the motion trajectory and the attitude change to vision sensor imaging and a disturbance error to a non-vision sensor measurement reference;

[0017] Generation of corresponding reverse compensation parameters according to the coordinate offset and the disturbance error, and online correction of the standard data.

[0018] Further, after the online correction of the standard data, the method further includes:

[0019] Comparison of the corrected standard data stream with a theoretical expected value calculated under an ideal static working condition in a unified coordinate system based on the digital twin model, formation of a residual feedback to adaptively adjust parameters of the motion prediction model, and optimization of subsequent compensation accuracy.

[0020] Further, the calculation of the pixel-level coordinate offset caused by the motion trajectory and the attitude change to vision sensor imaging and the disturbance error to the non-vision sensor measurement reference includes:

[0021] Based on installation position and orientation parameters of the vision sensor, conversion of the six-degree-of-freedom relative pose data into motion components in an image coordinate system, calculation of an image translation amount, a rotation angle, and a scale change rate caused by the relative motion, and obtaining of vision sensor pixel coordinate correction parameters;

[0022] Based on installation structure parameters and kinematic relationships of the weight sensor, calculation of a weight measurement deviation value caused by an inertial force component of the relative motion, and obtaining of weight sensor compensation coefficients;

[0023] According to installation geometric relationships of the spatial pose sensor, solving of a measurement reference offset caused by the relative motion, and obtaining of spatial pose sensor calibration parameters;

[0024] Combination of the vision sensor pixel coordinate correction parameters, the weight sensor compensation coefficients, and the spatial pose sensor calibration parameters to form a multi-sensor error compensation parameter set.

[0025] Further, the generation of the corresponding reverse compensation parameters according to the coordinate offset and the disturbance error includes:

[0026] generate an image geometric transformation matrix based on the visual sensor pixel coordinate correction parameter, the image geometric transformation matrix being used for coordinate transformation processing of image data collected by the visual sensor;

[0027] generate a weight reading compensation function based on the weight sensor compensation coefficient, the weight reading compensation function outputting a compensated weight value according to real-time collected original weight data;

[0028] generate a pose data correction mapping relationship based on the spatial pose sensor calibration parameter, the pose data correction mapping relationship being used for calibration processing of spatial pose measurement data;

[0029] encode the image geometric transformation matrix, the weight reading compensation function, and the pose data correction mapping relationship in time sequence to generate a sensor compensation instruction set synchronized with a clock of a spiral ship unloader control system;

[0030] input the sensor compensation instruction set to the digital twin model for compensation effect verification, and when a verification result shows that a deviation between data processed by the sensor compensation instruction set and an expected value of an ideal state after compensation simulated by the digital twin model exceeds a set threshold, readjust parameter values in the multi-sensor error compensation parameter set until a final compensation instruction is output after verification is passed.

[0031] Further, based on the job scene situation awareness result, drive the spiral ship unloader digital twin model to perform real-time simulation to obtain simulation results including future material accumulation space morphology, device running performance parameters, and / or potential risk probability, including:

[0032] input material geometric distribution, physical attributes, and category attributes contained in the job scene situation awareness result to the digital twin model, calculate material flow and accumulation process in a ship cabin in a next control period through discrete element simulation, and output a material accumulation morphology prediction map;

[0033] based on the environment pose relationship in the job scene situation awareness result and the current running state of the device, simulate the running process of the device under different control strategies through the digital twin model to obtain a device performance parameter prediction set including energy consumption, vibration amplitude, and structural stress;

[0034] according to the material accumulation morphology prediction map and the device performance parameter prediction set, combine abnormal event records under corresponding working conditions in a historical job database, calculate occurrence probability of each type of potential risk under a current job state through a pattern recognition algorithm, and generate a risk probability distribution map.

[0035] Further, based on the simulation result of the digital twin model, generate an optimized control instruction based on a preset decision rule, including:

[0036] According to the uneven material distribution area shown in the material accumulation form prediction map, a real-time adjustment instruction for generating a spiral head motion trajectory is generated, and the real-time adjustment instruction includes optimized settings of a deep excavation parameter and a horizontal movement path for a specific area;

[0037] Based on the energy consumption abnormality or vibration overrun shown in the device performance parameter prediction set, an adjustment instruction for device operation parameters is generated, and the adjustment instruction includes coordinated control parameters for spiral rotation speed, elevator working frequency and conveyor belt transmission rate;

[0038] For the high-risk area identified in the risk probability distribution map, a preventive treatment instruction is generated, and the preventive treatment instruction includes device obstacle avoidance path planning, emergency deceleration control and early warning signal triggering mechanism;

[0039] The real-time adjustment instruction, the adjustment instruction and the preventive treatment instruction are integrated according to the job priority and the time sequence relationship to obtain the optimization control instruction.

[0040] Further, after obtaining the optimization control instruction, the method further includes:

[0041] The optimization control instruction is input into the digital twin model for simulation execution to obtain device operation prediction parameters and material state change data;

[0042] The device operation prediction parameters are compared with the device performance parameter prediction set obtained by simulation of the digital twin model, and the material state change data are compared with the material accumulation form prediction map;

[0043] According to the comparison result, the control instruction deviating from the device performance parameter prediction set or the material accumulation form prediction map is adjusted in motion parameters;

[0044] Based on the adjusted control instruction, a safety execution instruction set is generated, and after the adjusted instruction is verified in the digital twin model, it is output to the spiral ship unloader control system for execution.

[0045] Correspondingly, a second aspect of the embodiment of the application provides an unattended spiral ship unloader material and ship control system, which controls the material and the ship based on the above-mentioned unattended spiral ship unloader material and ship control method, and includes:

[0046] A data acquisition module is configured to acquire multi-dimensional sensing data in real time through multiple types of sensors deployed on the spiral ship unloader, and the multi-dimensional sensing data includes visual images, material weight, material type and spatial pose;

[0047] a data fusion module configured to perform spatio-temporal registration and fusion on the collected multi-dimensional sensing data to obtain a job scene situation awareness result for comprehensively representing a real-time state of the unloading job scene;

[0048] a data simulation module configured to drive a digital twin model of the spiral unloader to perform real-time simulation based on the job scene situation awareness result, and obtain a simulation result including a future material accumulation space form, a device operation performance parameter, and / or a potential risk probability;

[0049] an optimization control module configured to generate an optimization control instruction based on a preset decision rule according to the simulation result of the digital twin model, so as to realize adaptive adjustment of unloading job parameters and intelligent disposal of abnormal working conditions.

[0050] Correspondingly, a third aspect of the embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the unmanned spiral unloader material and ship control method.

[0051] Correspondingly, a fourth aspect of the embodiment of the present application provides a computer readable storage medium having computer instructions stored thereon, the instructions being executed by a processor to implement the unmanned spiral unloader material and ship control method.

[0052] The above technical solutions of the embodiment of the present application have the following beneficial technical effects:

[0053] 1. Through the cooperative work of multiple types of sensors and the unique spatio-temporal registration and fusion algorithm, the defects of incomplete and inaccurate perception of single sensor in dust, light, shaking and other harsh working conditions are overcome; in particular, through dynamic compensation of six-degree-of-freedom relative motion between the ship and the unloader, the data deviation caused by measurement reference drift is greatly eliminated, and a comprehensive situation awareness result containing material geometry, physics, category and environmental pose relationship is generated, which provides an unprecedented accurate, stable and reliable data basis for subsequent intelligent decision-making, and fundamentally improves the adaptability of the system to complex dynamic environment;

[0054] 2. Mirror the real-time perceived job site situation to the high-fidelity digital twin model, and drive it to real-time simulation, which is not only a reproduction of the current state, but also a prediction of the future short-time domain working condition; it can predict the material accumulation form, equipment running performance (such as energy consumption, vibration) and potential risk probability distribution in advance, so as to change the control strategy from the traditional "after reaction" mode to the "before prediction" and "during intervention" mode, realize the leap from passive response to active optimization, and significantly improve the safety of the job and avoid potential failures;

[0055] 3. Form a closed-loop optimization and safety decision mechanism based on simulation verification, not directly executing the control instructions generated based on rules, but first performing "pre-execution" and verification of the instruction set in the digital twin model; by comparing the deviation of the prediction result from the expected target, the instruction is fine-tuned and optimized, and finally a safe execution instruction set verified by "sand table deduction" is generated; it is equivalent to adding a virtual "safety valve" to the control instruction, greatly avoiding the possibility of executing incorrect or high-risk instructions on real equipment due to sensing errors or model mismatches, ensuring that the control system's decision is both optimized and reliable, and ultimately realizing the adaptive and intelligent unmanned operation of the whole unloading process. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the flow chart of the material and ship control method of the unmanned spiral unloader provided by the embodiment of the present application;

[0057] Figure 2 is the module block diagram of the material and ship control system of the unmanned spiral unloader provided by the embodiment of the present application.

[0058] REFERENCE NUMERALS:

[0059] 1, data acquisition module, 2, data fusion module, 3, data simulation module, 4, optimization control module. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further described in detail below with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0061] Please refer to Figure 1 The first aspect of the embodiment of the present application provides a material and ship control method of an unmanned spiral unloader, comprising the following steps:

[0062] Step S100, real-time collection of multi-dimensional sensor data by multiple types of sensors deployed on the spiral ship unloader, the multi-dimensional sensor data including visual images, material weight, material type, and spatial pose.

[0063] By integrating high-resolution industrial cameras, high-precision weighing sensors, material composition spectrum analyzers, and inertial measurement units (IMU), GNSS receivers, laser radars, and other multi-sensor devices integrated into the unloader structure, the surface visual feature images of the materials in the ship cabin, the real-time weight data of the materials conveyed by the spiral conveyor, the material variety classification information obtained based on the material reflectance spectrum characteristics analysis, and the accurate spatial position and attitude data of each actuator of the unloader relative to the ship cabin are continuously obtained, thereby forming a multi-dimensional, heterogeneous sensor data stream covering the environment, load, attribute, and pose, laying a rich data foundation for subsequent intelligent processing.

[0064] Step S200, spatio-temporal registration and fusion processing of the collected multi-dimensional sensor data to obtain a job scene situation awareness result for comprehensively representing the real-time state of the unloading operation scene.

[0065] First, a unified world coordinate system and timestamp system are established to align all input sensor data in space and time. In particular, for the six-degree-of-freedom oscillation motion of the ship due to waves and other factors and the relative displacement and attitude change between the unloader, this step estimates the visual image pixel shift, weighing measurement disturbance, and pose measurement reference drift caused thereby in real time through the built-in kinematic model and filtering prediction algorithm, and performs dynamic compensation and correction to generate a spatio-temporally synchronized standard data stream. Subsequently, according to the confidence models of each sensor under different working conditions, appropriate fusion weights are assigned to the visual profile, weight reading, type discrimination result, and pose information, and through weighted fusion and cross-validation algorithms, a comprehensive situation awareness result that can accurately and comprehensively reflect the current material spatial distribution form, physical characteristics (such as density, angle of repose), material variety, and device-environment relative relationship is finally output.

[0066] Step S300, based on the job scene situation awareness result, driving the spiral ship unloader digital twin model to perform real-time simulation to obtain a simulation result, the simulation result including future material accumulation space form, device operation performance parameters, and / or potential risk probability.

[0067] The comprehensive state estimation results output by step S200, including the geometric distribution of the material, the physical properties, the category attributes, and the dynamic pose relationship of the environment, are loaded as boundary conditions and driving inputs into a pre-constructed high-fidelity ship unloader digital twin model; the digital twin model couples discrete element method (DEM) material flow simulation, multi-body dynamics simulation, and finite element analysis function modules, and through real-time operation, simulates the cutting, conveying, and accumulation process of the material under the action of the screw head in the cabin in the future one or more control periods, and predicts the detailed material surface morphology change; at the same time, the simulation calculation generates performance parameters such as power consumption, mechanical vibration, and structural stress when the equipment performs different actions under the current situation; and further combined with the historical fault database, the pattern recognition technology is used to evaluate the probability of abnormal conditions such as stall, overload, and collision that may occur under the current operating conditions, so as to output multi-level simulation results including the future material accumulation morphology diagram, the equipment performance parameter prediction set, and the risk probability distribution diagram.

[0068] Step S400, according to the simulation results of the digital twin model, generate optimization control instructions based on preset decision rules to realize adaptive adjustment of unloading operation parameters and intelligent disposal of abnormal conditions.

[0069] According to the predictive information provided by the digital twin simulation of step S300, the optimization decision rule library with efficiency, energy consumption, and safety as the core target is automatically generated; for example, for the uneven material distribution area predicted by simulation, generate trajectory optimization instructions including specific digging angle, cutting depth, and horizontal translation speed of the screw head; for the predicted energy consumption anomaly or vibration overrun, generate coordinated adjustment instructions for the screw head speed, elevator frequency, and belt conveyor speed; for the identified high-risk area or event, generate preventive disposal instructions including obstacle avoidance path replanning, equipment pre-deceleration, and even emergency shutdown; all these instructions will be sent into the digital twin model again for effect verification and parameter fine-tuning before being issued to the physical equipment for execution, forming an internal loop of “decision-simulation verification-optimization”, and finally ensuring that the output is a safe and efficient optimization control instruction set that has been fully verified, so as to realize the full-automatic adaptive adjustment of unloading operation parameters and the intelligent and forward-looking disposal of abnormal conditions.

[0070] The above deep fusion and dynamic compensation technology of multi-source sensing data solves the industry problem of inaccurate perception data in a harsh and dynamic environment; the digital twin technology is used to realize forward-looking simulation and virtual verification of the operation process, and the control mode is changed from passive reaction to active prediction and optimization; finally, a closed-loop intelligent control system is built, which can self-perceive, make autonomous decisions, automatically optimize, and pre-verify the action plan in a virtual space, thereby significantly improving the overall efficiency of bulk cargo unloading operations, the economy of equipment operation, and the level of intrinsic safety under the condition of no human intervention.

[0071] Specifically, the multi-dimensional sensing data collected in step S200 is subjected to space-time registration and fusion processing to obtain an operation scene situation awareness result for comprehensively representing the real-time state of the unloading operation scene, including:

[0072] In step S210, a unified time and space reference is used to align all multi-dimensional sensing data in space and time, a mapping model of image pixel coordinates and world coordinates is established, and the observation deviation caused by the relative motion of the ship and the unloading machine is dynamically compensated to obtain a standard data stream synchronized in space and time.

[0073] A stable and reliable space-time reference framework is established to eliminate the internal inconsistency of multi-source heterogeneous data due to different collection times and space reference systems. First, a unified UTC timestamp is provided by a high-precision Beidou / GNSS receiver, and nanosecond-level time synchronization is performed using industrial Ethernet to ensure that all sensor data is labeled with the same time. In space, a global world coordinate system is established with the unloading machine rotation center or the pre-marked ground-based laser radar station as the origin, and the accurate mapping relationship between the image pixel coordinate system and the world coordinate system of each vision sensor is established by pre-calibrating the internal and external parameters of each vision sensor, i.e., the conversion matrix from two-dimensional pixel points to three-dimensional world points is calculated through the camera calibration model. By fusing the IMU and GNSS unit data installed on the ship body and the unloading machine structure, the six-degree-of-freedom relative motion (pitch, roll, yaw, heave, surge, and sway) between the ship and the unloading machine caused by wind and waves is calculated and predicted in real time, and the pixel shift and distortion caused by the imaging perspective of the vision sensor, the inertial force interference of the weight sensor, and the influence on the reference plane of the laser radar pose measurement device are dynamically calculated according to the relative motion, and then inverse compensation parameters (such as image affine transformation matrix, weight reading dynamic filtering algorithm, and point cloud coordinate real-time transformation) are generated to correct the original data stream online, and finally a high-quality, space-time synchronous standard data stream is output, which is strictly aligned in time and space and effectively suppresses the relative motion interference.

[0074] Step S220, based on the standard data stream, according to the confidence level, timeliness and job progress of each data source, adaptive allocation of fusion weight, weighted fusion and mutual verification of visual material profile, weight information, type information and pose information, to obtain the comprehensive state estimation containing material geometric distribution, physical properties, category attributes and environmental pose relationship, as the job scene situation awareness result.

[0075] On the basis of clean and synchronized data provided in step S210, the fusion decision process of information extraction and sublimation is carried out. The above steps are not simply superimposed on all data sources, but through adaptive weighted fusion strategy, the weight allocation is dynamically dependent on three core factors: the confidence level of each data source in the current environment (for example, the confidence of visual sensor is reduced when dust is diffused, while the confidence of millimeter wave radar remains stable), the timeliness of the data itself (that is, the freshness of the data, the higher the weight of the more newly collected data), and the specific job process stage (for example, in the initial excavation stage, more attention is paid to the pose information for positioning, and in the stable conveying stage, more attention is paid to the weight information to control the flow). According to these real-time calculated weights, the material profile point cloud from vision, the instantaneous flow information from the weighing unit, the material type discrimination results from the spectrum analyzer, and the accurate pose data from IMU / GNSS are weighted and fused, and mutual verification and conflict resolution are carried out in the process (for example, the estimated material volume and weight information are cross-verified to determine whether the density is reasonable, or the material surface morphology recognized by vision is used to assist in verifying the rationality of the pose data), and finally a comprehensive and reliable comprehensive state estimation is generated, which accurately describes the three-dimensional geometric distribution and volume of the material, the physical properties (such as bulk density, humidity), the material variety classification, and the real-time pose relationship between the ship unloader and the ship hatch, the material pile body, which constitutes the job scene situation awareness result relied on by subsequent intelligent decision-making.

[0076] Through rigorous space-time registration and innovative dynamic motion compensation technology, the core pain points of multi-sensor data reference inconsistency and error interweaving in dynamic marine environment are fundamentally solved; then through adaptive weighted fusion mechanism, the complementary advantages of multi-source information are intelligently utilized, the limitations of single sensor in specific working conditions are suppressed, and finally a highly accurate, reliable and comprehensive job site "situation map" is generated, which provides an irreplaceable and high-quality data basis for high-fidelity simulation of digital twin and subsequent optimization control decision, and is the key prerequisite and core guarantee for the realization of intelligentization and self-automation of the whole system.

[0077] Further, the observation deviation caused by the relative motion of the ship and the ship unloader in step S210 is dynamically compensated, including:

[0078] Step S211, real-time acquisition of six-degree-of-freedom relative pose data between the ship and the ship unloader, input of the six-degree-of-freedom relative pose data into a motion prediction model, calculation of the motion trajectory and attitude change of the relative motion between the ship and the ship unloader in a future control period.

[0079] By deeply fusing the data of multiple sets of inertial measurement units (IMUs) and global navigation satellite system (GNSS) receivers installed on the ship body and the ship unloader, the relative pose changes in three translational degrees of freedom (surge, sway, and heave) and three rotational degrees of freedom (roll, pitch, and yaw) between the two are calculated in real time at a high frequency (usually hundreds of hertz). Subsequently, these high-dimensional six-degree-of-freedom relative pose data streams are input into a motion prediction model based on Kalman filtering or deep learning modeling. This model not only describes the current motion state, but also extrapolates and predicts the expected motion trajectory and attitude change of the ship relative to the ship unloader in a future control period (usually tens to hundreds of milliseconds), providing a key time lookahead for feedforward compensation.

[0080] Step S212, calculation of the pixel-level coordinate offset caused by the motion trajectory and attitude change to the vision sensor imaging and the disturbance error to the non-vision sensor measurement reference.

[0081] First, for the vision sensor, according to its pre-calibrated internal parameters (focal length, principal point) and external installation position and orientation, the predicted six-degree-of-freedom relative motion changes are decomposed and converted into specific effects in the image coordinate system through strict perspective projection geometric models. This includes calculating the image global translation deviation caused by relative translation, the image affine distortion and rotation caused by relative rotation, and the scale caused by relative distance change. Finally, the offset prediction of each pixel coordinate in the future image is obtained. For non-vision sensors such as weighing sensors, the additional disturbance error of weight measurement values caused by inertial forces of heave, pitch, and roll acceleration motions is calculated according to their installation structure and mechanical model; for laser radar and other pose sensors, the systematic measurement error of target point cloud coordinates caused by the pose changes of the measurement reference (i.e. the sensor itself) is calculated, so as to fully quantify the disturbance of relative motion to all perception dimensions.

[0082] Step S213, generation of corresponding inverse compensation parameters according to the coordinate offset and disturbance error, online correction of standard data.

[0083] The various error quantities calculated in step S212 are converted into specific, executable compensation actions. According to the calculated image pixel-level coordinate offset, this step generates a corresponding inverse geometric transformation matrix (for example, an affine or perspective transformation matrix that integrates translation, rotation, and scaling), and applies this matrix to the original image collected by the vision sensor for real-time resampling and transformation, thereby canceling the distortion caused by relative motion at the image level. For the weight sensor, according to the calculated inertial force interference error, a dynamic, real-time motion state-dependent compensation function or filtering algorithm is generated to subtract or filter out the inertial force component from the original weight reading, restoring the true material weight. For the pose point cloud data, coordinate transformation is applied to correct it to a stable world coordinate system. All these compensation parameters are encapsulated into instructions that are strictly synchronized with the control system clock, and are used to preprocess and correct the "standard data stream" output by the data acquisition module in real time.

[0084] Through the "prediction-quantization-compensation" feedforward control mechanism, the largest source of uncertainty in unloading operations, ship motion with waves, is actively canceled, causing systematic errors to the perception system. The motion interference is transformed from noise that needs to be filtered out into a deterministic quantity that can be predicted and compensated, thereby restoring the contaminated multi-source sensor data to the measurement level under the nearly static reference, greatly improving the accuracy and reliability of subsequent data fusion, situation awareness, and simulation decision-making, and laying an indispensable foundation for the entire system to maintain high-precision stable operation in a real dynamic marine environment.

[0085] Furthermore, after the online correction of the standard data in step S213, it also includes:

[0086] Step S214, by comparing the corrected standard data stream with the theoretical expected value calculated based on the digital twin model under ideal static working conditions in the unified coordinate system, a residual feedback is formed to adaptively adjust the parameters of the motion prediction model, and the subsequent compensation accuracy is optimized.

[0087] On the basis of performing feedforward compensation in step S213, a continuous self-learning and optimization cycle is constructed, and the "corrected standard data stream" generated after real-time dynamic compensation is taken as an evaluatable actual result, which is compared with a high-credibility reference benchmark; this reference benchmark is not from other sensors, but is provided by the digital twin model inside the system: the digital twin model calculates a set of theoretically expected data values in the unified global coordinate system, based on material characteristics, known motion instructions of the equipment, and assumed ideal static working conditions (i.e. no ship sway disturbance), through high-fidelity physical simulation, such as theoretically expected material surface three-dimensional point cloud coordinates, theoretically expected material weight values, and the precise pose at which the equipment should theoretically be located. The corrected real-time data is compared with the theoretically expected values frame by frame and point by point, and the difference between the two, i.e. the residual error, is calculated. These residual error signals are fed back to the motion prediction model in step S211 in real time as training data and optimization targets for online self-adaptation of the model parameters; through optimization algorithms such as recursive least squares, gradient descent, etc., the key parameters involved in the motion prediction model (such as the process noise and observation noise covariance matrices of the Kalman filter, or the weights of the deep learning prediction model) are dynamically adjusted, so that the model's prediction of future relative motion is more accurate next time, and thus the error calculation in step S212 and the compensation action in step S213 are closer to the true requirements, forming a negative feedback loop that continuously reduces the residual error and optimizes the compensation accuracy.

[0088] By upgrading an open-loop feedforward compensation system to an intelligent self-adaptive system with online self-calibration capability, and by using the reliable theoretical expectations provided by the digital twin model as a "ruler", the deficiencies of the feedforward compensation are continuously measured, and the errors are used as training signals to optimize the core parameters of the motion prediction model in the opposite direction, so that the entire dynamic compensation system can gradually adapt to the slow changes in the ship motion characteristics (such as changes in the rocking period caused by changes in load) and even the performance drift of the sensor itself, thereby ensuring that the compensation accuracy for the relative motion disturbance is maintained at a high level throughout the long unloading operation period, and even continuously converges and improves, greatly enhancing the long-term robustness and reliability of the entire perception system in complex time-varying environments.

[0089] Further, the calculation of the motion trajectory and attitude change in step S212 causes pixel-level coordinate shifts in the vision sensor and disturbance errors in the non-vision sensor measurement reference, including:

[0090] Step S212a: Based on the installation position and orientation parameters of the vision sensor, the six-degree-of-freedom relative pose data is converted into motion components in the image coordinate system, and the image translation, rotation angle and scale change rate caused by the relative motion are calculated respectively to obtain the vision sensor pixel coordinate correction parameters.

[0091] The predicted relative motion six-degree-of-freedom data is accurately converted into specific quantitative errors of various sensors, and error modeling for visual sensors is specially processed. According to the pre-precisely calibrated internal parameters (including focal length, image principal point, distortion coefficient) and external installation parameters (i.e. the accurate position of the optical center in the world coordinate system and the attitude angle of the three axes) of each visual sensor, the six-degree-of-freedom relative pose change quantity between the ship and the ship unloader predicted in step S211 is decomposed and mapped onto the two-dimensional image plane through strict perspective projection transformation and coordinate system conversion chain. This process accurately calculates the pixel translation amount of the entire image in the row and column directions caused by relative translation motion, the rotation angle of the image around the optical axis caused by relative rotation motion and the affine shear distortion caused thereby, and the image scale scaling ratio caused by the change in the relative distance between the two; finally, all these calculated geometric change quantities are integrated and encapsulated as a complete set of visual sensor pixel coordinate correction parameters, which can usually be expressed as a composite affine or perspective transformation matrix containing translation, rotation, scaling and shear components, which directly defines how to inversely correct the original image pixel coordinates to offset the motion influence.

[0092] In step S212b, the weight sensor compensation coefficient is calculated based on the installation structure parameters and kinematic relationship of the weight sensor, and the weight measurement deviation value caused by the inertial force component of the relative motion is obtained.

[0093] For the kinetic error calculation of the weight sensor (such as the weighing sensor installed on the lifting mechanism), based on the specific installation position, attitude of the weight sensor and its mechanical measurement principle (usually based on strain measurement), rigid body kinematics and Newton's second law are used to decompose the relative motion acceleration (especially the heave, pitch and roll accelerations) predicted in step S211 to the sensitive axes (measurement axes) of the weighing sensor, and calculate the size and direction of the inertial force component generated by these accelerations and superimposed on the real material weight. This inertial force constitutes the dynamic deviation value of the weight measurement, and accordingly a dynamic weight sensor compensation coefficient related to the real-time motion acceleration vector (usually a compensation amount proportional to the acceleration value or a parameter of a dynamic filtering algorithm) can be derived.

[0094] In step S212c, the measurement reference offset caused by the relative motion is solved according to the installation geometric relationship of the spatial pose sensor, and the spatial pose sensor calibration parameter is obtained.

[0095] Focusing on the spatial pose sensor (such as laser radar, GNSS receiver antenna), the error calculation is derived from a basic principle: the measurement value of the sensor is relative to its own installation reference, and the reference itself is in motion; according to the installation geometry of these pose sensors (i.e. the installation position and attitude of the sensor relative to the ship unloader reference frame), the six-degree-of-freedom motion of the ship unloader structure (i.e. the sensor installation foundation) predicted in step S211 is directly converted into systematic offset of the target measurement value (such as three-dimensional coordinates of a point, normal vector of a plane) through rigid body coordinate transformation, and the spatial pose sensor calibration parameters (usually a reverse rigid body transformation matrix or a set of coordinate offset) for the reverse correction of the original measurement value to the stable world coordinate system are obtained by solving the measurement reference offset.

[0096] Step S212d, combining the visual sensor pixel coordinate correction parameter, the weight sensor compensation coefficient and the spatial pose sensor calibration parameter to form a multi-sensor error compensation parameter set.

[0097] The integration and standardization step standardizes and time-stamp synchronizes the correction parameters (visual transformation matrix), compensation coefficients (weight dynamic compensation) and calibration parameters (pose transformation matrix) calculated in the foregoing steps for different physical principle sensors, and combines them to form a structured multi-sensor error compensation parameter set strictly aligned with the global clock of the control system, which is delivered to step S213 as a detailed "correction instruction set" for final online data correction.

[0098] By constructing a refined error modeling and solving process based on the unique physical principles and installation characteristics of each sensor, the macroscopic and overall six-degree-of-freedom relative mechanical motion is accurately decoupled and quantified to the microscopic measurement error of each specific sensor, achieving accurate mathematical mapping from "machine motion" to "pixel shift", "reading deviation" and "coordinate drift", thereby providing a unique and reliable quantitative input for subsequent execution of high-precision and differentiated sensor data compensation, which is the most critical core technology link for the entire dynamic compensation system to move from theoretical concept to engineering practice and achieve the expected effect.

[0099] Further, the generation of corresponding reverse compensation parameters according to the coordinate offset and disturbance error in step S213 includes:

[0100] Step S213a, generating an image geometric transformation matrix based on the visual sensor pixel coordinate correction parameter, and the image geometric transformation matrix is used for coordinate transformation processing of image data collected by the visual sensor.

[0101] For vision data compensation, receive the vision sensor pixel coordinate correction parameters (i.e. mathematical description containing translation, rotation, scaling and shear components) from step S212a, and instantiate it as a programmable executable image geometric transformation matrix (e.g. a specific affine transformation or perspective transformation matrix) which defines a one-to-one mapping relationship from the original image coordinates distorted by motion to the corrected ideal image coordinates, and integrate it into the image processing library for pixel-level coordinate transformation and resampling processing of each frame of image data collected by the vision sensor in real time, thereby directly eliminating the geometric distortion caused by relative motion at the image level.

[0102] Step S213b, based on the weight sensor compensation coefficient, generate a weight reading compensation function, which outputs the compensated weight value according to the real-time collected original weight data.

[0103] For weight data compensation, use the weight sensor compensation coefficient (usually a function or coefficient set related to motion acceleration) calculated in step S212b to construct a real-time weight reading compensation function; this function takes the real-time collected original weight reading and the synchronously acquired current motion acceleration data as input, and outputs the compensated, more accurate net weight value by built-in algorithms (such as directly subtracting the calculated inertial force component, or applying an acceleration feedback-based dynamic filter).

[0104] Step S213c, based on the spatial pose sensor calibration parameters, generate a pose data correction mapping relationship for calibration processing of spatial pose measurement data.

[0105] For pose data compensation, based on the spatial pose sensor calibration parameters (usually an inverse coordinate transformation matrix or offset vector) obtained in step S212c, define a pose data correction mapping relationship; the mapping relationship is applied to the original measurement data output by the spatial pose sensor (such as laser radar, GNSS / IMU combined navigation system), and through the application of an inverse rigid body transformation or coordinate offset, the measurement value drifted due to the motion of the sensor's own reference is corrected back to the stable global world coordinate system.

[0106] Step S213d, encode the image geometric transformation matrix, weight reading compensation function and pose data correction mapping relationship in time sequence to generate a sensor compensation instruction set synchronized with the spiral ship unloader control system clock.

[0107] The isomerized image geometric transformation matrix, weight reading compensation function and pose data correction mapping relationship generated in the foregoing steps are encapsulated and coded according to a unified timestamp to generate a structured sensor compensation instruction set that is strictly synchronized with the clock of the main control system of the spiral ship unloader, ensuring that all correction actions on the data are completely aligned in time and avoiding the introduction of new errors due to processing delays.

[0108] In step S213e, the sensor compensation instruction set is input to the digital twin model for compensation effect verification. When the verification result shows that the deviation between the data processed by the sensor compensation instruction set and the expected value of the ideal state after compensation simulated by the digital twin model exceeds the set threshold, the parameter values in the multi-sensor error compensation parameter set are adjusted again until the final compensation instruction is output after the verification passes.

[0109] By introducing the final safety verification link, the generated sensor compensation instruction set is first input to the digital twin model for execution, simulating the entire compensation process in a virtual environment, and comparing the compensated "data" with the theoretically perfect expected value after compensation derived by the digital twin model under ideal static working conditions. When the simulation verification shows that the deviation of some data exceeds the preset safety threshold (indicating insufficient compensation or overcompensation), a feedback loop is triggered to adjust the corresponding parameter values in the multi-sensor error compensation parameter set generated in step S212 (such as adjusting the gain of the motion-to-error mapping model), and a new instruction set is generated for verification, forming an iterative optimization process until the simulation verification passes, and finally the verified and reliable final compensation instruction is output to the actual real-time data correction unit for execution.

[0110] The final transformation from error theory calculation to engineering implementation not only generates concrete and executable compensation instructions, but more importantly, by introducing virtual verification and iterative optimization based on digital twins, a crucial "safety valve" and "optimizer" is added to the entire feedforward compensation system, ensuring the effectiveness and safety of the compensation action, greatly preventing the risk of control decision errors caused by model mismatch or parameter inaccuracy in the real system, thereby improving the reliability of dynamic compensation to the high standard requirements applicable to industrial unattended systems.

[0111] Specifically, based on the job scene situation awareness result in step S300, the spiral ship unloader digital twin model is driven for real-time simulation to obtain simulation results including future material accumulation space morphology, device running performance parameters and / or potential risk probability, including:

[0112] Step S310, input the material geometric distribution, physical properties and category attributes contained in the job scene situation awareness result into the digital twin model, calculate the flow and accumulation process of the material in the ship cabin in the next control period through discrete element simulation, and output a material accumulation form prediction map.

[0113] The high-precision situation awareness result is used to drive the digital twin model for forward-looking simulation to predict the job state in the future short time domain. First, the key information contained in the job scene situation awareness result output in step S200, including the material geometric distribution accurately described by the three-dimensional point cloud, the material physical properties (such as density, internal friction angle, and viscosity) obtained through multi-source fusion, and the material category attributes determined through spectral analysis, are input into the digital twin material simulation module constructed based on the discrete element method (DEM) as initial boundary conditions. The module simulates the dynamic whole process of the flow, shear, transportation and final accumulation of the material in the ship cabin in the next control period by calculating the motion and interaction of millions of material particles under the action of gravity, inter-particle force and mechanical force of the screw head, and outputs a material accumulation form prediction map that details the predicted surface form, internal stress distribution and density change.

[0114] Step S320, based on the environment pose relationship in the job scene situation awareness result and the current running state of the equipment, the running process of the equipment under different control strategies is simulated through the digital twin model to obtain a set of equipment performance parameter predictions including energy consumption, vibration amplitude and structural stress.

[0115] Parallelly, the equipment performance simulation is carried out, the environment pose relationship (i.e. the relative pose of the ship unloader and the ship, the material pile) in the situation awareness result is combined with the current actual running state of the equipment (such as the rotating speed and torque of each motor), and in the multi-body dynamics and finite element analysis module of the digital twin, different candidate control strategies (such as different screw head feeding speed and rotating speed combinations) are simulated. By solving the dynamic equation and structural mechanics equation of the system, the expected power consumption, vibration amplitude spectrum and structural stress distribution of key points of the equipment key components (such as the screw head, lifting mechanism and support structure) under various strategies are calculated, so as to obtain a set of equipment performance parameter predictions containing multi-dimensional performance indicators.

[0116] Step S330, according to the material accumulation form prediction map and the equipment performance parameter prediction set, combining the abnormal event records under the corresponding working conditions in the historical job database, the occurrence probability of each type of potential risk under the current job state is calculated through a pattern recognition algorithm to generate a risk probability distribution map.

[0117] The simulation outputs of the two steps are combined, i.e., the material accumulation pattern prediction map predicting the future material state and the equipment performance parameter prediction set predicting the future equipment state, as a comprehensive descriptor of the current operation state, and are matched with the recorded abnormal events (such as stall, overload, collision, abnormal vibration) in the historical operation database under the corresponding working conditions (corresponding material, ship type, sea state, equipment action) for pattern recognition; the conditional probability of triggering of various potential risk events under the current state is calculated through a machine learning algorithm (such as cluster analysis, Bayesian network), and finally a risk probability distribution map is generated, which is labeled on the operation area map and quantifies the risk probability at different spatial positions and under different operation types.

[0118] By combining high-fidelity physical simulation (discrete element, multi-body dynamics) with data-driven historical experience (abnormal event library), the evolution process of the "material-equipment-environment" complex system is advanced in the virtual space, thereby improving the control system from "visualization" that can only perceive the current state to "predictability" that can foresee the short-term future, providing unprecedented depth information support for generating optimal control instructions that are both efficient and safe, and is the core basis for realizing intelligent unmanned decision-making.

[0119] Further, the optimization control instruction generated based on the preset decision rule according to the simulation result of the digital twin model in step S400 includes:

[0120] In step S410, real-time adjustment instructions for the spiral head motion trajectory are generated according to the uneven material distribution area shown in the material accumulation pattern prediction map, and the real-time adjustment instructions include optimized settings of the depth excavation parameter and the horizontal movement path for the specific area.

[0121] According to the prediction of the future material state, a precise excavation strategy is formulated to optimize the material taking efficiency. This step receives and analyzes the material accumulation pattern prediction map from the digital twin simulation, identifies the uneven material distribution area shown in the map, such as local "material peak" with excessive accumulation or "material valley" with insufficient excavation; based on the spatial position, volume and physical properties of the material in these areas, the decision algorithm generates real-time adjustment instructions for the spiral head motion trajectory, which no longer simply changes the speed, but includes a series of fine operation parameters for the specific area, such as increasing the cutting depth, reducing the horizontal movement speed and excavating with higher torque for the "material peak" area, while for the "material valley" or sparse area, the instructions may include reducing the excavation depth and increasing the coverage speed to avoid empty excavation, thereby achieving efficient and balanced material taking on the entire operation surface, avoiding local overload or idling of the equipment, and maximizing the material delivery volume per unit time.

[0122] Step S420, based on the device performance parameter prediction, the abnormal energy consumption or vibration overrun situation of the centralized display is predicted, and the adjustment instruction of the device operation parameter is generated. The adjustment instruction includes the coordinated control parameters of the screw rotation speed, the elevator working frequency and the conveying belt conveying rate.

[0123] By monitoring and analyzing the predictive values of various indicators in the device performance parameter prediction, once it is predicted that some key parameters will approach or exceed the safety threshold in the future period (such as abnormal increase of drive motor energy consumption indicating possible stall, or vibration amplitude overrun indicating potential fatigue damage), the step will immediately generate adjustment instructions of device operation parameters. These instructions are not isolated adjustment of a single device, but include coordinated control parameters of a series of related actuators such as screw rotation speed, elevator working frequency and conveying belt conveying rate. For example, when it is predicted that the vibration will increase, the instructions will simultaneously slightly reduce the screw rotation speed and the elevator frequency, while adjusting the conveying belt speed to match the new flow, forming a systematic load reduction and vibration reduction control strategy, so as to ensure continuous production while maintaining the device operation state in the optimal interval of safety, high efficiency and low consumption.

[0124] Step S430, for the high-risk area identified in the risk probability distribution graph, a preventive treatment instruction is generated. The preventive treatment instruction includes device obstacle avoidance path planning, emergency deceleration control and early warning signal triggering mechanism.

[0125] Based on the risk prediction of the forward-looking safety protection, the risk probability distribution graph processed quantifies the risk levels of different operations and areas. For the high-risk areas (such as high-probability collision area) or high-risk operations (such as high-probability stall action) identified in the graph, this step does not wait for the occurrence of the abnormality to act, but generates preventive treatment instructions in advance. These instructions include a complete response mechanism, such as immediately re-planning the motion path of the screw head to avoid obstacles or high-risk areas, automatically triggering the emergency deceleration control program to smoothly reduce the device speed when approaching the high-risk area, and sending different levels of early warning signals (such as early warning, alarm) to the monitoring system to prompt potential risks. Thus, the traditional passive safety protection is transformed into an active and embedded safety strategy, greatly improving the intrinsic safety level of the system.

[0126] Step S440, integrate the real-time adjustment instruction, the adjustment instruction and the preventive treatment instruction according to the job priority and the time sequence relationship to obtain the optimized control instruction.

[0127] By receiving the multiple possible existing timing or logical conflict instruction drafts from S410, S420 and S430; according to the preset global optimization target (such as safety first, efficiency first) and strict job priority (such as obstacle avoidance instruction always takes priority over efficiency optimization instruction), these heterogeneous instructions are logically integrated and sorted according to their urgency, importance and execution time sequence, and the possible conflicts are resolved, and finally a coordinated and consistent optimized control instruction which can be safely and efficiently executed is packaged and output to the subsequent verification and execution module.

[0128] The above process realizes the closed-loop across from "perception-prediction" to "decision-action", and converts the advanced prediction information generated by the digital twin model into a series of specific, executable and coordinated optimization and control actions. These actions not only pursue the improvement of production efficiency, but also prioritize and proactively protect and prevent safety, so as to ensure that the entire unloading operation can be self-sufficient, safe, efficient and economical under harsh conditions without human intervention.

[0129] Further, after obtaining the optimized control instruction in step S440, it further includes:

[0130] Step S451, input the optimized control instruction into the digital twin model for simulation execution, and obtain the device running prediction parameter and material state change data.

[0131] After generating the preliminary optimized control instruction in step S440, first input the instruction as an input to drive the high-fidelity spiral unloader digital twin model again for a round of high-frequency real-time simulation. However, the purpose of this simulation is no longer prediction, but to simulate the execution of these pending control instructions. In this process, the digital twin model strictly follows the instructions and simulates every action of the device, and accurately calculates the expected changes of various operating parameters (such as motor current, vibration spectrum, stress distribution) of the device under these actions, as well as the dynamic response (such as evolution of stacking form, flow state) of the material due to these actions, thereby obtaining a detailed set of "device running prediction parameters" and "material state change data".

[0132] Step S452, compare the device running prediction parameters with the device performance parameter prediction set obtained by the digital twin model simulation, and compare the material state change data with the material stacking form prediction map.

[0133] A strict comparison and verification process is started, comparing the "device running prediction parameters" obtained from simulation execution with the "device performance parameter prediction set" obtained from the digital twin model based on previous situational awareness results in step S320, and comparing the "material state change data" obtained from simulation execution with the "material accumulation form prediction map" output in step S310. The core of this comparison is to verify whether the execution effect of the control instruction is consistent with the initial performance prediction and form prediction, whether there is a significant deviation caused by the instruction being too aggressive or conservative (for example, the vibration value obtained from simulation execution is much higher than the predicted safe range, or the actual material taking efficiency is much lower than expected).

[0134] In step S453, the control instruction deviating from the device performance parameter prediction set or the material accumulation form prediction map is adjusted in motion parameters according to the comparison result.

[0135] According to the comparison result in step S452, online optimization is performed. For those control instructions that cause the running parameters or material state to deviate significantly from the expected trajectory, the system will adaptively adjust its motion parameters (for example, slightly reduce the screw head feed speed for a certain material peak to reduce vibration, or fine-tune the horizontal movement path to more accurately match the predicted material profile), the goal is to make the simulation execution result converge to the initially predicted ideal state as much as possible.

[0136] In step S454, a safe execution instruction set is generated based on the adjusted control instruction, and after verification of the adjusted instruction in the digital twin model, it is output to the screw unloader control system for execution.

[0137] Based on the adjusted control instruction, an optimized safe execution instruction set with improved expected effect is regenerated, and it is again sent to the digital twin model for final effect verification; only when this verification confirms that all key indicators are within the acceptable error range, this final safe execution instruction set will be approved to be output to the real screw unloader control system for execution, otherwise a new round of adjustment and verification cycle will be triggered.

[0138] Before the control instruction is issued to the physical world, an absolutely safe "digital sandbox" is created, through repeated simulation, comparison and iterative optimization in this sandbox, it is ensured that the control strategy to be executed is not only theoretically optimal, but also safe, reliable and highly consistent with the expected effect in actual execution, so as to minimize the risk of equipment damage or operation interruption caused by decision-making errors, and provide the final and most critical safety guarantee for realizing truly unmanned and reliable operation.

[0139] Correspondingly, please refer to Figure 2The second aspect of the embodiment of the present application provides a kind of unmanned spiral ship unloader material and ship control system, based on the above-mentioned unmanned spiral ship unloader material and ship control method carries out material and ship control, it includes:

[0140] Data acquisition module 1, for real-time acquisition of multidimensional sensor data by multiple types of sensors deployed on spiral ship unloader, multidimensional sensor data includes visual image, material weight, material type and space pose.

[0141] Data fusion module 2, for the time and space registration and fusion processing of the collected multidimensional sensor data, obtain the operation scene situation awareness result for comprehensively representing the real-time state of unloading operation scene.

[0142] Data simulation module 3, for driving spiral ship unloader digital twin model based on operation scene situation awareness result to carry out real-time simulation, obtain simulation result, simulation result includes future material accumulation space form, equipment running performance parameter and / or potential risk probability.

[0143] Optimization control module 4, for generating optimization control instruction based on preset decision rule according to simulation result of digital twin model, to realize adaptive adjustment of unloading operation parameters and intelligent disposal of abnormal working condition.

[0144] Correspondingly, the third aspect of the embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory connected with the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to make the at least one processor execute the above-mentioned unmanned spiral ship unloader material and ship control method.

[0145] Correspondingly, the fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by processor to realize the above-mentioned unmanned spiral ship unloader material and ship control method.

[0146] The embodiment of the present application aims to protect a kind of unmanned spiral ship unloader material and ship control method and system, with the following effects:

[0147] 1. Through the cooperative work of multiple types of sensors and the innovative spatiotemporal registration and fusion algorithm, the defects of incomplete and inaccurate perception of single sensor under harsh working conditions such as dust, light, and shaking are overcome; in particular, through dynamic compensation of the six-degree-of-freedom relative motion of the ship and the ship unloader, the data deviation caused by the drift of the measurement reference is greatly eliminated, generating comprehensive situation awareness results containing the geometric, physical, and environmental pose relationship of the material, providing an unprecedented accurate, stable, and reliable data foundation for subsequent intelligent decision-making, and fundamentally improving the system's adaptability to complex dynamic environments.

[0148] 2. The real-time perceived operation scene situation is mirrored to a high-fidelity digital twin model, and the digital twin model is driven to perform real-time simulation, which not only reproduces the current state but also predicts the future short-term working conditions; the accumulation form of the material, the running performance of the equipment (such as energy consumption and vibration), and the probability distribution of potential risks can be predicted in advance, so that the control strategy is changed from the traditional "after-the-fact reaction" mode to the "before-the-fact prediction" and "during-the-process intervention" mode, realizing the leap from passive response to active optimization, significantly improving the safety of the operation and avoiding potential failures.

[0149] 3. A closed-loop optimization and safety decision-making mechanism based on simulation verification is formed, and instead of directly executing the control instructions generated based on rules, the instruction set is first "pre-executed" and verified in the digital twin model; by comparing the deviation between the prediction result and the expected target, the instructions are fine-tuned and optimized, and finally the safe execution instruction set verified by "sand table deduction" is generated; it is equivalent to adding a virtual "safety valve" to the control instructions, greatly avoiding the possibility of executing incorrect or high-risk instructions on real equipment due to sensing errors or model mismatches, ensuring that the control system's decisions are both optimized and reliable, and ultimately realizing the self-adaptive and intelligent unmanned operation of the entire unloading process.

[0150] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied therein.

[0151] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0152] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0153] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0154] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application. Although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for controlling materials and ships using an unmanned screw unloader, characterized in that, Includes the following steps: By deploying multiple types of sensors on the spiral unloader, multi-dimensional sensing data is collected in real time, including visual images, material weight, material type, and spatial pose. The collected multi-dimensional sensor data is spatiotemporally registered and fused to obtain the situational awareness results of the operation scenario, which are used to comprehensively characterize the real-time status of the unloading operation scenario. Based on the situational awareness results of the aforementioned work scenario, the digital twin model of the screw unloader is driven to perform real-time simulation to obtain simulation results, which include the future material accumulation space shape, equipment operating performance parameters and / or potential risk probabilities. Based on the simulation results of the digital twin model, optimized control commands are generated based on preset decision rules to achieve adaptive adjustment of unloading operation parameters and intelligent handling of abnormal working conditions. The process of spatiotemporal registration and fusion of the collected multi-dimensional sensor data yields situational awareness results for comprehensively characterizing the real-time status of the unloading operation scenario, including: Using a unified time and space reference, all the multi-dimensional sensor data are spatiotemporally aligned, a mapping model between image pixel coordinates and world coordinates is established, and the observation deviation caused by the relative motion of the ship and the unloader is dynamically compensated to obtain a standard data stream that is spatiotemporally synchronized. Based on the standard data stream, according to the confidence level, timeliness and operation progress of each data source, the fusion weight is adaptively allocated, and the visual material outline, weight information, type information and pose information are weighted, fused and cross-verified to obtain a comprehensive state estimate including material geometric distribution, physical attributes, category attributes and environmental pose relationship, which is used as the situational awareness result of the operation scene. The dynamic compensation for observational deviations caused by the relative motion between the ship and the unloading machine includes: The six-degree-of-freedom relative pose data between the ship and the unloader is acquired in real time. The six-degree-of-freedom relative pose data is input into the motion prediction model to calculate the motion trajectory and attitude change between the ship and the unloader in the next control cycle. Calculate the pixel-level coordinate offset caused by the motion trajectory and attitude change to the visual sensor imaging, as well as the perturbation error to the non-visual sensor measurement reference; Based on the coordinate offset and disturbance error, corresponding inverse compensation parameters are generated to perform online correction on the standard data.

2. The unmanned screw unloader material and ship control method according to claim 1, characterized in that, After performing online calibration on the standard data, the method further includes: By comparing the corrected standard data stream with the theoretical expected value calculated under ideal static conditions in a unified coordinate system based on the digital twin model, residual feedback is formed to adaptively adjust the parameters of the motion prediction model and optimize the subsequent compensation accuracy.

3. The unmanned screw unloader material and ship control method according to claim 1, characterized in that, Calculating the pixel-level coordinate offset caused by the motion trajectory and attitude change to the visual sensor imaging, as well as the perturbation error to the non-visual sensor measurement reference, includes: Based on the installation position and orientation parameters of the visual sensor, the six-degree-of-freedom relative pose data is converted into motion components in the image coordinate system. The image translation, rotation angle and scale change rate caused by the relative motion are calculated respectively to obtain the visual sensor pixel coordinate correction parameters. Based on the installation structure parameters and kinematic relationship of the weight sensor, the weight measurement deviation caused by the inertial force component generated by relative motion is calculated, and the weight sensor compensation coefficient is obtained. Based on the installation geometry of the spatial pose sensor, the offset of the measurement reference caused by relative motion is calculated to obtain the calibration parameters of the spatial pose sensor. The visual sensor pixel coordinate correction parameters, weight sensor compensation coefficients, and spatial pose sensor calibration parameters are combined to form a multi-sensor error compensation parameter set.

4. The unmanned screw unloader material and ship control method according to claim 3, characterized in that, Based on the coordinate offset and disturbance error, corresponding inverse compensation parameters are generated, including: Based on the pixel coordinate correction parameters of the vision sensor, an image geometric transformation matrix is ​​generated. The image geometric transformation matrix is ​​used to perform coordinate transformation processing on the image data acquired by the vision sensor. Based on the weight sensor compensation coefficient, a weight reading compensation function is generated, and the weight reading compensation function outputs the compensated weight value according to the real-time collected raw weight data; Based on the calibration parameters of the spatial pose sensor, a pose data correction mapping relationship is generated for calibrating the spatial pose measurement data. The image geometric transformation matrix, the weight reading compensation function, and the pose data correction mapping relationship are encoded according to the time sequence to generate a sensor compensation instruction set synchronized with the clock of the screw unloader control system; The sensor compensation instruction set is input into the digital twin model to verify the compensation effect. When the verification result shows that the deviation between the data processed by the sensor compensation instruction set and the expected value of the ideal state after compensation simulated by the digital twin model exceeds a set threshold, the parameter values ​​in the multi-sensor error compensation parameter set are readjusted until the verification is passed and the final compensation instruction is output.

5. The unmanned screw unloader material and ship control method according to any one of claims 1-4, characterized in that, Based on the situational awareness results of the aforementioned operational scenario, the digital twin model of the auger unloader is driven to perform real-time simulation, obtaining simulation results including the future material accumulation space shape, equipment operating performance parameters, and / or potential risk probabilities, including: The material geometric distribution, physical attributes and category attributes contained in the situational awareness results of the operation scenario are input into the digital twin model. The flow and accumulation process of the material in the ship's hold in the next control cycle is calculated through discrete element simulation, and the predicted material accumulation pattern is output. Based on the environmental pose relationship and the current operating status of the equipment in the situational awareness results of the work scenario, the operation process of the equipment under different control strategies is simulated in the digital twin model to obtain a set of predicted equipment performance parameters including energy consumption, vibration amplitude and structural stress. Based on the predicted material accumulation pattern and the predicted equipment performance parameters, combined with the abnormal event records under the corresponding working conditions in the historical operation database, the probability of occurrence of various potential risks under the current working condition is calculated by the pattern recognition algorithm, and a risk probability distribution map is generated.

6. The unmanned screw unloader material and ship control method according to claim 5, characterized in that, Based on the simulation results of the digital twin model, optimized control commands are generated according to preset decision rules, including: Based on the uneven material distribution area shown in the material accumulation pattern prediction map, a real-time adjustment instruction for the spiral head movement trajectory is generated. The real-time adjustment instruction includes optimized settings for depth mining parameters and horizontal movement path for specific areas. Based on the abnormal energy consumption or excessive vibration conditions predicted and displayed by the equipment performance parameters, adjustment instructions for equipment operating parameters are generated. These adjustment instructions include coordinated control parameters for screw rotation speed, elevator operating frequency, and conveyor belt conveying speed. For high-risk areas identified in the risk probability distribution map, preventive handling instructions are generated, which include equipment obstacle avoidance path planning, emergency deceleration control, and early warning signal triggering mechanisms. The real-time adjustment command, the regulation command, and the preventive action command are integrated according to the operation priority and time sequence to obtain the optimized control command.

7. The unmanned screw unloader material and ship control method according to claim 6, characterized in that, After obtaining the optimized control command, the method further includes: The optimized control commands are input into the digital twin model for simulation execution to obtain equipment operation prediction parameters and material state change data; The predicted parameters of equipment operation are compared with the predicted set of equipment performance parameters obtained by simulation of the digital twin model, and the material state change data are compared with the predicted material accumulation pattern. Based on the comparison results, the motion parameters of the control commands that deviate from the equipment performance parameter prediction set or the material accumulation pattern prediction diagram are adjusted. A safe execution instruction set is generated based on the adjusted control instructions, and after the adjusted instructions are verified in the digital twin model, they are output to the screw unloader control system for execution.

8. A material and ship control system for an unmanned screw unloader, characterized in that, Material and ship control based on the unmanned screw unloader material and ship control method as described in any one of claims 1-7 includes: The data acquisition module is used to collect multi-dimensional sensor data in real time through various types of sensors deployed on the spiral unloader. The multi-dimensional sensor data includes visual images, material weight, material type and spatial pose. The data fusion module is used to perform spatiotemporal registration and fusion processing on the collected multi-dimensional sensor data to obtain the situational awareness results of the operation scenario that comprehensively characterize the real-time status of the unloading operation scenario. The data simulation module is used to drive the digital twin model of the screw unloader to perform real-time simulation based on the situational awareness results of the operation scenario, and obtain simulation results, including the simulation results of the future material accumulation space shape, equipment operating performance parameters and / or potential risk probabilities. The optimization control module is used to generate optimization control commands based on the simulation results of the digital twin model and preset decision rules, so as to realize adaptive adjustment of unloading operation parameters and intelligent handling of abnormal working conditions.

Images