Customs port concentrate sampling robot monitoring method and system based on multi-modal digital twinning

Abstract

The invention relates to the technical field of data management, in particular to a customs port sampling robot monitoring method and system based on multi-modal digital twinning, and the method comprises the steps: building the real-time real mapping of a sampling robot in a virtual space through collected multi-modal data; therefore, the functions of equipment prediction and maintenance, fault simulation, remote inspection and sampling drill, path optimization, operation analysis and the like are realized, the purposes of real-time monitoring, full-life-cycle dynamic simulation and decision optimization in the sampling process are achieved, equipment faults and complex environment conditions are reduced, and the sampling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method and system for monitoring a concentrate sampling robot at a customs port based on a multimodal digital twin. Background Art

[0002] With the rapid development of global mineral trade, sampling robots are playing an increasingly critical role in quality control and trade settlement at customs ports. However, existing sampling robots mostly rely on single sensor data or fixed processes, are difficult to adapt to complex dynamic sampling environments, and lack the ability to deeply perceive the equipment status and external environment. The development of digital twin technology provides new possibilities for solving this problem.

[0003] Digital twin technology achieves equipment monitoring and control through the integration of virtual and real. It can map the multimodal data of physical equipment (such as vision, displacement, temperature, pressure, etc.) to the virtual space in real time, thereby realizing accurate equipment operation monitoring and predictive maintenance.

[0004] Multimodal digital twin technology is an emerging intelligent technology that aims to build a more comprehensive digital twin model by collecting, processing and analyzing multimodal data. Based on the traditional single data source (such as temperature, displacement, vision, etc.), multimodal digital twin technology integrates multiple data types and captures and integrates information of different modes through sensor networks and data fusion technology. Specifically, this multimodal data can include visual information, temperature data, displacement changes, pressure data, etc., so that the system can perceive the status of the equipment in all directions and realize multimodal perception.

[0005] For the monitoring application of sampling robots, multimodal digital twin technology can provide more detailed and dynamic status feedback. Through multimodal perception, the monitoring system can track the various parameters and operating status of the robot in real time. It can not only monitor the numerical changes such as mechanical displacement, temperature or pressure, but also identify environmental changes and capture abnormal behavior through visual information. With the integrated analysis of multimodal data, the digital twin system can achieve three-dimensional and precise status monitoring, providing strong data support for the comprehensive control of the equipment's operating status.

[0006] The application of multimodal digital twin technology can not only improve the monitoring system's comprehensive perception of the sampling robot's status, but also provide key support for optimizing operation strategies and improving system reliability. Multimodal digital twin technology can help predict possible failures and perform preventive maintenance, thereby effectively reducing the risk of equipment downtime and optimizing maintenance efficiency and resource allocation.

[0007] However, in the application of sampling robots, the research on digital twin technology is still in its infancy. How to integrate multiple sensor data and build a real-time dynamic robot monitoring system is the main challenge currently faced. Summary of the invention

[0008] In order to address the deficiencies in the prior art, the present invention provides a customs port concentrate sampling robot monitoring system based on multimodal digital twins. Through the collected multimodal data, a real-time mapping of the sampling robot is established in the virtual space, thereby realizing equipment predictive maintenance, fault simulation, remote inspection and sampling drills, path optimization, operation analysis and other functions, so as to achieve the purpose of real-time monitoring, dynamic simulation of the entire life cycle of the sampling process and decision optimization, reduce equipment failures and environmental complexities, and improve sampling efficiency.

[0009] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0010] According to one aspect of the present invention, a method for monitoring a concentrate sampling robot at a customs port based on a multimodal digital twin is provided, comprising the following steps:

[0011] S1. Conduct multi-modal real-time monitoring of sampling robots and concentrate environmental data, and integrate multi-source data;

[0012] S2. Based on real-time data and historical data, a virtual model of the sampling robot equipment is constructed in the virtual space from four dimensions: geometry, physics, behavior, and rules, forming a virtual simulation of the entire life cycle of the robot sampling process;

[0013] S3, based on the multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model, reflects the real-time status of the sampling operation and realizes the advance prediction of path optimization, providing data and model basis for global collaborative optimization in the sampling process;

[0014] S4. Based on the visualization engine, the sampling process of the sampling robot is visualized to intuitively present the state of the robot during sampling;

[0015] S5. Build a remote monitoring terminal;

[0016] S6. The decision optimization and control feedback results generated by the virtual model act on the sampling robot entity.

[0017] Furthermore, the multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model in S3 specifically includes:

[0018] S31, construct a multimodal sensor information transmission network for sampling robots;

[0019] S32. Construct a multimodal digital twin model of the sampling robot;

[0020] S33, Multimodal sensor information mining and state perception of sampling robots.

[0021] Further, S31, constructing a multimodal sensor information transmission network of a sampling robot adopts the following method:

[0022] A variety of sensors, depth cameras and microphones are installed on the sampling robot to collect information about the sampling robot, including environmental information, to form a perception of the full state of the sampling robot and provide multimodal perception data for the digital twin prototype.

[0023] Further, S32, constructing a multimodal digital twin model of the sampling robot adopts the following method:

[0024] S321. Use ADAMS software to build a 1:1 three-dimensional virtual model based on the physical dimensions of the sampling robot, and perform preliminary motion simulation of the model; export the model in ADAMS to a file format supported by Unity 3D, and after importing, confirm the accuracy of the dimensions, proportions, and hierarchical structure;

[0025] S322. Reset the assembly relationship between the various parts of the sampling robot in Unity 3D, and use Unity's parent-child hierarchy and joint components to simulate the mechanical connection in reality; configure Unity's physics engine, set the physical properties of the model such as mass, friction, elasticity, etc., write scripts to control the sampling robot, simulate the dynamic behavior of the sampling robot when working, and use Unity 3D's powerful rendering and animation tools to create an interactive sampling robot virtual prototype to ensure that the virtual prototype can simulate all possible motion postures of the physical entity;

[0026] S323. Sort out the multi-source and multi-modal data of various sensors installed on the sampling robot, as well as depth cameras and microphones, and design appropriate data interfaces to allow sensor data to be input into the virtual model in Unity 3D in real time; develop an interactive user interface to display the status and sensor data of the sampling robot, preliminarily drive the virtual prototype, write control logic that conforms to the movement of physical entities, ensure that the virtual prototype can respond to external stimuli in real time, and preliminarily build a digital twin model of the multi-modal sampling robot.

[0027] Further, S33, the sampling robot multimodal sensor information mining and state perception adopts the following method:

[0028] S331. Use a signal conditioning system to optimize and denoise the sensor data from a physical level, and use a digital filter to filter the sensor data from a software level to remove undesirable components in the data;

[0029] S332. Apply statistical analysis methods to identify and process outliers, calibrate sensors, ensure consistency of multi-source data in time series through timestamp alignment, and establish sensor response curves to adapt to different operating conditions; smooth time series data to reduce random fluctuations; use spatial filtering technology to filter image and depth camera data;

[0030] S333. Use the Z-score standardization method to convert all sensor data into a unified dimension and numerical range. Use the reinforcement learning algorithm and set up a reward mechanism to gradually learn and mine real sensor data information through interaction with the environment, and then optimize the virtual model of the concentrate sampling robot. At the information level, analyze the relationship between different sensor data, optimize the data processing process, and improve data processing efficiency and accuracy. Use the optimized data to drive the virtual prototype of the concentrate sampling robot, fully simulate its motion posture and operation strategy, and conduct further calibration and optimization through interactive verification between the virtual prototype and the physical concentrate sampling robot. Based on the optimized data and feedback information, enhance the perception ability of the concentrate sampling robot to form a highly intelligent multimodal information perception system.

[0031] According to another aspect of the present invention, there is provided a customs port sampling robot monitoring system based on multimodal digital twins, comprising a physical layer, a model layer, a data layer, a service layer, and an application layer;

[0032] The physical layer collects real-time data through the sampling robot and sensors installed in the sampling environment, and based on a unified data communication protocol and conversion method, realizes multi-source heterogeneous real-time data access and integration at the physical layer, forming the interconnection and integration of man-machine-object-environment;

[0033] The model layer integrates and fuses the multi-dimensional virtual models of the sampling robot and the sampling environment to establish a virtual model covering the entire life cycle of the sampling process for simulation;

[0034] The data layer is used to sense and process multimodal data;

[0035] The service layer is used for data-driven services, including pre-sampling, sampling and post-sampling services;

[0036] The application layer interacts with the service layer and the user, and the result data after the command execution is fed back to the virtual model for easy monitoring and operation.

[0037] Furthermore, the data layer includes an edge processing layer, a collection layer, and a cloud processing layer;

[0038] The edge processing layer includes a data preprocessing module and a local fault detection module. The data preprocessing module processes and analyzes sensor data in real time, filters useless information, and extracts key features. The local fault detection module quickly detects abnormal conditions in local devices to reduce data transmission delays and cloud computing loads.

[0039] The collection layer collects the edge-processed data and sends it to the remote server or cloud through a specific interface;

[0040] The cloud processing layer performs in-depth analysis, modeling, simulation, and prediction on the collected multimodal data.

[0041] Furthermore, the pre-sampling service uses multimodal digital twin technology to perform pre-operation simulation and human-computer interaction operations; the during-sampling service performs real-time modeling and simulation of each system of the running sampling robot to realize equipment fault diagnosis, sampling visualization, and path optimization functions, ensuring the healthy operation of the equipment during the sampling process and safe and efficient operation; the post-sampling service summarizes the experience of completed operations, automatically generates electronic logs, operation reports, etc., integrates the sampling database in the form of multimodal field data and knowledge, realizes data closure, and provides a basis for the optimization of subsequent operation plans.

[0042] Furthermore, the application layer includes a monitoring terminal, which supports 2D and 3D visualization interfaces and can display the operating status of the robot, the simulation results of the twin model, and the monitoring data of the sensor in real time.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. The present invention can effectively improve the accuracy of the concentrate sampling process, can timely discover and correct the deviation in the sampling process, improve the accuracy of sampling, and can adapt to environmental changes in real time, such as temperature, humidity and other external conditions, to ensure the stability and reliability of the sampling process in various environments;

[0045] 2. By integrating data from different sensors, we can fully understand the robot's working environment and status, enhance the intelligence of the system, and make decisions more accurate;

[0046] 3. Enhanced real-time performance: through the combination of multimodal sensors and digital twin models, all-round monitoring of the real-time status of the concentrate sampling robot can be achieved;

[0047] 4. Fault prediction and maintenance optimization: through multimodal twin data analysis and machine learning technology, robot failures can be predicted in advance, equipment downtime can be reduced, and maintenance costs can be reduced;

[0048] 5. Operational safety is improved. The system can detect changes in the operating environment in real time to ensure that the robot works in a safe state and avoid accidents.

[0049] 6. The user interaction experience is improved, and the staff can understand the working status of the robot more intuitively, thus improving the efficiency of human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Attached Figure 1 This is the block diagram of this system;

[0051] Attached Figure 2 It is a flow chart of multimodal data processing. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0053] According to one aspect of the present invention, a method for monitoring a concentrate sampling robot at a customs port based on a multimodal digital twin is provided, comprising the following steps:

[0054] S1. Conduct multimodal real-time monitoring of sampling robots and concentrate environmental data, and integrate multi-source data. The multimodal real-time monitoring of sampling robots and concentrate environmental data includes multimodal real-time monitoring of robots, concentrate environment and other data through various multimodal sensors and other data acquisition instruments;

[0055] S2. Based on real-time data and historical data, a virtual model of the sampling robot equipment is constructed in the virtual space from four dimensions: geometry, physics, behavior, and rules, forming a virtual simulation of the entire life cycle of the robot sampling process;

[0056] S3. Based on the multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model, the real-time status of the sampling operation is reflected, and the advance prediction of path optimization is realized, providing data and model basis for global collaborative optimization in the sampling process; at the same time, with the help of advanced computer technology, safe, efficient and low-cost simulation operations can be carried out on the virtual model to realize functions such as pre-sampling drills, sampling optimization and post-sampling analysis;

[0057] S4. Based on the visualization engine, the sampling process of the sampling robot is visualized to intuitively present the state of the robot during sampling;

[0058] S5. Through data, engineering and other multi-party interconnection, build a remote monitoring terminal, optimize the operation process and improve the sampling quality;

[0059] S6. Utilize human-computer interaction, intelligent control and other technologies to apply the decision optimization and regulation feedback results generated by the virtual model to the sampling robot entity, and ultimately achieve closed-loop regulation of the entire life cycle of the sampling process.

[0060] Specifically, the multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model in S3 includes:

[0061] S31, construct a multimodal sensor information transmission network for sampling robots;

[0062] Select appropriate locations on the concentrate sampling robot to install vibration sensors, torque sensors, component analysis sensors, gyroscopes, laser radars, and angle sensors for measuring the rotation and pitch angles of the robotic arm, and install depth cameras and microphones for collecting surrounding environment information to collect concentrate sampling robot information including environmental information;

[0063] Taking advantage of the characteristics of the concentrate sampling robot, tensile pressure sensors, displacement sensors and temperature sensors are installed to form a perception of the full state (vision / auditory / tactile) of the concentrate sampling robot, thereby providing multimodal perception data for the digital twin prototype.

[0064] S32. Construct a multimodal digital twin model of the sampling robot;

[0065] S321. Use ADAMS software to build a 1:1 three-dimensional virtual model based on the physical dimensions of the sampling robot and perform preliminary motion simulation of the model; export the model in ADAMS to a file format supported by Unity 3D, such as FBX or OBJ, and after importing, confirm the accuracy of the dimensions, proportions, and hierarchical structure;

[0066] S322. Reset the assembly relationship between the various parts of the sampling robot in Unity 3D, and use Unity's parent-child hierarchy and joint components to simulate the mechanical connection in reality; configure Unity's physics engine, set the physical properties of the model such as mass, friction, elasticity, etc., write scripts to control the sampling robot, simulate the dynamic behavior of the sampling robot when working, and use Unity 3D's powerful rendering and animation tools to create an interactive sampling robot virtual prototype to ensure that the virtual prototype can simulate all possible motion postures of the physical entity;

[0067] S323. Sort out multi-source and multi-modal data including pressure sensors, gyroscopes, vibration sensors, angle sensors, displacement sensors, temperature sensors, radar sensors, and environmental sensors (depth cameras, microphones), and design appropriate data interfaces to allow sensor data to be input into the virtual model in Unity 3D in real time; develop an interactive user interface to display the status and sensor data of the concentrate sampling robot, preliminarily drive the virtual prototype, write control logic that conforms to the movement of physical entities, ensure that the virtual prototype can respond to external stimuli in real time, and preliminarily build a digital twin model of the multi-modal concentrate sampling robot.

[0068] S33, sampling robot multimodal sensor information mining and state perception;

[0069] S331. In extremely harsh working environments, accurate collection and processing of sensor information is crucial. On the one hand, a signal conditioning system is used to optimize and denoise the sensor data from a physical level. On the other hand, a digital filter is used to filter the sensor data from a software level to remove undesirable components in the data, thereby preserving the most comprehensive sensor data information.

[0070] S332. Apply statistical analysis methods such as Z-score and box plots to identify and handle outliers, calibrate sensors, ensure consistency of multi-source data in time series through timestamp alignment, and establish sensor response curves to adapt to different operating conditions; apply moving average, exponential smoothing and other methods to smooth time series data to reduce random fluctuations, and apply spatial filtering techniques such as Gaussian filtering to filter spatial information such as images and depth data (depth cameras);

[0071] S333. Use the Z-score standardization method to convert all sensor data into a unified dimension and numerical range. Use the reinforcement learning algorithm and set up a reward mechanism to gradually learn and mine real sensor data information through interaction with the environment, and then optimize the virtual model of the concentrate sampling robot. At the information level, analyze the relationship between different sensor data, optimize the data processing process, and improve data processing efficiency and accuracy. Use the optimized data to drive the virtual prototype of the concentrate sampling robot, fully simulate its motion posture, operation strategy, etc., and conduct further calibration and optimization through interactive verification between the virtual prototype and the physical concentrate sampling robot. Based on the optimized data and feedback information, enhance the perception ability of the concentrate sampling robot to form a highly intelligent multimodal information perception system.

[0072] According to another aspect of the present invention, there is provided a customs port concentrate sampling robot monitoring system based on multimodal digital twins, comprising a physical layer, a model layer, a data layer, a service layer, and an application layer;

[0073] The physical layer collects real-time data through the sampling robot and sensors installed in the sampling environment, and based on a unified data communication protocol and conversion method, realizes multi-source heterogeneous real-time data access and integration at the physical layer, forming the interconnection and integration of man-machine-object-environment;

[0074] Multiple modalities and sensors include but are not limited to: visual modalities, environmental sensors (depth cameras, microphones), using depth cameras to capture environmental images in real time, monitor robot operations, and help identify obstacles and changes;

[0075] Vibration modal, vibration sensors, configured to monitor robot mechanical vibration and shock;

[0076] Pressure mode, pressure sensor, configured to detect the contact force of the sampling robot and the pressure change of the sampling head to ensure that no damage to the sample or equipment;

[0077] Temperature and humidity mode, temperature and humidity sensor, configured to monitor ambient temperature and humidity and the operating temperature of the equipment;

[0078] Sound modality, audio sensors, configured to capture mechanical operation sounds and analyze abnormal sounds to warn of equipment failures;

[0079] A gyro sensor configured to provide posture information of the robot;

[0080] A laser radar configured to locate the robot;

[0081] Composition analysis sensor, configured to detect the mineral composition and physical properties of collected samples online;

[0082] The significance of realizing these modes is that through diversified data collection and analysis, the sampling environment and equipment status can be fully understood, the accuracy and real-time performance of monitoring can be improved, risks can be reduced, and the efficiency and safety of the sampling process can be ensured.

[0083] The model layer integrates and fuses the multi-dimensional virtual models of the sampling robot and the sampling environment to establish a virtual model covering the entire life cycle of the sampling process for simulation; the model layer is a faithful mapping of the physical layer entities, and its core is modeling and simulation. For example, the digital twin model of the sampling robot equipment mainly includes descriptive elements and behavioral levels. The descriptive elements refer to the static feature modeling of the geometric structure and material properties of the equipment to establish a multi-scale three-dimensional model; the behavioral level refers to the dynamic model of the equipment's motion rules, such as mechanical power models, fatigue damage models, etc. The virtual model can provide low-cost working condition simulation and complex accident handling, is not restricted by time and space, and realizes multi-party information sharing. Integrating and fusing the multi-dimensional virtual models of the sampling robot and the sampling environment, and finally establishing a virtual model covering the entire life cycle of the sampling process, is an important carrier form of the digital twin of the sampling robot.

[0084] The data layer is used to perceive and process multimodal data; the data layer includes: an edge processing layer, a collection layer, and a cloud processing layer.

[0085] Edge processing layer: There are data preprocessing modules and local fault detection modules in the edge processing layer. The system processes and analyzes sensor data in real time, filters useless information, and extracts key features; it quickly detects abnormal conditions in local devices to reduce data transmission delays and cloud computing loads. This layer can effectively reduce data transmission bandwidth pressure, reduce delays through local computing, and provide preliminary anomaly detection functions;

[0086] Collection layer: The collection layer collects data processed by the edge and sends it to the remote server or cloud through a specific interface. The collection layer is responsible for data compression and format conversion to ensure that data can be interoperable between different systems;

[0087] Cloud processing layer: The cloud processing layer is the core computing layer of the entire system. It is responsible for in-depth analysis, modeling, simulation and prediction of the collected multimodal data. This layer can provide complex anomaly detection, decision support and system optimization functions;

[0088] The service layer is used for data-driven services, including pre-sampling, sampling and post-sampling services. Before sampling, multimodal digital twin technology can be used to perform pre-job simulation, human-computer interaction and other operations, realize low-cost job simulation in a virtual environment, further improve the design of sampling path plans, and improve sampling efficiency. The sampling service mainly performs real-time modeling and simulation on the various systems of the running sampling robot, realizes equipment fault diagnosis, sampling visualization, path optimization and other functions, and ensures the healthy operation of the equipment during the sampling process and safe and efficient operation. The post-sampling service mainly summarizes the experience of the completed operations, automatically forms electronic logs, operation reports, etc., integrates the sampling database in the form of multimodal field data and knowledge, realizes data closure, and provides a basis for the optimization of subsequent operation plans.

[0089] The application layer interacts with the service layer and the user, and the result data after the command execution is fed back to the virtual model for easy monitoring and operation. The application function covers the entire life cycle of the sampling process, including pre-operation drills, equipment monitoring and solution optimization, thereby improving the sampling efficiency and benefits. At the same time, the result data after the command execution will also be fed back to the virtual model for easy monitoring and operation.

[0090] Furthermore, the application layer includes a monitoring terminal, which supports 2D and 3D visualization interfaces and can display the operating status of the robot, the simulation results of the twin model, and the monitoring data of the sensor in real time.

[0091] Furthermore, in order to prevent obstacles in the environment (such as cargo stacking and equipment movement) from causing the robot's navigation and path planning to fail and affecting the execution of the sampling task, the monitoring system will use sensors such as lidar and cameras to monitor obstacles in real time, and dynamically update the digital twin model to adjust the robot's path in time to avoid new obstacles.

[0092] To prevent changes in environmental conditions such as temperature and humidity from affecting the accuracy of sensors and sampling quality, the monitoring system will regularly calibrate the temperature and humidity sensors and synchronize their data with the environmental conditions in the digital twin model. At the same time, it will use environmental sensors to monitor and record changes in environmental conditions in real time to adjust the sampling strategy.

[0093] In order to meet dynamic task requirements, that is, changes in task requirements (such as changes in sampling points and adjustments in sampling frequency), the concentrate sampling robot can adjust the sampling points and operation plans according to real-time needs, apply adaptive path planning algorithms, respond to changes in task requirements in real time, and ensure that sampling tasks are performed as needed.

[0094] To prevent changes in the status of equipment and robots, equipment wear or robot status (such as low battery) from affecting the stability of the system and sampling accuracy, the robot status can be monitored on the monitoring terminal, the status of the robot and equipment can be monitored in real time, abnormal situations can be detected and handled in a timely manner, and a maintenance early warning system can be set up to automatically remind maintenance or replace parts according to the equipment status.

[0095] In order to prevent the failure of the digital twin model and affect the decision-making and prediction capabilities of the system, the robot real-time monitoring system establishes a mechanism for regularly updating and correcting the digital twin model to ensure the real-time consistency between the model and the actual environment, and uses a feedback mechanism to obtain data from actual operations to automatically adjust and optimize the digital twin model.

[0096] The present invention can also be applied in other fields. For example, in manufacturing, it can optimize production processes and equipment maintenance; in urban management, it can improve the intelligence level of traffic and environmental monitoring; in the medical field, it can help with patient monitoring and treatment optimization. In general, multimodal digital twin technology has broad application potential in improving efficiency, reducing costs and enhancing decision-making capabilities.

[0097] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A customs port sampling robot monitoring method based on multimodal digital twins, characterized in that: The following steps are involved: S1. Conduct multi-modal real-time monitoring of sampling robots and concentrate environmental data, and integrate multi-source data; S2. Based on real-time data and historical data, a virtual model of the sampling robot equipment is constructed in the virtual space from four dimensions: geometry, physics, behavior, and rules, forming a virtual simulation of the entire life cycle of the robot sampling process; S3, based on the multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model, reflects the real-time status of the sampling operation and realizes the advance prediction of path optimization, providing data and model basis for global collaborative optimization in the sampling process; S4. Based on the visualization engine, the sampling process of the sampling robot is visualized to intuitively present the state of the robot during sampling; S5. Build a remote monitoring terminal; S6. The decision optimization and control feedback results generated by the virtual model act on the sampling robot entity.

2. According to claim 1, the method for monitoring the customs port sampling robot based on multimodal digital twins is characterized in that: The multimodal twin data generated by the synchronous operation of the sampling robot entity and the virtual model in S3 specifically includes: S31, construct a multimodal sensor information transmission network for sampling robots; S32. Construct a multimodal digital twin model of the sampling robot; S33, Multimodal sensor information mining and state perception of sampling robots.

3. The method for monitoring a customs port sampling robot based on multimodal digital twins according to claim 2 is characterized in that: S31. Constructing a multimodal sensor information transmission network for a sampling robot adopts the following method: A variety of sensors, depth cameras and microphones are installed on the sampling robot to collect information about the sampling robot, including environmental information, to form a perception of the full state of the sampling robot and provide multimodal perception data for the digital twin prototype.

4. The method for monitoring a customs port sampling robot based on multimodal digital twins according to claim 2 is characterized in that: S32. The following method is used to construct a multimodal digital twin model of the sampling robot: S321. Use ADAMS software to build a 1:1 three-dimensional virtual model based on the physical dimensions of the sampling robot, and perform preliminary motion simulation of the model; export the model in ADAMS to a file format supported by Unity 3D, and after importing, confirm the accuracy of the dimensions, proportions, and hierarchical structure; S322. Reset the assembly relationship between the various parts of the sampling robot in Unity 3D, and use Unity's parent-child hierarchy and joint components to simulate the mechanical connection in reality; configure Unity's physics engine, set the physical properties of the model such as mass, friction, elasticity, etc., write scripts to control the sampling robot, simulate the dynamic behavior of the sampling robot when working, and use Unity 3D's powerful rendering and animation tools to create an interactive sampling robot virtual prototype to ensure that the virtual prototype can simulate all possible motion postures of the physical entity; S323. Sort out the multi-source and multi-modal data of various sensors installed on the sampling robot, as well as depth cameras and microphones, and design appropriate data interfaces to allow sensor data to be input into the virtual model in Unity 3D in real time; develop an interactive user interface to display the status and sensor data of the sampling robot, preliminarily drive the virtual prototype, write control logic that conforms to the movement of physical entities, ensure that the virtual prototype can respond to external stimuli in real time, and preliminarily build a digital twin model of the multi-modal sampling robot.

5. The method for monitoring a customs port sampling robot based on multimodal digital twins according to claim 2 is characterized in that: S33, sampling robot multimodal sensor information mining, state perception adopts the following methods: S331. Use a signal conditioning system to optimize and denoise the sensor data from a physical level, and use a digital filter to filter the sensor data from a software level to remove undesirable components in the data; S332. Apply statistical analysis methods to identify and process outliers, calibrate sensors, ensure consistency of multi-source data in time series through timestamp alignment, and establish sensor response curves to adapt to different operating conditions; smooth time series data to reduce random fluctuations; use spatial filtering technology to filter image and depth camera data; S333. Use the Z-score standardization method to convert all sensor data into a unified dimension and numerical range. Use the reinforcement learning algorithm and set up a reward mechanism to gradually learn and mine real sensor data information through interaction with the environment, and then optimize the virtual model of the concentrate sampling robot. At the information level, analyze the relationship between different sensor data, optimize the data processing process, and improve data processing efficiency and accuracy. Use the optimized data to drive the virtual prototype of the concentrate sampling robot, fully simulate its motion posture and operation strategy, and conduct further calibration and optimization through interactive verification between the virtual prototype and the physical concentrate sampling robot. Based on the optimized data and feedback information, enhance the perception ability of the concentrate sampling robot to form a highly intelligent multimodal information perception system.

6. A customs port concentrate sampling robot monitoring system based on multimodal digital twins, characterized in that: Includes physical layer, model layer, data layer, service layer, and application layer; The physical layer collects real-time data through the sampling robot and sensors installed in the sampling environment, and based on a unified data communication protocol and conversion method, realizes multi-source heterogeneous real-time data access and integration at the physical layer, forming the interconnection and integration of man-machine-object-environment; The model layer integrates and fuses the multi-dimensional virtual models of the sampling robot and the sampling environment to establish a virtual model covering the entire life cycle of the sampling process for simulation; The data layer is used to sense and process multimodal data; The service layer is used for data-driven services, including pre-sampling, sampling and post-sampling services; The application layer interacts with the service layer and the user, and the result data after the command execution is fed back to the virtual model for easy monitoring and operation.

7. The customs port sampling robot monitoring system based on multimodal digital twin according to claim 6 is characterized by: The data layer includes an edge processing layer, a collection layer, and a cloud processing layer; The edge processing layer includes a data preprocessing module and a local fault detection module. The data preprocessing module processes and analyzes sensor data in real time, filters useless information, and extracts key features. The local fault detection module quickly detects abnormal conditions in local devices to reduce data transmission delays and cloud computing load; The collection layer collects the edge-processed data and sends it to the remote server or cloud through a specific interface; The cloud processing layer performs in-depth analysis, modeling, simulation, and prediction on the collected multimodal data.

8. The customs port sampling robot monitoring system based on multimodal digital twin according to claim 6 is characterized by: The pre-sampling service uses multimodal digital twin technology to perform pre-operation simulation and human-computer interaction operations; the during-sampling service performs real-time modeling and simulation of each system of the running sampling robot to realize equipment fault diagnosis, sampling visualization, and path optimization functions, ensuring the healthy operation of the equipment during the sampling process and safe and efficient operation; the post-sampling service summarizes the experience of completed operations, automatically generates electronic logs, operation reports, etc., integrates the sampling database in the form of multimodal field data and knowledge, realizes data closure, and provides a basis for the optimization of subsequent operation plans.

9. The customs port concentrate sampling robot monitoring system based on multimodal digital twin according to claim 6 is characterized in that: The application layer includes a monitoring terminal, which supports 2D and 3D visualization interfaces and can display the operating status of the robot, the simulation results of the twin model, and the monitoring data of the sensor in real time.

Images