Multi-sensor fusion and environment feature recognition method and system of digital twin equipment

Abstract

The invention provides a multi-sensor fusion and environment feature recognition method and system for digital twin equipment, and the method comprises the steps: constructing a digital twin model of the equipment in a visual platform, and setting a self-updating mechanism; deploying a multi-sensor array at a key installation part of the equipment physical entity, collecting multi-source sensing data and uploading the multi-source sensing data to a twin platform; carrying out noise reduction processing on the multi-source sensing data and completing data fusion to obtain fused data; constructing a diffusion model corresponding to a noise scheduling strategy, generating new sensing data and a corresponding label, and forming analysis data; a multi-channel deep learning model is constructed to extract feature parameters from analysis data, regression prediction is carried out, and an identification result is obtained; the system is integrated to a twin platform and is combined with a control decision function. According to the invention, real-time regression prediction of environment characteristic parameters based on sensing data is realized, deep fusion of environment perception and control decision of equipment is promoted, and perception capability and adaptive adjustment capability of a control decision system of the equipment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing and pattern recognition. Specifically, it relates to a multi-sensor fusion and environmental feature recognition method and system for digital twin equipment, mainly aiming at equipment working in complex environmental media, and promoting overall adaptive decision-making by improving its real-time perception ability. Background Art

[0002] With the development of the economic society and the improvement of the industrialization level, in order to replace manual labor and improve production efficiency, more and more major equipment is used in complex media with extreme environmental conditions, such as non-uniform geology, multiphase fluids, particulate media, etc. For equipment operating in such a complex environment, its mechanical response, motion characteristics, control accuracy and other performances will be significantly affected by the change of the medium. Therefore, the equipment must have real-time environmental perception and adaptation capabilities, continuously sense and identify environmental medium parameters during the working process, so as to adjust its own operating state in real time and ensure stable operation under changing working conditions.

[0003] Digital twin technology provides an effective solution for improving the performance robustness of equipment in complex environments. By constructing a virtual mapping of the equipment, digital twin can not only realize real-time monitoring of the equipment operating state, but also make optimal control decisions. This requires the digital twin body to have real-time environmental perception ability, that is, how to accurately extract the characteristics of the environment where the equipment is located.

[0004] In order to sense the environment, a sensor system of the digital twin body needs to be constructed, and a sensor array is used to collect data in real time. Sensing data in complex environments often has the characteristics of high non-linearity and strong noise, and is also prone to violent fluctuations due to the influence of the medium. Traditional linear filtering methods are difficult to effectively remove noise, and more robust non-linear filtering techniques need to be adopted to improve the reliability and time series consistency of sensing data. At the same time, the experimental cost of equipment in complex media is high, and the available sensing data is limited, which is difficult to meet the analysis requirements. Effective data enhancement is required. Existing data enhancement methods include simple interpolation, noise perturbation, etc., which are difficult to maintain the dynamic characteristics of time series data, and are also difficult to generate high-quality time series data and the corresponding labels of the sequence at the same time.

[0005] In addition, the deep feature extraction based on multi-source sensing data is the core link for the twin system to realize environmental feature recognition. Multi-source sensing data involves multi-dimensional signal features, and traditional feature engineering methods relying on manual design are difficult to comprehensively describe its key information. However, in the face of multi-source data collected in complex environments, how to design the structure of the deep learning model to ensure its stability and interpretability is still a challenge.

[0006] Meanwhile, the ultimate goal of improving the environmental perception ability is to deeply integrate with the control decision-making in the digital twin system to achieve intelligent adaptive adjustment of the equipment. Most current digital twin systems stay at the visualization stage of environmental information, lacking linkage with the control system, making it impossible for the perception results to effectively act on the operation optimization of the equipment. How to combine environmental perception, feature recognition, and equipment control to achieve decision-making optimization based on digital twins is a major bottleneck in the current technological development.

[0007] The patent document "A Digital Twin Modeling Method for Complex Equipment Based on Multi-Source Data Fusion" (CN116341396A) discloses improving data quality through multi-source data fusion technology, feature selection, and feature dimension reduction data. At the same time, the data is labeled through the K-means algorithm, which is more convenient in the subsequent use of the recurrent neural network. At the same time, the digital twin modeling model is obtained through model training using the recurrent neural network, reducing the modeling difficulty and improving the modeling accuracy. The patent document "A Method for Trustworthy Perception of the State of Equipment Components Based on Multi-Domain Mining of Twin Data" (CN118656758A) discloses accurately distinguishing multi-source sensing data and non-working state redundant data of the working state of equipment components through a working state recognition model, avoiding waste of big data computing resources; converting multi-source sensing data into time-frequency two-dimensional images, using long short-term graph neural networks, and extracting key sensing features in a spatio-temporal fusion manner to accurately identify abnormal sensing data. However, both do not involve the real-time extraction of complex environmental medium parameters, lack data enhancement capabilities and adaptive adjustment capabilities, and are not applicable to dynamic complex environments.

[0008] In view of the above situation, a real-time environmental feature recognition method based on multi-sensor fusion in a twin platform is proposed, focusing on improving the noise reduction and fusion effect of multi-sensor data, optimizing the multi-source data enhancement method based on the diffusion model, and constructing a feature extraction method based on a deep neural network. Summary of the Invention

[0009] Aiming at the defects in the prior art, the purpose of the present invention is to provide a multi-sensor fusion and environmental feature recognition method and system for digital twin equipment.

[0010] The multi-sensor fusion and environmental feature recognition method for digital twin equipment provided by the present invention includes:

[0011] Step S1, constructing a digital twin model of the equipment in the visualization platform to form a twin platform, and setting a self-update mechanism;

[0012] Step S2, deploying a multi-sensor array at key installation parts of the physical entity of the equipment, collecting multi-source sensing data according to the self-update mechanism, and uploading it to the twin platform;

[0013] Step S3: The twin platform performs noise reduction processing on multi-source sensing data and completes data fusion to obtain fused data;

[0014] Step S4: A diffusion model corresponding to the noise scheduling strategy is constructed for the fused data to generate new sensing data and corresponding labels, forming analysis data;

[0015] Step S5: A multi-channel deep learning model is constructed to extract feature parameters from the analysis data for regression prediction to obtain recognition results;

[0016] Step S6: The recognition results are integrated into the twin platform and combined with the control decision-making function to control and adjust the decision.

[0017] Preferably, in step S1, a three-dimensional geometric model and physical characteristic parameters of the equipment are imported into the visualization platform, a twin platform for constructing a digital twin model is adopted with a multi-level structure, and a data flow interface facing the sensor is set.

[0018] The multi-level includes a structure layer, a stress layer, and an environment layer.

[0019] The self-update mechanism is to obtain dynamically updated multi-source sensing data in real time through the acquisition and transmission path and visualize it on the visualization platform.

[0020] The acquisition and transmission path is the path between the physical entity and the digital twin model, including wired transmission and wireless transmission.

[0021] The multi-sensor array is an array formed by multi-channel and multi-category sensors, including displacement sensors, vibration sensors, and force sensors.

[0022] The key installation parts include drive joints and execution parts.

[0023] The multi-source sensing data includes vibration information, force information, and displacement information.

[0024] Preferably, the noise reduction processing is that the twin platform extracts and arranges multi-source sensing data, and after format conversion and preprocessing, it is sent into a non-linear filter. Sigma point sampling, transformation, and state prediction are performed for each multi-source sensing data to update the time series.

[0025] The preprocessing includes outlier removal, interpolation completion, and detrending processing.

[0026] The data fusion is to set a fixed characteristic frequency, perform interpolation and point filling on multi-source sensing data below the characteristic frequency, perform downsampling filtering on multi-source sensing data higher than or equal to the characteristic frequency, and arrange them according to the time series after alignment on the time axis.

[0027] Preferably, the noise scheduling strategy is an exponential β scheduling strategy for highly non-linear multi-source sensing data, and a cosine β scheduling strategy for multi-source sensing data based on physical characteristics. t For multi-source sensing data based on physical characteristics, a cosine β t scheduling strategy is adopted.

[0028] In step S4, based on the reverse denoising, forward diffusion, and learning prediction of the diffusion model, new time series sensing data and corresponding labels are generated.

[0029] The multi-channel deep learning model is a deep learning neural network model for extracting multi-channel time series data features, adopting a CNN-BiLSTM network architecture.

[0030] The extracted feature parameters are to standardize the analysis data. The multi-channel deep learning model extracts short-term information, local spatial features, and spatial distributions. Based on a sliding time window, the time series of the sensing data are segmented and fused into training samples, and then environmental feature labels are made. The data are input into the multi-channel deep learning model in sequence to construct a loss function.

[0031] The short-term information includes spectral information and sequence change trends.

[0032] The sliding time window is determined according to the sensor sampling rate and environmental change characteristics.

[0033] Preferably, in step S6, the multi-channel deep learning model is deployed to the twin platform, and the compressive strength and brittleness coefficient of the environment contacted by the equipment are displayed based on color mapping or dynamic graphs, and an interaction mechanism with the control decision-making system is established.

[0034] The combination with the control decision-making is to use the compressive strength and brittleness coefficient in the recognition result as the input quantity of the control decision-making system in the twin platform. At the same time, the twin platform visualizes the recognition result in real time.

[0035] According to the present invention, a multi-sensor fusion and environmental feature recognition system for digital twin equipment is provided, including:

[0036] Module M1: Construct a digital twin model of the equipment in the visualization platform to form a twin platform, and set a self-update mechanism;

[0037] Module M2: Deploy a multi-sensor array at the key installation parts of the equipment physical entity, and collect multi-source sensing data according to the self-update mechanism and upload it to the twin platform;

[0038] Module M3: The twin platform performs noise reduction processing on the multi-source sensing data and completes data fusion to obtain fusion data;

[0039] Module M4 constructs a diffusion model for the fused data to build a corresponding noise scheduling strategy, generates new sensing data and corresponding labels, and forms analysis data;

[0040] Module M5 constructs a multi-channel deep learning model to extract feature parameters from the analysis data, conducts regression prediction, and obtains recognition results;

[0041] Module M6 integrates the recognition results into the twin platform, combines with the control decision-making function, and controls and adjusts the decision.

[0042] Preferably, in Module M1, a three-dimensional geometric model and physical characteristic parameters of the equipment are imported into the visualization platform, a twin platform for the digital twin model is constructed using a multi-level structure, and a data flow interface facing the sensor is set.

[0043] The multi-level includes a structure layer, a stress layer, and an environment layer.

[0044] The self-update mechanism is to obtain dynamically updated multi-source sensing data in real time through the acquisition and transmission path and visualize it on the visualization platform.

[0045] The acquisition and transmission path is the path between the physical entity and the digital twin model, including wired transmission and wireless transmission.

[0046] The multi-sensor array is an array formed by multi-channel and multi-category sensors, including displacement sensors, vibration sensors, and force sensors.

[0047] The key installation parts include drive joints and trigger parts.

[0048] The multi-source sensing data includes vibration information, force information, and displacement information.

[0049] Preferably, the noise reduction processing is that the twin platform extracts and arranges the multi-source sensing data, after format conversion and preprocessing, it is sent into a non-linear filter, Sigma point sampling, transformation, and state prediction are performed for each multi-source sensing data, and the time series is updated.

[0050] The preprocessing includes outlier removal, interpolation completion, and detrending processing.

[0051] The data fusion is to set a fixed characteristic frequency, perform interpolation and point filling on the multi-source sensing data below the characteristic frequency, perform downsampling filtering on the multi-source sensing data higher than or equal to the characteristic frequency, and align on the time axis and arrange according to the time series.

[0052] Preferably, the noise scheduling strategy is an exponential β t scheduling strategy for highly non-linear multi-source sensing data, and a cosine β tScheduling strategy.

[0053] In the module M4, based on reverse denoising, forward diffusion, and learning prediction of the diffusion model, new time series sensing data and corresponding labels are generated.

[0054] The multi-channel deep learning model is a deep learning neural network model for extracting features of multi-channel time series data, adopting a CNN-BiLSTM network architecture.

[0055] The feature extraction parameters are to standardize the analysis data. The multi-channel deep learning model extracts short-term information, local spatial features, and spatial distributions. Based on a sliding time window, the time series of the sensing data is segmented and fused into training samples, and then environmental feature labels are made. They are input into the multi-channel deep learning model in sequence to construct a loss function.

[0056] The short-term information includes spectral information and sequence change trends.

[0057] The sliding time window is determined according to the sensor sampling rate and environmental change characteristics.

[0058] Preferably, in the module M6, the multi-channel deep learning model is deployed to the digital twin platform, and the compressive strength and brittleness coefficient of the environment contacted by the equipment are displayed based on color mapping or dynamic graphs, and an interaction mechanism with the control decision-making system is established.

[0059] The combination with the control decision-making is to use the compressive strength and brittleness coefficient in the recognition result as the input quantity of the control decision-making system in the digital twin platform. At the same time, the digital twin platform visualizes the recognition result in real time.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The present invention can effectively address problems such as insufficient environmental perception accuracy, serious influence of data noise, high data acquisition cost, and limited performance of deep learning models, promote the deep integration of equipment environmental perception and control decision-making, and improve the overall performance of equipment in complex media.

[0062] 2. The present invention deploys a multi-channel sensor array based on the digital twin, collects data in real time, and extracts deep features of the environmental medium in real time. Through the real-time perception ability of environmental features, the digital twin improves the perception ability and adaptive adjustment ability of its control decision-making system.

[0063] 3. The present invention designs methods for filtering fusion, data enhancement, and feature extraction for multi-source data, realizes real-time regression prediction of environmental feature parameters based on sensing data, and provides a basis for the control decision-making of the digital twin and the improvement of equipment performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objectives, and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0065] Figure 1 It is a schematic diagram of the overall framework for equipping multi-sensor fusion and environmental feature recognition.

[0066] Figure 2 It is a schematic block diagram of the construction of a multi-level self-updating digital twin platform.

[0067] Figure 3 It is a schematic block diagram of the sensor layout and data transmission of the equipment.

[0068] Figure 4 It is a schematic block diagram of the noise reduction and fusion of sensor data of the equipment.

[0069] Figure 5 It is a schematic block diagram of enhancing the diffusion model for sensor data of the equipment.

[0070] Figure 6 It is a schematic block diagram of extracting sensing data features using a deep neural network.

[0071] Figure 7 It is a schematic diagram of the structure of the deep neural network.

[0072] Figure 8 It is a schematic block diagram of integrating environmental perception and control decision-making for the digital twin of the equipment. Specific Embodiments

[0073] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all fall within the protection scope of the present invention.

[0074] According to the present invention, a method for multi-sensor fusion and environmental feature recognition of digital twin equipment is proposed, mainly for the digital twin of major equipment working in complex environmental media. Based on the fusion data of multiple sensors, through data enhancement and deep learning, the environmental perception ability of the equipment in complex media is improved. A multi-channel sensor array is deployed based on the digital twin to collect data in real time; methods for filtering fusion, data enhancement, and feature extraction are constructed for multi-source data, realizing real-time regression prediction of environmental feature parameters based on sensing data, and providing a basis for the control decision-making of the digital twin and the improvement of equipment performance. Specifically, taking Figure 1 as an example, it includes:

[0075] Step S1: Form a digital twin model. According to the internal and external structures and physical mechanisms of the equipment, construct a digital twin model of the equipment in the visualization platform, that is, the digital twin platform, and set up a self-update mechanism for virtual-real interaction.

[0076] Specifically, taking Figure 2 as an example, the construction of the digital twin platform of the equipment is based on the understanding of the physical characteristics and operating mechanisms of the equipment. Import its three-dimensional geometric model and physical characteristic parameters into the Unity3D platform, and construct a digital twin of the equipment based on visualization technologies such as model rendering, realizing the simulation of internal and external characteristics such as structural design, fluid transmission, and control circuits. Use visualization technologies to construct a digital twin covering the structure and principle of the equipment. Establish a self-update mechanism for virtual-real interaction in the digital twin, set up a data flow interface facing sensors, and establish a multi-level update mechanism to provide support at the system structure level for the dynamic real-time update of the digital twin.

[0077] The self-update mechanism for virtual-real interaction of the equipment is to build a sensor data acquisition and transmission path between its physical entity and the digital twin, and the digital twin can receive and display the direct observation data of the sensors of the object entity in real time.

[0078] The self-update mechanism is that the digital twin obtains the current sensor information in real time through the acquisition and transmission path, makes judgments and updates on the operating state of the equipment, collects the dynamically updated sensor information and observation and analysis information, and visualizes it on the Unity3D platform. For example, adjust the motion posture of the equipment to be consistent with the physical entity. The acquisition and transmission path is only one part of the self-update mechanism, and the purpose is to make the digital twin always consistent with the physical entity in dynamic working conditions.

[0079] In addition, when constructing the digital twin, adopt a multi-level structure, namely the structure layer, the force layer, and the environment layer, which respectively reflect the motion forms of the overall equipment mechanism, the force response of key parts of the equipment, and the characteristic parameters of the environment where the equipment is located. Multiple levels can carry out clear real-time updates based on sensor data and the analysis function of the digital twin.

[0080] Step S2: Deploy a multi-sensor array. Based on the actual working conditions and motion model of the equipment, deploy multi-channel and multi-category sensors at multiple key installation parts such as the mechanism joints of the physical entity of the equipment to form an array, collect data in real time and upload it to the twin platform.

[0081] Taking Figure 3 as an example, determine the compressive strength and brittleness coefficient as the key environmental characteristics to be identified, and deploy multiple sensors considering the complexity of their identification. Appropriately install linear variable differential transformers (LVDTs) as displacement sensors, accelerometers as vibration sensors, and strain gauges as force sensors in the equipment to form a sensor array.

[0082] Specifically, for the multi-sensor array of the equipment, it is first necessary to consider the working form and contact situation of the equipment in the environmental medium to determine the representative parts where sensors can be installed, such as the driving joints and executing parts of the equipment. Install various types and multi-channel sensors at these parts to collect the data required for environmental feature recognition, including acceleration for collecting vibration information, strain gauges for collecting force information, and linear displacement sensors for collecting displacement information, while ensuring that the collected data can be uploaded in real time.

[0083] Step S3: Denoise and fuse multi-source sensing data. In the digital twin platform, a non-linear filter is used to perform denoising processing on multi-source sensor data, and based on the resampling technology, the multi-source data, that is, multi-source sensing data, is aligned on the time axis to complete the data fusion operation.

[0084] Specifically, taking Figure 4 as an example, for the denoising of the sensor data of the equipment, the unscented Kalman filter (UKF) is selected as the non-linear filter to denoise the non-linear data of various sensors. This filter performs statistical linearization on the state of the sensor system based on the unscented transform and approximately propagates the state distribution of the system. The multi-source sensing data is sorted out in the digital twin platform and sent into the UKF filter. The filter performs Sigma point sampling, transformation, and state prediction for each piece of data, and finally updates the time series. Compared with the ordinary Kalman filter, this method can better handle the characteristics of high non-linearity and strong noise of the sensing data in a complex environmental medium and eliminate abnormal data.

[0085] For the sensor data fusion of the equipment, since the acquisition frequencies of different types of sensors are different, the multi-channel data uploaded to the platform is inconsistent in the time scale. Through the resampling technology of interpolation and downsampling filtering, the multi-source data with different frequencies is aligned on the time axis, and the aligned data is arranged in an orderly manner for subsequent analysis.

[0086] The resampling technology is used to adjust and match the time scales of multi-source and multi-channel sensor data, and a unique characteristic frequency is set. The data below this frequency is subjected to interpolation, and the data above this frequency is subjected to downsampling filtering. Ensure the alignment of different sensor data on the time axis and reduce the influence of time deviation on the analysis accuracy.

[0087] Step S4: Propose a diffusion model for data augmentation. For the different characteristic data collected by multi-source sensors, diffusion models with different noise scheduling strategies are constructed to generate multiple groups of new sensing data and corresponding labels.

[0088] After step S3 is completed, multi-source sensor data aligned in the time axis can be formed, that is, fused data. The data aligned in the time axis can be input into the diffusion model in a unified format for data augmentation to cope with the extreme environmental conditions with scarce experimental data. The feature data is equivalent to the multi-source sensor data. The multi-source sensor data has two aspects: one is that the types of sensors are different, and the other is that the same type of sensor is arranged at multiple points in the form of an array with different channels. Therefore, the features of the multi-source sensor data are also different, that is, multi-source data.

[0089] Specifically, taking Figure 5 as an example, a combined diffusion model with different noise scheduling strategies and applicable to multi-source non-linear time series is used for multi-sensor data augmentation of vibration and force sensors to form a rich data set to improve the richness of analysis samples. For the highly non-linear sensor data, an exponential β t scheduling strategy is adopted, and for the sensor data based on physical characteristics, a cosine β t scheduling strategy is adopted. Based on the reverse denoising and learning prediction of the diffusion model, high-quality time series and corresponding labels are generated.

[0090] For the sensor data augmentation of the equipment, in view of the differences in non-linearity degree, change trend, frequency spectrum distribution, etc. of different types of sensing data, a combined diffusion model is proposed to enhance the time series. In actual use, especially for the interaction process between the equipment and the complex extreme environment, the data of vibration sensors has a high degree of non-linearity and large random influence. An exponential noise scheduling is used to enhance its dynamic characteristics. The data of displacement and force sensors is based on a certain physical mechanism, and a more gentle cosine noise scheduling is used to maintain physical consistency. Through the reverse denoising and reconstruction of the model, this augmentation operation generates high-quality data and labels together.

[0091] Step S5: Construct a multi-channel deep learning model suitable for "end-to-end" feature extraction for feature extraction. Construct a multi-channel deep neural network to accurately extract feature parameters of the environment such as time-frequency information from the multi-sensor fusion data, that is, the enhanced multi-source data set, and perform regression recognition of the environmental feature parameters.

[0092] Specifically, taking Figure 6 , 7 as an example, for the environmental feature extraction of the equipment, a deep neural network is used to extract deep environmental features from multi-source sensing data, that is, a deep learning model for multi-channel time series data feature extraction, so as to perform real-time and accurate regression on the compressive strength and brittleness coefficient of the current environment.

[0093] The multi-channel time-series data feature extraction deep learning model adopts a CNN-BiLSTM network architecture. The data enhanced by the diffusion model is first segmented and fused into training samples based on a sliding time window, and then input into the model in sequence. The trained deep neural network can learn and perform regression prediction on environmental feature parameters.

[0094] In digital twins, deploy the neural network architecture of CNN-BiLSTM through ONNX technology, and determine the hyperparameters of the model according to the actual situation. CNN is responsible for capturing local time-frequency features and can better process multi-channel sensor array data and learn spatial characteristics. BiLSTM is used to analyze bidirectional long-term dependencies. The combination of the two can more effectively extract deep data features. Before data feature extraction, perform time series segmentation through a sliding time window and create environmental feature labels; after data feature extraction, construct a loss function for multi-source data and environmental recognition tasks to improve the prediction accuracy.

[0095] Step S6: Integrate the environmental feature recognition mechanism. Integrate the real-time perception of environmental features and the recognition results of environmental recognition into the digital twin body and combine them with the control decision-making function in the twin body, so that the twin system can adjust the operating state of the equipment in real time based on the environmental perception ability, and improve its intelligent level, overall working performance and robustness.

[0096] Specifically, taking Figure 8 as an example, perform the sensing and recognition of environmental feature parameters, integrate the environmental feature recognition results based on sensor observation data and the deep learning model, and display the compressive strength and brittleness coefficient of the environment contacted by the current equipment based on color mapping or dynamic graphs in the Unity3D platform to guide equipment decision-making. Establish an interaction mechanism with the equipment control decision-making function to form a closed-loop control driven by environmental features and realize the support for adaptive decision-making.

[0097] The combination of the environmental perception and control decision-making of the equipment is to use the model to perform real-time regression prediction of the compressive strength and brittleness coefficient of the environment where the equipment is located, which is used as the input quantity of the control decision-making system in the digital twin body to realize the linkage between perception and control. At the same time, the Unity3D platform visualizes the recognition results in real time, so as to realize adaptive adjustment.

[0098] In the embodiment, taking the cantilever roadheader cutting complex intercalated gangue coal seam that needs to face complex environmental medium changes when necessary as an example, it includes the steps:

[0099] Step S1: Construct the equipment digital twin platform. Import the internal and external structures and physical mechanisms of the equipment into the visualization platform Unity3D according to the actual working conditions, and establish a sensor data acquisition path and a digital twin multi-level self-update mechanism;

[0100] In more preferred examples, the application scenarios, environmental working conditions, its own motion mechanism, and physical characteristics of the equipment are clarified, the key analysis parts that need to be digitally twin modeled are determined, and a self-updating digital model for the virtual-real interaction of the equipment is built relying on the visualization environment in the Unity3D platform. First, import the 3D CAD model of the internal and external structure of the equipment into the platform, visualize the 3D model using the real-time rendering function of Mesh, and at the same time, using the function of modeling with the Rigidbody physics engine, model the physical mechanisms such as dynamics and mechanism of the equipment in the form of C# scripts. The Unity3D platform supports 3D interaction functions and provides functions such as multi-view switching, data curve display, and parameter adjustment. Users can view the changes in the equipment status in real time and intuitively analyze its operating conditions in complex environments; Next, build a complete digital twin sensor data acquisition path, build a complete sensor receiving port in the Unity3D platform, and initially design functions such as data extraction, format conversion, and visualization to support subsequent data analysis and processing.

[0101] Focus on establishing a multi-level self-updating mechanism for the digital twin platform. According to the function decomposition of the equipment under actual working conditions, the digital twin body is divided into a multi-level structure of an environment layer, a force layer, and a structure layer to clearly reflect the environmental working conditions and operating status of the equipment. The structure layer mainly describes the geometric structure and kinematic characteristics of the equipment to ensure that its dynamic behavior in the virtual environment is consistent with the actual equipment; the force layer mainly focuses on the force conditions of key parts, and subsequently integrates the force, strain, and vibration data collected by sensors to achieve visual monitoring of the force state of the equipment; the environment layer mainly focuses on the characteristics of the environmental medium contacted by the equipment, and subsequently integrates the data collected by sensors for feature extraction to establish the correlation between equipment behavior adjustment and environmental changes.

[0102] Step S2: Deployment of the equipment sensor array. Determine the positions of sensor deployment based on the working form of the equipment, and form a multi-source sensing array with multiple accelerometers, strain gauges, and LVDT sensors;

[0103] In more preferred examples, based on the working mode of the equipment in a specific environment, multiple types of sensors are used to form a multi-source and multi-channel sensor array. First, determine the types of sensor installations. To perform real-time identification of deep environmental characteristics such as compressive strength and brittleness coefficient, accelerometers are used to monitor the vibration signals of the equipment to reflect the dynamic characteristics during equipment operation, strain gauges are used to monitor the deformation of the equipment to reflect the magnitude of the force on key parts of the equipment, and linear variable differential transformers (LVDTs) are used to monitor the joint displacements of the equipment to reflect the attitude and position of the equipment; Next, analyze the operating conditions, force characteristics, and key parts of the equipment to determine the reasonable layout positions of the sensors. For the layout of accelerometers and strain gauges, positions with high stress and high deformation are preferably selected, such as bearing seats and operating arms. For the layout of LVDT sensors, necessary joints that drive the movement of the equipment are preferably selected. Multiple sensors are appropriately arranged at several positions to form a sensor array.

[0104] Regarding the stability and accuracy of sensor data, a combination of wireless and wired methods is used for data acquisition and transmission. LVDT sensors are deployed at high-speed moving parts such as complex joints and use wireless transmission to reduce wiring interference and wear. Accelerometers and strain gauges are deployed on stable structures or stressed components and use wired connections to ensure the stability and anti-interference ability of data transmission; Finally, the sensor array uses the Socket mechanism under the Unity3D platform to transmit data and uploads the real-time collected data to the digital twin. The digital twin extracts and converts the data format.

[0105] Step S3: Denoising and fusion of sensor data. In the digital twin, unscented Kalman filtering is used to process the non-linear data of the sensors, and resampling technology is used to ensure the alignment of the data on the time axis;

[0106] In more preferred examples, the data real-time collected by the sensors in the previous step is processed, and the noise interference in the data is reduced through filtering and resampling, and the multi-source data is aligned on the time axis, laying a foundation for further subsequent analysis. First, preprocess the multi-source sensing data collected, including outlier removal, interpolation completion, and detrending processing to improve the data quality and reduce measurement errors. Assume that the data collected by a certain sensor is X = {x1, x2, …… x n}, and the corresponding time series is T = {t1, t2, …… t n}, then there is: mean standard deviation

[0107] n represents the total number of time points of the sensor-collected data, i represents the serial number in the time series, x i corresponds to t i and ti The sensor data collected at a moment is x i .

[0108] Outlier rejection adopts the 3σ criterion based on the mean and standard deviation:

[0109] |x i - u| > 3

[0110] Lagrange interpolation is used for interpolation and completion, and an interpolation polynomial can be constructed:

[0111]

[0112] Among them, the Lagrange basis function L i (t) is:

[0113]

[0114] Detrending processing adopts moving average filtering. Let D t be the time series after detrending, and D t is calculated as follows:

[0115]

[0116] Next, the unscented Kalman filter (UKF) is used to denoise the non-linear sensing data. The UKF will generate a set of Sigma sampling points through the unscented transformation and propagate them in the non-linear system, thereby improving the accuracy of state estimation. The UKF approximates the state distribution through the sampling points and calculates the weighted mean through the non-linear transformation:

[0117]

[0118] Among them is the mean of the state estimation after non-linear transformation, χ i is the sampled Sigma point, is the corresponding mean weight, where m has no actual physical meaning, only indicating W i is the weight of "mean", and L is the dimension of the state vector. This formula is used to calculate the predicted mean of the state, avoiding the linearization error of the traditional Kalman filter and better adapting to data with a high degree of non-linearity.

[0119] Finally, to ensure the alignment of the time axes of different sensor data in the digital twin, a resampling technique is used to synchronize the time of multi-source signal data. Before processing, the sensor data with different sampling frequencies caused misalignment of the data on the time axis, affecting the accuracy of data fusion. Therefore, it is first necessary to determine a reference frequency, or characteristic frequency, and compensate the low-frequency sensing signals by interpolation (Lagrange interpolation), and resample the high-frequency sensing signals by downsampling filtering. Among them, Lagrange interpolation has been described above, and the complete process of downsampling filtering is as follows:

[0120]

[0121] Among them, z[m] is the signal after downsampling, D is the downsampling factor, y[n] is the filtered signal, x[n] is the original signal, h[k] is the low-pass filter coefficient, usually a Hann window, M is the filter order, m is the serial number of the time series after downsampling, and k is the serial number of the filter order. After processing, the signals of each channel are registered to the reference frequency, ensuring the synchronization of multi-source data in time.

[0122] Step S4: Data augmentation of time series and labels. A combined diffusion model is proposed to augment the multi-source sensor data, and high-quality time series and corresponding labels are generated simultaneously based on reverse denoising and prediction;

[0123] In more preferred examples, the equipment currently working in complex environmental media is often high-end and major equipment, and it is difficult to conduct experiments and data collection, with a long time cycle and high cost. Therefore, data augmentation is used to increase the number and richness of analysis samples. Considering the differences in non-linearity, change trend, and spectral distribution of different types of sensing data, different diffusion processes are designed. Specifically, the diffusion model includes two core processes: forward noise diffusion and reverse denoising reconstruction. The formula for forward diffusion is:

[0124]

[0125] where x t is the data at time step t, β t is the noise scheduling parameter, N(μ,σ2) is the Gaussian distribution, and I is the identity matrix, indicating that the noise is independent and identically distributed. The formula for reverse denoising is:

[0126]

[0127] where μ θ (x t ,t) is the mean predicted by the neural network, and σ θ is the learnable variance.

[0128] Due to the highly non - linear and strongly random nature of vibration sensor data, an exponential noise scheduling strategy is adopted for data augmentation to emphasize its dynamic characteristics and maintain the rationality of the data distribution:

[0129] β t = β min e t / T

[0130] In contrast, the displacement and force sensor data are physically more stable. To maintain their physical consistency, a cosine - type noise scheduling strategy is adopted to ensure that the samples after data augmentation still conform to the actual physical laws:

[0131] β t = β min + 0.5(β max - β min (1 + cos(πt / T))

[0132] In the forward diffusion stage, the original sensor data are subjected to multi - step random perturbations according to the set noise scheduling strategy to generate samples with different noise levels. In the reverse denoising reconstruction stage, using the trained diffusion probability model, the high - noise samples are gradually restored to clear synthetic data. This process is sampled based on the Markov Chain Monte Carlo (MCMC) method and combined with a conditional constraint strategy to ensure that the generated data is consistent with the original data distribution. In the above process, by constructing the joint distribution of "data - label", the label data corresponding to the augmented data, that is, the two environmental characteristics of compressive strength and brittleness coefficient, can also be predicted during the reverse generation of the time series. Finally, new "time series - label information" analysis data is generated, effectively improving the diversity and robustness of the sensor data.

[0133] Step S5: Based on the extraction of environmental characteristics by deep learning, design a CNN - BiLSTM deep neural network to extract features from the multi - source sensor time series passing through a sliding time window and perform regression prediction on the key environmental characteristic parameters;

[0134] In more preferred examples, after the data augmentation operation in the previous step, the digital twin already has rich analysis data. Next, a deep neural network based on CNN-BiLSTM is designed to extract environmental features. First, in the data processing stage, a sliding time window is used to segment the multi-source sensing data to ensure that the input data has a fixed time scale. The length of the time window is determined according to the sensor sampling rate and the characteristics of environmental changes. In addition, to improve the accuracy of environmental feature extraction, after the time series data is segmented, the data is normalized to reduce the impact of the dimensional differences between different physical quantities on model training. Next, the short-term information and local spatial features of the multi-channel time series are extracted. The data processed by the sliding time window is fed into the CNN convolutional neural network. On the one hand, CNN extracts the short-term information in the time series through one-dimensional convolution, including spectral information and sequence change trends. On the other hand, through the setting of its multi-channel convolutional kernels, CNN extracts the spatial distribution of multi-source data. At the same time, CNN combines batch normalization and the ReLU activation function to improve the training stability of the model.

[0135] Next, the long-term dependencies of the multi-channel time series are extracted. The local features extracted by CNN are input into the bidirectional long short-term memory (Bi-LSTM). BiLSTM consists of a forward LSTM and a backward LSTM, which can consider historical information and future trends simultaneously, thus greatly improving the ability to analyze environmental features. Compared with the traditional LSTM, the BiLSTM structure has better modeling ability when dealing with non-stationary time series and can more accurately identify the patterns of environmental characteristics changing over time. The data processed by Bi-LSTM will be fed into the regression prediction layer;

[0136] Finally, after the Bi-LSTM layer, for the regression tasks of two environmental characteristic parameters, compressive strength and brittleness coefficient, a loss function for multi-source feature extraction is constructed. This loss function combines the mean square error (MSE) of the two feature predictions and the overall structural loss (Structural Loss), which while ensuring the minimization of numerical errors, constrains the matching degree between the prediction results and the true physical laws. The loss function is expressed as follows:

[0137]

[0138] where, and are the compressive strength and brittleness coefficient predicted by the model, and are the true value labels, λ i is the weight hyperparameter, which is customized by the user according to the actual situation, and L struct is the structural loss.

[0139] In addition, this neural network uses Adam as the optimizer for training to achieve efficient gradient updates, adopts the Mini-Batch small-batch training strategy to improve computational efficiency, and uses cross-validation to evaluate the performance of the model to ensure the generalization ability of the model under different working conditions.

[0140] Step S6: Twin fusion of environmental perception and control decision-making. In the digital twin, the regression prediction result of environmental perception is used as the input of the control decision-making system to realize the real-time adjustment of the working state of the equipment in complex media.

[0141] In more preferred examples, the trained CNN-BiLSTM model in the previous step is deployed to the digital twin system and cross-platform calls are realized through ONNX (Open Neural Network Exchange) technology. First, based on the environmental feature recognition result of the deep neural network, in the digital twin, it will go through an adaptive correction mechanism, and the recognition result will be adjusted and compensated according to the historical data of environmental features recorded during the previous operation of the equipment to improve its stability and reliability; next, the recognized environmental feature parameters are input into the control decision-making system of the twin to optimize the operation strategy of the equipment. The control system directly uses the predicted compressive strength and brittleness coefficient as key decision variables to realize the dynamic adjustment of the operation state of the equipment; finally, the Unity3D platform will visualize the recognition result to intuitively display the real-time features of the environmental medium contacted by the equipment. The visualization interface also adopts a multi-level information display method, displaying the environmental features in the environmental layer and the adjusted operation state and decision-making situation of the equipment in the structure layer.

[0142] For the digital twin equipment working in complex media, through operations such as deploying a multi-source sensor array, noise reduction and enhancement of sensing data, and extraction of environmental features, the digital twin can directly collect data through sensors and extract the deep features of the environmental medium in real time. The digital twin improves the perception ability and adaptive adjustment ability of its control decision-making system through the real-time recognition of environmental features.

[0143] The present invention also provides a multi-sensor fusion and environmental feature recognition system for a digital twin equipment. The multi-sensor fusion and environmental feature recognition system for the digital twin equipment can be realized by executing the process steps of the multi-sensor fusion and environmental feature recognition method for the digital twin equipment. That is, those skilled in the art can understand the multi-sensor fusion and environmental feature recognition method for the digital twin equipment as a preferred implementation manner of the multi-sensor fusion and environmental feature recognition system for the digital twin equipment.

[0144] According to the present invention, there is provided a multi-sensor fusion and environmental feature recognition system for a digital twin equipment, including:

[0145] Module M1: Build a digital twin model of the equipment in the visualization platform to form a twin platform, and set up a self-update mechanism;

[0146] Module M2: Deploy a multi-sensor array at the key installation parts of the equipment physical entity, collect multi-source sensing data according to the self-update mechanism, and upload it to the twin platform;

[0147] Module M3: The twin platform performs noise reduction processing on the multi-source sensing data and completes data fusion to obtain fused data;

[0148] Module M4: Build a diffusion model corresponding to the noise scheduling strategy for the fused data, generate new sensing data and corresponding labels to form analysis data;

[0149] Module M5: Build a multi-channel deep learning model to extract feature parameters from the analysis data, perform regression prediction, and obtain recognition results;

[0150] Module M6: Integrate the recognition results into the twin platform, combine with the control decision-making function, and control and adjust the decision.

[0151] In more preferred examples, in Module M1, import the three-dimensional geometric model and physical characteristic parameters of the equipment into the visualization platform, build a twin platform for the digital twin model using a multi-level structure, and set up a data flow interface facing the sensor.

[0152] The multi-level includes a structure layer, a stress layer, and an environment layer.

[0153] The self-update mechanism is to obtain dynamically updated multi-source sensing data in real time through the acquisition and transmission path, and visualize it in the visualization platform.

[0154] The acquisition and transmission path is the path between the physical entity and the digital twin model, including wired transmission and wireless transmission.

[0155] The multi-sensor array is an array formed by multi-channel and multi-category sensors, including displacement sensors, vibration sensors, and force sensors.

[0156] The key installation parts include drive joints and trigger parts.

[0157] The multi-source sensing data includes vibration information, force information, and displacement information.

[0158] In more preferred examples, the noise reduction processing is that the twin platform extracts and arranges the multi-source sensing data, and after format conversion and preprocessing, it is sent to a non-linear filter. Sigma point sampling, transformation, and state prediction are performed for each multi-source sensing data to update the time series.

[0159] The preprocessing includes outlier removal, interpolation completion, and detrending processing.

[0160] The data fusion is to set a fixed characteristic frequency, interpolate and fill in the multi-source sensing data below the characteristic frequency, downsample and filter the multi-source sensing data above or equal to the characteristic frequency, and then arrange them according to the time series after alignment on the time axis.

[0161] In more preferred examples, the noise scheduling strategy is to adopt an exponential β t scheduling strategy for highly non-linear multi-source sensing data, and a cosine β t scheduling strategy for multi-source sensing data based on physical characteristics.

[0162] In the module M4, reverse denoising, forward diffusion, and learning prediction based on the diffusion model are performed to generate new time series sensing data and corresponding labels.

[0163] The multi-channel deep learning model is a deep learning neural network model for extracting the characteristics of multi-channel time series data, adopting a CNN-BiLSTM network architecture.

[0164] The extracted feature parameters are to standardize the analysis data. The multi-channel deep learning model extracts short-time information, local spatial features, and spatial distributions. Based on a sliding time window, the time series of the sensing data are segmented and fused into training samples, and then environmental feature labels are made. They are input into the multi-channel deep learning model in sequence to construct a loss function.

[0165] The short-time information includes spectral information and sequence change trends.

[0166] The sliding time window is determined according to the sensor sampling rate and environmental change characteristics.

[0167] In more preferred examples, in the module M6, the multi-channel deep learning model is deployed to the twin platform, and the compressive strength and brittleness coefficient of the environment contacted by the equipment are displayed based on color mapping or dynamic graphs, and an interaction mechanism with the control decision-making system is established.

[0168] The combination with the control decision is to use the compressive strength and brittleness coefficient in the recognition result as the input quantity of the control decision-making system in the twin platform, and at the same time, the twin platform visualizes the recognition result in real time.

[0169] Those skilled in the art know that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc., to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a kind of hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structures within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or structures within the hardware component.

[0170] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-sensor fusion and environmental feature recognition method for digital twin equipment, characterized in that Including: Step S1: Build a digital twin model of the equipment in the visualization platform to form a twin platform, and set up an auto-update mechanism. Step S2: Deploy a multi-sensor array at the key installation parts of the equipment physical entity, collect multi-source sensing data according to the auto-update mechanism, and upload it to the twin platform. Step S3: The twin platform performs noise reduction processing on the multi-source sensing data and completes data fusion to obtain fused data. Step S4: Build a diffusion model corresponding to the noise scheduling strategy for the fused data, generate new sensing data and corresponding labels to form analysis data. Step S5: Build a multi-channel deep learning model to extract feature parameters from the analysis data, perform regression prediction, and obtain recognition results. Step S6: Integrate the recognition results into the twin platform, combine with the control decision-making function, and control and adjust the decision.

2. The multi-sensor fusion and environmental feature recognition method for digital twin equipment according to claim 1, characterized in that, In step S1, import the 3D geometric model and physical characteristic parameters of the equipment into the visualization platform, build a twin platform of the digital twin model using a multi-level structure, and set up a data flow interface for sensors. The multi-level includes a structure layer, a stress layer, and an environment layer. The auto-update mechanism is to obtain multi-source sensing data with dynamic updates in real time through the acquisition and transmission path and visualize it in the visualization platform. The acquisition and transmission path is the path between the physical entity and the digital twin model, including wired transmission and wireless transmission. The multi-sensor array is an array formed by multi-channel and multi-category sensors, including displacement sensors, vibration sensors, and force sensors. The key installation parts include drive joints and execution parts. The multi-source sensing data includes vibration information, force information, and displacement information.

3. The multi-sensor fusion and environmental feature recognition method of the digital twin equipment according to claim 1, characterized in that, The noise reduction processing is that the twin platform extracts and arranges the multi-source sensing data, and after format conversion and preprocessing, it is sent into a non-linear filter. Sigma point sampling, transformation, and state prediction are performed for each multi-source sensing data to update the time series. The preprocessing includes outlier removal, interpolation completion, and detrending processing. The data fusion is to set a fixed characteristic frequency, perform interpolation and point completion on the multi-source sensing data below the characteristic frequency, perform downsampling filtering on the multi-source sensing data higher than or equal to the characteristic frequency, and align them on the time axis and arrange them according to the time series.

4. The multi-sensor fusion and environmental feature recognition method for digital twin equipment according to claim 1, characterized in that The noise scheduling strategy is an exponential β t scheduling strategy for highly nonlinear multi-source sensing data, and a cosine β t scheduling strategy for multi-source sensing data based on physical characteristics; In step S4, based on the reverse denoising, forward diffusion, and learning prediction of the diffusion model, generate new time series sensing data and corresponding labels. The multi-channel deep learning model is a multi-channel deep learning neural network model for extracting time series data features, using a CNN-BiLSTM network architecture. The extraction of feature parameters is to perform standardization processing on the analysis data. The multi-channel deep learning model extracts short-time information, local spatial features, and spatial distributions. Based on a sliding time window, the time series of the sensing data is segmented and fused into training samples, and then environmental feature labels are made. The training samples are input into the multi-channel deep learning model in sequence to construct a loss function. The short-time information includes spectral information and sequence change trends. The sliding time window is determined according to the sensor sampling rate and environmental change characteristics.

5. The multi-sensor fusion and environmental feature recognition method for digital twin equipment according to claim 1, characterized in that In step S6, the multi-channel deep learning model is deployed to the twin platform, and the compressive strength and brittleness coefficient of the environment contacted by the equipment are displayed based on color mapping or dynamic graphs, and an interaction mechanism with the control decision-making system is established; The combination with the control decision-making means that the compressive strength and brittleness coefficient in the recognition result are used as input quantities of the control decision-making system in the twin platform, and at the same time, the twin platform visualizes the recognition result in real time.

6. A multi-sensor fusion and environmental feature recognition system for digital twin equipment, characterized in that, It includes: Module M1: Build a digital twin model of the equipment in the visualization platform to form a twin platform, and set up a self-update mechanism; Module M2: Deploy a multi-sensor array at the key installation parts of the equipment physical entity, and collect multi-source sensing data according to the self-update mechanism and upload it to the twin platform; Module M3: The twin platform performs noise reduction processing on the multi-source sensing data and completes data fusion to obtain fused data; Module M4: Build a diffusion model corresponding to the noise scheduling strategy for the fused data, generate new sensing data and corresponding labels to form analysis data; Module M5: Build a multi-channel deep learning model to extract feature parameters from the analysis data, perform regression prediction, and obtain recognition results; Module M6: Integrate the recognition results into the twin platform, combine with the control decision-making function, and control and adjust the decision-making.

7. The multi-sensor fusion and environmental feature recognition system for digital twin equipment according to claim 6, wherein In module M1, import the three-dimensional geometric model and physical characteristic parameters of the equipment into the visualization platform, build a twin platform for the digital twin model using a multi-level structure, and set up a data flow interface facing the sensor; The multi-level includes a structure layer, a stress layer, and an environment layer; The self-update mechanism is to obtain dynamically updated multi-source sensing data in real time through the acquisition and transmission path, and visualize it in the visualization platform; The acquisition and transmission path is the path between the physical entity and the digital twin model, including wired transmission and wireless transmission; The multi-sensor array is an array formed by multi-channel and multi-category sensors, including displacement sensors, vibration sensors, and force sensors; The key installation parts include drive joints and trigger parts; The multi-source sensing data includes vibration information, force information, and displacement information.

8. The multi-sensor fusion and environmental feature recognition system of the digital twin equipment according to claim 6, characterized in that, The noise reduction processing is that the twin platform extracts and arranges the multi-source sensing data, and after format conversion and preprocessing, it is sent to a non-linear filter. Sigma point sampling, transformation, and state prediction are performed for each multi-source sensing data to update the time series; The preprocessing includes outlier removal, interpolation completion, and detrending processing; The data fusion is to set a fixed characteristic frequency, perform interpolation and filling for multi-source sensing data below the characteristic frequency, perform downsampling filtering for multi-source sensing data higher than or equal to the characteristic frequency, and arrange them according to the time series after aligning on the time axis.

9. The multi-sensor fusion and environmental feature recognition system of the digital twin equipment according to claim 6, characterized in that, The noise scheduling strategy is to use an exponential β for highly nonlinear multi-source sensor data. t The scheduling strategy adopts cosine type β for multi-source sensor data based on physical characteristics. t Scheduling strategy; In module M4, based on the reverse denoising, forward diffusion, and learning prediction of the diffusion model, generate new time series sensing data and corresponding labels; The multi-channel deep learning model is a multi-channel time series data feature extraction deep learning neural network model, using a CNN-BiLSTM network architecture; The extracted feature parameters are to perform standardization processing on the analysis data, and a multi-channel deep learning model extracts short-time information, local spatial features, and spatial distribution. Based on a sliding time window, the time series of the sensing data is segmented and fused into training samples, and then an environmental feature label is created. The training samples are input into the multi-channel deep learning model in sequence to construct a loss function; The short-time information includes spectral information and sequence change trends; The sliding time window is determined according to the sensor sampling rate and environmental change characteristics.

10. The multi-sensor fusion and environmental feature recognition system of the digital twin equipment according to claim 6, characterized in that, In the module M6, the multi-channel deep learning model is deployed to the twin platform, and based on color mapping or dynamic graphs, the compressive strength and brittleness coefficient of the environment contacted by the equipment are displayed, and an interaction mechanism with the control decision-making system is established; The combination with the control decision-making is to use the compressive strength and brittleness coefficient in the recognition results as input quantities for the control decision-making system in the twin platform. At the same time, the twin platform visualizes the recognition results in real time.

Images