Digital twinning method integrating multi-modal data

Abstract

The invention relates to the technical field of data integration, in particular to a digital twinning method for integrating multi-modal data, which comprises the steps of entity dynamic deformation field monitoring: capturing three-dimensional deformation data of a physical entity in real time, and generating a dynamic deformation field; constructing a non-rigid reference system; calculating a motion vector of each point on the surface of the entity, and generating a non-rigid reference system grid which is updated in real time along with deformation; binding a twinborn body reference system: dynamically binding a virtual coordinate system of the digital twinborn body with a non-rigid reference system grid; multi-modal data remapping: carrying out spatial transformation on the multi-modal sensing data acquired by the physical entity, and outputting a remapping data stream; and entity deformation suppression closed loop: detecting an abnormal deformation area based on the remapping data stream, generating an actuator control instruction to suppress entity deformation, and collecting suppressed data. According to the method, the abnormity identification precision and the response speed are improved, the predictive protection capability is achieved, and the stability and the safety margin of system operation are improved.

Description

Technical Field

[0001] The present invention relates to the field of data integration technology, and in particular to a digital twin method for integrating multimodal data. Background Art

[0002] During the operation of complex structural equipment (such as wind turbine blades, aviation airfoils, and large bridge structures), entities are often accompanied by nonlinear, multi-scale, and multi-source coupled deformation behaviors, and their spatiotemporal evolution states are highly dynamic and locally sudden. Traditional digital twin modeling methods are mostly based on static coordinate mapping or rigid approximation, which makes it difficult to accurately track the true state of the entity during non-rigid deformation. In addition, multimodal sensor data (such as fiber optic strain, thermal imaging, vibration, stress, etc.) are often distributed in different acquisition channels and spatiotemporal coordinate systems, resulting in inaccurate data fusion and insufficient collaborative interpretation, thereby restricting the real-time accuracy and controllability of the twin.

[0003] At the same time, when early signs of abnormal physical operation appear, such as sudden increases in local strain or sudden changes in curvature, traditional methods usually rely on post-processing analysis and are unable to achieve real-time identification and proactive intervention, resulting in delayed risk response. Existing control strategies are mostly open-loop operations and lack the high-precision suppression capabilities based on twin feedback closed loops. This makes it difficult to ensure system stability, especially when dealing with rapid dynamic disturbances and local deformations. Therefore, it is urgent to build a digital twin method that can integrate non-rigid reference frame binding, cross-modal data remapping, and actuator closed-loop control to achieve an intelligent closed-loop process from "precise modeling" to "active suppression." Summary of the Invention

[0004] The present invention provides a digital twin method for integrating multimodal data.

[0005] The digital twin method integrating multimodal data includes the following steps: S1, entity dynamic deformation field monitoring: through distributed fiber optic sensors and laser holographic scanning, the three-dimensional deformation data of the physical entity is captured in real time to generate a dynamic deformation field; S2, non-rigid reference system construction: based on the dynamic deformation field, the motion vector of each point on the solid surface is solved to generate a non-rigid reference system mesh that is updated in real time with the deformation; S3, twin reference frame binding: dynamically bind the virtual coordinate system of the digital twin to the non-rigid reference frame grid, and output the twin of the bound reference frame; S4, multimodal data remapping: the multimodal sensor data collected by the physical entity are spatially transformed according to the reference grid coordinates at the binding moment, and the remapped data stream is output; S5, entity deformation suppression closed loop: based on the remapped data stream, abnormal deformation areas are detected, actuator control instructions are generated to suppress entity deformation, and suppressed data is collected and returned to S1.

[0006] Optionally, the S1 includes: S11, optical fiber strain data calculation: wavelength offset is collected by distributed optical fiber sensors, and microstrain distribution is calculated based on the fiber Bragg grating strain sensing model; S12, holographic deformation field reconstruction: Obtain the phase difference of interference fringes through laser holographic scanning, and reconstruct the normal deformation based on the holographic interferometry model; S13, spatiotemporal data fusion: The optical fiber microstrain data and the holographic normal deformation are input into the spatiotemporal fusion model to generate a three-dimensional dynamic deformation field tensor.

[0007] Optionally, the S13 includes: S131, Grid Discretization: Define the discretized spatial grid coordinates; S132, using the three-point central difference scheme to approximate the calculation of the second-order partial derivative in the x direction; S133, using central difference discretization to calculate the second-order partial derivative in the y direction; S134, adding the differential formulas in the x and y directions to obtain a total curvature operator; S135, Boundary condition processing: Use the second-order forward / backward difference formula to process the boundary points.

[0008] Optionally, the S2 includes: S21, Motion vector field construction: Calculate the three-dimensional motion vector at the grid point based on the normal displacement component and strain component in the dynamic deformation field; S22, thermal deformation compensation: Considering the thermal expansion effect of the material, the motion vector is corrected by the thermal field; S23, reference frame grid generation: The discrete motion vector field is mapped into a continuous non-rigid reference frame grid using thin plate spline interpolation method.

[0009] Optionally, the S21 includes: S211, displacement gradient and curvature extraction: Based on the normal deformation obtained from the dynamic deformation field, the displacement gradient vector of the grid point is calculated using the central difference method; S212, motion vector integrated expression: the displacement gradient, curvature and microstrain value are combined to form a three-dimensional motion vector at the grid point.

[0010] Optionally, the S3 includes: S31, control point coordinate mapping: constructing a one-to-one mapping relationship between the virtual control point set and the physical control point set; S32, affine transformation solution: solve the optimal rigid transformation by minimizing the squared difference between the imaginary and real coordinates; S33, nonlinear residual compensation: calculate the affine transformation residual and construct a thin plate spline interpolation function based on the residual points; S34, binding function synthesis: constructing dynamic binding transformation function based on the constructed thin plate spline interpolation function; S35, full model binding: applying the transformation function to all twin model vertices.

[0011] Optionally, the S33 includes: S331, affine residual calculation: After the affine transformation is completed, for each pair of control points, the corresponding transformation residual vector is calculated; S332, residual interpolation function construction: Based on the spatial distribution of all residual points, the distributed sensor index positions are used as interpolation control points to construct a thin plate spline interpolation function.

[0012] Optionally, the S4 includes: S41, sensor parameter coordinate extraction: extracting the parameter coordinates of all sensors on the physical entity to form a parameter coordinate set; S42, real-time spatial position calculation: based on the current non-rigid reference frame grid function, calculate the real-time three-dimensional spatial position of the sensor; S43, binding position calculation: calling the grid snapshot function recorded at the binding moment to calculate the binding reference position of each sensor; S44, spatial displacement vector generation: calculating the spatial displacement of each sensor; S45, data remapping execution: spatially shifting the sensor raw data to eliminate the effect of entity deformation; S46, data stream reconstruction: aggregate all remapped data and output aligned remapped data stream.

[0013] Optionally, the S5 includes: S51, extracting a deformation-related feature set from the remapped data stream; S52, abnormal area identification: identifying abnormal areas according to a multi-level threshold strategy; S53, calculation of the inhibitory force field: calculating the required reverse inhibitory force according to the characteristic value of the abnormal area; S54, actuator instruction generation: generating a control instruction set according to the calculated restraining force; S55, closed-loop execution and feedback: After the control instruction is sent to the entity, a closed-loop feedback process is executed.

[0014] Optionally, the S54 includes: S541, hydraulic compensation command calculation: Generate a driving area control command for the hydraulic system based on the suppression force field and the normal vector of the abnormal area; S542, piezoelectric micro-vibration command calculation: using a piezoelectric actuator to generate a vibration control signal that matches the structural resonance frequency; S543, active cooling control instruction calculation: Based on the integral of the deformation variable and the safety threshold, a corresponding thermal control cooling power instruction is generated to reduce the source of thermally induced strain.

[0015] Beneficial effects of the present invention: The present invention significantly improves the adaptability of digital twins to the entity deformation process by introducing a non-rigid reference frame binding mechanism. Compared with traditional rigid or linear mapping methods, the present invention constructs a non-rigid grid that is updated in real time with the deformation based on the dynamic deformation field, and binds the virtual twin coordinate system to the grid through affine transformation and residual compensation, thereby achieving high-precision synchronous mapping; this mechanism effectively solves the problem of twin model deviation under working conditions such as large deformation and local distortion of the entity, so that the twin is always in a state of precise correspondence with the real physical scene.

[0016] The present invention introduces a closed-loop deformation suppression strategy driven by multiple source actuators into the digital twin framework. By extracting deformation features such as strain, curvature, and strain rate from the remapped data stream, a multi-dimensional anomaly detection indicator system is constructed, and a spatially distributed suppression force field is generated for abnormal areas, which is further converted into multi-mode execution instructions such as hydraulic compensation, piezoelectric vibration, and active cooling to form a highly responsive closed-loop control path. This mechanism not only improves the accuracy and response speed of anomaly identification, but also has predictive protection capabilities, significantly improving the stability and safety margin of system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of a method according to an embodiment of the present invention; Figure 2 This is a deformation suppression diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0020] It should be noted that references in the specification to "one embodiment," "an embodiment," "exemplary embodiments," "some embodiments," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not necessarily every embodiment will include such specific features, structures, or characteristics. Furthermore, when specific features, structures, or characteristics are described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement such features, structures, or characteristics in conjunction with other embodiments (whether or not explicitly described).

[0021] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0022] like Figure 1-Figure 2 As shown in the figure, the digital twin method integrating multimodal data includes the following steps: S1, entity dynamic deformation field monitoring: through distributed fiber optic sensors and laser holographic scanning, the three-dimensional deformation data of the physical entity is captured in real time to generate a dynamic deformation field; S2, non-rigid reference system construction: Based on the dynamic deformation field, the motion vector of each point on the solid surface is solved to generate a non-rigid reference system mesh that is updated in real time with the deformation; S3, twin reference frame binding: dynamically bind the virtual coordinate system of the digital twin to the non-rigid reference frame grid and output the twin of the bound reference frame; S4, multimodal data remapping: the multimodal sensor data collected by the physical entity are spatially transformed according to the reference grid coordinates at the binding moment, and the remapped data stream is output; S5, entity deformation suppression closed loop: detect abnormal deformation areas based on the remapped data stream, generate actuator control instructions to suppress entity deformation, and collect suppressed data and return it to S1.

[0023] S1 specifically includes: S11, Fiber Strain Data Determination: Wavelength Offset Acquisition via Distributed Fiber Sensors , the microstrain distribution is calculated based on the fiber Bragg grating strain sensing model and is expressed as: ; in, is the microstrain value, in units of , is the fiber Bragg grating reflection wavelength offset, in nm, is the initial center wavelength of the fiber Bragg grating, in nm, is the strain sensitivity coefficient, which is 0.78; S12, Holographic deformation field reconstruction: Obtaining interference fringe phase difference through laser holographic scanning , based on the holographic interferometry model, the normal deformation is reconstructed and expressed as: ; in, is the normal deformation, in units of , is the phase difference of the interference fringes, in rad, is the laser wavelength in nm, is the laser incident angle, in units of ; when (i.e. vertical incidence), the formula is expressed as: ; S13, spatiotemporal data fusion: integrating optical fiber microstrain data and holographic normal deformation Input the spatiotemporal fusion model to generate a three-dimensional dynamic deformation field tensor, which is expressed as: ; in, is the dynamic deformation field tensor, is the entity's three-dimensional space coordinate, is the timestamp, Is the curvature operator of the normal deformation variable, which represents the local surface shape change rate. The curvature operator is defined as: ; In the discretization process, the central difference method can be used to approximately solve the above second-order partial derivatives, including: (1) Grid discretization: Define the discretized spatial grid coordinates, expressed as: ; ; in, are the grid spacing in the x and y directions, in millimeters (mm), is the grid point The normal deformation value at , in microns ; (2) Discrete second-order partial derivative in the x-direction: The three-point central difference format is used for approximate calculation, which can be expressed as: ; (3) Directional second-order partial derivative discretization: central difference discretization is used, expressed as: ; (4) Curvature operator synthesis expression: The difference formula of the direction is added to obtain the total curvature operator, which is expressed as: ; when hour, ; (5) Boundary condition processing: For boundary points, the second-order forward / backward difference formula is used, including: 1) Left border ( ): ; 2) Right border ( ): ; 3) Upper / lower boundaries or .

[0024] S2 specifically includes: S21, Motion Vector Field Construction: Based on Dynamic Deformation Field The normal displacement component in and strain components , calculate the grid points The three-dimensional motion vector at is expressed as: ; ; ; in, Grid Points The motion vector at The displacement gradient is calculated using central difference discretization. is the microstrain value, in units of (parts per million strain), is the curvature value in units of ; S22, thermal deformation compensation: Considering the thermal expansion effect of the material, the motion vector is corrected by thermal field to obtain the correction vector, which is expressed as: ; in, is the thermal expansion coefficient of the material (unit: ), is the temperature at the grid point obtained by infrared thermal imaging (unit: ), is the reference temperature (unit: ), is the temperature-corrected motion vector; S23, Reference frame grid generation: The discrete motion vector field is mapped into a continuous non-rigid reference frame grid using thin plate spline interpolation method. The position is expressed as: ; ; in, Indicates parameter point Corresponding three-dimensional space coordinates, control points The three-dimensional position of , the interpolation weight is in vector form , is the thin plate spline basis function, , is the control point position, For the spline interpolation weights, by solving the linear system get, are the elements of the thin plate spline basis function, , indicating the The first and The radial basis function response between the control points, , is the affine term, .

[0025] S3 specifically includes: S31, Control point coordinate mapping: Constructing a virtual control point set With physical control point set One-to-one mapping relationship between them, are the predefined anchor coordinates in the digital twin, is the spatial coordinate of the corresponding point of the physical entity, indexing distributed sensor grids; S32, affine transformation solution: solve the optimal rigid transformation by minimizing the square difference between the imaginary and real coordinates, expressed as: ; ; in, is the linear transformation matrix, is the translation vector; S33, nonlinear residual compensation: Calculate the affine transformation residual and construct a thin plate spline interpolation function based on the residual points, expressed as: ; ; in, is the thin plate spline basis function, and its domain is , Control weights for interpolation; S34, binding function synthesis: construct a dynamic binding transformation function, expressed as: ; in, is a dynamic binding transformation function, It means that the three-dimensional point Projected onto the parameter plane of the non-rigid reference mesh, is the value of the residual field at that point; S35, full model binding: for all twin model vertices Apply the transformation function, expressed as: ; in, After the dynamic reference system is bound The vertex positions are calculated to generate a digital twin model that is dynamically consistent with the non-rigid reference frame of the physical entity.

[0026] S4 specifically includes: S41, sensor parameter coordinate extraction: extract the parameter coordinates of all sensors on the physical entity to form a parameter coordinate set, which is expressed as: ; in, It is The fixed coordinates of the sensors on the entity parameter plane, is the total number of sensors; S42, real-time spatial position calculation: based on the current non-rigid reference frame grid function , calculate the real-time three-dimensional spatial position of the sensor , expressed as: ; S43, Binding position calculation: Call the grid snapshot function recorded at the binding moment , calculate the binding reference position of each sensor , expressed as: ; S44, Spatial displacement vector generation: Calculate the spatial displacement of each sensor , expressed as: ; in, ; S45, data remapping execution: sensor raw data Spatial translation is used to eliminate the influence of entity deformation, which is expressed as: ; in, is the sensor data after remapping; S46, data stream reconstruction: aggregate all remapped data and output aligned remapped data stream , expressed as: .

[0027] S5 specifically includes: S51, extract the deformation-related feature set from the remapped data stream, expressed as: ; in, is the remapped microstrain (unit: ), is the remap curvature (unit: ), is the strain change rate (unit: ), is the Laplace gradient of curvature (unit: ); S52, abnormal area identification: Identify abnormal areas based on multi-level threshold strategy , expressed as: ; in, is the material yield strain threshold, and its value range is , is the safe curvature threshold, ranging from , is the strain change rate threshold, and its value range is , is the threshold of curvature mutation space, and its value range is ; S53, Inhibition force field calculation: Calculate the required reverse inhibition force based on the characteristic value of the abnormal area , expressed as: ; in, is the proportional gain coefficient (unit: ), the value range is 0.1~1.0, is the differential gain coefficient (unit: ), the value range is 0.001~0.05, is the curvature compensation gain (unit: ), the value range is 0.5~5.0; S54, actuator command generation: Generate a control command set based on the calculated restraining force , expressed as: ; Among them, the definitions of each sub-instruction of the control instruction specifically include: (1) Hydraulic compensation command: ; (2) Piezoelectric micro-vibration command: ; (3) Active cooling instructions: ; in, is the surface normal vector of the abnormal area, is the system resonant frequency (unit: Hz), is the conversion coefficient of various actuators, , , ; S55, closed-loop execution and feedback: After the control instruction is sent to the entity, the closed-loop feedback process is executed, specifically including: (1) Execution control instructions ; (2) Waiting delay , the conditions are expressed as: ;in, is the hydraulic response time, is the piezoelectric response time, is the cooling response time; (3) Collect deformation data again and feed it back to step S1.

[0028] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0029] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A digital twin method integrating multimodal data, characterized by: The following steps are involved: S1, entity dynamic deformation field monitoring: through distributed fiber optic sensors and laser holographic scanning, the three-dimensional deformation data of the physical entity is captured in real time to generate a dynamic deformation field; S2, non-rigid reference system construction: based on the dynamic deformation field, the motion vector of each point on the solid surface is solved to generate a non-rigid reference system mesh that is updated in real time with the deformation; S3, twin reference frame binding: dynamically bind the virtual coordinate system of the digital twin to the non-rigid reference frame grid, and output the twin of the bound reference frame; S4, multimodal data remapping: the multimodal sensor data collected by the physical entity are spatially transformed according to the reference grid coordinates at the binding moment, and the remapped data stream is output; S5, entity deformation suppression closed loop: based on the remapped data stream, abnormal deformation areas are detected, actuator control instructions are generated to suppress entity deformation, and suppressed data is collected and returned to S1.

2. The digital twin method for integrating multimodal data according to claim 1, characterized in that: Said S1 comprises: S11, optical fiber strain data calculation: wavelength offset is collected by distributed optical fiber sensors, and microstrain distribution is calculated based on the fiber Bragg grating strain sensing model; S12, holographic deformation field reconstruction: Obtain the phase difference of interference fringes through laser holographic scanning, and reconstruct the normal deformation based on the holographic interferometry model; S13, spatiotemporal data fusion: The optical fiber microstrain data and the holographic normal deformation are input into the spatiotemporal fusion model to generate a three-dimensional dynamic deformation field tensor.

3. The digital twin method for integrating multimodal data according to claim 2, characterized in that: The S13 includes: S131, Grid Discretization: Define the discretized spatial grid coordinates; S132, using the three-point central difference scheme to approximate the calculation of the second-order partial derivative in the x direction; S133, using central difference discretization to calculate the second-order partial derivative in the y direction; S134, adding the differential formulas in the x and y directions to obtain a total curvature operator; S135, Boundary condition processing: Use the second-order forward / backward difference formula to process the boundary points.

4. The digital twin method for integrating multimodal data according to claim 3, characterized in that: The S2 includes: S21, Motion vector field construction: Calculate the three-dimensional motion vector at the grid point based on the normal displacement component and strain component in the dynamic deformation field; S22, thermal deformation compensation: Considering the thermal expansion effect of the material, the motion vector is corrected by the thermal field; S23, reference frame grid generation: The discrete motion vector field is mapped into a continuous non-rigid reference frame grid using thin plate spline interpolation method.

5. The digital twin method of integrating multimodal data according to claim 4, characterized in that: The S21 includes: S211, displacement gradient and curvature extraction: Based on the normal deformation obtained from the dynamic deformation field, the displacement gradient vector of the grid point is calculated using the central difference method; S212, motion vector integrated expression: the displacement gradient, curvature and microstrain value are combined to form a three-dimensional motion vector at the grid point.

6. The digital twin method of integrating multimodal data according to claim 5, characterized in that: The S3 includes: S31, control point coordinate mapping: constructing a one-to-one mapping relationship between the virtual control point set and the physical control point set; S32, affine transformation solution: solve the optimal rigid transformation by minimizing the squared difference between the imaginary and real coordinates; S33, nonlinear residual compensation: calculate the affine transformation residual and construct a thin plate spline interpolation function based on the residual points; S34, binding function synthesis: constructing dynamic binding transformation function based on the constructed thin plate spline interpolation function; S35, full model binding: applying the transformation function to all twin model vertices.

7. The digital twin method of integrating multimodal data according to claim 6, characterized in that: The S33 includes: S331, affine residual calculation: After the affine transformation is completed, for each pair of control points, the corresponding transformation residual vector is calculated; S332, residual interpolation function construction: Based on the spatial distribution of all residual points, the distributed sensor index positions are used as interpolation control points to construct a thin plate spline interpolation function.

8. The digital twin method of integrating multimodal data according to claim 7, characterized in that: The S4 includes: S41, sensor parameter coordinate extraction: extracting the parameter coordinates of all sensors on the physical entity to form a parameter coordinate set; S42, real-time spatial position calculation: based on the current non-rigid reference frame grid function, calculate the real-time three-dimensional spatial position of the sensor; S43, binding position calculation: calling the grid snapshot function recorded at the binding moment to calculate the binding reference position of each sensor; S44, spatial displacement vector generation: calculating the spatial displacement of each sensor; S45, data remapping execution: spatially shifting the sensor raw data to eliminate the effect of entity deformation; S46, data stream reconstruction: aggregate all remapped data and output aligned remapped data stream.

9. The digital twin method of integrating multimodal data according to claim 8, characterized in that: The S5 includes: S51, extracting a deformation-related feature set from the remapped data stream; S52, abnormal area identification: identifying abnormal areas according to a multi-level threshold strategy; S53, calculation of the inhibitory force field: calculating the required reverse inhibitory force according to the characteristic value of the abnormal area; S54, actuator instruction generation: generating a control instruction set according to the calculated restraining force; S55, closed-loop execution and feedback: After the control instruction is sent to the entity, a closed-loop feedback process is executed.

10. The digital twin method of integrating multimodal data according to claim 8, characterized in that: The S54 includes: S541, hydraulic compensation command calculation: Generate a driving area control command for the hydraulic system based on the suppression force field and the normal vector of the abnormal area; S542, piezoelectric micro-vibration command calculation: using a piezoelectric actuator to generate a vibration control signal that matches the structural resonance frequency; S543, active cooling control instruction calculation: Based on the integral of the deformation variable and the safety threshold, a corresponding thermal control cooling power instruction is generated to reduce the source of thermally induced strain.

Images