Multi-mode-based digital twinborn interaction method and device for wind power plant and storage medium

Abstract

The invention discloses a multi-modal-based wind power plant digital twinborn interaction method and device and a storage medium, and the method comprises the steps: collecting a real-time image of a wind turbine generator, completing the calibration of an image pose, forming a three-dimensional scene with a NeRF voxel grid, importing CFD flow field simulation data, mapping a flow field scalar value to the NeRF voxel grid, obtaining an enhanced neural scene, and carrying out the simulation of the flow field. The global continuity of simulation data is considered at low wind speed, wake flow distribution of the whole wind power plant can be clearly seen, local turbulence of a single unit can be accurately positioned, a multi-modal model is deployed, professional terms in the field of wind power operation and maintenance are preset, after an instruction is sent, the model combines state information of an enhanced neural scene, the instruction is analyzed into a recognizable sequence, and the recognition efficiency of the wind power operation and maintenance is improved. The geometric structure and physical field distribution of a neural scene are enhanced through sequence combination, an optimal observation path is generated, the track smoothness is adjusted according to the eye movement angular velocity of a user, the correction amplitude is amplified during high eye movement, and view angle switching lagging is avoided through focusing point anchoring.

Description

Technical Field

[0001] This invention relates to the field of wind farm digital interaction technology, specifically to a multimodal wind farm digital twin interaction method, device, and storage medium. Background Technology

[0002] A wind turbine is a complete set of electromechanical equipment system that converts wind energy into electrical energy. Its core function is to capture the kinetic energy of natural wind flow through the wind turbine, convert it into electrical energy through the transmission system and generator, and finally connect it to the power grid.

[0003] A wind farm is a power generation system that centrally installs multiple wind turbine generators to convert wind energy into electrical energy on a large scale and connect it to the power grid. Its core function is to capture wind energy through clustered wind turbine generators, and then provide society with stable clean electricity through energy conversion and grid connection. It is one of the core infrastructures for global energy transition.

[0004] Existing wind farm models only show the external appearance of the tower and blades, failing to reflect implicit conditions such as internal blade stress and nacelle oil temperature. Maintenance personnel cannot predict faults in advance. Virtual modeling of the external appearance, tower structure measurements by lidar, and thermal distribution data from infrared scanning are all independent. Whether it is low wind speed steady state or high wind speed turbulence, the same standard is used to fuse simulation and measured data, resulting in a sharp drop in data accuracy under complex operating conditions. When observing holographic images, the perspective switching trajectory is abrupt. When maintenance personnel are focusing on blade cracks, the old perspective switching is choppy, making it impossible to accurately locate the fault point and affecting inspection efficiency. Summary of the Invention

[0005] The present invention proposes a multimodal digital twin interaction method, device, and storage medium for wind farms to address the challenges in existing industrial scenarios where dust and water mist often exhibit non-uniform, high-concentration distribution characteristics. Fixed physical model parameters are difficult to adapt to complex scenarios, leading to problems such as incomplete defogging, image color distortion, and errors in the dynamic medium morphology restoration. These issues prevent the provision of high-quality 2D visual input for subsequent 3D reconstruction. Furthermore, existing 3D reconstruction algorithms only focus on the restoration of scene geometry and visual effects, without quantitatively evaluating the uncertainty of the reconstruction results point by point, resulting in a significant reduction in the accuracy of subsequent holographic reconstruction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The multimodal wind farm digital twin interaction method of the present invention includes: S1. Acquire real-time RGB images of the wind turbine and complete image pose calibration. Input the calibrated image pose data into NeRF and train it through the Instant-NGP model to implicitly store the geometric shape and real texture of the wind farm scene, forming a three-dimensional scene with NeRF voxel mesh. S2. Import the CFD flow field simulation data calculated by OpenFOAM, map the flow field scalar values ​​to the NeRF voxel mesh, and additionally encode physical properties in the color output channel of the NeRF to obtain an enhanced neural scene integrating geometry, texture and physics. S3. Deploy a multimodal model as the interaction hub. The multimodal model is pre-defined with professional terms in the field of wind power operation and maintenance. After issuing a command, the model combines the state information of the enhanced neural scene to parse the command into an API call sequence that the rendering engine can recognize. S4. By combining the API call sequence with the geometric structure and physical field distribution of the enhanced neural scene, the optimal observation path and camera movement trajectory are generated. S5. The optimal observation path presents the holographic image of the enhanced neural scene in real time through the XR device, while simultaneously collecting the spatial displacement and rotation angle of the user's gestures and mapping the gestures into transformation commands for virtual objects. S6. The instruction is processed through permission verification and instruction conversion, and then output to the wind turbine.

[0007] Preferably, in step S1, an Instant-NGP model with piecewise linear density encoding is trained based on the image pose data, and the scene geometry, texture and multiple physical properties are implicitly stored through the NeRF volume rendering equation. The NeRF volume rendering equation is as follows: ; in, For along the ray The output multi-dimensional fused feature values For piecewise linear transmittance, For piecewise linear volume density, For texture weights, For spatial points In the direction of observation The basic RGB color values ​​below, For the first Physics-like field weighting coefficients For spatial points First scalar values ​​of physics fields and The volume density is the endpoint density of the interval.

[0008] Preferably, in S3, a multimodal model is deployed as the interaction hub, and professional terms in the field of wind power operation and maintenance are integrated to complete the instruction embedding. The instructions are parsed into an API call sequence that can be recognized by the rendering engine through a mechanism-enhanced intent parsing formula. The intent parsing formula for the mechanism enhancement is as follows: ; in, To fuse feature vectors, For the feature fusion weight matrix, User commands BERT embedding vectors, For the current scenario The state embedding vector, Vectors for embedding professional terms in the field of wind power operation and maintenance. This is the feature fusion bias term.

[0009] Preferably, in step S2, during the mapping process, the static CFD flow field simulation data calculated by OpenFOAM and the real-time sensing data of the wind farm are dynamically fused using the CFD simulation-real-time sensing dynamic calibration formula to obtain an enhanced neural scene integrating geometry, texture and physical field. The CFD simulation-real-time sensing dynamic calibration formula is as follows: ; ; ; in, for Time-space coordinates Adaptive calibration of physical field values ​​under operating conditions For the three-dimensional spatial coordinates of the wind farm, for Time-space coordinates The initial CFD simulation physics values ​​at the location, for Time-space coordinates Measured physical field values ​​of edge sensing at the location For spatiotemporal dual-dimensional calibration coefficients, For time-calibrated sensitivity coefficients, The time convergence threshold, This is a correction factor for wind speed conditions. This represents the real-time wind speed at the wheel hub.

[0010] Preferably, in the optimal observation path generation stage of S4, an initial observation trajectory is first generated using the Bezier trajectory-eye-tracking motion sickness suppression formula, and then the high curvature segment is smoothed and corrected using the same formula, while limiting the camera movement speed to no more than a safe threshold.

[0011] Preferably, the formula for suppressing motion sickness based on the Bezier trajectory-eye movement association is as follows: ; ; ; in, Standard third-order Bessel initial trajectory Camera spatial position at any given moment The starting point of the trajectory, As the focus of the trajectory, For the trajectory transition point, The endpoint of the trajectory, The optimal smoothing point for eye-tracking adaptation. This is a trajectory smoothing correction factor. for Gradient of curvature of the trajectory at any moment )for Camera movement speed at all times The safe moving speed threshold for the camera, This is the eye movement angular velocity correction factor. Provides real-time eye movement angular velocity for users. These are the trajectory time parameters.

[0012] Preferably, in step S6, the instruction is input into the operation instruction security verification formula, the device indicators after the instruction is executed are predicted by the digital twin model, and then a security score is calculated. If the score is lower than the threshold, the instruction is intercepted by the security gateway. The security verification formula for the operation command is as follows: ; ; ; in, The operation instructions are evaluated based on a comprehensive safety score. The total number of security verification indicators, The first instruction executed Predicted values ​​for each indicator For the first Safety benchmark values ​​for each indicator For the first The upper and lower limits of the safety threshold for each indicator For the first Safety weight of each indicator As the basic weight of the indicator, As an indicator of historical fault correlation factors, The threshold for safe execution of instructions. This is a control flag for instruction execution.

[0013] Preferably, in step S5, a holographic image of the enhanced neural scene is presented through an XR device, and gesture actions are mapped into transformation instructions for virtual objects based on a gesture interaction mapping formula with physical constraints. ; in, This refers to the virtual object transformation matrix updated after a gesture operation. This is the original virtual object transformation matrix before the gesture operation. For gesture-based rotation angle The constructed 3×3 rotation matrix, The moment of inertia constraint coefficient, This is the interaction sensitivity scaling factor. Let be the displacement vector of the gesture in the physical world coordinate system. This is the force constraint coefficient.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described method.

[0015] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0016] As can be seen from the above technical solution, this invention provides a multimodal wind farm digital twin interaction method, device, and storage medium. Compared with the prior art, this invention has the following advantages: By acquiring real-time images of wind turbine generators and completing image pose calibration, the images are input into NeRF, implicitly storing the geometric shape and real texture of the wind farm scene, forming a three-dimensional scene with NeRF voxel mesh. This not only restores the rust texture of the tower and the fine cracks of the blades, but also imports implicit data such as stress and temperature into the model. Furthermore, weights are assigned according to data reliability to avoid contradictions in multi-source data. CFD flow field simulation data is imported, and the flow field scalar values ​​are mapped to the NeRF voxel mesh to obtain enhanced neural networks. In scenarios with low wind speeds, the simulation data maintains global continuity, clearly showing the wake distribution of the entire wind farm while accurately locating the local turbulence of a single turbine. A multimodal model is deployed, pre-defined with professional terminology from the wind power operation and maintenance field. Upon issuing commands, the model combines the state information of the augmented neural network scenario to parse the commands into a recognizable sequence. This sequence, combined with the geometric structure and physical field distribution of the augmented neural network scenario, generates the optimal observation path. The trajectory smoothness is adjusted based on the user's eye-tracking angular velocity, with the correction amplitude amplified during high eye-tracking events. Anchoring at the point of interest prevents stuttering during viewpoint switching. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal wind farm digital twin interaction method of the present invention. Figure 2 This is an architecture diagram of the multimodal wind farm digital twin interaction method of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0019] like Figure 1 and 2 As shown in this embodiment, the multimodal wind farm digital twin interaction method includes: S1. Collect real-time RGB images of wind turbines through a cluster of drones and complete image pose calibration. Input the calibrated image pose data into NeRF and train it through the Instant-NGP model to implicitly store the geometric shape and real texture of the wind farm scene, forming a three-dimensional scene with NeRF voxel mesh. S2. Import the CFD flow field simulation data calculated by OpenFOAM, map the flow field scalar values ​​to the NeRF voxel mesh, and additionally encode physical properties in the color output channel of NeRF to obtain an enhanced neural scene that integrates geometry, texture and physics. S3. Deploy a multimodal model as the interaction hub. The multimodal model is pre-defined with professional terms in the field of wind power operation and maintenance. After issuing instructions, the model combines the state information of the augmented neural scene to parse the instructions into an API call sequence that the rendering engine can recognize. S4. By combining the geometric structure and physical field distribution of the enhanced neural scene with the API call sequence, the optimal observation path and camera movement trajectory are generated. S5. The optimal observation path uses XR devices to present holographic images of enhanced neural scenes in real time, while simultaneously collecting the spatial displacement and rotation angle of user gestures and mapping the gestures into transformation commands for virtual objects. S6. After authorization verification and instruction conversion, the instruction is output to the wind turbine.

[0020] The present invention provides a multimodal wind farm digital twin interaction method, in which an Instant-NGP model with piecewise linear density encoding is trained based on image pose data in S1, and scene geometry, texture and multiple physical properties are implicitly stored through NeRF volume rendering equations. The NeRF volume rendering equation is as follows: ; in, For along the ray The output multi-dimensional fused feature values For piecewise linear transmittance, For piecewise linear volume density, For texture weights, For spatial points In the direction of observation The basic RGB color values ​​below, For the first Physics-like field weighting coefficients For spatial points First scalar values ​​of physics fields and The volume density is the endpoint density of the interval.

[0021] In practical applications, it is necessary to carry out "blade crack inspection + high wind speed wake analysis + unit reset operation" for 15 2.5MW units in an onshore wind farm. Real-time RGB images of the wind turbine units are collected by a drone swarm, and data such as stress and temperature are detected by edge sensors on the units. The maintenance team needs to rebuild an integrated virtual model of "appearance + stress + heat distribution" for Unit 15 (which had previously experienced microcracks in its blades) for subsequent remote inspections. The camera's X-ray parameters are as follows: the starting point of the X-ray is selected from the observation point of the XR equipment in the cabin. The unit vector in the direction of observation is The integral interval points to the crack in the blade. for The step size range of the ray in the leaf region, and the density of the leaf body is the volume density at the endpoints of the interval. ( ), ( Texture weights Physical field weights are divided into stress. and temperature The leaf texture color was determined from the RGB images captured by the camera. Since the leaves are painted gray, the vector of the captured RGB image is... The physical field scalar values ​​are divided into stress at the crack. and ; The final calculation yielded Among them, the first 3 dimensions The RGB texture corresponding to the crack area is darker, which matches the actual wear and tear of the paint at the crack. (Later 2D) Corresponding to normalized stress and temperature (stress dimension values ​​are significantly higher than in the normal region, allowing for intuitive identification of crack stress concentration). As can be seen from the above, by adding stress and temperature dimensions in addition to RGB texture, we can "see the appearance and read the hidden state". Maintenance personnel can simultaneously judge the appearance location of cracks and the degree of internal damage without switching interfaces.

[0022] The present invention provides a multimodal wind farm digital twin interaction method, in which a multimodal model is deployed in S3 as the interaction hub, and professional terms in the field of wind power operation and maintenance are integrated to complete the instruction embedding. The instructions are parsed into an API call sequence that can be recognized by the rendering engine through a mechanism-enhanced intent parsing formula. The formula for mechanism-enhanced intent parsing is as follows: ; in, To fuse feature vectors, For the feature fusion weight matrix, User commands BERT embedding vectors, For the current scenario The state embedding vector, Vectors for embedding professional terms in the field of wind power operation and maintenance. This is the feature fusion bias term.

[0023] Maintenance personnel issue a voice command, "Check the hub stress of Unit 15." The system needs to interpret the intent and ensure operational safety. At this point, the command... BERT embedding vectors Scene State embedding vector Wind power operation and maintenance terminology embedding vector fusion weight Feature fusion bias term ; Calculation ; The multimodal wind farm digital twin interaction method provided by this invention, in S2, during the mapping process, uses the CFD simulation-real-time sensing dynamic calibration formula to dynamically fuse the static CFD flow field simulation data calculated by OpenFOAM with the real-time sensing data of the wind farm to obtain an enhanced neural scene integrating geometry-texture-physical field. The CFD simulation-real-time sensor dynamic calibration formula is as follows: ; ; ; in, for Time-space coordinates Adaptive calibration of physical field values ​​under operating conditions For the three-dimensional spatial coordinates of the wind farm, for Time-space coordinates The initial CFD simulation physics values ​​at the location, for Time-space coordinates Measured physical field values ​​of edge sensing at the location For spatiotemporal dual-dimensional calibration coefficients, For time-calibrated sensitivity coefficients, The time convergence threshold, This is a correction factor for wind speed conditions. This represents the real-time wind speed at the wheel hub.

[0024] At this moment, Unit 3 suddenly 13m / s High wind speeds require calibration of wake wind speed data to provide a basis for operation and maintenance decisions. The initial CFD simulation wind speed was obtained from OpenFOAM offline simulation. Wind speed measured by cabin sensors ,time ,Depend on It can be known High wind speed conditions Calculate the time calibration factor: ; Calculate the actual wind speed after calibration ; Under high wind speed conditions, the system prioritizes matching measured data, reducing wind speed deviation from the traditional 15% to less than 3%. This allows the enhanced neural network scenario to dynamically adapt to the operating conditions, preserving the global wake distribution of CFD while accurately reflecting the real-time wind speed of a single unit, thus solving the problem of disconnect between static simulation and dynamic measurement. The multimodal wind farm digital twin interaction method provided by this invention, in the optimal observation path generation stage of S4, first generates an initial observation trajectory through the Bezier trajectory-eye-tracking correlation motion sickness suppression formula, and then uses the formula to smooth and correct the high curvature segment, while limiting the camera movement speed to not exceed the safety threshold.

[0025] Its key feature is that the Bezier trajectory-eye-tracking motion sickness suppression formula is as follows: ; ; ; in, Standard third-order Bessel initial trajectory Camera spatial position at any given moment The starting point of the trajectory, As the focus of the trajectory, For the trajectory transition point, The endpoint of the trajectory, The optimal smoothing point for eye-tracking adaptation. This is a trajectory smoothing correction factor. for Gradient of curvature of the trajectory at any moment )for Camera movement speed at all times The safe moving speed threshold for the camera, This is the eye movement angular velocity correction factor. Provides real-time eye movement angular velocity for users. These are the trajectory time parameters.

[0026] In practical applications, the system needs to generate the optimal observation path from the nacelle to the blade crack of Unit 15 to avoid dizziness for maintenance personnel. Bezier trajectory control point: cabin start point Cracks as a concern transition point ,end ; Trajectory Time Parameters Eye movement angular velocity (Exceed Trajectory smoothing correction factor ,speed ; Then calculate the initial trajectory points. ; Then calculate the optimal smoothing point. ; Assumption Substituting into The trajectory is smoother; As shown above, by combining eye-tracking data to enhance smoothness, the incidence of motion sickness in users decreased from 62% to 9.3%, which can support long-term inspections of more than 30 minutes. Then, by anchoring the crack focus, the visualization accuracy of the target area is improved by 20%, and microcracks as small as 0.5 mm can be clearly seen.

[0027] The multimodal wind farm digital twin interaction method provided by this invention inputs the instruction into the operation instruction security verification formula in S6, predicts the equipment indicators after the instruction is executed through the digital twin model, and then calculates the security score. If the score is lower than the threshold, the instruction is intercepted by the security gateway. The formula for verifying the security of operating instructions is as follows: ; ; ; in, The operation instructions are evaluated based on a comprehensive safety score. The total number of security verification indicators, The first instruction executed Predicted values ​​for each indicator For the first Safety benchmark values ​​for each indicator For the first The upper and lower limits of the safety threshold for each indicator For the first Safety weight of each indicator As the basic weight of the indicator, As an indicator of historical fault correlation factors, The threshold for safe execution of instructions. This is a control flag for instruction execution.

[0028] In practical applications, when maintenance personnel issue a reset command for Unit 15 in virtual space, they must first verify the security of the command before sending it to the wind turbine. According to the operation and maintenance system, five indicators need to be verified: speed, torque, oil temperature, vibration, and voltage. Therefore, what is the total number of safety verification indicators? Yes, this unit has experienced an oil temperature failure before. Taking oil temperature calibration as an example, a comprehensive safety score of 0.82 or higher is sufficient to pass the calibration. ,in, Replace with a symbol representing oil temperature ,so Oil temperature prediction value Safety benchmark value threshold Oil temperature base weight Calculate All other indicators scored (The safety scores for speed, torque, vibration, and voltage are 0.85, 0.9, 0.9, and 0.92, respectively). Then calculate the score for the oil temperature item. ; Final calculation ; It can be known Therefore, instructions can be issued; As shown above, by pre-verifying multiple indicators, the instruction execution failure rate was reduced from 7% to 0; by increasing the weight of historical failure indicators, the failure prediction accuracy reached 98%, thus avoiding the recurrence of similar failures.

[0029] The present invention provides a multimodal wind farm digital twin interaction method, in which holographic images of enhanced neural scenes are presented through XR devices in S5, and gesture actions are mapped into transformation instructions of virtual objects based on a gesture interaction mapping formula with physical constraints. ; in, This refers to the virtual object transformation matrix updated after a gesture operation. This is the original virtual object transformation matrix before the gesture operation. For gesture-based rotation angle The constructed 3×3 rotation matrix, The moment of inertia constraint coefficient, This is the interaction sensitivity scaling factor. Let be the displacement vector of the gesture in the physical world coordinate system. This is the force constraint coefficient.

[0030] In practical applications, maintenance personnel need to disassemble the gearbox of Unit 15 in the XR equipment to verify the status of internal components; The given conditions are as follows: Original virtual object transformation matrix During the operation, the maintenance personnel rotated their hand gestures by 15°, that is... ; so Because the gearbox has a large moment of inertia, which restricts rotation, the moment of inertia constraint coefficient is... To simulate heavy damping, a force constraint coefficient is set. The gesture displacement vector of maintenance personnel Sensitivity scaling factor ; Calculate the transformation matrix again ; First calculate the rotating part: ; Displacement part: ; final It achieves slow rotation and small displacement of the gearbox, without the distortion of "lifting the gearbox with one hand" operation; As can be seen from the above, due to the addition of physical constraints, the operation feels more like real heavy equipment, and the interaction distortion rate is reduced from 28% to 0; the gesture delay is only 8ms, which can accurately complete the disassembly and inspection of delicate parts such as gearbox bolts.

[0031] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described method.

[0032] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0033] As can be seen from the above technical solution, this invention provides a holographic enhancement system for strong interference environments based on reverse diffusion generation. Compared with the prior art, this invention has the following advantages: By acquiring real-time images of wind turbine generators and completing image pose calibration, the images are input into NeRF, implicitly storing the geometric shape and real texture of the wind farm scene, forming a three-dimensional scene with NeRF voxel mesh. This not only restores the rust texture of the tower and the fine cracks of the blades, but also imports implicit data such as stress and temperature into the model. Furthermore, weights are allocated according to the data's reliability to avoid contradictions between multi-source data. CFD flow field simulation data is imported, and the flow field scalar values ​​are mapped to the NeRF voxel mesh to obtain enhanced neural networks. In low-wind-speed scenarios, this invention maintains the global continuity of simulation data, clearly showing the wake distribution of the entire wind farm while accurately locating the local turbulence of individual turbines. It deploys a multimodal model, pre-programs wind power operation and maintenance terminology, and, upon issuing commands, the model, combined with the state information of the augmented neural network scenario, parses the commands into a recognizable sequence. This sequence, combined with the geometric structure and physical field distribution of the augmented neural network scenario, generates the optimal observation path and adjusts the trajectory smoothness according to the user's eye-tracking angular velocity, amplifying the correction amplitude during high-eye-tracking scenarios. Anchoring the focus point avoids stuttering during viewpoint switching. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0034] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0036] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0037] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The multimodal wind farm digital twin interaction method of the present invention is characterized in that, include: S1. Acquire real-time RGB images of the wind turbine and complete image pose calibration. Input the calibrated image pose data into NeRF and train it through the Instant-NGP model to implicitly store the geometric shape and real texture of the wind farm scene, forming a three-dimensional scene with NeRF voxel mesh. S2. Import the CFD flow field simulation data calculated by OpenFOAM, map the flow field scalar values ​​to the NeRF voxel mesh, and additionally encode physical properties in the color output channel of the NeRF to obtain an enhanced neural scene integrating geometry, texture and physics. S3. Deploy a multimodal model as an interaction hub. The multimodal model is pre-defined with professional terms in the field of wind power operation and maintenance. After issuing instructions, the model combines the state information of the enhanced neural scene to parse the instructions into an API call sequence that the rendering engine can recognize. S4. By combining the API call sequence with the geometric structure and physical field distribution of the enhanced neural scene, the optimal observation path and camera movement trajectory are generated. S5. The optimal observation path presents the holographic image of the enhanced neural scene in real time through the XR device, while simultaneously collecting the spatial displacement and rotation angle of the user's gestures and mapping the gestures into transformation commands for virtual objects. S6. The instruction is processed through permission verification and instruction conversion, and then output to the wind turbine.

2. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 1, characterized in that: In S1, an Instant-NGP model with piecewise linear density encoding is trained based on image pose data, and scene geometry, texture and multiple physical properties are implicitly stored through the NeRF volume rendering equation. The NeRF volume rendering equation is as follows: ； in, For along the ray The output multi-dimensional fused feature values, For piecewise linear transmittance, For piecewise linear volume density, For texture weights, For spatial points In the direction of observation The basic RGB color values ​​below, For the first Physics-like field weighting coefficients For spatial points First scalar values ​​of physics fields and The volume density is the endpoint density of the interval.

3. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 1, characterized in that: The S3 deploys a multimodal model as the interaction hub, integrates professional terms in the field of wind power operation and maintenance to complete the instruction embedding, and uses a mechanism-enhanced intent parsing formula to parse the instructions into an API call sequence that the rendering engine can recognize. The intent parsing formula for the mechanism enhancement is as follows: ； in, To fuse feature vectors, For the feature fusion weight matrix, User commands BERT embedding vectors, For the current scenario The state embedding vector, Vectors for embedding professional terms in the field of wind power operation and maintenance. This is the feature fusion bias term.

4. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 1, characterized in that: In S2, during the mapping process, the static CFD flow field simulation data calculated by OpenFOAM and the real-time sensing data of the wind farm are dynamically fused using the CFD simulation-real-time sensing dynamic calibration formula to obtain an enhanced neural scene integrating geometry-texture-physical field. The CFD simulation-real-time sensing dynamic calibration formula is as follows: ； ； ； in, for Time-space coordinates Adaptive calibration of physical field values ​​under operating conditions For the three-dimensional spatial coordinates of the wind farm, for Time-space coordinates The initial CFD simulation physics values ​​at the location, for Time-space coordinates Measured physical field values ​​of edge sensing at the location For spatiotemporal dual-dimensional calibration coefficients, For time-calibrated sensitivity coefficients, The time convergence threshold, This is a correction factor for wind speed conditions. This represents the real-time wind speed at the wheel hub.

5. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 1, characterized in that: In the optimal observation path generation stage of S4, the initial observation trajectory is first generated using the Bezier trajectory-eye-tracking motion sickness suppression formula, and then the high curvature segment is smoothed and corrected using the same formula, while limiting the camera movement speed to no more than a safe threshold.

6. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 5, characterized in that: The formula for suppressing motion sickness based on the Bessel trajectory-eye movement association is as follows: ； ； ； in, Standard third-order Bessel initial trajectory Camera spatial position at any given moment The starting point of the trajectory, As the focus of the trajectory, As the trajectory transition point, The endpoint of the trajectory, The optimal smoothing point for eye-tracking adaptation. This is a trajectory smoothing correction factor. for Gradient of curvature of the trajectory at any moment )for Camera movement speed at all times The safe moving speed threshold for the camera, This is the eye movement angular velocity correction factor. Provides real-time eye movement angular velocity for users. These are the trajectory time parameters.

7. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 6, characterized in that: In step S6, the instruction is input into the operation instruction security verification formula, the device indicators after the instruction is executed are predicted by the digital twin model, and then the security score is calculated. If the score is lower than the threshold, the instruction is intercepted by the security gateway. The security verification formula for the operation command is as follows: ； ； ； in, The operation instructions are given a comprehensive safety score. The total number of security verification indicators, The first [number] instruction executed Predicted values ​​for each indicator For the first Safety benchmark values ​​for each indicator For the first The upper and lower limits of the safety threshold for each indicator For the first Safety weight of each indicator As the basic weight of the indicator, As an indicator of historical fault correlation factors, The threshold for safe execution of instructions. This is a control flag for instruction execution.

8. The holographic enhancement system for strong interference environments based on reverse diffusion generation according to claim 1, characterized in that: In S5, the holographic image of the enhanced neural scene is presented through an XR device, and the gesture action is mapped into the transformation command of the virtual object based on the gesture interaction mapping formula with physical constraints. ； in, This refers to the virtual object transformation matrix updated after a gesture operation. This is the original virtual object transformation matrix before the gesture operation. For gesture-based rotation angle The constructed 3×3 rotation matrix, The moment of inertia constraint coefficient, This is the interaction sensitivity scaling factor. Let be the displacement vector of the gesture in the physical world coordinate system. This is the force constraint coefficient.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images