Multi-modal data processing method and device, equipment, medium and product

Abstract

The invention provides a multi-modal data processing method and device, equipment, a medium and a product, and relates to the technical field of virtual reality. Comprising the steps of obtaining multi-modal data in an industrial environment, wherein the multi-modal data comprises sensor data, image data and text data; time sequence features of the sensor data are extracted through a preset recurrent neural network, and time sequence feature vectors are obtained; extracting spatial features of the image data through a preset convolutional neural network to obtain spatial feature vectors; performing semantic coding on the text data through a pre-training language learning model to obtain a text semantic vector; performing feature fusion on the time sequence feature vector, the spatial feature vector and the text semantic vector to generate a multi-modal fusion feature vector; and driving the dynamic update of the virtual scene according to the multi-modal fusion feature vector. According to the method provided by the invention, the multi-modal data in the industrial environment is fused, the virtual scene is dynamically updated, the dynamic change of the industrial equipment is reflected in time, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a multimodal data processing method, apparatus, device, medium and product. Background Technology

[0002] With the rapid development of industry and intelligent manufacturing, industrial digital twin platforms have become a key technology for achieving deep connection between the physical and digital worlds.

[0003] In existing technologies, industrial digital twin platforms typically employ a modular architecture, including modules for data acquisition, data processing, virtual scene construction, and user interaction. The data acquisition module obtains operating parameters and log records from industrial equipment; the data processing module cleans, formats, and performs preliminary analysis of these parameters and log records; the virtual scene construction module uses 3D modeling tools to build digital models and visualizes them through a rendering engine; and the user interaction module provides basic functions such as chart display and model manipulation.

[0004] However, existing industrial digital twin platforms suffer from delayed virtual scene updates, failing to reflect the dynamic changes in industrial equipment in a timely manner. Summary of the Invention

[0005] This application provides a multimodal data processing method, apparatus, equipment, medium, and product, which can dynamically update virtual scenes by fusing and processing multimodal data in an industrial environment, timely reflect the dynamic changes of industrial equipment, realize the synchronous update of virtual scenes and the physical world, and thus improve user experience.

[0006] Firstly, this application provides a multimodal data processing method, the method comprising:

[0007] Acquire multimodal data in industrial environments, including sensor data, image data, and text data;

[0008] The temporal features of sensor data are extracted by a pre-set recurrent neural network to obtain a temporal feature vector;

[0009] Spatial features of image data are extracted using a pre-defined convolutional neural network to obtain spatial feature vectors;

[0010] Text semantic vectors are obtained by semantically encoding text data using a pre-trained language learning model.

[0011] The temporal feature vector, spatial feature vector, and text semantic vector are fused to generate a multimodal fused feature vector;

[0012] The virtual scene is dynamically updated based on the multimodal fusion feature vector.

[0013] In one possible implementation, temporal feature vectors, spatial feature vectors, and text semantic vectors are fused to generate a multimodal fused feature vector, including:

[0014] The temporal feature vector, spatial feature vector, and text semantic vector are normalized respectively to obtain normalized temporal feature vector, normalized spatial feature vector, and normalized text semantic vector.

[0015] The normalized temporal feature vector, the normalized spatial feature vector, and the normalized text semantic vector are concatenated into a joint feature vector;

[0016] The joint feature vector is input into a preset multilayer perceptron to obtain a multimodal fusion feature vector.

[0017] In one possible implementation, the virtual scene is dynamically updated based on a multimodal fusion feature vector, including...

[0018] The virtual scene is divided into a static layer and a dynamic layer. The static layer is used to display the structural attributes of the industrial equipment, and the dynamic layer is used to display the state attributes of the industrial equipment.

[0019] Multimodal fusion feature vectors are mapped to state parameters of industrial equipment, including operating temperature, vibration amplitude, visual appearance and failure probability.

[0020] Based on the status parameters, update the visual attributes of the corresponding virtual device in the dynamic layer. The visual attributes include geometry, color, animation effects, and alarm indicators.

[0021] In one possible implementation, after driving the dynamic update of the virtual scene based on the multimodal fusion feature vector, the method further includes:

[0022] Multimodal fusion feature vectors are overlaid onto the AR view of industrial equipment using an augmented reality (AR) rendering engine;

[0023] Gesture recognition technology is used to identify the user's gestures and obtain operation instructions;

[0024] Adjust the content displayed in the AR view according to the operation instructions.

[0025] In one possible implementation, after acquiring multimodal data from an industrial environment, the process further includes:

[0026] Denoising and imputing missing values ​​in sensor data;

[0027] Perform illumination correction and target region cropping on the image data;

[0028] The text data is processed by word segmentation, stop word filtering, and entity standardization.

[0029] In one possible implementation, it also includes:

[0030] When the device state represented by the multimodal fusion feature vector exceeds a preset safety threshold, at least one of the following operations is triggered:

[0031] Highlight abnormal devices in a virtual scene;

[0032] Generate maintenance work orders and push them to the management terminal;

[0033] Send load reduction control commands or shutdown control commands to industrial equipment.

[0034] Secondly, this application provides a multimodal data processing apparatus, the apparatus comprising:

[0035] The acquisition module is used to acquire multimodal data in the industrial environment, including sensor data, image data, and text data.

[0036] The feature extraction module is used to extract the temporal features of sensor data through a preset recurrent neural network to obtain a temporal feature vector;

[0037] The feature extraction module is also used to extract spatial features of image data through a preset convolutional neural network to obtain spatial feature vectors;

[0038] The feature extraction module is also used to semantically encode text data using a pre-trained language learning model to obtain text semantic vectors;

[0039] The feature fusion module is used to fuse temporal feature vectors, spatial feature vectors, and text semantic vectors to generate multimodal fused feature vectors.

[0040] The update module is used to drive the dynamic update of the virtual scene based on the multimodal fusion feature vector.

[0041] In one possible implementation, the feature fusion module is further used to normalize the temporal feature vector, spatial feature vector, and text semantic vector respectively to obtain normalized temporal feature vector, normalized spatial feature vector, and normalized text semantic vector.

[0042] The feature fusion module is also used to concatenate the normalized temporal feature vector, the normalized spatial feature vector, and the normalized text semantic vector into a joint feature vector;

[0043] The feature fusion module is also used to input the joint feature vector into a preset multilayer perceptron to obtain a multimodal fusion feature vector.

[0044] In one possible implementation, the update module is further configured to divide the virtual scene into a static layer and a dynamic layer, wherein the static layer is used to display the structural attributes of the industrial equipment and the dynamic layer is used to display the state attributes of the industrial equipment.

[0045] The update module is also used to map multimodal fusion feature vectors to state parameters of industrial equipment, including operating temperature, vibration amplitude, visual appearance status and failure probability.

[0046] The update module is also used to update the visual attributes of the corresponding virtual device in the dynamic layer based on the status parameters. The visual attributes include geometric shape, color, animation effects, and alarm indicators.

[0047] In one possible implementation, the update module is also configured to overlay multimodal fused feature vectors onto the AR view of the industrial equipment via an augmented reality (AR) rendering engine;

[0048] The update module is also used to recognize the user's gesture operations through gesture recognition technology to obtain operation instructions;

[0049] The update module is also used to adjust the content displayed in the AR view based on operation instructions.

[0050] In one possible implementation, the acquisition module is also used to denoise and impute missing values ​​in the sensor data;

[0051] The acquisition module is also used to perform illumination correction and target region cropping on the image data;

[0052] The acquisition module is also used for word segmentation, stop word filtering, and entity standardization of text data.

[0053] In one possible implementation, the update module is further configured to trigger at least one of the following operations when the device status represented by the multimodal fusion feature vector exceeds a preset safety threshold: highlighting the abnormal device in the virtual scene; generating a maintenance work order and pushing it to the management terminal; or sending a load reduction control command or a shutdown control command to the industrial equipment.

[0054] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor.

[0055] The memory stores the instructions that the computer executes.

[0056] The processor executes computer execution instructions stored in memory to implement a multimodal data processing method according to the first aspect of the invention.

[0057] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a multimodal data processing method according to the first aspect of the invention.

[0058] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a multimodal data processing method according to the first aspect of the invention.

[0059] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0060] This application provides a multimodal data processing method, apparatus, device, medium, and product, comprising: first, acquiring multimodal data in an industrial environment, including sensor data, image data, and text data; then, extracting temporal features from the sensor data using a pre-set recurrent neural network to obtain a temporal feature vector; extracting spatial features from the image data using a pre-set convolutional neural network to obtain a spatial feature vector; and semantically encoding the text data using a pre-trained language learning model to obtain a text semantic vector; then, fusing the temporal feature vector, spatial feature vector, and text semantic vector to generate a multimodal fused feature vector; and finally, driving the dynamic updating of a virtual scene based on the multimodal fused feature vector. The following technical effects were achieved: By acquiring multimodal data from the industrial environment, and extracting temporal features from sensor data using a pre-set recurrent neural network, extracting spatial structural features from image data using a pre-set convolutional neural network, and semantically encoding text data using a pre-trained language model, the extracted three types of feature vectors were then fused to generate a unified multimodal fusion feature vector. Finally, based on this multimodal fusion feature vector, the dynamic update of the virtual scene was driven, thereby achieving millisecond-level response and high-fidelity virtual-real synchronization on the industrial digital twin platform. This breakthrough overcomes the dependence of traditional digital twin platforms on a single data source and static update mechanism. Through multimodal perception and intelligent fusion, the sensitivity, expressive richness, and real-time update of the virtual scene to dynamic changes in the physical world were significantly improved, thereby enhancing the user experience. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 1 ;

[0064] Figure 2 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 2 ;

[0065] Figure 3 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 3 ;

[0066] Figure 4 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 4 ;

[0067] Figure 5 This is a schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application;

[0068] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0069] Figure label:

[0070] 510 - Acquisition Module; 520 - Feature Extraction Module; 530 - Feature Fusion Module; 540 - Update Module;

[0071] 610 - Processor; 620 - Memory; 630 - Communication components; 640 - Bus. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply difference. It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0074] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the multimodal data processing method provided in the embodiments of this application is merely an example; a multimodal data processing method may include more or fewer elements.

[0075] With the rapid development of industry and intelligent manufacturing, industrial digital twin platforms, as a core enabling technology connecting the physical world and the digital space, have been widely used in scenarios such as equipment monitoring, predictive maintenance, and production simulation. A typical industrial digital twin platform usually adopts a modular architecture, mainly including: a data acquisition module, a data processing module, a virtual scene construction module, and a user interaction module.

[0076] The data acquisition module obtains operating parameters (such as temperature, pressure, and speed) and operation logs of industrial equipment through sensor or log interfaces. The data processing module cleans, standardizes, and performs preliminary analysis on the raw data. The virtual scene construction module builds digital models of industrial equipment or production lines using 3D modeling tools and uses a graphics rendering engine to achieve visualization, providing users with an intuitive virtual scene. The user interaction module provides basic operation functions such as chart display, model rotation / zoom, and status query, facilitating user interaction with the virtual scene.

[0077] However, existing industrial digital twin platforms have significant shortcomings in practical applications. On the one hand, the data sources are limited, relying heavily on structured sensor data and lacking effective utilization of unstructured information such as images and text. On the other hand, the virtual scene update mechanism is lagging, typically employing timed polling or event-triggered update strategies, making it difficult to respond in real time to rapid changes in the state of industrial equipment (such as sudden vibrations, external damage, fault alarms, etc.). This leads to a mismatch between the virtual scene and the industrial equipment, weakening its credibility and practicality in real-time monitoring and decision support.

[0078] Therefore, there is an urgent need for a method that can integrate multi-source heterogeneous data and achieve low-latency, high-fidelity dynamic updates of virtual scenes, so as to truly achieve synchronous mapping between the physical world and the virtual scene.

[0079] Based on this, embodiments of this application propose a multimodal data processing method, apparatus, device, medium, and product, which can be used in the field of virtual reality technology and aims to solve the above-mentioned technical problems of the prior art. By acquiring multimodal data from an industrial environment, the multimodal data includes at least time-series signals from industrial sensors, image data collected by industrial cameras or monitoring equipment, and text data from operation logs, work order records, or speech-to-text transcription. A preset recurrent neural network is used to extract the time-series features of the sensor data to obtain a time-series feature vector; a preset convolutional neural network is used to extract the spatial structure features of the image data to obtain a spatial feature vector; a pre-trained language model is used to semantically encode the text data to obtain a text semantic vector; then, the above three types of feature vectors are fused to generate a unified multimodal fused feature vector; finally, based on this multimodal fused feature vector, the dynamic update of the virtual scene is driven, such as real-time adjustment of the geometric deformation, surface color, animation status, or alarm indicators of the digital model, thereby achieving millisecond-level response and high-fidelity virtual-real synchronization on an industrial digital twin platform. Breaking away from the reliance of traditional industrial digital twin platforms on a single data source and static update mechanism, this technology significantly improves the sensitivity, expressive richness, and real-time update speed of virtual scenes to dynamic changes in the physical world through multimodal perception and intelligent fusion, thereby enhancing the user experience.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 1 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 1The execution entity in this embodiment can be a data processing server or other devices with data processing capabilities, such as laptops, personal computers, tablets, etc. The data processing server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server, etc., without specific limitations. For ease of description, this application embodiment uniformly describes the execution entity of a multimodal data processing method as a server. Figure 1 As shown, the method includes:

[0082] S101. Acquire multimodal data in the industrial environment.

[0083] In this embodiment, multimodal data includes sensor data, image data, and text data.

[0084] Specifically, multimodal data can originate from heterogeneous sensing devices deployed in industrial environments. Sensor data refers to continuous time-series signals collected by industrial sensing units such as temperature sensors, vibration sensors, current transformers, and pressure transmitters. The sampling frequency is typically from 10 Hz to 1 kHz, reflecting the dynamic changes in the operating status of industrial equipment.

[0085] Image data refers to visible light / thermal imaging images captured by industrial cameras, infrared thermal imagers, or surveillance cameras, with a resolution of no less than 640 pixels × 480 pixels, used to capture visual information such as the appearance of industrial equipment, instrument readings, and smoke leaks.

[0086] Text data refers to unstructured text derived from operation logs, work order systems, speech recognition transcription results, or alarm messages from Supervisory Control and Data Acquisition (SCADA) systems. It includes descriptions of industrial equipment status, maintenance records, or fault keywords (such as abnormal bearing noise, motor overheating, planned shutdown, etc.).

[0087] S102. Extract the temporal features of the sensor data through a preset recurrent neural network to obtain a temporal feature vector.

[0088] Specifically, sensor data is input into a pre-defined recurrent neural network (RNN), such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU) structure, to effectively model long-term dependencies in time series.

[0089] The server can divide sensor data into multiple time steps in chronological order, inputting the sensor data into a pre-defined recurrent neural network in the form of a time series. Each time step corresponds to a sensor value or sensor vector at a sampling moment. After processing by multiple recurrent units, the pre-defined recurrent neural network can output a fixed-dimensional time-series feature vector (e.g., 256-dimensional). This time-series feature vector comprehensively encodes the operating trend, abnormal fluctuations, and periodic patterns of industrial equipment within the observation window, and can be used to characterize the dynamic evolution pattern of the operating status of industrial equipment.

[0090] S103. Extract spatial features from image data using a pre-defined convolutional neural network to obtain spatial feature vectors.

[0091] Specifically, for image data, the input is fed into a pre-defined convolutional neural network (CNN). For example, a backbone network architecture such as Residual Network (ResNet), Efficient Network (EfficientNet), or Lightweight Mobile Network (MobileNet) can be used to construct the pre-defined CNN. It can contain multiple convolutional layers, batch normalization layers, and activation functions (ReLU). The image data is processed layer by layer by the pre-defined CNN to extract spatial features from local to global levels. Finally, a spatial feature vector (e.g., 512-dimensional) is generated through global average pooling. This spatial feature vector contains key visual semantics such as the integrity of the industrial equipment's appearance, abnormal thermal distribution, and component occlusion, used to encode the appearance of the industrial equipment and the contextual information of the industrial environment.

[0092] S104. Semantically encode the text data using a pre-trained language learning model to obtain a text semantic vector.

[0093] Specifically, for text data, the input is fed into a pre-trained language learning model, such as Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Approach (RoBERTa), or A Lite BERT (ALBERT).

[0094] Text data can first be segmented and serialized by a tokenizer to form a sequence of sub-words composed of tokens.

[0095] Subsequently, context-aware semantic encoding is performed by inputting into a pre-trained language learning model to extract the hidden states corresponding to special classification tokens (CLS tokens) in the word sequence, or mean pooling is performed on the hidden states of all words to generate a fixed-dimensional text semantic vector (e.g., 768-dimensional), thereby effectively capturing the semantic intent, fault description, operation instructions and abnormal clues in operation logs and alarm information.

[0096] S105. The temporal feature vector, spatial feature vector and text semantic vector are fused to generate a multimodal fused feature vector.

[0097] Specifically, the server can then perform multimodal feature fusion on the three types of feature vectors mentioned above. First, L2 normalization is applied to each feature vector to eliminate dimensional differences, i.e., to eliminate scale differences between modalities. Then, a joint representation is formed by concatenating these vectors, i.e., a joint feature vector. This joint feature vector is then input into a multilayer perceptron (MLP) consisting of two fully connected layers for cross-modal interaction modeling, ultimately outputting a unified multimodal fused feature vector (e.g., 1024 dimensions). This multimodal fused feature vector integrates the temporal behavior, visual state, and semantic context of industrial equipment, possessing stronger discriminative and representational capabilities.

[0098] S106. Drive the dynamic update of the virtual scene based on the multimodal fusion feature vector.

[0099] Specifically, the server can map multimodal fusion feature vectors to a set of state parameters for industrial equipment, such as real-time operating temperature, vibration amplitude, visual damage level, and failure probability. These state parameters are then passed to the rendering engine of the industrial digital twin platform, triggering dynamic attribute updates to the virtual equipment model. In other words, based on these state parameters, the corresponding visualization attributes of the virtual equipment are adjusted in real-time within the industrial digital twin platform. For example, when the vibration amplitude exceeds a threshold, a shaking animation is triggered in the virtual equipment model; when the failure probability is higher than 80%, the virtual equipment's casing color changes from green to red and a flashing alarm icon is overlaid; when a text log mentions oil leakage and an oil stain area is detected in the image, a stain texture is rendered at the corresponding location in the virtual equipment model; and when the temperature rises, the heat map visualization layer is updated synchronously. This achieves millisecond-level response and high-fidelity reproduction of the virtual scene to dynamic changes in the physical world, significantly improving the real-time performance, reliability, and human-computer interaction experience of the industrial digital twin platform.

[0100] This embodiment provides a multimodal data processing method. First, multimodal data from an industrial environment is acquired, including sensor data, image data, and text data. Next, temporal features of the sensor data are extracted using a pre-set recurrent neural network to obtain a temporal feature vector. Spatial features of the image data are extracted using a pre-set convolutional neural network to obtain a spatial feature vector. Text data is semantically encoded using a pre-trained language learning model to obtain a text semantic vector. Then, the temporal feature vector, spatial feature vector, and text semantic vector are fused to generate a multimodal fused feature vector. Finally, the dynamic update of the virtual scene is driven based on the multimodal fused feature vector.

[0101] The following technical effects were achieved: By acquiring multimodal data from the industrial environment, and extracting temporal features from sensor data using a pre-set recurrent neural network, extracting spatial structural features from image data using a pre-set convolutional neural network, and semantically encoding text data using a pre-trained language model, the extracted three types of feature vectors were then fused to generate a unified multimodal fusion feature vector. Finally, based on this multimodal fusion feature vector, the dynamic update of the virtual scene was driven, thereby achieving millisecond-level response and high-fidelity virtual-real synchronization on the industrial digital twin platform. This breakthrough overcomes the dependence of traditional digital twin platforms on a single data source and static update mechanism. Through multimodal perception and intelligent fusion, the sensitivity, expressive richness, and real-time update of the virtual scene to dynamic changes in the physical world were significantly improved, thereby enhancing the user experience.

[0102] Figure 2 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 2 In one possible implementation, such as Figure 2 As shown, S105, the temporal feature vector, spatial feature vector, and text semantic vector are fused to generate a multimodal fusion feature vector, including:

[0103] S201. Normalize the temporal feature vector, spatial feature vector, and text semantic vector respectively to obtain normalized temporal feature vector, normalized spatial feature vector, and normalized text semantic vector.

[0104] S202. Concatenate the normalized temporal feature vector, the normalized spatial feature vector, and the normalized text semantic vector into a joint feature vector.

[0105] S203. Input the joint feature vector into a preset multilayer perceptron to obtain a multimodal fusion feature vector.

[0106] Specifically, the server can first normalize the three types of original feature vectors—temporal feature vectors, spatial feature vectors, and text semantic vectors—to eliminate feature bias caused by differences in dimensions, numerical ranges, or distributions between different modalities.

[0107] Specifically, L2 normalization is performed on the temporal feature vector (e.g., a 256-dimensional vector output by LSTM) to obtain a normalized temporal feature vector.

[0108] The spatial feature vector (e.g., the 512-dimensional vector output by the CNN backbone network) is also L2 normalized to obtain the normalized spatial feature vector.

[0109] L2 normalization is also applied to the text semantic vector (e.g., the 768-dimensional vector output by BERT) to obtain a normalized text semantic vector.

[0110] Normalization can ensure that each modality feature has comparability and equal contribution weight in the subsequent fusion process, and avoid high-dimensional or large-value-range features dominating the fusion result.

[0111] Secondly, the server can concatenate the three normalized feature vectors along their respective feature dimensions to form a high-dimensional joint feature vector. For example, if the dimensions of the normalized temporal feature vector, the normalized spatial feature vector, and the normalized textual semantic vector are 256, 512, and 768 respectively, then the dimension of the concatenated joint feature vector is 256 + 512 + 768 = 1536. This joint feature vector fully preserves the original semantic information of the three modalities and constructs a preliminary cross-modal association structure.

[0112] Finally, the server can input the joint feature vector into a pre-defined Multilayer Perceptron (MLP). This pre-defined MLP can contain two fully connected layers. The first layer maps the 1536-dimensional input to a 1024-dimensional hidden layer. The activation function between layers can be a Gaussian Error Linear Unit (GELU) or a Rectified Linear Unit (ReLU) to introduce non-linear expressive power. The second layer further compresses or expands the 1024-dimensional hidden layer to the target dimension (e.g., 512-dimensional or 1024-dimensional). Optionally, a dropout mechanism, such as a dropout rate of 0.2, can be applied to enhance generalization performance. The vector output after processing by the pre-defined MLP is the multimodal fusion feature vector.

[0113] This multimodal fusion feature vector not only preserves the temporal dynamics of sensor data, the spatial structure of image data, and the semantic context of text data, but also achieves cross-modal feature interaction and complementarity through parameter learning within the MLP. This results in the generation of a compact, stable, and semantically rich unified representation, providing high-quality input for subsequent virtual scene updates, fault diagnosis, or decision support.

[0114] Figure 3 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 3 In one possible implementation, such as Figure 3 As shown, S106, based on the multimodal fusion feature vector, drives the dynamic update of the virtual scene, including:

[0115] S301. Divide the virtual scene into a static layer and a dynamic layer.

[0116] S302. Map the multimodal fusion feature vector to the state parameters of industrial equipment.

[0117] S303. Update the visual attributes of the corresponding virtual device in the dynamic layer according to the status parameters.

[0118] In this embodiment, the static layer displays the structural properties of the industrial equipment, while the dynamic layer displays its state properties. State parameters include operating temperature, vibration amplitude, visual appearance, and failure probability. Visual attributes include geometric shape, color, animation effects, and alarm indicators.

[0119] Specifically, the server can first divide the virtual scene in the industrial digital twin platform into static and dynamic layers to achieve a balance between rendering efficiency and expressiveness.

[0120] Static layers are used to display the structural attributes of industrial equipment, including its geometry, fixed components, installation location, and basic material textures—elements that do not change frequently over time. Static layers can be loaded during scene initialization and are only updated when the industrial equipment is physically modified or the model is upgraded, offering high stability and a low update frequency.

[0121] Dynamic layers can be overlaid on static layers and are specifically used to present the status attributes of industrial equipment, such as operating status, health, and environmental interaction effects—information that changes in real time. Dynamic layers can support high-frequency refresh rates and are a key carrier for achieving virtual-real synchronization.

[0122] Secondly, the server can convert the multimodal fusion feature vector into a set of interpretable industrial equipment state parameters through a preset mapping model. This mapping model can be a lightweight regression network, a fully connected neural network, or a rule engine. The output state parameters include, but are not limited to: the operating temperature of the industrial equipment (in degrees Celsius), reflecting the thermal load status of the equipment; vibration amplitude (in millimeters per second), characterizing the smoothness of mechanical operation; visual appearance status (such as discrete labels or damage scores for normal, rust, oil leakage, or deformation); and failure probability (ranging from 0 to 1), representing the probability of functional failure of the industrial equipment within a preset time window.

[0123] Finally, the server can update the visualization attributes of the corresponding virtual devices in the dynamic layer in real time based on the aforementioned status parameters. Specifically, when the operating temperature rises, the color of the heat map of the virtual device's surface material is adjusted (e.g., a gradient from blue, green, to red). When the vibration amplitude exceeds a preset safety threshold, a slight shaking or swaying animation of the virtual device model is triggered to simulate real physical vibration. When the visual appearance indicates oil leakage or corrosion, a semi-transparent stain or corrosion texture is overlaid on the corresponding area of ​​the virtual device. When the failure probability is higher than a preset alarm threshold (e.g., 0.75), a flashing alarm indicator (e.g., a red exclamation mark icon) is displayed above or on the side of the virtual device, and can be linked to sound prompts or log recordings.

[0124] Through the aforementioned layered architecture and parameter-driven mechanism of state parameters, while ensuring efficient rendering of static virtual models, millisecond-level response and high-fidelity visualization of the dynamic state of industrial equipment can be achieved, significantly improving the real-time performance, immersive experience, and operation and maintenance assistance capabilities of the industrial digital twin platform.

[0125] Figure 4 A flowchart illustrating a multimodal data processing method provided in this application embodiment. Figure 4 In one possible implementation, such as Figure 4 As shown, after driving the dynamic update of the virtual scene based on the multimodal fusion feature vector, it also includes:

[0126] S401: The multimodal fusion feature vector is overlaid onto the AR view of industrial equipment through the augmented reality (AR) rendering engine.

[0127] S402. The user's gesture operation is recognized through gesture recognition technology to obtain operation instructions.

[0128] S403. Adjust the content displayed in the AR view according to the operation instructions.

[0129] Specifically, the server can first use an augmented reality (AR) rendering engine to overlay the state parameters mapped by the multimodal fusion feature vectors onto the real AR view of the industrial equipment in the form of visual information.

[0130] Specifically, users can wear AR glasses and use the camera of the AR device to obtain real-time information about the industrial environment. The AR rendering engine can accurately align the digital twin of the corresponding industrial equipment to its physical location based on spatial positioning and industrial equipment recognition technology. The aforementioned status parameters, such as operating temperature values, vibration level icons, fault probability progress bars, and highlighted visual damage areas, are then overlaid in real-time on the top of or on the surface of the corresponding industrial equipment in the AR view in the form of semi-transparent information cards, thermal layers, dynamic labels, or three-dimensional (3D) indicator arrows, to achieve an immersive operation and maintenance experience where what you see is what you get.

[0131] Secondly, user gestures can be recognized in real time using built-in or external gesture recognition modules to generate corresponding operation commands. Gesture recognition technology can be based on depth cameras, structured light sensors, or video frame analysis algorithms, supporting interactive actions including but not limited to finger taps (e.g., selecting specific industrial equipment to expand a detailed status panel), palm panning (e.g., rotating or panning the AR scene view), two-finger zooming (e.g., zooming in / out of a device model to view local details), fist dragging (e.g., moving an information window), and drawing circles in the air (e.g., triggering historical data review of industrial equipment or initiating a diagnostic process).

[0132] The server can convert the recognition results into structured operation instructions through a preset gesture and command mapping table.

[0133] Finally, the content displayed in the AR view is dynamically adjusted based on the user's input. For example, when a user clicks on a specific wind turbine model, the AR interface automatically displays the real-time temperature curve, the three most recent vibration spectrum graphs, and a list of related work orders for that industrial equipment. When the user zooms in, the view switches to a high-resolution local model and overlays an infrared thermal pseudo-color image of the bearing area. When the user makes a gesture to retrieve history, the AR engine loads the animation of the past 24 hours' status evolution from the backend database and plays it back on the real device in a timeline format.

[0134] Through the aforementioned AR overlay, gesture recognition, and dynamic response mechanisms, this implementation not only achieves seamless integration of digital twin information with the physical site, but also provides a hands-free, natural, and intuitive human-computer interaction method, significantly improving the inspection efficiency, fault location speed, and decision-making accuracy of field engineers.

[0135] In one possible implementation, after acquiring multimodal data in an industrial environment, the process further includes: denoising and imputing missing values ​​in the sensor data; performing illumination correction and target region cropping on the image data; and performing word segmentation, stop word filtering, and entity standardization on the text data.

[0136] Specifically, after acquiring multimodal data from an industrial environment, the server can perform targeted preprocessing on the raw multimodal data to improve the accuracy and stability of subsequent feature extraction.

[0137] Specifically, this includes, firstly, denoising and interpolation of missing values ​​in sensor data. Specifically, sensor signals in industrial environments are often affected by electromagnetic interference, line jitter, or aging industrial equipment, easily introducing high-frequency noise or sudden jumps. Therefore, wavelet transform or sliding window midpoint filtering can be used to smooth and denoise the original time-series signal. For missing values ​​caused by communication interruptions or sampling failures, an interpolation strategy is selected based on the data characteristics. If the missing percentage is low (e.g., less than 5%), linear interpolation or forward padding can be used. If continuous missing segments exist, intelligent interpolation can be performed using historical periodic patterns or LSTM-based time-series prediction models to ensure the temporal continuity and numerical rationality of the sensor data.

[0138] Secondly, illumination correction and target region cropping can be performed on the image data. Specifically, images captured by industrial cameras under different lighting conditions (such as strong reflections, low light, and shadow occlusion) may have uneven brightness or contrast distortion. Therefore, adaptive histogram equalization (AHE) can be used for illumination correction to enhance the visibility of image details. Subsequently, the main area of ​​the industrial equipment can be automatically identified based on prior location information (such as CAD model projection coordinates) or through target detection models, and the image can be precisely cropped to remove irrelevant backgrounds (such as walls, walkways, and other equipment), focusing on the key components to be analyzed (such as motor housings, instrument panels, and pipe interfaces), thereby reducing redundant calculations and improving the focus and accuracy of subsequent visual feature extraction.

[0139] Finally, the text data can be tokenized, stop-word filtered, and entity standardized. Specifically, the original text data (such as logs, work orders, speech transcriptions) usually contains unstructured natural language and needs to be linguistically preprocessed. First, a Chinese tokenization tool adapted to the industrial field, such as Jieba combined with a custom device dictionary, can be used for accurate tokenization. Subsequently, stop words such as "de", "le", "jinxing" (which have no actual semantic contribution) are removed. Further, entity standardization is performed on the key terms. For example, "motor overheating", "motor getting hot", and "high motor temperature" are uniformly mapped to a standard fault code, or "Pump 1#", "Pump No. 1", and "PUMP-A" are normalized to the unique device identifier "PUMP-001". This process can be achieved through a pre-built industrial knowledge graph or a rule matching engine to ensure the consistency and computability of the text semantics.

[0140] Through the above multi-modal data preprocessing process, the quality and structuring degree of the original data can be effectively improved, laying a reliable data foundation for subsequent deep learning-based feature extraction and multi-modal fusion, and significantly enhancing the perception accuracy and decision reliability of the entire digital twin system.

[0141] Furthermore, in a possible implementation, the edge computing nodes can be reasonably set near the data source, and the data processing microservices deployed in a containerized manner are used to perform preliminary processing on the original data. These microservices can adopt efficient algorithms and data structures to perform preliminary processing on the original data, including data cleaning, format unification, and timestamp synchronization. For example, during the data cleaning process, an abnormal data detection method based on statistical analysis is used to accurately identify and remove abnormal data; during the format unification process, a standardized data format conversion tool is adopted to ensure the consistency of the data format; and timestamp synchronization is achieved through an accurate clock synchronization mechanism to ensure the time consistency of the data.

[0142] Stream processing technologies such as distributed message queues (Kafka), streaming processing engines (Storm), etc. can also be used to perform real-time analysis on the original data, quickly process the data stream, timely detect the abnormalities and trends in the data, and provide real-time support for decision-making.

[0143] In a possible implementation, it further includes: when the device state represented by the multi-modal fusion feature vector exceeds the preset safety threshold, triggering at least one of the following operations: highlighting the abnormal device in the virtual scene; generating an operation and maintenance work order and pushing it to the management terminal; sending a load reduction control instruction or a shutdown control instruction to the industrial device.

[0144] Specifically, the method also includes an abnormal state response mechanism. When the state parameters of industrial equipment (such as operating temperature, vibration amplitude, failure probability, etc.) represented by the multimodal fusion feature vector exceed the preset safety threshold, the server can automatically trigger at least one of the following emergency or early warning operations to ensure production safety and improve operation and maintenance efficiency.

[0145] First, highlight abnormal equipment. Specifically, in the virtual scene of the industrial digital twin platform, visually enhance the identification of equipment models with abnormal status. For example, switch the virtual equipment's casing material to a striking red or orange; overlay a pulsed flashing border or a rotating alarm icon; display real-time exceeding parameter values ​​(such as "Temperature: 98℃ > Threshold 85℃") above the virtual equipment; and simultaneously link with sound prompts (such as a buzzer effect) to attract the operator's attention. This measure ensures that abnormal information is quickly identified in the 3D scene.

[0146] Second, maintenance work orders are automatically generated and pushed to the management terminal. Specifically, the server can automatically create structured electronic work orders based on the anomaly type, severity level, and device attribution. The content includes: device identifier (ID), anomaly parameter details, occurrence time, multimodal evidence summary (such as image snapshots of the anomaly period, key log fragments, vibration spectrum diagrams), and suggested handling measures (such as checking bearing wear and cleaning heat dissipation channels). This maintenance work order can be pushed in real-time to the designated maintenance personnel's mobile terminal or work order management system via message middleware (such as Kafka) or communication interfaces (such as email interfaces), and supports status recording and closed-loop feedback, achieving automated workflow from anomaly detection to task assignment.

[0147] Third, send load reduction control commands or shutdown control commands to industrial equipment. Specifically, when the abnormal state reaches a high-risk level (such as a failure probability > 0.95 or vibration amplitude exceeding the mechanical safety limit), in order to prevent damage to industrial equipment or safety accidents, the server, after authorization verification, can send remote control commands to the programmable logic controller (PLC) or distributed control system (DCS) controller in the field through industrial communication protocols.

[0148] Specifically, if the accident is controllable, a load reduction control command (such as reducing motor speed or load current) can be sent first to put the industrial equipment into a safe operating mode.

[0149] In case of an emergency, a shutdown control command can be sent directly to cut off the power source and trigger the safety interlock mechanism.

[0150] All control operations can be recorded in the audit log and must meet industrial safety standards to ensure the standardization and traceability of control actions.

[0151] The above response mechanism can be flexibly configured and combined with strategies according to actual application scenarios to form a proactive security protection system with hierarchical early warning, intelligent linkage, and closed-loop handling, thereby significantly improving the security and intelligence level of industrial systems.

[0152] This application embodiment can divide an electronic device or main control device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0153] Figure 5 This is a schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application. Figure 5 As shown, the device includes: an acquisition module 510, a feature extraction module 520, a feature fusion module 530, and an update module 540.

[0154] The acquisition module 510 is used to acquire multimodal data in an industrial environment, including sensor data, image data, and text data.

[0155] The feature extraction module 520 is used to extract the temporal features of sensor data through a preset recurrent neural network to obtain a temporal feature vector.

[0156] The feature extraction module 520 is also used to extract spatial features of image data through a preset convolutional neural network to obtain a spatial feature vector.

[0157] The feature extraction module 520 is also used to semantically encode text data through a pre-trained language learning model to obtain text semantic vectors.

[0158] The feature fusion module 530 is used to fuse temporal feature vectors, spatial feature vectors and text semantic vectors to generate multimodal fusion feature vectors.

[0159] The update module 540 is used to drive the dynamic update of the virtual scene based on the multimodal fusion feature vector.

[0160] In one possible implementation, the feature fusion module 530 is further used to normalize the temporal feature vector, spatial feature vector, and text semantic vector respectively to obtain normalized temporal feature vector, normalized spatial feature vector, and normalized text semantic vector.

[0161] The feature fusion module 530 is also used to concatenate the normalized temporal feature vector, the normalized spatial feature vector, and the normalized text semantic vector into a joint feature vector.

[0162] The feature fusion module 530 is also used to input the joint feature vector into a preset multilayer perceptron to obtain a multimodal fusion feature vector.

[0163] In one possible implementation, the update module 540 is further configured to divide the virtual scene into a static layer and a dynamic layer, wherein the static layer is used to display the structural attributes of the industrial equipment and the dynamic layer is used to display the state attributes of the industrial equipment.

[0164] The update module 540 is also used to map the multimodal fusion feature vector to the state parameters of industrial equipment, including operating temperature, vibration amplitude, visual appearance status and failure probability.

[0165] The update module 540 is also used to update the visual attributes of the corresponding virtual device in the dynamic layer according to the status parameters. The visual attributes include geometric shape, color, animation effect and alarm indicator.

[0166] In one possible implementation, the update module 540 is also configured to overlay multimodal fused feature vectors onto the AR view of the industrial equipment via an augmented reality (AR) rendering engine.

[0167] The update module 540 is also used to recognize the user's gesture operations through gesture recognition technology to obtain operation instructions.

[0168] The update module 540 is also used to adjust the content displayed in the AR view according to operation instructions.

[0169] In one possible implementation, the acquisition module 510 is also used to denoise and impute missing values ​​in the sensor data.

[0170] The acquisition module 510 is also used to perform illumination correction and target region cropping on the image data.

[0171] The acquisition module 510 is also used for word segmentation, stop word filtering and entity standardization of text data.

[0172] In one possible implementation, the update module 540 is further configured to trigger at least one of the following operations when the device status represented by the multimodal fusion feature vector exceeds a preset safety threshold: highlighting the abnormal device in the virtual scene; generating a maintenance work order and pushing it to the management terminal; sending a load reduction control command or a shutdown control command to the industrial equipment.

[0173] This embodiment provides a multimodal data processing device that can execute a multimodal data processing method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0174] In a specific implementation of the aforementioned multimodal data processing device, each module can be implemented as a processor, which can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned multimodal data processing method.

[0175] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device includes at least one processor 610 and a memory 620. The electronic device also includes a communication component 630. The processor 610, memory 620, and communication component 630 are connected via a bus 640.

[0176] In the specific implementation process, at least one processor 610 executes computer execution instructions stored in memory 620, causing at least one processor 610 to execute a multimodal data processing method as executed on the electronic device side as described above.

[0177] The specific implementation process of processor 610 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0178] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0179] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0180] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0181] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0182] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the multimodal data processing method described above.

[0183] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0184] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0185] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in the above embodiments.

[0186] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0187] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal data processing method, characterized in that, include: Acquire multimodal data in an industrial environment, including sensor data, image data, and text data; The temporal features of the sensor data are extracted using a pre-defined recurrent neural network to obtain a temporal feature vector; Spatial features of the image data are extracted using a pre-defined convolutional neural network to obtain a spatial feature vector; The text data is semantically encoded using a pre-trained language learning model to obtain a text semantic vector; The temporal feature vector, the spatial feature vector, and the text semantic vector are fused to generate a multimodal fusion feature vector; The virtual scene is dynamically updated based on the multimodal fusion feature vector.

2. The method according to claim 1, characterized in that, The step of fusing the temporal feature vector, the spatial feature vector, and the text semantic vector to generate a multimodal fusion feature vector includes: The temporal feature vector, the spatial feature vector, and the text semantic vector are normalized respectively to obtain normalized temporal feature vector, normalized spatial feature vector, and normalized text semantic vector; The normalized temporal feature vector, the normalized spatial feature vector, and the normalized text semantic vector are concatenated into a joint feature vector; The joint feature vector is input into a preset multilayer perceptron to obtain the multimodal fusion feature vector.

3. The method according to claim 1, characterized in that, The step of driving the dynamic update of the virtual scene based on the multimodal fusion feature vector includes: The virtual scene is divided into a static layer and a dynamic layer. The static layer is used to display the structural attributes of the industrial equipment, and the dynamic layer is used to display the state attributes of the industrial equipment. The multimodal fusion feature vector is mapped to the state parameters of industrial equipment, including operating temperature, vibration amplitude, visual appearance, and failure probability. Based on the state parameters, update the visualization attributes of the corresponding virtual device in the dynamic layer. The visualization attributes include geometric shape, color, animation effects, and alarm indicators.

4. The method according to claim 1, characterized in that, After driving the dynamic update of the virtual scene based on the multimodal fusion feature vector, the method further includes: The multimodal fused feature vectors are overlaid onto the AR view of the industrial equipment using an augmented reality (AR) rendering engine. Gesture recognition technology is used to identify the user's gestures and obtain operation instructions; Adjust the content displayed in the AR view according to the operation instructions.

5. The method according to claim 1, characterized in that, After acquiring multimodal data from the industrial environment, the process also includes: The sensor data is denoised and missing values ​​are interpolated. The image data is subjected to illumination correction and target region cropping; The text data is processed by word segmentation, stop word filtering, and entity standardization.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: When the device state represented by the multimodal fusion feature vector exceeds a preset security threshold, at least one of the following operations is triggered: Highlight abnormal devices in a virtual scene; Generate maintenance work orders and push them to the management terminal; Send load reduction control commands or shutdown control commands to industrial equipment.

7. A multimodal data processing device, characterized in that, include: The acquisition module is used to acquire multimodal data in an industrial environment, including sensor data, image data, and text data. The feature extraction module is used to extract the temporal features of the sensor data through a preset recurrent neural network to obtain a temporal feature vector; The feature extraction module is also used to extract spatial features of the image data through a preset convolutional neural network to obtain a spatial feature vector; The feature extraction module is also used to perform semantic encoding on the text data through a pre-trained language learning model to obtain a text semantic vector; The feature fusion module is used to fuse the temporal feature vector, the spatial feature vector, and the text semantic vector to generate a multimodal fusion feature vector; The update module is used to drive the dynamic update of the virtual scene based on the multimodal fusion feature vector.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images