Multi-sensor edge system and method based on cross-modal dynamic attention

Abstract

The invention discloses a multi-sensor edge system and method based on cross-modal dynamic attention. The method comprises the following steps: S1, acquiring environment data based on multiple sensors and multi-modal input data based on an interaction module; s2, preprocessing the acquired environment data and the multi-modal input data, and performing feature extraction to obtain multi-modal feature data after feature extraction; s3, carrying out space-time dimension enhancement on the multi-modal feature data; s4, dynamically optimizing the configuration of each modal resource based on the enhanced multi-modal characteristic data obtained in the step S3; and S5, inputting the dynamically optimized modal resource information into the generative model to generate an output result. According to the method provided by the invention, the data processing flow is optimized, the data transmission delay is reduced, efficient integration and utilization of information are realized through a cross-modal dynamic attention mechanism, and the accuracy and robustness of decision making are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing and is a multi-sensor edge system and method based on cross-modal dynamic attention. Background Art

[0002] Although existing large multimodal models have made some progress in fusing multi-source data such as text, images, and speech, they still face major challenges in the fine coordination and complementary use of multimodal information in complex situations. Such models are usually limited to static or predefined modal interaction mechanisms, and it is difficult to dynamically and flexibly implement effective attention allocation between various types of sensor data based on the different needs of real-time input content. This limitation seriously hinders the model from deeply exploring the information associations between cross-modal data, and limits its ability to create high-precision and highly adaptable cross-modal outputs, especially in resource-constrained edge computing environments.

[0003] In actual industrial application scenarios, it is necessary to analyze and integrate massive data from various sensors in real time on edge devices, and to intelligently adjust the flow of attention between modalities to ensure that in complex and changing application scenarios, information that is critical to the task can be accurately captured and enhanced, while promoting the synergy between different modal signals, enhancing the large model's understanding of deep information and the quality of cross-modal content generation. Therefore, it is of great practical significance and application value to explore how to study and design an innovative multi-sensor edge system and method based on cross-modal dynamic attention under the condition of limited computing resources to achieve efficient and intelligent fusion of multi-modal data. Summary of the invention

[0004] The present invention is to solve the problems existing in the above-mentioned prior art and provide a multi-sensor edge computing method based on cross-modal dynamic attention, which is characterized by comprising the following steps: S1. Acquire environmental data based on multiple sensors and multimodal input data based on interaction modules; S2. After preprocessing the acquired environmental data and multimodal input data, feature extraction is performed to obtain multimodal feature data after feature extraction; S3. Enhance the spatiotemporal dimensions of multimodal feature data; S4. Based on the enhanced multimodal feature data obtained in step S3, dynamically optimize the configuration of each modal resource; S5. Input the dynamically optimized modal resource information into the generative model to generate output results.

[0005] Furthermore, the feature extraction method in step S2 is to extract through a cross-modal fusion algorithm based on self-attention and cross-attention.

[0006] Furthermore, the self-attention expression is:

[0007] in X ( i ) is the original feature of mode i, Q ( i )= X ( i ) Wq ( i ), K ( i )= X ( i ) Wk ( i ), V ( i )= X ( i ) Wv ( i ) are the query matrix, key matrix and value matrix respectively, Wq ( i ), Wk ( i ), Wv ( i ) is the corresponding weight matrix, dk is the dimension of the key vector, used to scale the dot product to stabilize the calculation of the softmax function.

[0008] The cross-attention expression of modality i based on modality j is:

[0009] The cross-modal fusion algorithm is:

[0010] in, and is a learnable weight parameter used to balance the importance of self-attention and cross-attention.

[0011] Furthermore, the method for enhancing the spatiotemporal dimension of the fused data in step S3 is to introduce the temporal attention weight g t i and context vector s t Affects modal features and weights, specifically:

[0012]

[0013] in,f t i Indicates i Mode in time t The characteristic representation of w t i For the i Mode in time t The weights assigned to them satisfy , ⊙ represents element-wise multiplication, g t i It can be calculated through the temporal attention mechanism to capture the changing patterns of features over time.

[0014] Furthermore, the method for dynamically optimizing the configuration of each modal resource in step S4 is: Every moment t Aggregate state of all modal eigenvectors

[0015] The action space of all possible sets of modal weight vectors

[0016] Reward Function

[0017] The output is the weight of each mode:

[0018] Among them, NN θ Is with parameters θ The softmax ensures that the weight sum is 1, and the reward function participates in auxiliary training in the process of approximating the optimal policy function training.

[0019] Furthermore, the generative model is the LLaVA-1.5-13B open source multimodal generative large model.

[0020] The present invention also provides a multi-sensor edge system based on cross-modal dynamic attention, which is used in the above-mentioned edge computing method, including: A multi-sensor access module 10, wherein the multi-sensor access module 10 includes a plurality of environmental sensors 100, and is used to obtain different environmental data; A multimedia interaction module 12, the multimedia interaction module 12 comprising a voice transceiver device 121 and a text input device 122, for inputting multimodal data; The large model reasoning edge module 11 includes a plurality of large model reasoning devices 110 , which are used to perform edge computing on the environmental data of the multi-sensor access module 10 and the multi-modal data of the multimedia interaction module 12 .

[0021] Furthermore, the multimedia interaction module 12 also includes a display device for displaying the modal data after edge calculation by the large model reasoning edge module 11.

[0022] The present invention has the following beneficial effects: The present invention provides a multi-sensor edge computing method based on cross-modal dynamic attention, which not only optimizes the data processing process and reduces data transmission delay, but also realizes efficient integration and utilization of information through the cross-modal dynamic attention mechanism, and enhances the accuracy and robustness of decision-making. By deploying the processing logic at the edge, the cloud computing pressure is effectively reduced, the data transmission cost and the risk of privacy leakage are reduced, and a low-latency, high-security multi-modal interactive experience is provided to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a multi-sensor edge computing method for cross-modal dynamic attention of the present invention;

[0024] Figure 2 This is a structural diagram of the multi-sensor edge system for cross-modal dynamic attention of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings.

[0026] Embodiment 1: This embodiment provides a multi-sensor edge computing method for cross-modal dynamic attention, including the following steps: S1. Acquire environmental data based on multiple sensors and multimodal input data based on human-computer interaction; use a high-performance multi-sensor access module to comprehensively collect multi-sensor data from the physical environment; at the same time, the multimedia interaction module acquires rich multimodal input for interaction between the user and the system, including but not limited to text, voice and gesture information. In an embodiment of the present invention, the current production status data is acquired through multi-sensor data, the current production plan is acquired through a text input device, and a temporarily added production plan is input through a voice transceiver device.

[0027] S2. After preprocessing the acquired environmental data and multimodal input data, feature extraction and fusion are performed.

[0028] The large model inference edge module deployed on the edge performs preliminary data preprocessing operations and performs feature extraction. This step aims to extract highly abstract and semantically rich feature representations, and through the cross-modal fusion algorithm, ensure the effective integration and complementarity of various modal information, laying a solid foundation for subsequent analysis; the cross-modal fusion algorithm used combines self-attention and cross-attention to adjust modal features and obtain enhanced feature representation. For modality i, self-attention can be expressed as:

[0029] in, Q ( i )= X ( i ) Wq ( i ), K ( i )= X ( i ) Wk ( i ), V ( i )= X ( i ) Wv ( i ) are the query matrix, key matrix and value matrix, which are obtained by X ( i ) and the corresponding weight matrix Wq ( i ), Wk ( i ), Wv ( i ) dot multiplication; dk is the dimension of the key vector, used to scale the dot product to stabilize the calculation of the softmax function.

[0030] The cross-attention mechanism is used to capture the interaction between different modalities. If the attention of modality i based on modality j is expressed as:

[0031] Use additional learning weights to complete the fusion of the two

[0032] and is a learnable parameter used to balance the importance of self-attention and cross-attention. Through this process, modality i can more effectively utilize modality j The contextual information provided improves the performance in cross-modal tasks.

[0033] S3. Enhance the spatiotemporal dimensions of the data fused in step S2.

[0034] The large model reasoning edge module integrates the system into time series analysis and environmental context understanding functions, adds dynamic time dimensions and environmental adaptability parameters to the pre-processed data, and enhances the model's accurate analysis and in-depth insight into data in complex dynamic scenarios and specific situations.

[0035] The specific approach is as follows: Introducing temporal attention weight g t i and context vector s t Influencing modal features and weights:

[0036]

[0037] in, f t i Indicates i Mode Z (i) enhanced In time t The characteristic representation of w t i For the i Mode in time t The weights assigned to them satisfy , ⊙ represents element-wise multiplication, g t i It can be calculated through the temporal attention mechanism to capture the changing pattern of features over time. There are different fixed context vectors in different contexts. s t , which can be obtained through training. πθ is the weight distribution parameter used for dimensionality reduction.

[0038] S4. Based on the enhanced multimodal feature data obtained in step S3, dynamically optimize the configuration of each modal resource.

[0039] Based on the comprehensive feature representation and real-time task requirements obtained in the first two steps, the large model inference edge module implements advanced strategy decisions and dynamically optimizes the configuration of each modal resource. This intelligent scheduling mechanism ensures the efficient use of computing resources and optimizes the information processing flow for the current task. The specific approach is as follows: Assume that the multimodal data consists of M modes, and at each moment t Status St is the aggregation of all modal eigenvectors and can be expressed as:

[0040] in, f t i Indicates i Mode in time t The feature representation of .

[0041] Define the set of all possible modal weight vectors as the action space At Right now:

[0042] in, w t i For the i Mode in time t The weights assigned to them satisfy Design the reward function based on the performance indicators of the model on the current task (such as accuracy, loss function value, etc.) Rt :

[0043] Using neural networks to approximate the optimal policy function πθ ( St ), the output is the weight of each mode:

[0044] Among them, NN θ Is with parameters θ The softmax ensures that the weight sum is 1. Finally, an optimal w t i Weighted combination.

[0045] S5. Input the dynamically optimized modal resource information into the generative model to generate output results.

[0046] The large model reasoning edge module inputs the strategically optimized information into a high-performance generative model, which synthesizes high-quality, multi-dimensional output results based on the current context, achieving a creative transformation from data to meaningful multimodal content. In an embodiment of the present invention, the LLaVA-1.5-13B open source multimodal generative large model after fine-tuning and quantization is used as a benchmark to generate high-quality production scheduling change results, including inventory adjustments, production line adjustments, and personnel arrangements.

[0047] After the calculation is completed, the above algorithm can also be responsible for displaying the multimodal results generated by the large model reasoning edge module through the multimedia interaction module in real time on a suitable human-computer interaction device through an intuitive and user-friendly interface, ensuring a seamless and highly immersive user experience.

[0048] Embodiment 2: This embodiment provides a multi-sensor edge with cross-modal dynamic attention, which is used to implement the edge computing method of embodiment 1, including: Multi-sensor access module 10, multi-sensor access module 10 and multimedia interaction module 12, multi-sensor access module 10 includes multiple different environmental sensors 100, used to obtain different environmental data, such as production line operation status sensor, inventory quantity sensor, abnormal condition monitoring camera, temperature and humidity sensor. Multimedia interaction module 12 includes voice transceiver device 121, text input device 122, used to input multi-modal data, including but not limited to text, voice and gesture information.

[0049] The large model reasoning edge module 11 includes a plurality of large model reasoning devices 110, which are used to perform edge computing on the environmental data of the multi-sensor access module 10 and the multi-modal data of the multimedia interaction module 12. In the embodiment, a single-card RTX4090 computing box is used to perform edge large model reasoning.

[0050] The multimedia interaction module 12 may also include a display device for displaying the modal data after edge calculation by the large model inference edge module 11.

Claims

1. A multi-sensor edge computing method based on cross-modal dynamic attention, characterized in that: The steps include: S1. Acquire environmental data based on multiple sensors and multimodal input data based on human-computer interaction; S2. After preprocessing the acquired environmental data and multimodal input data, feature extraction and fusion are performed; S3. Enhance the spatiotemporal dimension of the data fused in step S2; S4. Based on the enhanced multimodal feature data obtained in step S3, dynamically optimize the configuration of each modal resource; S5. Input the dynamically optimized modal resource information into the generative model to generate output results.

2. According to claim 1, a multi-sensor edge computing method based on cross-modal dynamic attention is characterized in that: The feature extraction and fusion method described in step S2 is to extract through a cross-modal fusion algorithm based on self-attention and cross-attention.

3. A multi-sensor edge computing method based on cross-modal dynamic attention according to claim 2, characterized in that: The self-attention expression is: ； in X ( i ) is the original feature of mode i, Q ( i )= X ( i ) Wlq ( i ), K ( i )= X ( i ) Wk ( i ), V ( i )= X ( i ) W ( i ) are the query matrix, key matrix and value matrix respectively, Wlq ( i ), Wk ( i ), W ( i ) is the corresponding weight matrix, dk is the dimension of the key vector, used to scale the dot product to stabilize the calculation of the softmax function; The cross-attention expression of modality i based on modality j is: ； The cross-modal fusion algorithm is: ； in, and is a learnable weight parameter used to balance the importance of self-attention and cross-attention.

4. The multi-sensor edge computing method based on cross-modal dynamic attention according to claim 1, characterized in that: The method for enhancing the spatiotemporal dimension of the fused data in step S3 is to introduce the temporal attention weight g t i and context vector s t Influence modal features and weights, specifically: ； ； in, f t i Indicates i Mode in time t The characteristic representation of w t i For the i Mode in time t The weights assigned to them satisfy , ⊙ represents element-wise multiplication, π θ Assign parameters to weights.

5. The multi-sensor edge computing method based on cross-modal dynamic attention according to claim 1, characterized in that: The method for dynamically optimizing the configuration of each modal resource in step S4 is: Calculate each moment t Aggregate state of all modal eigenvectors: ； Get the action space of all possible sets of modal weight vectors: ； Set the reward function: ； The output is the weight of each mode: ； Among them, NN θ With parameters θ In a neural network, softmax ensures that the sum of the weights is 1.

6. The multi-sensor edge computing method based on cross-modal dynamic attention according to claim 1, characterized in that: The generative model is the LLaVA-1.5-13B open source multimodal generative large model.

7. A multi-sensor edge system based on cross-modal dynamic attention, used to implement the method according to any one of claims 1 to 6, characterized in that: include A multi-sensor access module (10), the multi-sensor access module (10) comprising a plurality of environmental sensors (100) for acquiring different environmental data; A multimedia interaction module (12), the multimedia interaction module (12) comprising a voice transceiver device (121) and a text input device (122), and being used for inputting multimodal data; A large model reasoning edge module (11), the large model reasoning edge module (11) comprising a plurality of large model reasoning devices (110) for performing edge computing on environmental data of a multi-sensor access module (10) and multi-modal data of a multimedia interaction module (12).

8. The multi-sensor edge system based on cross-modal dynamic attention according to claim 7, characterized in that: The multimedia interaction module (12) also includes a display device for displaying the modal data after edge calculation by the large model inference edge module (11).

Images