Application method and application device of Transform-based large-scale pre-training model in intelligence analysis and decision support
Through large-scale pre-trained models and multi-sensor fusion technology based on Transformer, the difficulties of multimodal data fusion and analysis are solved, efficient intelligence analysis and decision-making support are achieved, and the real-time and accuracy of the system are improved.
Patent Information
- Application Number
- CN202411129029.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing Transformer model is mainly used for the processing of single modal data, and it is difficult to efficiently integrate and analyze multimodal data, resulting in limited intelligence analysis efficiency and accuracy, and there are problems such as insufficient computing resources, data transmission delay and poor real-time performance.
Using a large-scale pre-trained model based on Transformer, multiple types of data are collected through multi-sensor fusion technology, preprocessing and feature extraction, and fusion and analysis of multimodal data is used by self-attention mechanism and multi-head attention mechanism, and combined with The edge computing and backend server work together to achieve real-time intelligence analysis and decision-making support.
It significantly improves the processing efficiency of intelligence data and the accuracy of decision support, reduces costs, enhances the flexibility and security of the system, and can quickly process and analyze complex intelligence data, suitable for changing intelligence environments.
Smart Images

Figure BDA0004997294820000101 
Figure BDA0004997294820000111 
Figure BDA0004997294820000121
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of artificial intelligence, machine learning and intelligence analysis, and in particular to an application method and an application device of a large-scale pre-training model based on Transformer in intelligence analysis and decision support. Background Art
[0002] With the rapid development of information technology, intelligence analysis and decision support systems are increasingly being used in various fields. Traditional intelligence analysis systems mainly rely on the processing and analysis of single-modal data, which makes it difficult to fully utilize the complementary information between different types of data, resulting in limited efficiency and accuracy of intelligence analysis. In addition, traditional systems often face problems such as insufficient computing resources, data transmission delays, and poor real-time performance when processing large-scale data.
[0003] In recent years, deep learning technology has made significant progress in computer vision, natural language processing, and data analysis. In particular, pre-trained models represented by Transformer have significantly improved the model's ability to handle complex data tasks through technologies such as self-attention mechanism and multi-head attention mechanism. However, the existing Transformer model is mainly used for processing single-modal data. How to expand its application to the fusion and analysis of multi-modal data is still a technical problem that needs to be solved urgently.
[0004] Multimodal data fusion technology can provide more comprehensive and accurate information by integrating data from different sensors. However, the differences in spatiotemporal distribution and data characteristics of different modal data bring challenges to data synchronization and fusion. In addition, existing technologies also have many shortcomings in real-time processing and decision support of multimodal data, such as high data transmission delay, poor real-time performance, and low processing efficiency.
[0005] Therefore, there is an urgent need for an intelligence analysis and decision support system that can efficiently integrate multimodal data and use an improved Transformer model for real-time processing and analysis, so as to improve the processing efficiency of intelligence data and the accuracy of decision support and meet the application needs in various complex intelligence environments. Summary of the invention
[0006] The present invention proposes a method and device for applying a large-scale pre-trained model based on Transformer in intelligence analysis and decision support. The method collects various types of intelligence data, including video, image, text and sensor data, and pre-processes and extracts features from these data. The improved Transformer model is used to train and analyze the pre-processed multimodal data, thereby realizing real-time processing of intelligence data and decision support.
[0007] Specifically, in the technical solution of the system of the present invention, its operation logic includes the following steps:
[0008] Multimodal data acquisition:
[0009] Data sources: The system collects intelligence data through devices such as cameras, infrared sensors, lidars, and microphones. These devices can be installed on drones or fixed monitoring devices to capture high-resolution videos, images, thermal imaging data, 3D point cloud data, and environmental audio. Through the combination of multiple devices, comprehensive intelligence information can be obtained.
[0010] Multi-sensor fusion technology: Multi-sensor fusion technology is used to achieve high-precision capture and synchronization of data. The complementarity of sensor data is utilized to perform preliminary processing through edge computing devices to reduce data transmission delays and ensure the real-time and integrity of data. This technology can also effectively filter noise and improve data reliability.
[0011] Time synchronization: The system uses high-precision clocks to synchronize the data of different devices to ensure the consistency of various types of data in time. Through timestamp alignment, various data can be accurately mapped to the same time point, providing high-quality basic data for subsequent processing.
[0012] Data security: During data collection and transmission, the system uses the Advanced Encryption Standard (AES) and Transport Layer Security (TLS) protocols to encrypt data to ensure data security and integrity and prevent data from being stolen or tampered with during transmission.
[0013] Data preprocessing:
[0014] Data cleaning and denoising: Clean the collected multimodal data to remove noise and irrelevant information. Specific methods include median filtering, Gaussian filtering, etc., to denoise the image and point cloud data to improve the clarity and accuracy of the data. Text data is preprocessed through technologies such as word segmentation and stop word removal.
[0015] Format conversion: Convert data from different sensors into a unified format, such as at least one of PEG, PNG, PLY, and WAV, to facilitate subsequent data fusion and analysis. Use tools such as Numpy and Pandas to generate feature matrices to provide high-quality data input for the Transformer model, ensuring that the model can fully learn the characteristics of the data.
[0016] Multimodal matrix information merging: The feature matrices of different modalities are spliced in the feature dimension to form a high-dimensional comprehensive feature matrix. This comprehensive feature matrix contains the key features of all modal data, providing rich and complete data input for subsequent deep learning models.
[0017] Convolution processing: Convolution processing is performed on the preprocessed data to reduce the volume of the data matrix and reduce the amount of calculation. The image after convolution processing is segmented, but the coordinate information of the image is retained so that the subsequent model can understand the spatial information. The point cloud data is also processed for dimensionality reduction to retain key features.
[0018] Horizontal encoding arrangement: Positional encoding technology is used to horizontally encode and arrange the convolutional data, and then input it into the Transformer neural network. Positional encoding ensures that the Transformer model can effectively understand and process the spatiotemporal information in the data, improving the processing ability of the model.
[0019] Multimodal Transformer model training and application:
[0020] Feature extraction and fusion: Extract and fuse high-level features from multimodal data through the self-attention mechanism. The self-attention mechanism can capture important information in the data and fuse data of different modalities. The multi-head attention mechanism is used in the feature fusion process to enhance the model's understanding of the complex relationship between different data modalities and improve the overall performance of the model.
[0021] Model training: Use pre-processed multimodal data for supervised learning, define the loss function and use optimization algorithms (such as Adam) to train the model and update the model parameters. During the training process, a strategy combining pre-training and fine-tuning is adopted to improve the generalization ability and application effect of the model. The pre-trained model can effectively use large-scale data to improve the initial performance of the model, while fine-tuning is optimized for specific application scenarios.
[0022] Model application: Input new data into the trained model to generate intelligence analysis results. In the model application stage, dynamic adjustment technology is used to adaptively adjust model parameters according to actual conditions to ensure the accuracy and adaptability of the model. Through real-time monitoring and feedback mechanisms, the model can be guaranteed to operate efficiently in different environments.
[0023] Real-time intelligence analysis:
[0024] Edge computing: handles simple intelligence analysis tasks and returns decision suggestions to people and other terminal devices in JSON format. Edge computing devices have preliminary analysis and filtering capabilities to ensure low latency and efficient response. Edge computing can also reduce the burden on central servers and improve the overall response speed of the system.
[0025] Complex decision analysis: complex decisions are handed over to the backend server for large-scale computing, and in-depth analysis is performed using large-scale pre-trained models at the trillion level. Prompt optimization, RAG (Retrieval-Augmented Generation) technology and knowledge graph are combined to improve the accuracy and richness of decision analysis. The backend server has powerful computing power, can handle complex analysis tasks, and provide detailed intelligence analysis results.
[0026] Intelligence Generation and Distribution:
[0027] Report generation: Based on the results of intelligence analysis, intelligence reports are automatically generated, including text descriptions, charts, and key indicators. The report generation process integrates natural language generation (NLG) technology to ensure the readability and accuracy of the report content. Reports can be customized according to user needs and are divided into visual reports for human reading and JSON format reports for terminal devices to meet the needs of different scenarios.
[0028] Report distribution: Reports are distributed to relevant decision-makers and commanders through secure communication channels, and JSON format reports are sent to relevant terminal devices through the function calling interface (Function Calling) to achieve automated and intelligent intelligence distribution. The system supports multiple communication protocols and security encryption technologies to ensure the security and reliability of data transmission. Through a flexible distribution mechanism, it ensures that intelligence information can be delivered to each required terminal in a timely and accurate manner.
[0029] Through the above system, the present invention significantly improves the efficiency and accuracy of intelligence analysis, reduces costs, and enhances the flexibility and security of the system. This method can quickly process and analyze a large amount of complex intelligence data, is applicable to various complex and changeable intelligence environments, and provides efficient and accurate decision support. The system's multimodal data fusion and deep learning technology enable it to perform well in processing high-dimensional and diversified data, significantly improving the overall performance of intelligence analysis and decision support. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0031] Figure 1 The figure is a schematic diagram of the overall architecture of the system of the present invention. The figure shows the entire process from data collection, preprocessing, model training and application to real-time intelligence analysis and intelligence generation and distribution.
[0032] Figure 2 It is a detailed structural diagram of the multimodal data acquisition module. The diagram illustrates the arrangement and data synchronization mechanism of multiple data sources (such as cameras, infrared sensors, lidar, and microphones) and edge computing devices.
[0033] Figure 3 This is a detailed flow chart of the data preprocessing module. The figure shows the specific steps of data cleaning and denoising, format conversion, convolution processing, and horizontal coding arrangement.
[0034] Figure 4 This is a detailed structural diagram of the multimodal Transformer model training and application module. This diagram illustrates the specific methods and steps of feature extraction and fusion, model training, and model application.
[0035] Figure 5 This is a functional diagram of the real-time intelligence analysis module. This diagram shows how edge computing devices handle simple intelligence analysis tasks and how backend servers perform complex decision analysis.
[0036] Figure 6 This is a flowchart of the intelligence generation and distribution module. This diagram illustrates the specific steps of automatically generating reports based on intelligence analysis results and distributing reports through secure communication channels. DETAILED DESCRIPTION
[0037] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0038] Figure 1 The overall system architecture of the application method of large-scale pre-trained models based on Transformer in intelligence analysis and decision support is demonstrated. The following is a detailed description of each module and its implementation method:
[0039] Data acquisition module
[0040] Data source:
[0041] The system collects intelligence data through a variety of sensor devices, including cameras, infrared sensors, lidar and microphones.
[0042] These devices can be mounted on drones or fixed surveillance installations to capture high-resolution video, images, thermal imaging data, 3D point cloud data, and ambient audio.
[0043] Multi-sensor fusion technology:
[0044] The system uses multi-sensor fusion technology to achieve high-precision capture and synchronization of data.
[0045] After the data from each sensor is collected, it is synchronized through timestamps and spatial locations to ensure the consistency of the data in time and space.
[0046] Initial processing of edge computing devices:
[0047] After data collection, it is initially processed by edge computing devices to reduce data transmission delays and ensure the real-time and integrity of the data.
[0048] Preliminary processing includes data denoising, filtering and preliminary analysis.
[0049] Data Encryption:
[0050] During data collection and transmission, the system uses the Advanced Encryption Standard (AES) and Transport Layer Security (TLS) protocols to encrypt data to ensure data security and integrity.
[0051] Data preprocessing module
[0052] Data cleaning and denoising:
[0053] The collected data is cleaned to remove noise and irrelevant information.
[0054] Median filtering and Gaussian filtering are used to denoise image and point cloud data to improve data clarity and accuracy.
[0055] Format conversion:
[0056] Convert data from different sensors into a unified format, which should be at least one of PEG, PNG, PLY, and WAV.
[0057] Use Numpy and Pandas to generate feature matrices to provide high-quality data input for the Transformer model.
[0058] Multimodal matrix information merging:
[0059] The feature matrices generated by different sensors are merged to form a comprehensive feature matrix.
[0060] Multimodal data are uniformly arranged and aligned, and data fusion is achieved through matrix operations to ensure the consistency of different modal data in time and space.
[0061] Convolution processing:
[0062] The preprocessed data is convolved to reduce the volume of the data matrix and the amount of calculation.
[0063] The image after convolution is segmented, but the coordinate information of the image is retained for subsequent processing.
[0064] Position encoding technology:
[0065] Positional Encoding technology is used to encode and arrange the convolutional data horizontally, and then input it into the Transformer neural network.
[0066] Positional encoding ensures that the Transformer model can effectively understand and process the spatial and temporal information in the data.
[0067] Multimodal Transformer model training and application module
[0068] Feature extraction and fusion:
[0069] Extract and fuse high-level features from multimodal data through the Self-Attention Mechanism.
[0070] The multi-head attention mechanism is used to enhance the model's understanding of the complex relationships between different data modalities and improve the overall performance of the model.
[0071] Model training:
[0072] Use the preprocessed multimodal data for supervised learning, define the loss function and use the optimization algorithm (such as Adam) to train the model and update the model parameters.
[0073] During the training process, a strategy combining pre-training and fine-tuning is adopted to improve the generalization ability and application effect of the model.
[0074] Model Application:
[0075] Input new data into the trained model to generate intelligence analysis results.
[0076] Dynamic adjustment technology is used to adaptively adjust model parameters according to actual conditions to ensure the accuracy and adaptability of the model.
[0077] Real-time intelligence analysis module
[0078] Edge computing (mobile):
[0079] Process simple intelligence analysis tasks and return decision recommendations in JSON format to people and other terminal devices.
[0080] Edge computing devices have preliminary analysis and filtering capabilities to ensure low latency and efficient response.
[0081] Real-time intelligence analysis (server side):
[0082] Complex decisions are delegated to backend servers for large-scale computing, and in-depth analysis is performed using trillion-level large-scale pre-trained models.
[0083] Combining Prompt optimization, RAG (Retrieval-Augmented Generation) technology and Knowledge Graph, the accuracy and richness of decision analysis are improved.
[0084] Intelligence generation and decision distribution module
[0085] Report Generation:
[0086] Based on the intelligence analysis results, intelligence reports are automatically generated, including text descriptions, charts and key indicators.
[0087] The report generation process integrates natural language generation (NLG) technology to ensure the readability and accuracy of the report content.
[0088] Report Distribution:
[0089] The reports are distributed to relevant decision makers and commanders through secure communication channels, and the JSON format reports are sent to relevant terminal devices through the function calling interface (Function Calling), realizing automated and intelligent intelligence distribution.
[0090] The system supports multiple communication protocols and security encryption technologies to ensure the security and reliability of data transmission.
[0091] Figure 2 The detailed structure of the multimodal data acquisition module is shown, including the various sensors, mobile devices, edge computing devices, data synchronization and data encryption processes. The following is a detailed description of each component and its implementation method:
[0092] Data acquisition module
[0093] Camera (high resolution images and video):
[0094] Use high-resolution cameras to capture video and still images. The camera is mounted on a mobile device (such as a drone) or a fixed surveillance device. Video data and image data are captured by the camera, providing rich visual information.
[0095] The acquired image data can be stored in standard JPEG or PNG format, and the video data can be stored in standard MP4 or AVI format.
[0096] LiDAR (3D point cloud data):
[0097] Use LiDAR to capture 3D point cloud data. LiDAR provides highly accurate spatial information by emitting laser beams and receiving reflected signals to measure distances. LiDAR data is used to build a 3D environment model, providing detailed spatial structure information.
[0098] The collected point cloud data can be stored in PLY or LAS formats, which can efficiently represent three-dimensional point cloud data.
[0099] Infrared sensor (thermal imaging data):
[0100] Capture thermal imaging data using infrared sensors. Infrared sensors can detect the temperature distribution on the surface of an object and generate thermal imaging images. Thermal imaging data is used to identify the temperature characteristics of the target object, which is especially important at night or in low-visibility environments.
[0101] Thermal imaging data is usually stored in TIFF format, which can effectively preserve the thermal information of the image.
[0102] Microphone (ambient audio):
[0103] Use microphones to capture ambient audio data. Microphones are installed on mobile devices or fixed devices and can collect sound information from the surrounding environment. Audio data is used to analyze ambient sound characteristics and identify specific sound sources or events.
[0104] Audio data can be stored using WAV or MP3 formats, which can save audio information efficiently.
[0105] Mobile device (data collection):
[0106] The above sensors are installed on mobile devices (such as drones) to collect data. Mobile devices can move flexibly in different environments, expanding the scope and flexibility of data collection.
[0107] The use of mobile devices allows the system to cover a wide area and collect data in more dimensions.
[0108] Preliminary data processing
[0109] Edge computing devices (preliminary data processing):
[0110] During the data collection process, edge computing devices perform preliminary processing on the data. Edge computing devices are installed on mobile devices and can process the collected data in real time. Preliminary processing includes data denoising, filtering, and preliminary analysis to reduce data transmission delays and ensure the real-time and integrity of the data.
[0111] An important step in preliminary processing is data cleaning and denoising, usually using median filtering and Gaussian filtering. The median filter retains the edge features of the image while removing noise. The Gaussian filter reduces the impact of noise through smoothing.
[0112] Data synchronization (high-precision clock):
[0113] Use high-precision clocks to synchronize the data from different sensors. Ensure the consistency of various types of data in time and provide high-quality basic data for subsequent fusion processing. Time synchronization enables data from different sensors to be processed within the same time frame by aligning timestamps.
[0114] Assume that the data collection time of each sensor is t v , t l , t i , t m These times are aligned to a unified time t by time synchronization s Then we have:
[0115]
[0116] This time synchronization method ensures that all sensor data are consistent at the same time point, providing a basis for subsequent multimodal data fusion.
[0117] Data encryption (AES, TLS):
[0118] During data collection and transmission, the system uses the Advanced Encryption Standard (AES) and Transport Layer Security (TLS) protocols to encrypt data. Data encryption ensures the security and integrity of data and prevents data from being stolen or tampered with during transmission.
[0119] The AES encryption algorithm is a symmetric encryption algorithm, the core idea of which is to use a key to encrypt and decrypt data. The TLS protocol is used to protect the security of data during transmission, and through encryption and authentication mechanisms, it ensures the confidentiality and integrity of data during network transmission.
[0120] Output preliminary data:
[0121] The data after preliminary processing, synchronization and encryption are integrated and output to provide basic data for subsequent processing and analysis. The output preliminary data includes processed high-resolution images, videos, 3D point cloud data, thermal imaging data and environmental audio data.
[0122] These preliminary data will be further processed to extract useful features and information, providing high-quality data input for subsequent model training and application.
[0123] Figure 3 The detailed process of the data preprocessing module is shown, including data cleaning and denoising, format conversion, multimodal data merging, convolution processing and position encoding technology. The following is a detailed description of each step and its implementation method:
[0124] Data preprocessing module
[0125] Data cleaning and denoising:
[0126] Clean and denoise the collected multimodal data to remove noise and irrelevant information. Common methods include median filtering and Gaussian filtering.
[0127] Median filtering: Median filtering is a nonlinear filtering technique that can effectively remove salt and pepper noise by replacing the value of each pixel with the median of the neighboring pixel values while retaining the edge information of the image. Let x[i, j] represent the value of the i, jth pixel in the image. The value y[i, j] after median filtering can be expressed as:
[0128] y[i,j]=median{x[i+k,j+l]-m≤k,l≤m}
[0129] Gaussian filtering: Gaussian filtering can smooth the image and reduce high-frequency noise by performing a convolution operation on the image and taking Gaussian distribution as the weight function to perform weighted averaging on the pixels. The convolution kernel G of the Gaussian filter is defined as:
[0130]
[0131] G(x, y): This is the output value of the Gaussian function, which represents the weight at the xXX and yyy coordinate positions of the image. This weight is used in the convolution operation and determines the importance of the corresponding pixel in the smoothing process.
[0132] x, y: These are spatial coordinates in the image that determine the location of the convolution kernel relative to the image. They represent the distance between the currently considered pixel and the center of the Gaussian kernel.
[0133] σ: This is the standard deviation of the Gaussian distribution and determines the width of the Gaussian kernel. A larger σ value will produce a wider distribution, which makes the Gaussian filter have a stronger smoothing effect on the image. Conversely, a smaller σ value will produce a narrower distribution, resulting in a sharper image.
[0134] exp: This is the exponential function, which represents the core part of the Gaussian function and is used to calculate the value at the spatial coordinates (x, y).
[0135] 2πσ 2 : This is the normalization coefficient of the Gaussian function, ensuring that the area of the entire Gaussian kernel is 1, thereby keeping the total intensity of the image unchanged during the convolution process.
[0136] Format conversion:
[0137] Convert data from different sensors into a unified format, such as at least one of PEG, PNG, PLY, and WAV, to facilitate subsequent data fusion and analysis.
[0138] Use Numpy and Pandas to generate feature matrices to provide high-quality data input for subsequent Transformer models.
[0139] Feature matrix generation: Through format conversion, images, videos, point clouds, and audio data are converted into operable numerical matrices. These feature matrices will contain the key features of various modal data to ensure the consistency of data formats.
[0140] Multimodal data merging:
[0141] The feature matrices generated by different sensors are combined to form a comprehensive feature matrix. Specific methods include splicing and weighted averaging.
[0142] Feature matrix integration: The feature matrices of different modalities are spliced in the feature dimension to form a high-dimensional comprehensive feature matrix. For example, suppose the video feature matrix is F v , the image feature matrix is F i , the thermal imaging feature matrix is F t , the point cloud feature matrix is F p , the audio feature matrix is F a , then the comprehensive feature matrix F c It can be expressed as:
[0143] F c =[F v , F i , F t , F p , F a ]
[0144] Weighted average: The features of different modes are weighted averaged, taking into account the importance and data quality of different sensors. Assume that the weights of the features of each mode are w v , w i , w t , w p , w a , then the comprehensive feature matrix F c It can be expressed as:
[0145]
[0146] F c =w v ·F v +w i ·F i +w t ·F t +w p ·F p +w a ·F a ,
[0147] Unified arrangement and alignment: Unified arrangement and alignment of multimodal data to ensure the consistency of all data in time and space. Use timestamps and spatial position coordinates for alignment so that data from different sensors can be processed in the same time and space framework. Assume that the data collection time of each sensor is t v , t l , t i , t m Through time synchronization, these times are aligned to the same time t s Then we have:
[0148] Convolution processing:
[0149] Convolution is performed on the preprocessed data to reduce the volume of the data matrix and the amount of calculation while retaining important spatial information.
[0150] Convolution operation: Convolutional neural network (CNN) is used to perform convolution processing on image and point cloud data to extract spatial features. Convolution operation can effectively reduce the data dimension and extract high-level features in the data. Assuming the convolution kernel is K, the input data is I, and the output after convolution is O, then:
[0151] O[i, j] = ∑ m ∑ n I[i+m,j+n]·K[m,n]
[0152] O[i, j]: This is the pixel value of the output image at position (i, j). This value is calculated by the convolution operation between the input image and the convolution kernel.
[0153] ∑ m ∑ n : This is a double summation symbol, which represents the sum of all values of m and n. Here m and n usually represent the size range of the convolution kernel K, such as from -1 to 1, or other defined ranges. I[i+m, j+n]: This is the pixel value of the input image at position I at position (i+m, j+n). By translating the offset of m and n in the image, the pixel values at different positions can be obtained. K[m,n]: This is the weight of the convolution kernel at position (m,n). The convolution kernel K is a small matrix used to extract local features of the input image.
[0154] Preserve coordinate information: While performing convolution processing, the coordinate information of the image is retained to facilitate subsequent spatial information processing.
[0155] Position encoding technology:
[0156] Positional Encoding technology is used to encode and arrange the convolutional data horizontally, and then input it into the Transformer neural network.
[0157] Position encoding: Position encoding is to introduce position information into data features, so that the Transformer model can effectively understand and process the spatiotemporal information in the data. Specifically, by adding a position encoding vector to each data point, the model can distinguish features at different positions. Assuming the position encoding vector is P and the data feature is F, the encoded data is:
[0158] F′=F+P
[0159] Coding arrangement: Arrange the data with positional encoding horizontally to form an input sequence, ensuring that the temporal and spatial structural information of the data is preserved.
[0160] Through the above steps, the data preprocessing module realizes the cleaning, conversion, merging and convolution of multimodal data, providing high-quality input data for subsequent Transformer model training and application. This not only improves the efficiency of data processing, but also enhances the generalization ability and application effect of the model.
[0161] Figure 4 The detailed process of the multimodal Transformer model training and application module is shown, including the process of feature extraction and fusion, model training and model application. The following is a detailed description of each step and its implementation method:
[0162] Multimodal Transformer model training and application module
[0163] Feature extraction and fusion:
[0164] Self-Attention Mechanism: The self-attention mechanism is the core of the Transformer model. It extracts important feature information by calculating the correlation between each position in the input sequence and other positions. Suppose the input sequence is X = {x 1 , x 2 , ..., x n}, the attention weight matrix A of each position is calculated through the self-attention mechanism, and its calculation formula is:
[0165]
[0166] Where Q represents the query matrix, K represents the key matrix, and d k is the dimension of the key matrix and A is the attention weight matrix.
[0167] Multi-Head Attention: The multi-head attention mechanism captures multiple features in the input data by computing multiple attention heads in parallel. Each attention head is computed independently, and then the results are concatenated together and linearly transformed. Suppose there are h attention heads, and the output of each attention head is O i , the output of the multi-head attention mechanism is:
[0168] O=[O 1 , O 2 , ..., O h ]W O
[0169] Among them, [O 1 , O2 , ..., O h ] represents the concatenation of the outputs of all attention heads, W O is the linear transformation matrix.
[0170] Extract and fuse high-level features from multimodal data: Using the above attention mechanism, key features are extracted from multimodal data and fused. By inputting different modal data into the Transformer model, high-level fusion features are extracted.
[0171] Model training:
[0172] Supervised learning with processed multimodal data: Perform supervised learning on the preprocessed multimodal data, define the loss function and use the optimization algorithm to train the model.
[0173] Define loss function: Common loss functions include cross entropy loss and mean square error. Suppose the predicted value is The true value is y, the loss function It can be expressed as:
[0174]
[0175] Where N is the number of samples.
[0176] Use optimization algorithms for training: Common optimization algorithms include Adam and SGD. The optimization algorithm iteratively updates the model parameters to minimize the loss function. The parameter update formula of the Adam optimization algorithm is:
[0177] m t =β 1 m t-1 +(1-β 1 ) t
[0178]
[0179]
[0180] Among them, g t is the gradient, m t and t They are the first-order moment estimate and the second-order moment estimate of the gradient, β 1 and β 2 is the momentum parameter, α is the learning rate, and ∈ is a small amount to prevent division by zero.
[0181] Pre-training and fine-tuning strategy: A strategy combining pre-training and fine-tuning is used to improve the generalization ability and application effect of the model. First, the model is pre-trained on large-scale data, and then fine-tuned on specific tasks.
[0182] Model Application:
[0183] Input new data into the trained model to generate intelligence analysis results: After the model training is completed, input new data into the trained model to generate intelligence analysis results.
[0184] Dynamic adjustment technology: adaptively adjust model parameters according to actual conditions to ensure the accuracy and adaptability of the model. Specifically, by real-time monitoring of the model's performance indicators, dynamically adjust the model's hyperparameters, such as learning rate and regularization parameters.
[0185] Adaptive adjustment of model parameters: Adaptive optimization techniques such as AdaGrad and RMSprop are used to automatically adjust model parameters according to data characteristics to improve the adaptability of the model.
[0186] Real-time monitoring and feedback mechanism: During the model application process, a real-time monitoring and feedback mechanism is established to ensure the efficient operation of the model in different environments. By monitoring the output results and performance indicators of the model, the model strategy can be adjusted in time to optimize the model performance.
[0187] Through the above steps, the multimodal Transformer model training and application module realizes the complete process from feature extraction and fusion to model training and application, providing efficient and accurate technical means for intelligence analysis and decision support. This module makes full use of the powerful feature extraction and multimodal data fusion capabilities of the Transformer model, significantly improving the accuracy and efficiency of intelligence analysis.
[0188] Figure 5 The detailed process of the real-time intelligence analysis module is shown, including the process of edge computing and complex decision analysis. The following is a detailed description of each step and its implementation method:
[0189] Real-time intelligence analysis module
[0190] Edge computing:
[0191] Processing simple intelligence analysis tasks: Edge computing devices are used to process simple intelligence analysis tasks. These tasks include preliminary analysis and filtering of data, and identifying some simple patterns and anomalies.
[0192] Preliminary analysis and filtering: Preliminary analysis and filtering of data on edge computing devices can effectively reduce the amount of data and improve processing efficiency. Preliminary analysis includes noise filtering, data smoothing, and feature extraction.
[0193] Ensure low latency and efficient response: Edge computing devices have low network latency and can quickly process data locally to ensure efficient response. By performing calculations near the data collection point, data transmission time is reduced and the real-time performance of the system is improved.
[0194] Return decision suggestions in JSON format: After processing, the edge computing device returns the analysis results to the decision maker or other terminal devices in JSON format. JSON format has good readability and scalability, which facilitates data transmission and parsing.
[0195] Complex decision analysis:
[0196] Backend servers perform large-scale computing: For complex intelligence analysis tasks, data will be transferred to backend servers for large-scale computing. Backend servers have powerful computing capabilities and are able to handle large-scale data and complex analysis tasks.
[0197] Leverage trillion-level large-scale pre-trained models for in-depth analysis: The backend server uses pre-trained Transformer models for in-depth analysis. The pre-trained models have powerful feature extraction and pattern recognition capabilities and can extract valuable information from large-scale data.
[0198] Prompt optimization: During the model reasoning process, the analysis accuracy and efficiency of the model can be improved by optimizing the prompt. Prompt optimization includes selecting appropriate input templates and parameter settings.
[0199] RAG (Retrieval-Augmented Generation): Combined with RAG technology, the results generated by the model are enhanced by retrieving relevant information. The RAG model combines the generation model with the retrieval model to make the generated results more accurate and rich.
[0200] Knowledge Graph: Using knowledge graph technology, we can integrate and represent complex relationships and knowledge structures, and improve the model’s ability to understand and reason about data. Knowledge graphs represent entities and relationships through nodes and edges, and can provide structured knowledge representation.
[0201] Improve the accuracy and richness of decision analysis: Through the above technologies, the backend server can conduct in-depth analysis of data and generate accurate and rich decision suggestions. These decision suggestions will be passed on to relevant decision makers to assist them in making scientific and reasonable decisions.
[0202] Through the above steps, the real-time intelligence analysis module realizes the complete process from edge computing to complex decision analysis, providing efficient and accurate technical means for intelligence analysis and decision support. This module makes full use of the low latency and efficient response advantages of edge computing, as well as the large-scale computing and deep analysis capabilities of the back-end server, significantly improving the real-time and accuracy of intelligence analysis.
[0203] Figure 6 The detailed process of the intelligence generation and distribution module is shown, including the process of report generation and report distribution. The following is a detailed description of each step and its implementation method:
[0204] Intelligence generation and distribution module
[0205] Report Generation:
[0206] Generate reports based on intelligence analysis results: The system automatically generates intelligence reports based on the analysis results provided by the intelligence analysis module. Intelligence reports include text descriptions, charts, and key indicators, which fully reflect the analysis results and intelligence data.
[0207] Text description: The report contains detailed text descriptions that explain the analysis results and key findings. These text descriptions can help decision makers understand the information and trends behind the intelligence data.
[0208] Charts and key indicators: The report also includes various charts and key indicators, such as line charts, bar charts, pie charts, etc., which intuitively show the trend and comparison of data changes. The key indicator section lists important numerical information to help decision makers quickly obtain key information.
[0209] Integrated Natural Language Generation (NLG) technology: The report generation process integrates Natural Language Generation (NLG) technology. NLG technology can automatically generate easy-to-read text content to improve the readability and accuracy of reports. Through preset templates and rules, NLG technology can generate text descriptions that meet report specifications.
[0210] Ensure the readability and accuracy of the report content: During the report generation process, the system will check the generated text and charts to ensure the accuracy and logic of the report content. At the same time, the readability of the report is improved by adjusting the layout of the text and charts.
[0211] Report Distribution:
[0212] Distribute reports to relevant decision makers through secure communication channels: The generated intelligence reports are distributed to relevant decision makers through secure communication channels. The secure communication channel can be an encrypted network transmission channel to ensure the security of data during transmission.
[0213] Send JSON format reports to relevant terminal devices through the function calling interface (Function Calling): The system sends the generated JSON format reports to relevant terminal devices through the function calling interface (Function Calling), realizing automatic and intelligent intelligence distribution. JSON format reports have good scalability and are easy to parse and use by different terminal devices.
[0214] Automated and intelligent intelligence distribution: The report distribution process is highly automated and intelligent. The system can automatically select distribution channels and target devices based on preset rules and conditions to ensure that reports are delivered to relevant personnel in a timely manner.
[0215] Support multiple communication protocols and security encryption technologies: The system supports multiple communication protocols and security encryption technologies, such as HTTPS, TLS, etc., to ensure the security and reliability of data transmission. Through encrypted transmission and identity authentication, data leakage and tampering during transmission can be prevented.
[0216] Ensure the security and reliability of data transmission: During the report distribution process, the system will monitor and verify the communication channels and transmission process to ensure the stability and security of data transmission. If an abnormal situation is detected, the system will take timely measures to ensure the integrity of data transmission.
[0217] Through the above steps, the intelligence generation and distribution module realizes the complete process from report generation to report distribution, providing decision makers with accurate, timely and secure intelligence information support. This module uses natural language generation technology and secure communication technology to significantly improve the efficiency and security of intelligence generation and distribution.
Claims
1. A method for applying a large-scale pre-trained model based on Transformer in intelligence analysis and decision support. It is characterized in that The following steps are involved: Multimodal data collection, including: Collect intelligence data through devices such as cameras, infrared sensors, lidar, and microphones, which are mounted on drones or fixed surveillance devices to capture high-resolution video, images, thermal imaging data, 3D point cloud data, and ambient audio; Adopt multi-sensor fusion technology to achieve high-precision data capture and synchronization, and perform preliminary processing through edge computing devices to reduce data transmission delays and ensure the real-time and integrity of data; Use high-precision clocks to synchronize the data of different devices to ensure the consistency of data in time; During data collection and transmission, the data is encrypted using the Advanced Encryption Standard and Transport Layer Security Protocol to ensure data security and integrity; Data preprocessing, including: Clean the collected data, remove noise and irrelevant information, and use median filtering, Gaussian filtering and other methods to denoise the image and point cloud data; Convert data from different sensors into a unified format, at least one of PEG, PNG, PLY, and WAV, and generate feature matrices using Numpy and Pandas; The feature matrices of different modalities are concatenated in the feature dimension to form a high-dimensional comprehensive feature matrix. This comprehensive feature matrix contains the key features of all modal data and provides rich and complete data input for subsequent deep learning models. Perform convolution processing on the preprocessed data to reduce the volume of the data matrix, reduce the amount of calculation, and retain the coordinate information of the image; Using positional encoding technology, the convolutional data is horizontally encoded and arranged, and then input into the Transformer neural network; Multimodal Transformer model training and application, including: Extract and fuse high-level features from multimodal data through self-attention mechanism, and use multi-head attention mechanism to enhance the model's understanding of complex relationships between different data modalities; Use preprocessed multimodal data for supervised learning, define loss functions and use optimization algorithms to train models and update model parameters; A strategy combining pre-training and fine-tuning is adopted to improve the generalization ability and application effect of the model; Input new data into the trained model to generate intelligence analysis results, and use dynamic adjustment technology to adaptively adjust model parameters according to actual conditions; Real-time intelligence analysis, including: Process simple intelligence analysis tasks through edge computing devices and return decision recommendations in JSON format to ensure low latency and efficient response; Hand over complex decisions to backend servers for large-scale computing, use large-scale pre-trained models for in-depth analysis, and combine prompt optimization, RAG technology, and knowledge graphs; Intelligence generation and dissemination, including: Automatically generate intelligence reports based on intelligence analysis results, including text descriptions, charts, and key indicators, integrating natural language generation technology; The reports are distributed to relevant decision makers and commanders through secure communication channels, and the JSON format reports are sent to relevant terminal devices through the function call interface.
2. The method according to claim 1, It is characterized in that In the multimodal data collection step, preliminary processing is performed through edge computing equipment to reduce data transmission delays and ensure the real-time and integrity of the data.
3. The method according to claim 1, It is characterized in that The data preprocessing step includes: The collected data is cleaned to remove noise and irrelevant information, and median filtering, Gaussian filtering and other methods are used to denoise the image and point cloud data.
4. The method according to claim 1, It is characterized in that In the data preprocessing step, the data of different sensors are converted into a unified format, which is at least one of PEG, PNG, PLY, and WAV, and a feature matrix is generated using Numpy and Pandas; then the feature matrices of different modalities are spliced in the feature dimension to form a high-dimensional comprehensive feature matrix.
5. The method according to claim 1, It is characterized in that In the multimodal Transformer model training and application steps, a self-attention mechanism is used to extract and fuse high-level features from multimodal data, and a multi-head attention mechanism is used to enhance the model's understanding of the complex relationships between different data modalities.
6. The method according to claim 1, It is characterized in that In the real-time intelligence analysis step, the intelligence analysis task is processed by the edge computing device, and the decision recommendations are returned in JSON format.
7. The method according to claim 1, It is characterized in that In the intelligence generation and distribution step, an intelligence report is automatically generated based on the intelligence analysis results, including text descriptions, charts and key indicators, integrating natural language generation technology.
8. A device for applying a large-scale pre-trained model based on Transformer in intelligence analysis and decision support. It is characterized in that include: units used to collect intelligence data through devices such as cameras, infrared sensors, lidar and microphones; Units for cleaning, denoising, format conversion and convolution of collected data; A unit for inputting processed data into the Transformer neural network for training and analysis; A unit for processing simple intelligence analysis tasks through edge computing devices and returning decision recommendations; A unit used to generate and distribute intelligence reports.
Citation Information
Cited By
Aircraft information acquisition service design method
CN120013274A
