Multi-modal data classification method and system, computer equipment and storage medium

Abstract

The invention provides a multi-modal data classification method and system, computer equipment and a storage medium, and the method comprises the steps: receiving multi-modal data through edge equipment, and carrying out the preprocessing of the multi-modal data; aligning to the same latitude through linear transformation or a projection layer; a comprehensive information representation vector is generated through fusion of an attention mechanism or a Transform architecture; quantifying the multi-modal large model, and deploying the multi-modal large model in edge equipment; sending the comprehensive information representation vector to a multi-modal large model for reasoning, and processing the comprehensive information representation vector through a multi-layer Transform architecture; and generating a classification result according to task requirements. According to the method, the real-time performance, the accuracy and the resource utilization efficiency of the system are improved through localized deployment on the edge equipment, sensitive data can be processed locally, data privacy and safety are guaranteed, cross-modal feature extraction and alignment are achieved, and the recognition capability in a complex scene is improved. Dependence on large-scale annotation data is reduced, and adaptability is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically related to technologies such as computer vision, natural language processing, speech recognition, and edge computing. In particular, it relates to a multi-modal data classification method, system, computer device, and storage medium. Background Art

[0002] The existing technologies mainly rely on traditional deep learning methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and their variants (LSTMs and GRUs), for multimedia information recognition and classification tasks to achieve spatio-temporal feature extraction and analysis of video data. Although these technologies have made some progress in recognizing objects, behaviors, and events in videos, the accuracy is not high. And the current solution is to deploy large models in the cloud for multimedia information recognition and classification. However, this method requires uploading data to a remote server, which not only increases the latency of data transmission but also may cause data privacy and security issues.

[0003] In addition, existing artificial intelligence video analysis systems usually require a large amount of high-quality and diverse training data for training to achieve good performance. For unseen scenarios or abnormal behavior patterns, the model is prone to misjudgment or missed judgment. Since training a traditional deep neural network model requires a large amount of labeled data sets, in real life, such data is very limited and time-consuming. Therefore, the system often lacks the ability of personalized customization for specific users or scenarios and is difficult to meet specific requirements. On the one hand, when a convolutional neural network (CNN) processes image data, it mainly relies on the extraction of local features and has a weak ability to capture global structural information. While traditional recurrent neural networks (RNNs) are prone to problems of gradient vanishing or gradient explosion when dealing with long-term dependencies, which will lead to poor recognition effects for some behaviors or events with a long duration.

[0004] On the other hand, when deploying artificial intelligence large models on resource-constrained edge devices, existing technical solutions often face the problem of low inference efficiency, which not only increases the demand for hardware resources but also may lead to insufficient real-time performance. And existing large models mainly process single-modal data and lack the ability to fuse multi-modal data. This limits the performance of the model in complex scenarios, especially in tasks that require comprehensive decision-making based on multiple modal information. Summary of the Invention

[0005] The present invention aims to solve the problems that when the existing artificial intelligence model processes multimodal data, there are relatively high risks of data leakage and privacy infringement, high dependence on training data, limited customization ability and feature extraction ability, and low real-time performance and resource efficiency, and provides a multimodal data classification method, system, computer device, and storage medium.

[0006] The present invention provides a multimodal data classification method, including the following steps:

[0007] Receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features; wherein, the multimodal data includes text data, image data, audio data, and video data; the multimodal features include word embedding vectors, image feature vectors, audio feature vectors, and high-dimensional feature vectors;

[0008] Aligning the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector to the same dimension through a linear transformation or a projection layer;

[0009] Fusing the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector;

[0010] Quantifying a preset multimodal large model and deploying it in the edge device;

[0011] Sending the comprehensive information representation vector to the multimodal large model for inference and processing the comprehensive information representation vector through multiple layers of the Transformer architecture;

[0012] Generating a classification result according to task requirements.

[0013] Further, in the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes:

[0014] Receiving the input text of the user on the edge device;

[0015] Segmenting the input text into words, phrases, or characters through a preset tokenizer and mapping them into word embedding vectors; wherein, the word embedding vectors include the semantic information of the input text.

[0016] Further, in the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes:

[0017] Converting the image data into image analysis features through a preset image encoder to obtain the image feature vector.

[0018] Further, in the step of converting the image data into image analysis features through a preset image encoder, it includes:

[0019] Preprocess the input image; wherein, the preprocessing includes resizing to a unified size and normalization operation;

[0020] Generate visual tokens through a preset vision encoder;

[0021] Select high-resolution images;

[0022] Divide the high-resolution image into multiple slices through an adaptive image chunking algorithm, wherein the size of each slice matches the pre-training settings of the vision encoder;

[0023] Perform dimensionality reduction on the visual tokens through a compression layer.

[0024] Further, in the step of receiving multi-modal data by an edge device and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0025] Convert the audio data into model features through a preset Whisper model to generate audio vectors;

[0026] Align the output data of the Whisper model with a preset large language model at the token level.

[0027] Further, in the step of aligning the output data of the Whisper model with a preset large language model at the token level, it includes:

[0028] Align through forward KL divergence constraint.

[0029] Further, in the step of receiving multi-modal data by an edge device and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0030] Read video frames of the video through a preset tool and extract frames at fixed intervals according to the average frame rate of the video to obtain extracted frames;

[0031] Calculate the number of frames of the extracted frames;

[0032] Judge whether the number of frames exceeds a preset maximum value;

[0033] If so, screen the extracted frames through a uniform sampling algorithm to obtain screened frames;

[0034] Convert the screened frames into image objects and send them to the vision encoder to generate high-dimensional vectors.

[0035] Further, after the step of reading video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames, the method further includes:

[0036] Screen out high-resolution frames;

[0037] Divide the high-resolution image object into multiple sub-regions through an adaptive block algorithm so that the sub-regions match the pre-training settings of the visual encoder, and obtain high-resolution frame vectors;

[0038] Perform dimensionality reduction processing on the high-resolution frame vectors through the compression layer.

[0039] Further, in the step of fusing the word embedding vector, the image feature vector, the audio vector and the high-dimensional vector through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector, the method includes:

[0040] Integrate the word embedding vector, the image feature vector, the audio vector and the high-dimensional vector into the same semantic space to obtain the comprehensive information representation vector.

[0041] The present invention also provides a multi-modal data classification system, including an edge device, where a tokenizer, a visual encoder, an audio encoder, a multi-layer perceptron and an image encoder are deployed on the edge device; the tokenizer is electrically connected to the audio encoder; the visual encoder is electrically connected to the image encoder; the image encoder is electrically connected to the multi-layer perceptron; the tokenizer is used to split the input text into words, sub-words or characters and map them into word embedding vectors; the visual encoder is used to extract image features and generate image feature vectors or visual tokens; the audio encoder is used to extract audio features and generate audio feature vectors; the multi-layer perceptron is used to map multiple input data sets to a single output data set; the image encoder is used to convert image data into a feature representation for model analysis.

[0042] The present invention also provides a computer device, including a memory and a processor, where a computer program is stored in the memory, and the processor executes the computer program to implement the steps in any one of the above methods.

[0043] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in any one of the above methods are implemented.

[0044] The present invention provides a multi-modal data classification method, system, computer device and storage medium, which has the following beneficial effects:

[0045] This application receives multimodal data through an edge device, preprocesses the multimodal data, aligns the multimodal data to the same dimension through a linear transformation or a projection layer, fuses the vectors of multiple modalities through an attention mechanism or a Transformer architecture, and deploys the quantized multimodal large model on the edge device for inference, achieving efficient recognition and classification of multimedia information. By integrating data features of different modalities, the multimodal large model can significantly improve the ability to understand complex scenarios. By performing local deployment on the edge device, not only can the real-time performance, accuracy, and resource utilization efficiency of the system be improved, but also the latency and bandwidth consumption during data transmission can be effectively reduced. In addition, sensitive data can be directly processed locally without uploading to the cloud, thus better protecting data privacy and security.

[0046] This application processes data of multiple modalities through a multimodal large model, realizes cross-modal feature extraction and alignment, thereby improving the recognition ability in complex scenarios. It not only significantly reduces the dependence on large-scale labeled data, but also improves the adaptability and robustness of the model in new scenarios. In addition, by performing local fine-tuning on the multimodal large model, customizing the configuration according to user needs, and improving the flexibility and adaptability of the system, and optimizing the model engine through model quantization, the model can operate efficiently on devices with low power consumption and low storage to achieve this. The multimodal large model can simultaneously extract local details and global context information, thereby improving the recognition ability for scenarios with unclear features or complex backgrounds, and enhancing the feature extraction and modeling ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the method steps of a multimodal data classification method in the present invention;

[0048] Figure 2 It is a block diagram of the structure of a multimodal data classification system in the present invention;

[0049] Figure 3 It is a block diagram of the structure of a computer device of the present invention;

[0050] Figure 4 It is a schematic diagram of the steps of an embodiment of a multimodal data classification method in the present invention;

[0051] Figure 5 It is a schematic diagram of the workflow of an edge device preprocessing input data in an embodiment of a multimodal data classification method of the present invention.

[0052] Marking description: Edge device 10, tokenizer 20, audio encoder 30, image encoder 40, visual encoder 50, multi-layer perceptron 60. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0054] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] Existing non-localized systems rely on cloud computing, and user data needs to be uploaded to a remote server for processing, which increases the risk of data leakage and privacy infringement. Moreover, a large amount of high-quality and diverse training data is required to achieve good performance, and there is a high dependence on large-scale labeled data. When deploying large models on resource-constrained edge devices, there will be problems of low inference efficiency and insufficient real-time performance. In addition, existing large models mainly process single-modal data and lack the ability to fuse multi-modal data, which limits the performance of the model in complex scenarios, especially in tasks that require comprehensive decision-making based on multiple modal information.

[0056] Refer to the attached Figure 1 , a multi-modal data classification method in an embodiment of the present invention, includes:

[0057] S1. Receive multi-modal data through the edge device 10 and preprocess the multi-modal data to obtain multi-modal features; wherein, the multi-modal data includes text data, image data, audio data, and video data; the multi-modal features include word embedding vectors, image feature vectors, audio feature vectors, and high-dimensional feature vectors;

[0058] S2. Align the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors to the same dimension through a linear transformation or a projection layer;

[0059] S3. Fuse the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector;

[0060] S4. Quantify a preset multi-modal large model and deploy it in the edge device 10;

[0061] S5. Send the comprehensive information representation vector to the multi-modal large model for inference and process the comprehensive information representation vector through a multi-layer Transformer architecture;

[0062] S6. Generate a classification result according to the task requirements.

[0063] In the above steps, first, the edge device 10 receives multimodal data and preprocesses the multimodal data to obtain multimodal features. Among them, the multimodal data includes text data, image data, audio data, and video data; the multimodal features include word embedding vectors, image feature vectors, audio feature vectors, and high-dimensional feature vectors. The edge device 10 can be a mobile phone, a tablet, a sensor, a camera, and an industrial robot. Then, the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector are aligned to the same dimension through a linear transformation or a projection layer. Among them, the linear transformation changes the data dimension through matrix multiplication and maintains the linear structure of the data. In a specific embodiment, the dimension of the word embedding vector is 300, and the image feature vector is 512. Then, the two vectors are aligned to 256 dimensions through a linear transformation. Specifically, the 300×256 matrix is multiplied by the word embedding vector to obtain a new 256-dimensional vector; similarly, the 512×256 matrix is multiplied by the image feature vector to obtain a new 256-dimensional vector, so as to linearly transform two vectors with different dimensions to the same dimension. The projection layer can project the data to a specific dimension. In a specific embodiment, the audio feature vector is 1024 dimensions, and the high-dimensional vector is 2048 dimensions. Then, the two vectors are projected to 512 dimensions through the projection layer. Aligning the multimodal data through the linear transformation and the projection layer facilitates the subsequent fusion and processing stages.

[0064] Then, the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector are fused through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector. Among them, the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector are in the same semantic space. The attention mechanism can capture the important parts in the input data, thereby improving the accuracy. In a specific embodiment, the attention mechanism dynamically assigns attention weights by calculating the correlation between each element in the input data sequence and each element in the output sequence, and outputs the important parts or key content. For example, in a translation task, the input English sentence is “The cat sat on the mat.” When translating to “sat”, the attention mechanism will make the model focus on the previous “cat”, so as to clarify that “sat” means “the cat sat down” to grasp the key content and improve the accuracy of the generated content.

[0065] The Transformer architecture consists of an encoder and a decoder, each including multiple identical layers. The encoder is responsible for converting the input sequence into an intermediate representation for the decoder to use. Each encoder layer includes a multi-head self-attention mechanism (Multi-Head Attention): This mechanism allows multiple parallel attention processes to occur simultaneously, capturing relationships at different levels and directions in the input sequence. Each attention head: generates an attention weight matrix by calculating the correlations between elements in the input sequence. The results of multiple attention heads are integrated through concatenation and linear transformation to generate the final multi-head attention output. A feed-forward neural network (Feed-Forward Neural Network, FFN): is used to perform a non-linear transformation on the output of the multi-head attention. The feed-forward network typically consists of two linear transformations and an activation function (such as ReLU). Layer normalization (Layer Normalization): is applied separately after the multi-head self-attention and the feed-forward neural network to accelerate training and improve the stability of the model. Residual connections (Residual Connections): add the input directly to the output of the sub-layer, helping to alleviate the vanishing gradient problem and enhancing the training effect. The decoder generates the output sequence based on the intermediate representation generated by the encoder. Each decoder layer includes masked multi-head self-attention (Masked Multi-Head Attention): The first multi-head self-attention mechanism in the decoder has a masking mask (Mask), ensuring that the generated output sequence is autoregressive, i.e., the output at each position depends only on the previously generated part. Multi-head self-attention mechanism (Multi-Head Attention): The second multi-head self-attention mechanism in the decoder is used to capture the relationship between the encoder output and the current decoder input. Through interaction with the encoder, the decoder can generate more accurate outputs using the context information of the input sequence. Feed-forward neural network (Feed-Forward Neural Network, FFN): is used to perform a non-linear transformation on the output of the attention mechanism. Layer normalization (Layer Normalization): is applied separately after the multi-head self-attention and the feed-forward neural network to accelerate training and improve the stability of the model. Residual connections (Residual Connections): add the input directly to the output of the sub-layer, helping to alleviate the vanishing gradient problem and enhancing the training effect. Since the self-attention mechanism itself does not contain positional information, Transformer provides a unique encoding for each position in the input sequence through positional encoding (Positional Encoding). The positional encoding can be an embedding vector calculated in advance or through learnable parameters.In a specific embodiment, the positional encoding is added to the input embedding to obtain the final input representation.

[0066] In a specific embodiment, the input sequence is first converted into embedding vectors, and the input at each position is mapped to a vector space of a fixed dimension through an embedding layer. The embedding vectors are added to the positional encoding to obtain the input representation containing positional information. Then, the encoder converts the input sequence into an intermediate representation through multiple layers of multi-head self-attention and feed-forward neural networks. Each encoder layer captures the relationships between positions in the input sequence through the multi-head self-attention mechanism. Multiple parallel attention heads can capture information at different levels and in different directions. The output of the multi-head attention is further non-linearly transformed through a feed-forward network to further enhance the expressive power of the model. The decoder generates the output sequence based on the intermediate representation generated by the encoder. The first layer of the decoder uses the masked multi-head self-attention mechanism to ensure that the generated output depends only on previous positions. The second layer of the decoder uses the multi-head self-attention mechanism to capture the relationship between the encoder output and the current decoder input. The output of the attention mechanism is again non-linearly transformed through a feed-forward network to generate the final output representation. The output of the decoder undergoes a linear transformation and a softmax function to generate the final prediction result. Allowing the calculations at all positions to be performed in parallel significantly improves the computational efficiency. It can effectively capture long-range dependencies and solve the limitations of traditional recurrent neural networks (RNNs) in processing long sequences.

[0067] Next, the preset multi-modal large model is quantized and deployed in the edge device 10. Among them, the multi-modal large model can be CLIP (Contrastive Language-Image Pretraining), BLIP (Buildind Language ImagePrior), or Perceiver. The quantization can be fixed-point quantization, dynamic quantization, and post-training quantization. In a specific embodiment, the quantization is fixed-point quantization. For example, a 32-bit floating-point model is converted into a lower-bit integer model, such as an 8-bit integer model, and the model size is significantly reduced, and the inference speed is improved. Quantizing the trained multi-modal model can reduce the complexity and computational requirements of the model to adapt to the operation of small hardware devices, such as mobile phones, effectively reducing memory occupancy and computational time, while reducing energy consumption and extending the service life of the device.

[0068] After completing the independent encoding of text, images, audio, and video, the edge device 10 enters the multi-modal fusion and inference stage, which is to integrate the feature representations of different modalities into a unified semantic space to generate a representation vector that can comprehensively reflect the comprehensive information of the input data.

[0069] More specifically, first, the system aligns the feature vectors from different modalities to the same dimension through a linear transformation or a projection layer to ensure that the features of different modalities can interact in the same space. In a specific embodiment, the audio feature vector, the high-dimensional feature vector, and the word embedding vector are mapped into a shared high-dimensional space to facilitate subsequent multi-modal fusion operations. Finally, the feature vectors after multi-modal fusion are fed into a large language model for inference. The large language model further processes the input features through a multi-layer Transformer architecture and generates prediction results according to the task requirements.

[0070] Specifically, the multi-modal large model can significantly improve the ability to understand complex scenarios by integrating data features of different modalities. For example, in the content review task, the system can simultaneously analyze the visual content in the video, the speech information in the audio, and the caption text, so as to more comprehensively evaluate potential risks. In a specific embodiment, the task requirement is to identify the user's emotions, such as anger and happiness, based on the input text data, audio data, and image data of the user, and generate corresponding classification results. In another specific embodiment, the task requirement is to identify the current scene, such as a work scene, based on the image data and audio data.

[0071] Please refer to Figure 4 , in an embodiment, the multi-modal information is sent to the edge device, and the data is preprocessed on the edge device. The data can be text data, image data, audio data, and video data. NPU is the Neural Processing Unit, and the "1" in NPU1 is the model or version of the neural processing unit. The multi-modal large model and multiple neural processing units detect and identify the data, and finally obtain the classification result.

[0072] Please refer to Figure 5 , in an embodiment, first, the input image and audio data are preprocessed. The image data is sent to a multi-layer perceptron after being processed by an image encoder. Among them, the multi-layer perceptron (MLP, Multilayer Perceptron) is used to map multiple input data sets to a single output data set. The image encoder is used to convert the image data into a feature representation that can be analyzed by the large language model. At the same time, the audio data is converted into a feature representation that can be understood by the large language model through an audio encoder. Finally, the large language model further processes the input features through a multi-layer Transformer architecture and generates prediction results according to the task requirements. The prediction results can be other recognition results in addition to the classification results.

[0073] The present invention is provided with a multimodal large model that can simultaneously extract local details and global context information, thereby enhancing the recognition ability for scenes with unclear features or complex backgrounds, and achieving efficient recognition and classification of multimedia information. Among them, the multimedia information includes but is not limited to images, videos, audios, and texts. By performing local deployment on the edge device 10, not only the real-time performance, accuracy, and resource utilization efficiency of the system are significantly improved, but also the latency and bandwidth consumption in the data transmission process are effectively reduced. In addition, sensitive data can be directly processed locally without uploading to the cloud, thus better protecting data privacy and security. It is applicable to scenarios that require fast response speed and high privacy protection, such as smart home monitoring, industrial automation detection, and public security monitoring.

[0074] In one embodiment, in the step of receiving multimodal data by the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0075] Receiving the input text of the user on the edge device 10;

[0076] Segmenting the input text into words, phrases, or characters by a preset tokenizer 20 and mapping them into word embedding vectors; among them, the word embedding vectors include the semantic information of the input text.

[0077] In this embodiment, the user inputs text on the edge device 10. Among them, the input text can be dialogue text or query text. In a specific embodiment, the dialogue text is "The weather is very nice today. Do you want to go for a walk together?" In another specific embodiment, the query text is "How to make lemon tea?" Then, the input text is segmented into words, phrases, and characters by the tokenizer 20. For example, "How to make lemon tea?" is segmented into "How to make", "lemon tea", and "?" And the segmented words, phrases, or characters are mapped into word embedding vectors. Among them, mapping refers to the correspondence relationship from one set to another set, so that each element in the original set can correspond to an element in the target set. The mapping can be injective, surjective, or bijective. The word embedding vector is the representation of converting words, phrases, or characters into vectors of a fixed length, and the word embedding vector captures the semantic information of the input text.

[0078] In one embodiment, in the step of receiving multimodal data by the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0079] Converting the image data into image analysis features through a preset image encoder 40 to obtain image feature vectors.

[0080] In this embodiment, after receiving the image data, the image encoder 40 converts the image data into image analysis features to obtain an image feature vector. The image encoder 40 is used to convert the image data into a feature representation for further analysis by the model. In a specific embodiment, the user uploads an image file at the client, such as an image in JPG or PNG format, and the image content is a natural scenery photo including a river, mountains, and trees. Then the extracted image features are the colors of the river, mountains, and trees, the texture of the trees, and the objects.

[0081] In one embodiment, in the step of converting the image data into image analysis features through the preset image encoder 40, it includes:

[0082] Preprocess the input image; wherein, the preprocessing includes resizing to a unified size and normalization operation;

[0083] Generate visual tokens through the preset visual encoder 50;

[0084] Screen out high-resolution images;

[0085] Divide the high-resolution image into multiple slices through an adaptive image chunking algorithm, where the size of each slice matches the pre-training settings of the visual encoder 50;

[0086] Perform dimensionality reduction processing on the visual tokens through a compression layer.

[0087] In this embodiment, in the process of converting the image data into image analysis features, first preprocess the input image. The preprocessing includes resizing the image to a unified size and normalization operation to ensure data consistency. Among them, normalization refers to scaling the data proportionally to a specific interval or converting it into a dimensionless form to eliminate the dimensional difference and improve the training efficiency and stability of the model. Among them, the dimension refers to the unit or measurement method of the data, indicating the physical meaning or numerical range of the data. For example, the dimension of the image size is pixels, and the dimension of the pixel value is 0 - 255, representing the color intensity; while the dimensionless form means removing the dimension of the data to make it a pure numerical value without a unit. For example, normalizing the pixel value from 0 - 255 to 0 - 1, at this time the data does not have the unit of pixels and is just a proportional value.

[0088] Next, a visual marker is generated by the visual encoder 50, where the visual marker can capture the key feature information in the image. Then, high-resolution images are screened out, and the input image is divided into multiple slices through an adaptive image chunking method, and the size of each slice matches the pre-training settings of the visual encoder 50, thereby improving the efficiency and accuracy of feature extraction. Specifically, before using the visual encoder 50, the visual encoder 50 is pre-trained and the slice size is set. More specifically, the pre-training settings include the dataset, model architecture, training parameters, data augmentation, and loss function; among them, the dataset can be ImageNet, COCO, and OpenImages. ImageNet contains 1.4 million images in 1000 categories; COCO contains object detection and segmentation data in 80 categories; OpenImages contains millions of images. The model architecture includes but is not limited to ResNet-50, ViT (Vision Transformer), and EfficientNet. The training parameters can be the learning rate and the number of training epochs. The data augmentation can be random cropping and horizontal flipping. Random cropping is to randomly crop a piece from the image to simulate different perspectives, and the image is horizontally flipped to increase the diversity of the data; horizontal flipping can be to horizontally flip the image to increase the diversity of the data. The loss function can be cross-entropy loss and Dice loss, and Dice loss is a loss function converted by the Dice coefficient.

[0089] In a specific embodiment, the dataset is ImageNet, the model architecture is ResNet-50, a residual network with 50 layers in depth, training parameters: the learning rate is 0.1, the batch size is 256, the number of training epochs is 100, the optimizer is SGD, the momentum is set to 0.9, random cropping is used for data augmentation, the cropping size is 220 multiplied by 220, and the cross-entropy loss is used as the loss function. After the settings are completed, the ResNet-50 model is pre-trained on the ImageNet dataset.

[0090] Finally, the number of visual markers is reduced through the compression layer, reducing memory occupancy and computer resource consumption to improve the inference speed and efficiency. Among them, the compression layer refers to the layer or component used to compress data to reduce its volume during data processing, storage, or transmission. In a specific embodiment, if the compression layer is image compression, the JPG format and PNG format are used to compress the image to reduce the file size.

[0091] In one embodiment, in the step of receiving multi-modal data by the edge device 10 and pre-processing the multi-modal data to obtain multi-modal features, it includes:

[0092] Converting the audio data into model features through a preset Whisper model to generate an audio vector;

[0093] Align the output data of the Whisper model with a preset large language model at the token level.

[0094] In this embodiment, when preprocessing the audio data, the audio features are extracted by a pre-trained audio model to generate audio feature vectors. Specifically, the audio model is the Whisper model, which can convert audio signals into text form. In a specific embodiment, the user inputs a piece of speech. For example, the speech content is "There will be a meeting this afternoon", then the semantic information of the audio signal is extracted to convert the audio signal into text form.

[0095] Based on the Transformer architecture, the Whisper model can process long-sequence speech data through the self-attention mechanism. More specifically, the Whisper model consists of an encoder and a decoder. The encoder is used to convert the input speech waveform into a high-level representation, and the decoder is used to generate the corresponding text based on the representation generated by the encoder and output it. Moreover, the output data of the audio model needs to be aligned with the large model at the token level to ensure that the content converted from the audio signal into text form can accurately reflect the semantic information in the original speech. Among them, a token represents the smallest unit of meaning that the model can understand and generate, which is the basic unit of the model and can represent a word, a part of a word, or a character.

[0096] In one embodiment, in the step of aligning the output data of the Whisper model with a preset large language model at the token level, it includes:

[0097] Align through forward KL divergence constraint.

[0098] In this embodiment, aligning the output data of the Whisper model with a preset large language model through forward KL divergence constraint optimizes the generation behavior of each token, thereby improving the model's understanding ability of complex scenarios. Among them, forward KL divergence (Forward KL Divergence) is used to measure the difference between two probability distributions (P) and (Q). When considering the conversion from the true data distribution (P(x)) to the approximate distribution (Q(x|\theta)), the forward KL divergence is defined as:

[0099] [D_{\text{KL}}(P||Q)=\intp(x)\log{\frac{p(x)}{q(x|\theta)}}dx];

[0100] where (P(x)) is the true distribution of the target, and (Q(x|\theta)) represents the parameterized estimated distribution.

[0101] In one embodiment, in the step of receiving multimodal data through the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0102] Read the video frames of the video through a preset tool, and extract frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames;

[0103] Calculate the number of frames of the extracted frames;

[0104] Determine whether the number of frames exceeds a preset maximum value;

[0105] If so, screen the extracted frames through a uniform sampling algorithm to obtain the screened frames;

[0106] Convert the screened frames into image objects and send them to the visual encoder 50 to generate high-dimensional vectors.

[0107] In this embodiment, first, read the video frames of the video through a preset tool, and extract frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames, so as to reduce the computational burden. Among them, the tool can be OpenCV (Open Source Computer Vision Library) and FFmpeg. Then calculate the number of frames of the extracted frames and judge the number of frames of the extracted frames. If the number of frames of the extracted frames exceeds the preset maximum value, screen the extracted frames through a uniform sampling algorithm to obtain the screened frames, so as to ensure that the sampled extracted frames can be evenly distributed on the time axis. Then convert the screened frames into image objects and send them to the visual encoder 50 to generate high-dimensional vectors, which are used to capture key information such as colors, textures, and shapes in the images.

[0108] In one embodiment, after the step of reading the video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames, it further includes:

[0109] Screen out the high-resolution frames;

[0110] Divide the high-resolution image objects into multiple sub-regions through an adaptive block algorithm to make the sub-regions match the pre-training settings of the visual encoder 50, and obtain high-resolution frame vectors;

[0111] Perform dimensionality reduction processing on the high-resolution frame vectors through a compression layer.

[0112] In this embodiment, for high-resolution frames, an adaptive block algorithm is used to divide the image into multiple sub-regions to match the pre-training settings of the encoder, thereby improving the efficiency and accuracy of feature extraction. Finally, dimensionality reduction processing is performed on the high-dimensional vectors of the compression layer to reduce memory occupancy and improve the inference speed while retaining the core features of the video content. In a specific embodiment, if the compression layer is for streaming media compression, the video is processed through AAC to reduce the amount of data and the bandwidth requirements. Among them, dimensionality reduction processing is the process of mapping high-dimensional data, such as high-resolution vectors, to a low-dimensional space through mathematical transformation or model compression. In a video encoder, dimensionality reduction processing can not only reduce the dimension of the feature vectors, occupy less memory, and speed up the inference speed of the model, but also retain the core features of the video content.

[0113] In one embodiment, in the step of fusing word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector, it includes:

[0114] Integrate the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors into the same semantic space to obtain a comprehensive information representation vector.

[0115] In this embodiment, after the independent encoding of text, image, audio, and video is completed, the edge device 10 enters the multi-modal fusion and inference stage. The core goal of this stage is to integrate the feature representations of different modalities into a unified semantic space, thereby generating a representation vector that can comprehensively reflect the comprehensive information of the input data.

[0116] Refer to the appendix Figure 2 , a multi-modal data classification system, including an edge device 10, on which a tokenizer 20, a visual encoder 50, an audio encoder 30, a multi-layer perceptron 60, and an image encoder 40 are deployed; the tokenizer 20 is electrically connected to the audio encoder 30; the visual encoder 50 is electrically connected to the image encoder 40; the image encoder 40 is electrically connected to the multi-layer perceptron 60; the tokenizer 20 is used to split the input text into words, sub-words, or characters and map them into word embedding vectors; the visual encoder 50 is used to extract image features and generate image feature vectors or visual tokens; the audio encoder 30 is used to extract audio features and generate audio feature vectors; the multi-layer perceptron 60 is used to map multiple input data sets to a single output data set; the image encoder 40 is used to convert image data into a feature representation for model analysis.

[0117] In this embodiment, a tokenizer 20 is deployed on the edge device 10, which is used to split the input text into words, sub-words or characters and map them into word embedding vectors; a visual encoder 50 is used to extract image features and generate image feature vectors or visual tokens; an audio encoder 30 is used to extract audio features and generate audio feature vectors; a multi-layer perceptron 60 maps multiple input data sets to a single output data set; an image encoder 40 is used to convert image data into a feature representation for model analysis. The tokenizer 20 is electrically connected to the audio encoder 30; the visual encoder 50 is electrically connected to the image encoder 40; the image encoder 40 is electrically connected to the multi-layer perceptron 60. By performing local deployment on the edge device 10, not only can the real-time performance, accuracy and resource utilization efficiency of the system be significantly improved, but also the delay and bandwidth consumption in the data transmission process can be effectively reduced. It can be applied to scenarios that require fast response and high privacy protection, such as smart home monitoring, industrial automation detection, and public security monitoring, etc.

[0118] Refer to the appendix Figure 3 , in the embodiment of the present application, a computer device is further provided. This computer device can be a server, and its internal structure can be as Figure 3 shown. This computer device includes a processor, a memory, a network interface and a database. Among them, the processor of this computer device is used to provide computing and control capabilities. The memory of this computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system, computer program and database in the non-volatile storage medium. The database of this computer device is used to store data such as templates, tables, and preset fields. The network interface of this computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a multi-modal data classification method, including the following steps:

[0119] Receive multi-modal data through the edge device 10 and preprocess the multi-modal data to obtain multi-modal features; among them, the multi-modal data includes text data, image data, audio data and video data; the multi-modal features include word embedding vectors, image feature vectors, audio feature vectors and high-dimensional feature vectors;

[0120] Align the word embedding vectors, image feature vectors, audio vectors and high-dimensional vectors to the same dimension through a linear transformation or a projection layer;

[0121] Fuse the word embedding vectors, image feature vectors, audio vectors and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector;

[0122] Quantize a pre-set multi-modal large model and deploy it in the edge device 10;

[0123] Send the comprehensive information representation vector to the multi-modal large model for inference, and process the comprehensive information representation vector through a multi-layer Transformer architecture;

[0124] Generate a classification result according to the task requirements.

[0125] In one embodiment, in the step of receiving multi-modal data by the edge device 10 and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0126] Receive the input text of the user on the edge device 10;

[0127] Segment the input text into words, phrases or characters through a pre-set tokenizer 20 and map it to a word embedding vector; wherein, the word embedding vector includes the semantic information of the input text.

[0128] In one embodiment, in the step of receiving multi-modal data by the edge device 10 and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0129] Convert the image data into image analysis features through a pre-set image encoder 40 to obtain an image feature vector.

[0130] In one embodiment, in the step of converting the image data into image analysis features through a pre-set image encoder 40, it includes:

[0131] Preprocess the input image; wherein, the preprocessing includes resizing to a unified size and normalization operation;

[0132] Generate visual tokens through a pre-set vision encoder 50;

[0133] Filter out high-resolution images;

[0134] Divide the high-resolution image into multiple slices through an adaptive image tiling algorithm, wherein the size of each slice matches the pre-training setting of the vision encoder 50;

[0135] Perform dimensionality reduction processing on the visual tokens through a compression layer.

[0136] In one embodiment, in the step of receiving multi-modal data by the edge device 10 and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0137] Convert the audio data into model features through a pre-set Whisper model to generate an audio vector;

[0138] Align the output data of the Whisper model with a preset large language model at the token level.

[0139] In one embodiment, in the step of aligning the output data of the Whisper model with a preset large language model at the token level, it includes:

[0140] Align through forward KL divergence constraint.

[0141] In one embodiment, in the step of receiving multimodal data through the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0142] Read the video frames of the video through a preset tool, and extract frames at fixed intervals according to the average frame rate of the video to obtain extracted frames;

[0143] Calculate the number of frames of the extracted frames;

[0144] Judge whether the number of frames exceeds a preset maximum value;

[0145] If so, screen the extracted frames through a uniform sampling algorithm to obtain screened frames;

[0146] Convert the screened frames into image objects and send them to the visual encoder 50 to generate high-dimensional vectors.

[0147] In one embodiment, after the step of reading the video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain extracted frames, it further includes:

[0148] Screen out high-resolution frames;

[0149] Divide the high-resolution image objects into multiple sub-regions through an adaptive block algorithm to make the sub-regions match the pre-training settings of the visual encoder 50 to obtain high-resolution frame vectors;

[0150] Perform dimensionality reduction processing on the high-resolution frame vectors through a compression layer.

[0151] In one embodiment, in the step of fusing word embedding vectors, image feature vectors, audio vectors and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector, it includes:

[0152] Integrate the word embedding vectors, image feature vectors, audio vectors and high-dimensional vectors into the same semantic space to obtain a comprehensive information representation vector.

[0153] Those skilled in the art can understand, Figure 3The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied.

[0154] An embodiment of the present application also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, a multi-modal data classification method is implemented, including the following steps:

[0155] Receive multi-modal data through the edge device 10, and preprocess the multi-modal data to obtain multi-modal features; wherein, the multi-modal data includes text data, image data, audio data, and video data; the multi-modal features include word embedding vectors, image feature vectors, audio feature vectors, and high-dimensional feature vectors;

[0156] Align the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors to the same dimension through a linear transformation or a projection layer;

[0157] Fuse the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector;

[0158] Quantize a preset multi-modal large model and deploy it in the edge device 10;

[0159] Send the comprehensive information representation vector to the multi-modal large model for inference, and process the comprehensive information representation vector through a multi-layer Transformer architecture;

[0160] Generate a classification result according to the task requirements.

[0161] In one embodiment, in the step of receiving multi-modal data through the edge device 10 and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0162] Receive the input text of the user on the edge device 10;

[0163] Segment the input text into words, phrases, or characters through a preset tokenizer 20 and map them into word embedding vectors; wherein, the word embedding vectors include the semantic information of the input text.

[0164] In one embodiment, in the step of receiving multi-modal data through the edge device 10 and preprocessing the multi-modal data to obtain multi-modal features, it includes:

[0165] Convert the image data into image analysis features through a preset image encoder 40 to obtain image feature vectors.

[0166] In one embodiment, in the step of converting the image data into image analysis features through a preset image encoder 40, it includes:

[0167] Preprocess the input image; wherein, the preprocessing includes resizing to a unified size and normalization operations;

[0168] Generate visual tokens through a preset visual encoder 50;

[0169] Select high-resolution images;

[0170] Divide the high-resolution image into multiple slices through an adaptive image tiling algorithm, wherein the size of each slice matches the pre-training settings of the visual encoder 50;

[0171] Reduce the dimensionality of the visual tokens through a compression layer.

[0172] In one embodiment, when receiving multimodal data through the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0173] Convert the audio data into model features through a preset Whisper model to generate audio vectors;

[0174] Align the output data of the Whisper model with a preset large language model at the token level.

[0175] In one embodiment, when aligning the output data of the Whisper model with a preset large language model at the token level, it includes:

[0176] Align through forward KL divergence constraint.

[0177] In one embodiment, when receiving multimodal data through the edge device 10 and preprocessing the multimodal data to obtain multimodal features, it includes:

[0178] Read the video frames of the video through a preset tool and extract frames at fixed intervals according to the average frame rate of the video to obtain extracted frames;

[0179] Calculate the number of frames of the extracted frames;

[0180] Determine whether the number of frames exceeds a preset maximum value;

[0181] If so, screen the extracted frames through a uniform sampling algorithm to obtain screened frames;

[0182] Convert the screened frames into image objects and send them to the visual encoder 50 to generate high-dimensional vectors.

[0183] In one embodiment, after the step of reading the video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain extracted frames, it further includes:

[0184] Select high-resolution frames;

[0185] Divide the high-resolution image object into multiple sub-regions through an adaptive block algorithm to match the sub-regions with the pre-training settings of the visual encoder 50, obtaining high-resolution frame vectors;

[0186] Perform dimensionality reduction processing on the high-resolution frame vectors through a compression layer.

[0187] In one embodiment, in the step of fusing word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector, it includes:

[0188] Integrate the word embedding vectors, image feature vectors, audio vectors, and high-dimensional vectors into the same semantic space to obtain a comprehensive information representation vector.

[0189] In summary, for the multi-modal data classification method, system, computer device, and storage medium provided in the embodiments of the present application, the present application utilizes a multi-modal large model and an edge device 10 to achieve efficient recognition and classification of multimedia information. The multi-modal large model can significantly improve the ability to understand complex scenarios by integrating data features of different modalities. By performing local deployment on the edge device 10, it can not only improve the real-time performance, accuracy, and resource utilization efficiency of the system, but also effectively reduce the latency and bandwidth consumption during data transmission. In addition, sensitive data can be directly processed locally without uploading to the cloud, thus better protecting data privacy and security. The present application processes data of multiple modalities through a multi-modal large model to achieve cross-modal feature extraction and alignment, thereby improving the recognition ability in complex scenarios. It not only significantly reduces the dependence on large-scale labeled data, but also improves the adaptability and robustness of the model in new scenarios. In addition, by performing local fine-tuning on the multi-modal large model and making customized configurations according to user needs, the flexibility and adaptability of the system are improved, and the model engine is optimized through model quantization, enabling the model to run efficiently on devices with low power consumption and low storage. The multi-modal large model can simultaneously extract local details and global context information, thereby improving the recognition ability for scenarios with unclear features or complex backgrounds and enhancing the feature extraction and modeling ability.

[0190] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-described method embodiments. Among them, any reference to a memory, storage, database, or other medium provided in this application and used in the embodiments can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0191] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, apparatus, article, or method that includes such element.

[0192] The above are only the preferred embodiments of this application, and do not limit the patent scope of this application accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of this application.

Claims

1. A multi-modal data classification method, characterized in that, Including the following steps: Receiving multimodal data through an edge device, and preprocessing the multimodal data to obtain multimodal features; wherein, the multimodal data includes text data, image data, audio data, and video data; the multimodal features include word embedding vectors, image feature vectors, audio feature vectors, and high-dimensional feature vectors; Aligning the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector to the same dimension through a linear transformation or a projection layer; Fusing the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector through an attention mechanism or a Transformer architecture to generate a comprehensive information representation vector; Quantizing a preset multimodal large model and deploying it in the edge device; Sending the comprehensive information representation vector to the multimodal large model for inference, and processing the comprehensive information representation vector through multiple layers of the Transformer architecture; Generating a classification result according to the task requirements.

2. The multimodal data classification method according to claim 1, wherein In the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes: Receiving the input text of the user on the edge device; Segmenting the input text into words, phrases, or characters through a preset tokenizer and mapping them into word embedding vectors; wherein, the word embedding vectors include the semantic information of the input text.

3. The multimodal data classification method according to claim 1, wherein In the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes: Converting the image data into image analysis features through a preset image encoder to obtain the image feature vector.

4. The multimodal data classification method according to claim 1, characterized in that In the step of converting the image data into image analysis features through a preset image encoder, it includes: Preprocessing the input image; wherein, the preprocessing includes resizing to a unified size and normalization operations; Generating visual tokens through a preset vision encoder; Selecting high-resolution images; Dividing the high-resolution image into multiple slices through an adaptive image tiling algorithm, where the size of each slice matches the pre-training settings of the vision encoder; Performing dimensionality reduction processing on the visual tokens through a compression layer.

5. The multimodal data classification method according to claim 1, wherein In the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes: Converting the audio data into model features through a preset Whisper model to generate an audio vector; Aligning the output data of the Whisper model with a preset large language model at the token level.

6. The multimodal data classification method according to claim 5, characterized in that In the step of aligning the output data of the Whisper model with a preset large language model at the token level, it includes: Aligning through forward KL divergence constraint.

7. The multimodal data classification method according to claim 4, characterized in that In the step of receiving multimodal data through an edge device and preprocessing the multimodal data to obtain multimodal features, it includes: Reading the video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames; Calculating the number of frames of the extracted frames; Determine whether the number of frames exceeds a preset maximum value; If so, screen the extracted frames through a uniform sampling algorithm to obtain screened frames; Convert the screened frames into image objects and send them to the visual encoder to generate high-dimensional vectors.

8. The multimodal data classification method according to claim 7, wherein After the step of reading video frames of the video through a preset tool and extracting frames at fixed intervals according to the average frame rate of the video to obtain the extracted frames, it further includes: Screen out high-resolution frames; Divide the high-resolution image object into multiple sub-regions through an adaptive block algorithm to match the sub-regions with the pre-training settings of the visual encoder, obtaining high-resolution frame vectors; Perform dimensionality reduction processing on the high-resolution frame vectors through the compression layer.

9. The multimodal data classification method according to claim 1, wherein In the step of fusing the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector through the attention mechanism or the Transformer architecture to generate a comprehensive information representation vector, it includes: Integrate the word embedding vector, the image feature vector, the audio vector, and the high-dimensional vector into the same semantic space to obtain the comprehensive information representation vector.

10. A multimodal data classification system, characterized in that, It includes an edge device, and the edge device is deployed with a tokenizer, a visual encoder, an audio encoder, a multi-layer perceptron, and an image encoder; the tokenizer is electrically connected to the audio encoder; the visual encoder is electrically connected to the image encoder; the image encoder is electrically connected to the multi-layer perceptron; The tokenizer is used to split the input text into words, sub-words, or characters and map them into word embedding vectors; The visual encoder is used to extract image features and generate image feature vectors or visual tokens; The audio encoder is used to extract audio features and generate audio feature vectors; The multi-layer perceptron is used to map multiple input data sets to a single output data set; The image encoder is used to convert image data into a feature representation for model analysis.

11. A computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

Images