Big model-based multi-modal data analysis method and device

Abstract

The application provides a large model-based multi-modal data analysis method and device, the method comprising: constructing multi-modal data, including image data, text data and knowledge graph data; extracting an image feature vector according to the image data, and extracting a text feature vector according to the text data; performing cross-modal alignment and fusion on the image feature vector and the text feature vector through a preset mechanism to generate a cross-modal feature vector; extracting a time sequence feature vector from the image data, performing time sequence semantic joint modeling based on the cross-modal feature vector and the time sequence feature vector, inputting the features after joint modeling into a time sequence classifier to obtain a recognition result; and dynamically optimizing the large model based on an online learning and feedback mechanism. The method and device effectively eliminate the cross-modal semantic gap through multi-modal deep alignment and dynamic weight fusion, improve the accuracy and robustness of event recognition in complex scenarios, and support lightweight deployment and online adaptive optimization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for multimodal data analysis based on large models. Background Technology

[0002] With the deepening application of artificial intelligence technology in fields such as industrial security and intelligent transportation, the demand for vision-based dynamic event analysis is growing rapidly. Traditional analysis methods mainly rely on manually designed feature extraction methods (such as SIFT and HOG) combined with single-task classification models (such as CNN) to process image tasks through a fixed process. This approach has significant limitations: First, manually designed features are poorly robust to occlusion and lighting changes in complex scenes, making it difficult to capture high-level semantic relationships; second, traditional methods rely only on single visual modal data and cannot combine multi-source information such as text descriptions and knowledge graphs for deep semantic reasoning; third, sub-tasks such as detection, recognition, and reasoning require independent models, resulting in high system complexity and poor coordination; furthermore, existing solutions generally lack adaptive mechanisms for temporal information and scene changes, making it difficult to effectively handle complex dynamic events in video streams. These shortcomings make it difficult for traditional methods to meet practical application requirements in terms of accuracy, generalization ability, and system efficiency.

[0003] Therefore, there is a lack of existing technologies for a method and device for large-model multimodal data analysis that can deeply integrate multimodal information, has temporal modeling capabilities, and supports end-to-end inference. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and apparatus for multimodal data analysis based on a large model to address the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a multimodal data analysis method based on a large model, comprising:

[0006] Construct multimodal data, including image data, text data, and knowledge graph data;

[0007] Extract image feature vectors from image data and extract text feature vectors from text data;

[0008] Cross-modal feature vectors are generated by aligning and fusing image feature vectors and text feature vectors across modalities through a preset mechanism.

[0009] Temporal feature vectors are extracted from image data, and temporal semantic joint modeling is performed based on cross-modal feature vectors and temporal feature vectors. The jointly modeled features are then input into a temporal classifier to obtain the recognition results.

[0010] Based on online learning and feedback mechanisms, large models are dynamically optimized.

[0011] Optionally, the construction of multimodal data includes:

[0012] Acquire a continuous video frame segment centered on the original video frame as image data to capture the spatiotemporal context of dynamic events;

[0013] The predefined structured text description corresponding to the type of event to be identified is invoked as text data to provide clear semantic rules and judgment logic for the content analysis of continuous video frame segments. The structured text description includes at least event definition, judgment rules, exclusion rules, judgment basis and scene label.

[0014] By calling knowledge graphs or domain knowledge bases, entities, attributes, and their relationships are extracted into structured data and used as knowledge graph data. This data is then used to associate relevant entities, attributes, and relationships in the knowledge graph data with scene and target information extracted from continuous video frame segments and rules in text descriptions, thus injecting domain common sense and reasoning constraints into multimodal data analysis.

[0015] Optionally, the step of extracting image feature vectors from image data and extracting text feature vectors from text data includes:

[0016] The frame difference method is used to process the image data and initially extract the dynamic regions that have changed;

[0017] Within the initially extracted dynamic region, the motion vectors of pixels are calculated using the optical flow method to accurately extract the dynamic target region;

[0018] The dynamic target region is input into the image encoder of the CLIP model to extract the first image feature vector;

[0019] The dynamic target region is input into the ViT model to extract the second image feature vector;

[0020] The first image feature vector and the second image feature vector are fused to generate an image feature vector for cross-modal alignment and fusion;

[0021] Pre-defined text data related to the dynamic target region is pre-processed and then input into the text encoder of the CLIP model to generate text feature vectors.

[0022] Optionally, the step of performing cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors includes:

[0023] Using a bidirectional attention mechanism, the first correlation weight matrix of the image feature vector and the text feature vector in the image-text direction and the second correlation weight matrix of the image feature vector and the text feature vector in the text-image direction are calculated respectively, and the first correlation weight matrix and / or the second correlation weight matrix are used as the correlation weight matrix.

[0024] If the correlation weight matrix is ​​the first correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is fused with the weighted text feature vector to obtain the cross-modal feature vector;

[0025] If the association weight matrix is ​​the second association weight matrix, the image feature vector is weighted using the second association weight matrix to obtain the weighted image feature vector, and the text feature vector is fused with the weighted image feature vector to obtain the cross-modal feature vector;

[0026] If the correlation weight matrix includes both the first correlation weight matrix and the second correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is weighted using the second correlation weight matrix to obtain the weighted image feature vector. The weighted text feature vector and the weighted image feature vector are then fused to obtain the cross-modal feature vector.

[0027] Optionally, before calculating the first correlation weight matrix of the image feature vector and the text feature vector in the image-to-text direction and the second correlation weight matrix of the image feature vector and the text feature vector in the text-to-image direction using a bidirectional attention mechanism, and before using the first correlation weight matrix and / or the second correlation weight matrix as the correlation weight matrix, the method further includes:

[0028] Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0029] Calculate the first similarity between the image feature vector and the knowledge graph embedding vector, and calculate the first corrected weight matrix based on the first similarity; calculate the second similarity between the text feature vector and the knowledge graph embedding vector, and calculate the second corrected weight matrix based on the second similarity.

[0030] The image enhancement weighting matrix is ​​calculated based on the first modified weighting matrix, and the text enhancement weighting matrix is ​​calculated based on the second modified weighting matrix.

[0031] Based on the image enhancement weighting matrix, feature enhancement is performed on the image feature vector to obtain the enhanced image feature vector. Similarly, based on the text enhancement weighting matrix, feature enhancement is performed on the text feature vector to obtain the enhanced text feature vector.

[0032] Optionally, the step of performing cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors is implemented through a dynamic weighted feature fusion network, including:

[0033] Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0034] Obtain the task feature vector corresponding to the type of event to be identified;

[0035] Based on the task feature vector, the dynamic weights corresponding to the image feature vector, text feature vector and knowledge graph embedding vector are calculated respectively.

[0036] Based on dynamic weights, the image feature vector, text feature vector, and knowledge graph embedding vector are weighted and summed to obtain the cross-modal feature vector.

[0037] Optionally, the step of extracting temporal feature vectors from image data and performing joint temporal semantic modeling based on cross-modal feature vectors and temporal feature vectors includes:

[0038] A 3D convolutional neural network is used to encode image data and extract temporal feature vectors;

[0039] The cross-modal feature vector, temporal feature vector, and features obtained through the high-low point video linkage strategy are fused together. The features obtained through the high-low point video linkage strategy include: high-resolution texture feature vectors of dynamic target regions extracted from the low point view, and scene semantic feature vectors obtained from the high point view through semantic segmentation and knowledge graph reasoning.

[0040] By utilizing the cross-attention mechanism, joint modeling and spatial semantic alignment are performed on cross-modal feature vectors, temporal feature vectors, and feature vectors obtained through high and low point perspectives, thus completing the joint modeling of temporal and semantic aspects.

[0041] Optionally, the dynamic optimization of the large model based on the online learning and feedback mechanism includes:

[0042] Based on the difference between the recognition result generated by the temporal semantic joint modeling and the expected result, the model loss is calculated. The model loss includes the contrastive loss, which is calculated using the following formula: ,in, For similarity calculation function, For the model to predict features, For labeling features, For temperature parameters, Features of negative samples;

[0043] The parameters of the large model are updated based on the model loss.

[0044] Optionally, the dynamic optimization of the large model based on the online learning and feedback mechanism, implemented using an incremental learning strategy, includes:

[0045] Only update the parameters of the dynamic weight feature fusion network and / or the temporal classifier in the large model;

[0046] Maintain an incremental sample buffer to store samples with feedback annotations and their multimodal features;

[0047] Fine-tuning of the dynamic weight feature fusion network and / or temporal classifier based on samples in the buffer.

[0048] Secondly, the present invention provides a multimodal data analysis device based on a large model, comprising:

[0049] The multimodal data construction module is used to build multimodal data, including image data, text data, and knowledge graph data;

[0050] The feature extraction module, connected to the multimodal data construction module, is used to extract image feature vectors from image data and text feature vectors from text data.

[0051] The cross-modal alignment and fusion module, connected to the feature extraction module, is used to perform cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors;

[0052] The temporal semantic joint modeling module, connected to the cross-modal alignment and fusion module, is used to extract temporal feature vectors from image data, perform temporal semantic joint modeling based on cross-modal feature vectors and temporal feature vectors, and input the jointly modeled features into the temporal classifier in the temporal semantic joint modeling module to obtain the recognition result;

[0053] The dynamic optimization module is connected to the cross-modal alignment and fusion module and the temporal semantic joint modeling module, respectively, and is used to dynamically optimize large models based on online learning and feedback mechanisms.

[0054] The multimodal data analysis method and apparatus based on large models provided by this invention effectively overcomes the limitations of traditional methods in feature representation and the modal semantic gap in complex scenarios by introducing dynamic temporal features and a bidirectional attention mechanism, combined with multi-level cross-modal alignment and dynamic weight fusion strategies. It significantly improves the accuracy and robustness of dynamic event recognition, achieving efficient and deep fusion of image, text, and knowledge graph information. Furthermore, through joint temporal semantic modeling and scene reasoning enhancement mechanisms, it supports end-to-end multi-task parallel analysis and semantic reasoning. Moreover, by leveraging a lightweight adaptive framework and online learning mechanism, it greatly reduces system complexity and computational overhead, possessing good generalization ability and scene adaptability, making it suitable for edge computing scenarios with high real-time requirements such as industrial security and intelligent transportation. Attached Figure Description

[0055] Figure 1a This is a flowchart illustrating a multimodal data analysis method based on a large model provided in an embodiment of the present invention.

[0056] Figure 1b This is another flowchart illustrating the multimodal data analysis method based on a large model provided in an embodiment of the present invention;

[0057] Figure 1c This is another flowchart illustrating the multimodal data analysis method based on a large model provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the circuit module structure of the multimodal data analysis device based on a large model provided in an embodiment of the present invention;

[0059] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0061] like Figure 1a As shown, this invention provides a multimodal data analysis method based on a large model, including:

[0062] Step S11: Construct multimodal data, including image data, text data, and knowledge graph data;

[0063] Specifically, such as Figure 1b As shown, the construction of multimodal data in step S11 may include:

[0064] Step S111: Obtain a continuous video frame segment centered on the original video frame as image data to capture the spatiotemporal context of dynamic events;

[0065] Regarding the length, frame rate, and specific extraction method of the continuous video frame segments in step S111, those skilled in the art can flexibly set them according to the actual application scenario and the time span of the event to be identified. For example, for fast-moving events (such as vehicle collisions, intrusions, etc.), a higher frame rate can be used to capture the complete motion trajectory; for slowly changing events (such as smoke diffusion, equipment temperature rise, etc.), a lower frame rate can be used to reduce data redundancy and computational overhead. In addition, continuous video frame segments can be symmetrically extracted with the original video frame as the center, or asymmetrically extracted according to the start time of the event to be identified.

[0066] Step S112: Call the predefined structured text description corresponding to the event type to be identified as text data to provide clear semantic rules and judgment logic for the content analysis of continuous video frame segments. The structured text description includes at least event definition, judgment rules, exclusion rules, judgment basis and scene label.

[0067] For example, taking the "conveyor belt blockage detection" event as an example, its corresponding structured text description can be defined using the following JSON format:

[0068] json

[0069] {

[0070] Event Definition: "An abnormal state where material accumulation at the conveyor belt outlet exceeds a set threshold, preventing normal material discharge."

[0071] "Judgment Rules":[

[0072] "When the material height in the discharge port area continues to rise and exceeds the preset height threshold, it is determined as a material blockage event."

[0073] "A material blockage alarm will be triggered when the duration of the blockage exceeds a preset time threshold (e.g., 5 seconds)."

[0074] ],

[0075] "Exclusion Rules":[

[0076] "No material blockage detection is performed during normal equipment shutdown."

[0077] "No material blockage detection is performed during system self-check mode."

[0078] ],

[0079] "Judgment basis":[

[0080] "The material accumulation height in the discharge port area exceeds the preset threshold".

[0081] "Conveyor belt motor current abnormally high"

[0082] "The material has stopped moving forward."

[0083] ],

[0084] Scene Tags: ["Conveyor Belt", "Discharge Port", "Material Accumulation", "Material Blockage Alarm"]

[0085] }

[0086] In the structured text description above, "event definition" is used to clarify the essential meaning of the event to be identified, providing a semantic understanding basis for the large model; "judgment rules" are used to specify the specific conditions for the event to be triggered, providing a decision basis for the time series classifier; "exclusion rules" are used to define special scenarios that do not need to be detected, so as to reduce the false alarm rate; "judgment basis" is used to list multi-dimensional evidence to assist the large model in making judgments; "scene labels" are used to semantically label the current scene, which facilitates knowledge transfer and parameter adaptation of the large model between different scenes.

[0087] It should be noted that the above JSON format is only an exemplary implementation of the present invention. Those skilled in the art can flexibly adopt other structured data formats such as XML and YAML according to actual application scenarios and task requirements, or add, subtract and adjust the above fields. No limitation is made here.

[0088] Step S113: Call the knowledge graph or domain knowledge base to extract the entities, attributes and their relationships into structured data, which is used as knowledge graph data. This data is then used to associate the relevant entities, attributes and relationships in the knowledge graph data with the scene, target information and rules in the text description extracted from continuous video frame segments, thus injecting domain common sense and reasoning constraints into multimodal data analysis.

[0089] Step S12: Extract image feature vectors from image data and extract text feature vectors from text data;

[0090] Optionally, such as Figure 1c As shown, step S12 specifically includes:

[0091] Step S121: The image data is processed using the frame difference method to initially extract the dynamic regions that have changed;

[0092] Step S122: Within the initially extracted dynamic region, the motion vector of the pixel is calculated using the optical flow method to accurately extract the dynamic target region;

[0093] In steps S121 and S122 of this invention, the frame difference method is first used to quickly and coarsely locate the dynamic regions that are changing in the image data (i.e., continuous video frame segments), greatly narrowing the scope of subsequent fine processing. Then, computationally intensive optical flow is applied only within the initially extracted dynamic regions to accurately calculate pixel motion vectors and extract target contours. This cascaded processing flow from coarse to fine overcomes the limitations of a single method: on the one hand, it effectively shields static background interference through the frame difference method, improving the signal-to-noise ratio; on the other hand, it avoids optical flow calculations across the entire image range, improving computational efficiency and meeting the stringent requirements of high real-time application scenarios. Simultaneously, the two methods complement each other; the optical flow method effectively compensates for the shortcomings of the frame difference method in target contour extraction, jointly ensuring the accuracy and completeness of dynamic target region extraction, laying a solid foundation for subsequent high-quality cross-modal feature alignment and fusion.

[0094] Step S123: Input the dynamic target region into the image encoder of the CLIP model to extract the first image feature vector;

[0095] Step S124: Input the dynamic target region into the ViT model to extract the second image feature vector;

[0096] Among them, the CLIP (Contrastive Language-Image Pre-training) model and the ViT (Vision Transformer) model adopt the corresponding models in the existing technology. Those skilled in the art can flexibly choose according to actual needs, and there is no limitation here.

[0097] Step S125: Fuse the first image feature vector with the second image feature vector to generate an image feature vector for cross-modal alignment and fusion;

[0098] In steps S123 to S125 of this invention, the CLIP model, trained on massive amounts of image-text pairs, is first used to extract a first image feature vector from the dynamic target region. This first vector possesses strong semantic generalization capabilities and cross-modal affinity. Its advantage lies in its ability to directly map the content of consecutive video frame segments to a high-level semantic space aligned with the text. Subsequently, the ViT model performs secondary encoding on the dynamic target region, focusing on capturing local fine features and structural details that CLIP might overlook, forming a second image feature vector with complementary information. The final image feature vector generated by the fusion of these two models contains both the abstract semantic information from the CLIP model, which facilitates interaction with text features, and the specific visual details provided by the ViT model, which are crucial for target recognition and classification. This collaborative strategy effectively overcomes the feature limitations of a single model, laying a more comprehensive and higher-quality visual foundation for subsequent cross-modal alignment and fusion, thereby improving the overall accuracy of dynamic event analysis in complex scenarios.

[0099] Let's take the detection of "vehicle fire" in highway surveillance video as an example. Suppose the surveillance camera captures a continuous video frame segment, in which smoke is coming out of the hood of a car.

[0100] First, the dynamic target region containing the car and smoke is input into the CLIP model's image encoder. Leveraging its powerful semantic understanding capabilities trained on massive image-text pairs, the CLIP model immediately captures the most core and global semantic information of this dynamic target region. The resulting first image feature vector essentially represents a visual scene highly correlated with concepts such as vehicles, smoke, and fire. This first image feature vector is inherently aligned with the text space, making it easy to associate it with textual descriptions such as "vehicle spontaneous combustion" and "fire" in subsequent steps. This quickly identifies the semantic category of the event, laying a solid foundation for cross-modal alignment and fusion.

[0101] Secondly, the same dynamic target area is input into the ViT model. The ViT model utilizes its powerful visual Transformer architecture to focus on analyzing image details. The second image feature vector generated by the ViT model encodes more refined visual information, such as: "The smoke is continuously billowing from the gaps in the hood, rather than water vapor; the vehicle's brand and model are XX sedans; there is color distortion in the hood due to localized high temperatures." These details are crucial for determining the authenticity and severity of the event. The ViT model compensates for the CLIP model's shortcomings in fine-grained visual features, helping to distinguish between "real fire" and similar situations such as "exhaust pipe water vapor" and "reflected halos," significantly reducing the false alarm rate.

[0102] Finally, the first image feature vector output by the CLIP model (i.e., semantic features: this is a vehicle on fire) is fused with the second image feature vector output by the ViT model (i.e., detail features: the fire is thick smoke coming from the hood). The resulting image feature vector is an extremely information-rich vector representation. It not only ensures the understanding of macroscopic events but also allows for the clear visualization of microscopic evidence, thus enabling faster, more accurate, and more reliable judgments when faced with complex and ambiguous visual scenes.

[0103] The method of fusing the first image feature vector output by the CLIP model with the second image feature vector output by the ViT model can be a combination of concatenation and linear transformation. Of course, those skilled in the art can also use other methods, which are not limited here.

[0104] Specifically, firstly, the first image feature vector (Assuming the dimension is 512) and the second image feature vector (Assuming a dimension of 768) They are directly concatenated to form a longer joint vector. (Dimensionality is 512 + 768 = 1280); then, this high-dimensional joint vector is projected onto a new, fixed-dimensional feature space (e.g., 512-dimensional) through an existing trainable fully connected layer (linear layer). The role of this fully connected layer is to learn how to most effectively combine information from the CLIP and ViT models and remove redundancy. According to the formula: ,in, For image feature vectors, This indicates that the first image feature vector With the second image feature vector To splice, For a trainable weight matrix, is a trainable bias vector.

[0105] Assume the feature vector of the first image is a 3-dimensional vector, i.e. The second image feature vector is a 5-dimensional vector, that is... .

[0106] First, the two image feature vectors are directly concatenated to obtain an 8-dimensional joint vector. ,Right now [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8].

[0107] Then, this high-dimensional joint vector is passed through an existing trainable fully connected layer (linear layer) to project it onto a new, fixed-dimensional feature space. Suppose we want to project an 8-dimensional joint vector onto a 2-dimensional fused feature vector, we define trainable parameters: the weight matrix. The size is 2×8, and the bias vector is... Its size is 2×1. For example, The weights of the first row are [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]. The weights of the second row are [0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6]. .

[0108] Finally, according to the formula Calculate the first dimension: 0.1×0.1+0.2×0.2+0.3×0.3+0.4×0.4+0.5×0.5+0.6×0.6+0.7×0.7+0.8×0.8+0.01=2.05, and the second dimension: 0.9×0.1+1.0×0.2+1.1×0.3+1.2×0.4+1.3×0.5+1.4×0.6+1.5×0.7+1.6×0.8+0.02=4.94, to obtain the fused 2D image feature vector. .

[0109] Step S126: Input the predefined text data related to the dynamic target region into the text encoder of the CLIP model after preprocessing to generate text feature vectors.

[0110] The text data used in step S126 originates from the "structured text description" defined in step S112. The system has a built-in standardized text rule base for various recognition events (such as fires, intrusions, etc.). When the system initiates a specific event recognition task, it automatically calls the corresponding text data according to the task type. Since the CLIP model's text encoder accepts natural language statements as input, rather than structured data formats, the system needs to preprocess the text data: extracting fields such as event definitions, judgment rules, exclusion rules, judgment criteria, and scene labels from the structured text description, combining them into a complete natural language description statement according to a preset template, then processing it into a word sequence by the CLIP model's tokenizer before inputting it into the text encoder, finally outputting the corresponding text feature vector. This method ensures that the text modality provides stable and reliable prior knowledge, providing an accurate semantic benchmark for subsequent cross-modal alignment and fusion.

[0111] Step S13: Perform cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors;

[0112] In an optional embodiment of the present invention, step S13 specifically includes:

[0113] Step S131: Through the bidirectional attention mechanism, calculate the first correlation weight matrix of the image feature vector and the text feature vector in the image-text direction and the second correlation weight matrix of the image feature vector and the text feature vector in the text-image direction, and use the first correlation weight matrix and / or the second correlation weight matrix as the correlation weight matrix.

[0114] The first correlation weight matrix is ​​calculated using the following formula: ,in, This is the first correlation weight matrix. For image feature vectors, Let be the first parameter matrix that can be trained. For text feature vectors, The dimension of the feature vector. This is the scaling factor.

[0115] The second correlation weight matrix is ​​calculated using the following formula: ,in, This is the second correlation weight matrix. For image feature vectors, Let be the trainable second parameter matrix. For text feature vectors, The dimension of the feature vector. This is the scaling factor.

[0116] The following applies to the trainable first parameter matrix and the trainable second parameter matrix The iterative training process, based on the mechanism of forward and backward propagation, belongs to a well-known neural network training paradigm in this field. Specifically, the trainable first parameter matrix... and the trainable second parameter matrix It is obtained through an end-to-end model training process, and the specific training mechanism is as follows:

[0117] (1) Initialization: At the start of training, and Initial values ​​are assigned using random initialization or standard initialization strategies (such as Xavier initialization, He initialization, etc.) to ensure the stability of gradient propagation.

[0118] (2) Forward propagation: The image feature vectors of the current batch are... and text feature vectors Input bidirectional attention layer, utilizing the current and Calculate the first correlation weight matrix With the second correlation weight matrix This process generates cross-modal feature vectors, which are then fed into the subsequent feature fusion module and temporal classifier to obtain the model prediction results.

[0119] (3) Loss calculation and gradient update: Input the model prediction results and the true labels into the preset loss function (such as cross-entropy loss, contrastive loss, etc.) to calculate the model loss. Subsequently, based on the automatic differentiation and backpropagation algorithm of the deep learning framework, the loss function is calculated. For parameter matrix and The gradient is calculated strictly following the chain rule. The gradient signal originates from the loss layer and propagates backward layer by layer through the temporal classifier, feature fusion module, cross-modal feature generation process, and bidirectional attention layer, finally returning to the first parameter matrix. Second parameter matrix This allows for the full utilization of gradient information from each intermediate module.

[0120] (4) Parameter iteration: Use an optimizer (such as Adam, SGD, etc.) to iterate the parameters based on the backpropagated gradients. and Update the model. Repeat the forward propagation and backward iteration process described above until the model converges. After training is complete, and That is, the optimal cross-modal alignment transformation relationship between image semantics and text semantics has been learned.

[0121] Step S132: If the correlation weight matrix is ​​the first correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is fused with the weighted text feature vector to obtain the cross-modal feature vector;

[0122] If the association weight matrix is ​​the second association weight matrix, the image feature vector is weighted using the second association weight matrix to obtain the weighted image feature vector, and the text feature vector is fused with the weighted image feature vector to obtain the cross-modal feature vector;

[0123] If the correlation weight matrix includes both the first correlation weight matrix and the second correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is weighted using the second correlation weight matrix to obtain the weighted image feature vector. The weighted text feature vector and the weighted image feature vector are then fused to obtain the cross-modal feature vector.

[0124] Suppose an image is encoded into two feature blocks, one corresponding to "person" and the other to "kitchen". Its image feature vector... The text description corresponding to the image was also encoded into two words: "person" and "in the kitchen," resulting in a text feature vector. .

[0125] First, according to the formula Calculate the first correlation weight matrix between the image feature vector and the text feature vector in the image-text direction. In this embodiment, assume the dimension of the feature vector is... ,therefore Meanwhile, to facilitate the explanation of the calculation process, it is assumed that the first parameter matrix can be trained after sufficient training. It has converged to (This value is for example only; the actual parameter matrix after training depends on the specific training data.)

[0126] Specifically, calculation and calculate Apply the Softmax method to each row so that the sum of each row is 1 to obtain the association weight matrix. Correlation weight matrix The meaning is as follows: The first row represents the "person" area of ​​the image, with a correlation of 0.495 with the text word "person" and a correlation of 0.505 with "in the kitchen"; the second row represents the "kitchen" area of ​​the image, with a correlation of 0.504 with the text word "person" and a correlation of 0.496 with "in the kitchen".

[0127] Then, based on the association weight matrix, the weighted text feature vector is calculated. Specifically, the weighted text feature vector... = .

[0128] Finally, the image feature vector is fused with the weighted text feature vector to obtain the cross-modal feature vector. For example, methods such as concatenation or element-wise addition can be used to fuse the image feature vector with the weighted text feature vector. If the concatenation method is used, continuing the previous example, the cross-modal feature vector... .

[0129] This embodiment uses the first correlation weight matrix as the correlation weight matrix. Those skilled in the art will understand, based on this embodiment, the case where the second correlation weight matrix is ​​used as the correlation weight matrix, and will not be described in detail here.

[0130] It is important to note that when both the first and second correlation weight matrices are used as correlation weight matrices, cross-modal feature vectors are fused through concatenation. Specifically, the text feature vectors are first weighted using the first correlation weight matrix to obtain the weighted text feature vectors in the image space. Simultaneously, the image feature vector is weighted using the second correlation weight matrix to obtain the weighted image feature vector in the text space. Then, and By concatenating along the feature dimension, we obtain the fused cross-modal feature vector: ,Right now: ,in, This is the first correlation weight matrix. For text feature vectors, This is the second correlation weight matrix. Let be the image feature vector, [ ;] represent the concatenation operation along the feature dimension, and ReLU is the activation function.

[0131] Optionally, the method may further include the following steps before step S131:

[0132] Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0133] Calculate the first similarity between the image feature vector and the knowledge graph embedding vector, and calculate the first corrected weight matrix based on the first similarity; calculate the second similarity between the text feature vector and the knowledge graph embedding vector, and calculate the second corrected weight matrix based on the second similarity.

[0134] The image enhancement weighting matrix is ​​calculated based on the first modified weighting matrix, and the text enhancement weighting matrix is ​​calculated based on the second modified weighting matrix.

[0135] Based on the image enhancement weighting matrix, feature enhancement is performed on the image feature vector to obtain the enhanced image feature vector. Similarly, based on the text enhancement weighting matrix, feature enhancement is performed on the text feature vector to obtain the enhanced text feature vector.

[0136] For example, suppose the image feature vector , representing the "human" region; text feature vector , representing the word "person"; knowledge graph embedding vector set ,in, Representing the entity of "humanity", Represents the "tool" entity. Indicates the physical entity of "kitchen". This refers to the attribute of "human height". This indicates a relationship of "humans using tools". Specifically:

[0137] (1) Calculate the image feature vector With knowledge graph embedding vector set The first similarity of each vector in the matrix is ​​calculated, where the first similarity includes similarity 1 to similarity 5: similarity 1 = [0.8, 0.2]·[0.1, 0.1] = 0.10; similarity 2 = [0.8, 0.2]·[0.2, 0.2] = 0.20; similarity 3 = [0.8, 0.2]·[0.3, 0.3] = 0.30; similarity 4 = [0.8, 0.2]·[0.15, 0.15] = 0.15; similarity 5 = [0.8, 0.2]·[0.25, 0.25] = 0.25. Based on the first similarity, the first corrected weight matrix β is calculated. img =softmax([0.10,0.20,0.30,0.15,0.25])=[0.16,0.20,0.24,0.18,0.22].

[0138] (2) According to the first corrected weight matrix β img Calculate the image enhancement weighting matrix: =0.16×[0.1,0.1]+0.20×[0.2,0.2]+0.24×[0.3,0.3]+0.18×[0.15,0.15]+0.22×[0.25,0.25]=[0.210,0.210].

[0139] (3) Based on the image enhancement weighting matrix For image feature vectors Feature enhancement is performed to obtain the enhanced image feature vector: This method of introducing knowledge graph embedding vectors into image feature vectors integrates comprehensive semantic information about entities, attributes, and relationships from a knowledge graph into image feature vectors that contain only visual information.

[0140] (4) Similarly, following the above (1) to (3), the text enhancement weighting matrix can be obtained. and the enhanced text feature vector .

[0141] It should be noted that once the enhanced image feature vector and the enhanced text feature vector are obtained, steps S131 and S132 can be executed.

[0142] In another optional embodiment of the present invention, step S13 is implemented through a dynamic weighted feature fusion network, which specifically includes:

[0143] Step S131': Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0144] Step S132': Obtain the task feature vector corresponding to the type of event to be identified;

[0145] The task feature vectors are obtained through one of two methods. First, a task-specific embedding method is used: the system pre-defines a trainable task embedding table, mapping event types such as "fire detection" and "equipment fault diagnosis" to corresponding vectors. After initialization, this vector is optimized along with the network parameters during model training, thereby learning the essential features of each task. Second, a task text encoding method is used: the text description of the task (e.g., "Please detect flames or smoke in the scene") is input into a pre-trained text encoder (e.g., CLIP's text encoder), semantically encoding its meaning into a task feature vector. Both methods can generate vector representations that guide the model to focus on key task information.

[0146] Step S133': Based on the task feature vector, calculate the dynamic weights corresponding to the image feature vector, text feature vector, and knowledge graph embedding vector respectively;

[0147] The dynamic weights in step S133' are calculated using the following formula: ,in, Take respectively To calculate the dynamic weights for each modality accordingly; , , These are the importance coefficients for the visual modality, the text modality, and the knowledge graph embedding vector, respectively. This represents the task feature vector.

[0148] Step S134': Based on the dynamic weights, perform a weighted summation of the image feature vector, text feature vector, and knowledge graph embedding vector to obtain the cross-modal feature vector.

[0149] Specifically, the cross-modal feature vector is calculated using the following formula: ,in, For cross-modal feature vectors, For visual modality dynamic weights, For image feature vectors, For text modal dynamic weights, For text feature vectors, For the dynamic weights of the spectral modes, Embed vectors for knowledge graphs.

[0150] In one specific embodiment of the present invention, the calculation process of the dynamic weight feature fusion network is as follows: (1) Obtain the task feature vector corresponding to the event type to be identified, and read the importance coefficients of the visual modality, text modality, and knowledge graph embedding vector in the current model. These importance coefficients are usually initialized to the same value in the early stage of training (any value can be taken, such as 1.0, which will be adjusted through training in the later stage). (2) At the beginning of the calculation, the importance coefficient of each modality is multiplied by the task feature vector to obtain the original score of each modality; by exponentializing and normalizing these original scores, the dynamic weight of each modality is generated. This step ensures that the sum of all weights is 1, and the contribution ratio of each modality can be dynamically adjusted according to the task requirements. (3) The calculated dynamic weights are weighted and summed with the corresponding feature vectors. Specifically, the visual feature vector, text feature vector, and knowledge graph embedding vector are multiplied by their corresponding dynamic weights and then added together to finally generate a cross-modal feature vector that integrates multimodal information. (4) During training, the importance coefficients of each modality are continuously optimized through backpropagation. When the model prediction deviates, these importance coefficients will be automatically adjusted so that the modalities that contribute more in a specific task will receive higher weights. After multiple iterations of training, the model can autonomously learn the optimal modal weight allocation strategy for different tasks, thereby achieving more accurate multimodal data analysis.

[0151] It should be noted that steps S131, S132, and S131' to S134' are for illustrative purposes only and are not shown in the figures.

[0152] Step S14: Extract temporal feature vectors from the image data, perform joint temporal semantic modeling based on cross-modal feature vectors and temporal feature vectors, and input the jointly modeled features into a temporal classifier to obtain the recognition result;

[0153] Optionally, step S14, which involves extracting temporal feature vectors from the image data and performing joint temporal semantic modeling based on cross-modal feature vectors and temporal feature vectors, specifically includes:

[0154] A 3D convolutional neural network is used to encode image data and extract temporal feature vectors;

[0155] The cross-modal feature vector, temporal feature vector, and features obtained through the high-low point video linkage strategy are fused together. The features obtained through the high-low point video linkage strategy include: high-resolution texture feature vectors of dynamic target regions extracted from the low point view, and scene semantic feature vectors obtained from the high point view through semantic segmentation and knowledge graph reasoning.

[0156] By utilizing the cross-attention mechanism, joint modeling and spatial semantic alignment are performed on cross-modal feature vectors, temporal feature vectors, and feature vectors obtained through high and low point perspectives, thus completing the joint modeling of temporal and semantic aspects.

[0157] For example, taking the identification of "abnormal equipment vibration" in an industrial plant monitoring scenario as an example, we can illustrate the specific implementation process of temporal semantic joint modeling.

[0158] First, acquire continuous video frame segments centered on the target device (e.g., 10 frames per second, lasting 2 seconds). Extract temporal feature vectors using a 3D-ResNet network, which captures dynamic patterns such as device vibration frequency and swaying trajectory through 3D convolutional kernels. Simultaneously, a high-low point video linkage strategy is activated: a low-point camera deployed near the device captures high-resolution texture features, extracting vibration texture changes on the motor surface, thus obtaining a high-resolution texture feature vector; a high-point camera located on the factory roof identifies scene elements such as "conveyor belt area" and "restricted area" through semantic segmentation, and combines this with knowledge from a knowledge graph, such as "maximum amplitude threshold for this model of equipment" and "continuous shaking may cause breakage," to generate a scene semantic feature vector.

[0159] Next, a cross-attention mechanism is used for multi-feature fusion: the temporal feature vector is used as the query vector, the cross-modal feature (which has been fused with the "device jitter" text rule) is used as the key vector, and the feature vector obtained from high and low point perspectives is used as the value vector. By calculating attention weights, the model automatically focuses on the spatiotemporal region related to the semantics of "abnormal jitter" (such as vibrations at specific frequencies and amplitudes exceeding thresholds), while suppressing irrelevant background interference.

[0160] Finally, the features obtained from the joint modeling are input into the time-series classifier to accurately identify the "abnormal equipment vibration" event and trigger an alarm.

[0161] Step S15: Dynamically optimize the large model based on online learning and feedback mechanisms.

[0162] Optionally, step S15 specifically includes:

[0163] Based on the difference between the recognition result generated by the temporal semantic joint modeling and the expected result, the model loss is calculated. The model loss includes the contrastive loss, which is calculated using the following formula: ,in, For similarity calculation function, For the model to predict features, For labeling features, For temperature parameters, Features of negative samples;

[0164] The parameters of the large model are updated based on the model loss.

[0165] Among them, labeled features Negative sample features and temperature parameters All parameters are preset, and those skilled in the art can select them according to actual needs; no limitations are imposed here. Furthermore, those skilled in the art can also flexibly select updated large model parameters according to actual needs; no limitations are imposed here.

[0166] Optionally, step S15 is implemented using an incremental learning strategy, specifically including:

[0167] Only update the parameters of the dynamic weight feature fusion network and / or the temporal classifier in the large model;

[0168] Maintain an incremental sample buffer to store samples with feedback annotations and their multimodal features;

[0169] Fine-tuning of the dynamic weight feature fusion network and / or temporal classifier based on samples in the buffer.

[0170] Taking the scenario of conveyor belt blockage recognition as an example, the online learning and feedback optimization process is illustrated:

[0171] For example, the system maintains an incremental sample buffer containing 1000 samples, including: difficult sample samples: visually similar but actually different scenarios (such as stacked materials vs. normal stacking); new scenario samples: samples with new material types and changing lighting conditions; 200 samples are sampled from the incremental sample buffer every week, and only the dynamic weight feature fusion network and temporal classifier are fine-tuned.

[0172] The multimodal data analysis method based on a large model provided by this invention effectively overcomes the limitations of traditional methods in feature representation and the modal semantic gap in complex scenarios by introducing dynamic temporal features and a bidirectional attention mechanism, combined with multi-level cross-modal alignment and dynamic weight fusion strategies. It significantly improves the accuracy and robustness of dynamic event recognition, achieving efficient and deep fusion of image, text, and knowledge graph information. Furthermore, through joint temporal semantic modeling and scene reasoning enhancement mechanisms, it supports end-to-end multi-task parallel analysis and semantic reasoning. Moreover, by leveraging a lightweight adaptive framework and online learning mechanism, it significantly reduces system complexity and computational overhead, possessing good generalization ability and scene adaptability, making it suitable for edge computing scenarios with high real-time requirements such as industrial security and intelligent transportation.

[0173] Based on the same inventive concept, embodiments of the present invention also provide a large-model-based multimodal data analysis device for implementing the aforementioned large-model-based multimodal data analysis method. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more embodiments of the large-model-based multimodal data analysis device provided below can be found in the limitations of the large-model-based multimodal data analysis method described above, and will not be repeated here.

[0174] like Figure 2 As shown, this invention provides a multimodal data analysis device based on a large model, comprising: a multimodal data construction module 21, a feature extraction module 22, a cross-modal alignment and fusion module 23, a temporal semantic joint modeling module 24, and a dynamic optimization module 25; wherein,

[0175] Multimodal data construction module 21 is used to construct multimodal data, including image data, text data and knowledge graph data;

[0176] The feature extraction module 22, connected to the multimodal data construction module 21, is used to extract image feature vectors from image data and extract text feature vectors from text data.

[0177] The cross-modal alignment and fusion module 23, connected to the feature extraction module 22, is used to perform cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors;

[0178] The temporal semantic joint modeling module 24 is connected to the cross-modal alignment and fusion module 23. It is used to extract temporal feature vectors from image data, perform temporal semantic joint modeling based on cross-modal feature vectors and temporal feature vectors, and input the jointly modeled features into the temporal classifier in the temporal semantic joint modeling module 24 to obtain the recognition result.

[0179] The dynamic optimization module 25 is connected to the cross-modal alignment and fusion module 23 and the temporal semantic joint modeling module 24, respectively, and is used to dynamically optimize the large model based on online learning and feedback mechanisms.

[0180] Optionally, the multimodal data construction module 21 is specifically used for:

[0181] Acquire a continuous video frame segment centered on the original video frame as image data to capture the spatiotemporal context of dynamic events;

[0182] The predefined structured text description corresponding to the type of event to be identified is invoked as text data to provide clear semantic rules and judgment logic for the content analysis of continuous video frame segments. The structured text description includes at least event definition, judgment rules, exclusion rules, judgment basis and scene label.

[0183] By calling knowledge graphs or domain knowledge bases, entities, attributes, and their relationships are extracted into structured data and used as knowledge graph data. This data is then used to associate relevant entities, attributes, and relationships in the knowledge graph data with scene and target information extracted from continuous video frame segments and rules in text descriptions, thus injecting domain common sense and reasoning constraints into multimodal data analysis.

[0184] Optionally, the feature extraction module 22 is specifically used for:

[0185] The frame difference method is used to process the image data and initially extract the dynamic regions that have changed;

[0186] Within the initially extracted dynamic region, the motion vectors of pixels are calculated using the optical flow method to accurately extract the dynamic target region;

[0187] The dynamic target region is input into the image encoder of the CLIP model to extract the first image feature vector;

[0188] The dynamic target region is input into the ViT model to extract the second image feature vector;

[0189] The first image feature vector and the second image feature vector are fused to generate an image feature vector for cross-modal alignment and fusion;

[0190] Pre-defined text data related to the dynamic target region is pre-processed and then input into the text encoder of the CLIP model to generate text feature vectors.

[0191] Optionally, the cross-modal alignment and fusion module 23 is specifically used for:

[0192] Using a bidirectional attention mechanism, the first correlation weight matrix of the image feature vector and the text feature vector in the image-text direction and the second correlation weight matrix of the image feature vector and the text feature vector in the text-image direction are calculated respectively, and the first correlation weight matrix and / or the second correlation weight matrix are used as the correlation weight matrix.

[0193] If the correlation weight matrix is ​​the first correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is fused with the weighted text feature vector to obtain the cross-modal feature vector;

[0194] If the association weight matrix is ​​the second association weight matrix, the image feature vector is weighted using the second association weight matrix to obtain the weighted image feature vector, and the text feature vector is fused with the weighted image feature vector to obtain the cross-modal feature vector;

[0195] If the correlation weight matrix includes both the first correlation weight matrix and the second correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is weighted using the second correlation weight matrix to obtain the weighted image feature vector. The weighted text feature vector and the weighted image feature vector are then fused to obtain the cross-modal feature vector.

[0196] Optionally, the cross-modal alignment and fusion module 23 is also used for:

[0197] Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0198] Calculate the first similarity between the image feature vector and the knowledge graph embedding vector, and calculate the first corrected weight matrix based on the first similarity; calculate the second similarity between the text feature vector and the knowledge graph embedding vector, and calculate the second corrected weight matrix based on the second similarity.

[0199] The image enhancement weighting matrix is ​​calculated based on the first modified weighting matrix, and the text enhancement weighting matrix is ​​calculated based on the second modified weighting matrix.

[0200] Based on the image enhancement weighting matrix, feature enhancement is performed on the image feature vector to obtain the enhanced image feature vector. Similarly, based on the text enhancement weighting matrix, feature enhancement is performed on the text feature vector to obtain the enhanced text feature vector.

[0201] Optionally, cross-modal alignment and fusion of image feature vectors and text feature vectors are performed through a preset mechanism to generate cross-modal feature vectors. This is achieved through a dynamic weighted feature fusion network, including:

[0202] Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors;

[0203] Obtain the task feature vector corresponding to the type of event to be identified;

[0204] Based on the task feature vector, the dynamic weights corresponding to the image feature vector, text feature vector and knowledge graph embedding vector are calculated respectively.

[0205] Based on dynamic weights, the image feature vector, text feature vector, and knowledge graph embedding vector are weighted and summed to obtain the cross-modal feature vector.

[0206] Optionally, the temporal semantic joint modeling module 24 is specifically used for:

[0207] A 3D convolutional neural network is used to encode image data and extract temporal feature vectors;

[0208] The cross-modal feature vector, temporal feature vector, and features obtained through the high-low point video linkage strategy are fused together. The features obtained through the high-low point video linkage strategy include: high-resolution texture feature vectors of dynamic target regions extracted from the low point view, and scene semantic feature vectors obtained from the high point view through semantic segmentation and knowledge graph reasoning.

[0209] By utilizing the cross-attention mechanism, joint modeling and spatial semantic alignment are performed on cross-modal feature vectors, temporal feature vectors, and feature vectors obtained through high and low point perspectives, thus completing the joint modeling of temporal and semantic aspects.

[0210] Optionally, the dynamic optimization module 25 is specifically used for:

[0211] Based on the difference between the recognition result generated by the temporal semantic joint modeling and the expected result, the model loss is calculated. The model loss includes the contrastive loss, which is calculated using the following formula: ,in, For similarity calculation function, For the model to predict features, For labeling features, For temperature parameters, Features of negative samples;

[0212] The parameters of the large model are updated based on the model loss.

[0213] Optionally, the dynamic optimization module 25 is implemented using an incremental learning strategy, specifically for:

[0214] Only update the parameters of the dynamic weight feature fusion network in the cross-modal alignment and fusion module 23 and / or the temporal classifier in the temporal semantic joint modeling module 24 in the large model;

[0215] Maintain an incremental sample buffer to store samples with feedback annotations and their multimodal features;

[0216] Fine-tuning of the dynamic weight feature fusion network and / or temporal classifier based on samples in the buffer.

[0217] The multimodal data analysis device based on a large model provided by this invention effectively overcomes the limitations of traditional methods in feature representation and the modal semantic gap in complex scenarios by introducing dynamic temporal features and a bidirectional attention mechanism, combined with multi-level cross-modal alignment and dynamic weight fusion strategies. It significantly improves the accuracy and robustness of dynamic event recognition, achieving efficient and deep fusion of image, text, and knowledge graph information. Furthermore, through temporal semantic joint modeling and scene reasoning enhancement mechanisms, it supports end-to-end multi-task parallel analysis and semantic reasoning. Moreover, by leveraging a lightweight adaptive framework and online learning mechanism, it significantly reduces system complexity and computational overhead, possessing good generalization ability and scene adaptability, making it suitable for edge computing scenarios with high real-time requirements such as industrial security and intelligent transportation.

[0218] It should be noted that "multiple" in this invention includes two or more.

[0219] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0220] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0221] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data required for or generated by the aforementioned large-model-based multimodal data analysis method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a large-model-based multimodal data analysis method.

[0222] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multimodal data analysis method based on a large model. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0223] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0224] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0225] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0226] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0228] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal data analysis method based on a large model, characterized in that, include: Construct multimodal data, including image data, text data, and knowledge graph data; Extract image feature vectors from image data and extract text feature vectors from text data; Cross-modal feature vectors are generated by aligning and fusing image feature vectors and text feature vectors across modalities through a preset mechanism. Temporal feature vectors are extracted from image data, and temporal semantic joint modeling is performed based on cross-modal feature vectors and temporal feature vectors. The jointly modeled features are then input into a temporal classifier to obtain the recognition results. Based on online learning and feedback mechanisms, large models are dynamically optimized.

2. The method according to claim 1, characterized in that, The construction of multimodal data includes: Acquire a continuous video frame segment centered on the original video frame as image data to capture the spatiotemporal context of dynamic events; The predefined structured text description corresponding to the type of event to be identified is invoked as text data to provide clear semantic rules and judgment logic for the content analysis of continuous video frame segments. The structured text description includes at least event definition, judgment rules, exclusion rules, judgment basis and scene label. By calling knowledge graphs or domain knowledge bases, entities, attributes, and their relationships are extracted into structured data and used as knowledge graph data. This data is then used to associate relevant entities, attributes, and relationships in the knowledge graph data with scene and target information extracted from continuous video frame segments and rules in text descriptions, thus injecting domain common sense and reasoning constraints into multimodal data analysis.

3. The method according to claim 1, characterized in that, The step of extracting image feature vectors from image data and extracting text feature vectors from text data includes: The frame difference method is used to process the image data and initially extract the dynamic regions that have changed; Within the initially extracted dynamic region, the motion vectors of pixels are calculated using the optical flow method to accurately extract the dynamic target region; The dynamic target region is input into the image encoder of the CLIP model to extract the first image feature vector; The dynamic target region is input into the ViT model to extract the second image feature vector; The first image feature vector and the second image feature vector are fused to generate an image feature vector for cross-modal alignment and fusion; Pre-defined text data related to the dynamic target region is pre-processed and then input into the text encoder of the CLIP model to generate text feature vectors.

4. The method according to claim 1, characterized in that, The step of performing cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors includes: Using a bidirectional attention mechanism, the first correlation weight matrix of the image feature vector and the text feature vector in the image-text direction and the second correlation weight matrix of the image feature vector and the text feature vector in the text-image direction are calculated respectively, and the first correlation weight matrix and / or the second correlation weight matrix are used as the correlation weight matrix. If the correlation weight matrix is ​​the first correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is fused with the weighted text feature vector to obtain the cross-modal feature vector; If the association weight matrix is ​​the second association weight matrix, the image feature vector is weighted using the second association weight matrix to obtain the weighted image feature vector, and the text feature vector is fused with the weighted image feature vector to obtain the cross-modal feature vector; If the correlation weight matrix includes both the first correlation weight matrix and the second correlation weight matrix, the text feature vector is weighted using the first correlation weight matrix to obtain the weighted text feature vector, and the image feature vector is weighted using the second correlation weight matrix to obtain the weighted image feature vector. The weighted text feature vector and the weighted image feature vector are then fused to obtain the cross-modal feature vector.

5. The method according to claim 4, characterized in that, Before calculating the first correlation weight matrix and the second correlation weight matrix of the image feature vector and the text feature vector in the image-to-text direction, respectively, through the bidirectional attention mechanism, and using the first correlation weight matrix and / or the second correlation weight matrix as the correlation weight matrix, the method further includes: Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors; Calculate the first similarity between the image feature vector and the knowledge graph embedding vector, and calculate the first corrected weight matrix based on the first similarity; calculate the second similarity between the text feature vector and the knowledge graph embedding vector, and calculate the second corrected weight matrix based on the second similarity. The image enhancement weighting matrix is ​​calculated based on the first modified weighting matrix, and the text enhancement weighting matrix is ​​calculated based on the second modified weighting matrix. Based on the image enhancement weighting matrix, feature enhancement is performed on the image feature vector to obtain the enhanced image feature vector. Similarly, based on the text enhancement weighting matrix, feature enhancement is performed on the text feature vector to obtain the enhanced text feature vector.

6. The method according to claim 1, characterized in that, The method of cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors is achieved through a dynamic weighted feature fusion network, including: Vectorize the entities, attributes, and relationships in the knowledge graph data to generate a set of knowledge graph embedding vectors; Obtain the task feature vector corresponding to the type of event to be identified; Based on the task feature vector, the dynamic weights corresponding to the image feature vector, text feature vector and knowledge graph embedding vector are calculated respectively. Based on dynamic weights, the image feature vector, text feature vector, and knowledge graph embedding vector are weighted and summed to obtain the cross-modal feature vector.

7. The method according to claim 1, characterized in that, The extraction of temporal feature vectors from image data, and the joint modeling of temporal semantics based on cross-modal feature vectors and temporal feature vectors, include: A 3D convolutional neural network is used to encode image data and extract temporal feature vectors; The cross-modal feature vector, temporal feature vector, and features obtained through the high-low point video linkage strategy are fused together. The features obtained through the high-low point video linkage strategy include: high-resolution texture feature vectors of dynamic target regions extracted from the low point view, and scene semantic feature vectors obtained from the high point view through semantic segmentation and knowledge graph reasoning. By utilizing the cross-attention mechanism, joint modeling and spatial semantic alignment are performed on cross-modal feature vectors, temporal feature vectors, and feature vectors obtained through high and low point perspectives, thus completing the joint modeling of temporal and semantic aspects.

8. The method according to claim 1, characterized in that, The aforementioned online learning and feedback mechanism for dynamically optimizing large models includes: Based on the difference between the recognition result generated by the temporal semantic joint modeling and the expected result, the model loss is calculated. The model loss includes the contrastive loss, which is calculated using the following formula: ,in, For similarity calculation function, For the model to predict features, For labeling features, For temperature parameters, Features of negative samples; The parameters of the large model are updated based on the model loss.

9. The method according to claim 1, characterized in that, The dynamic optimization of the large model based on online learning and feedback mechanisms, implemented using an incremental learning strategy, includes: Only update the parameters of the dynamic weight feature fusion network and / or the temporal classifier in the large model; Maintain an incremental sample buffer to store samples with feedback annotations and their multimodal features; Based on samples in the buffer, the dynamic weight feature fusion network and / or temporal classifier are fine-tuned.

10. A multimodal data analysis device based on a large model, characterized in that, include: The multimodal data construction module is used to build multimodal data, including image data, text data, and knowledge graph data; The feature extraction module, connected to the multimodal data construction module, is used to extract image feature vectors from image data and text feature vectors from text data. The cross-modal alignment and fusion module, connected to the feature extraction module, is used to perform cross-modal alignment and fusion of image feature vectors and text feature vectors through a preset mechanism to generate cross-modal feature vectors; The temporal semantic joint modeling module, connected to the cross-modal alignment and fusion module, is used to extract temporal feature vectors from image data, perform temporal semantic joint modeling based on cross-modal feature vectors and temporal feature vectors, and input the jointly modeled features into the temporal classifier in the temporal semantic joint modeling module to obtain the recognition result; The dynamic optimization module is connected to the cross-modal alignment and fusion module and the temporal semantic joint modeling module, respectively, and is used to dynamically optimize large models based on online learning and feedback mechanisms.

Images