Data analysis method and device based on cross-modal data fusion, equipment and medium

Abstract

The application discloses a data analysis method and device based on cross-modal data fusion, equipment and medium, and relates to the field of data processing. The method comprises the following steps: acquiring multi-modal data associated with a data analysis target; performing time alignment on each modal data, and performing spatial alignment based on each modal data after time alignment; acquiring feature information corresponding to each modal in the spatially aligned multi-modal data, and identifying associated feature information between the feature information of each two modalities to obtain an associated feature information identification result between the multi-modal data; filtering the feature information of each modal based on the associated feature information identification result, performing weighted fusion processing on the filtered feature information of each modal to obtain a fusion result, and performing data analysis based on the fusion result to obtain an analysis result corresponding to the data analysis target. The application improves the efficiency of data fusion, the comprehensiveness and accuracy of data analysis.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data analysis method, apparatus, device and medium based on cross-modal data fusion. Background Technology

[0002] With the rapid development of information technology, multimodal data is increasingly widely used in many fields such as medical diagnosis, intelligent monitoring, and autonomous driving. Multimodal data usually includes various types such as text, images, audio, and video. Different modal data describe target objects or events from different dimensions, and their fusion analysis can provide more comprehensive information support for decision-making.

[0003] However, existing cross-modal data fusion analysis methods still have significant limitations in practical applications. On the one hand, they fail to delve into the intrinsic relationships between different modalities, and the fusion results cannot fully reflect the complete characteristics of the target object or event, leading to a one-sided data analysis and affecting its accuracy. On the other hand, the lack of consideration for the time dimension means that the data integration process cannot reflect the characteristics of data changes over time, resulting in the forced fusion of time-mismatched data, which further affects the reliability of the data analysis results. Summary of the Invention

[0004] This application provides a data analysis method, apparatus, device, and medium based on cross-modal data fusion, which solves the problems of existing cross-modal data fusion analysis methods that fail to deeply explore the intrinsic correlation between different modal data and lack consideration of the time dimension, resulting in incomplete and inaccurate data analysis results. It improves the efficiency of data fusion and enhances the comprehensiveness and accuracy of data analysis.

[0005] Firstly, this application provides a data analysis method based on cross-modal data fusion, including:

[0006] Upon detecting a data analysis target, acquire multimodal data that is correlated with the data analysis target;

[0007] The temporal feature sequence of each modality in the multimodal data is obtained, the dynamic time warping distance is determined based on the temporal feature sequence, the time offset mapping matrix is ​​determined based on the dynamic time warping distance, the temporal alignment of each modality is performed based on the time offset mapping matrix, and the spatial alignment of each temporally aligned modality is performed to obtain spatially aligned multimodal data.

[0008] The feature information corresponding to each modality in the spatially aligned multimodal data is obtained, and the correlation feature information between the feature information of every two modalities is identified to obtain the correlation feature information identification result between the multimodal data.

[0009] Based on the identification results of the associated feature information, the feature information of each modality is filtered to obtain the filtered feature information of each modality. The filtered feature information of each modality is then weighted and fused to obtain a fusion result. Based on the fusion result, data analysis is performed to obtain the analysis result corresponding to the data analysis target.

[0010] Optionally, determining the time offset mapping matrix based on the dynamic time warping distance includes:

[0011] Obtain any two modal data points and use them as the first modal data and the second modal data;

[0012] Obtain the first semantic feature vector and the first temporal feature vector corresponding to each time frame in the first modal data, and the second semantic feature vector and the second temporal feature vector corresponding to each time frame in the second modal data;

[0013] Determine multiple time frame pairs consisting of each time frame of the first modality data and each time frame of the second modality data, calculate the semantic similarity between the first semantic feature vector and the second semantic feature vector corresponding to each time frame pair, and the Euclidean distance between the first temporal feature vector and the second temporal feature vector corresponding to each time frame pair;

[0014] The Euclidean distance is weighted based on the similarity to obtain the weighted distance value for each time frame pair. The weighted distance values ​​of multiple time frame pairs are arranged in rows and columns to obtain a weighted distance matrix.

[0015] The optimal regularization path is determined based on the weighted distance matrix. The time frame index of the modal data corresponding to each node on the optimal regularization path is mapped to a corresponding relationship. A time offset mapping matrix is ​​generated based on the corresponding relationship.

[0016] Optionally, determining the optimal normalized path based on the weighted distance matrix includes:

[0017] The path search range is limited to a preset range along the main diagonal of the weighted distance matrix;

[0018] When the semantic similarity of a time frame pair is lower than a preset threshold, the path is prohibited from passing through the matrix position corresponding to the time frame pair in the weighted distance matrix;

[0019] Within the path search range, the path with the minimum cumulative cost is searched as the optimal regularization path.

[0020] Optionally, the spatial alignment based on each of the time-aligned modal data includes:

[0021] Select a modality with global spatial reference capability from the multimodal data as the reference modality data, and obtain the first spatial feature vector of the reference modality data;

[0022] Obtain the second spatial feature vector corresponding to each non-benchmark modality data;

[0023] The spatial transformation matrix is ​​determined based on the first spatial feature vector and each of the second spatial feature vectors;

[0024] Based on the spatial transformation matrix, a spatial transformation is performed on each of the multimodal data to obtain spatially aligned multimodal data.

[0025] Optionally, the step of identifying the associated feature information between the feature information of each pair of said modalities includes:

[0026] Obtain the temporal feature sequence of the feature information of each modality to obtain a plurality of temporal feature sequences of the feature information;

[0027] The temporal feature sequences of multiple feature information are combined pairwise to obtain multiple cross-modal joint features, wherein each cross-modal joint feature corresponds to a modality combination;

[0028] The association feature information recognition result between the multimodal data is determined based on the joint features of the multiple cross-modal data.

[0029] Optionally, determining the association feature information recognition result among the multimodal data based on the multiple cross-modal joint features includes:

[0030] Based on a preset sliding window size, each of the cross-modal joint features is segmented in the time dimension to obtain multiple local joint features corresponding to each modality combination;

[0031] Cross-modal feature interaction analysis is performed on each local joint feature of each modal combination to determine the local correlation features of the interaction between the two modes in each modal combination;

[0032] Aggregate all the local association features of each modality combination to obtain the global association features of each modality combination;

[0033] The identification result of the association feature information between the multimodal data is determined based on the global association features of each modality combination.

[0034] Optionally, the step of performing weighted fusion processing on the feature information after each modality screening to obtain the fusion result includes:

[0035] Determine the weights corresponding to the feature information after each modality screening;

[0036] Based on the feature information after each modality is filtered and the weight corresponding to the feature information after each modality is filtered, the weighted feature information of each modality is obtained;

[0037] The weighted feature information of each modality is fused to obtain the fusion result.

[0038] Secondly, this application provides a data analysis device based on cross-modal data fusion, comprising:

[0039] The data acquisition module is used to acquire multimodal data that is associated with the data analysis target when the data analysis target is detected;

[0040] The data alignment module is used to acquire the temporal feature sequence of each modality in the multimodal data, determine the dynamic time warping distance based on the temporal feature sequence, determine the time offset mapping matrix based on the dynamic time warping distance, perform time alignment on each modality based on the time offset mapping matrix, and perform spatial alignment on each time-aligned modality to obtain spatially aligned multimodal data.

[0041] The association information recognition module is used to acquire the feature information corresponding to each modality in the spatially aligned multimodal data, and to perform association feature information recognition between the feature information of every two modalities to obtain the association feature information recognition result between the multimodal data;

[0042] The data fusion module is used to filter the feature information of each modality based on the identification results of the associated feature information, obtain the filtered feature information of each modality, perform weighted fusion processing on the filtered feature information of each modality to obtain the fusion result, and perform data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target.

[0043] Thirdly, this application provides a data analysis device based on cross-modal data fusion, comprising:

[0044] One or more processors;

[0045] A memory that stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the data analysis method based on cross-modal data fusion as described in the first aspect.

[0046] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the data analysis method based on cross-modal data fusion as described in the first aspect.

[0047] In summary, compared with the prior art, the effective effects of the technical solution provided in this application include at least the following:

[0048] In this application, upon detecting a data analysis target, multimodal data associated with the data analysis target is acquired; the temporal feature sequence of each modality in the multimodal data is acquired; a dynamic time warping distance is determined based on the temporal feature sequence; a time offset mapping matrix is ​​determined based on the dynamic time warping distance; each modality is time-aligned based on the time offset mapping matrix; and each time-aligned modality is spatially aligned to obtain spatially aligned multimodal data. Feature information corresponding to each modality in the spatially aligned multimodal data is acquired, and correlation feature information is identified between the feature information of every two modalities to obtain the correlation feature information identification result between multimodal data. Based on the correlation feature information identification result, the feature information of each modality is filtered to obtain the filtered feature information of each modality; the filtered feature information of each modality is weighted and fused to obtain a fusion result; and data analysis is performed based on the fusion result to obtain the analysis result corresponding to the data analysis target. Among the above technical means, by performing time and spatial alignment on multimodal data, the reliability of data analysis results is avoided due to the misalignment of time and space. Then, based on the aligned multimodal data, the correlation information between the feature information of different modal data is further mined. Finally, based on the identification results of the correlation feature information, the feature information of each modality is filtered, and the filtered feature information is weighted and fused to improve the efficiency of data fusion and the comprehensiveness and accuracy of data analysis. Attached Figure Description

[0049] Figure 1 This is a flowchart of a data analysis method based on cross-modal data fusion provided in an embodiment of this application;

[0050] Figure 2 This is a flowchart illustrating the determination of the time offset mapping matrix provided in an embodiment of this application;

[0051] Figure 3 This is a flowchart of determining the optimal regularization path provided in an embodiment of this application;

[0052] Figure 4 This is a flowchart of spatial alignment of each modal data provided in an embodiment of this application;

[0053] Figure 5This is a flowchart of the process for identifying associated feature information provided in an embodiment of this application;

[0054] Figure 6 This is a flowchart illustrating the identification results of the correlation feature information between multimodal data provided in an embodiment of this application;

[0055] Figure 7 This is a flowchart illustrating the determination of the fusion result provided in an embodiment of this application;

[0056] Figure 8 This is a schematic diagram of the structure of a data analysis device based on cross-modal data fusion provided in an embodiment of this application;

[0057] Figure 9 This is a schematic diagram of the structure of a data analysis device based on cross-modal data fusion provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0059] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0060] Existing cross-modal data fusion analysis methods still have significant limitations in practical applications. On the one hand, they fail to delve into the intrinsic relationships between different modalities, and the fusion results cannot fully reflect the complete characteristics of the target object or event, leading to a relatively one-sided data analysis and thus affecting the accuracy of the data analysis. On the other hand, the lack of consideration for the time dimension makes it impossible for the data integration process to reflect the characteristics of data changes over time, resulting in the forced fusion of time-mismatched data, which further affects the reliability of the data analysis results.

[0061] To address the shortcomings of existing cross-modal data fusion analysis methods, such as insufficient exploration of the intrinsic relationships between different modalities and lack of consideration for the time dimension, which leads to incomplete and inaccurate data analysis results, this application provides a data analysis method based on cross-modal data fusion. By performing time and spatial alignment on multimodal data, the reliability of data analysis results is avoided due to misalignment. Then, based on the aligned multimodal data, the correlation features between the feature information of different modalities are explored in depth. Finally, based on the identification results of the correlation feature information, the feature information of each modality is filtered, and the filtered feature information is weighted and fused to improve the efficiency of data fusion and the comprehensiveness and accuracy of data analysis.

[0062] The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0063] Figure 1 A flowchart of a data analysis method based on cross-modal data fusion provided in an embodiment of this application is given. Please refer to... Figure 1 The data analysis method based on cross-modal data fusion specifically includes:

[0064] S110. If a data analysis target is detected, acquire multimodal data that is associated with the data analysis target.

[0065] The data analysis objective refers to the goal to be achieved through data analysis; then, data in multiple modalities related to this data analysis objective is collected. Multimodal data refers to data from different data sources or of different types that can describe the same target object or event from multiple dimensions; optionally, multimodal data includes, but is not limited to, text data, image data, video data, and sensor data.

[0066] S120. Obtain the temporal feature sequence of each modality in the multimodal data, determine the dynamic time warping distance based on the temporal feature sequence, determine the time offset mapping matrix based on the dynamic time warping distance, perform time alignment on each modality based on the time offset mapping matrix, and perform spatial alignment on each modality after time alignment to obtain spatially aligned multimodal data.

[0067] Time alignment refers to calibrating data from different sources or modalities along the time dimension, ensuring that data from different modalities correspond to the correct time points. Time alignment allows each modal data point to have corresponding feature information at the same time point, enabling accurate analysis of the relationships or dynamic changes between multimodal data. In this embodiment, the temporal feature sequence of each modality in the multimodal data is extracted, then the dynamic time warping distance is determined based on the temporal feature sequence, a time offset mapping matrix is ​​determined based on the dynamic time warping distance, and finally, time alignment is performed on each modality data based on the time offset mapping matrix. Dynamic time warping (DTW) distance is a metric for measuring the similarity between two time series; it finds the optimal non-linear time warping path between two sequences, minimizing the cumulative distance between the two sequences along that path. The time offset mapping matrix describes the temporal correspondence between different modal data. Specifically, the elements in the time offset mapping matrix represent the offset between one modality data point and another modality data point at a given time. For example, assuming there is a time series of mode A and a time series of mode B, after determining the optimal normalization path between them, we can determine the position of each time frame in mode A in the corresponding time frame in mode B, thus obtaining a time offset mapping matrix. This time offset mapping matrix can help understand the temporal misalignment of data from different modes, so as to perform the next step of time alignment operation.

[0068] Optional, Figure 2 This is a flowchart illustrating the determination of the time offset mapping matrix provided in an embodiment of this application. For example... Figure 2 As shown, the steps for determining the time offset mapping matrix specifically include S1201-S1205:

[0069] S1201. Obtain any two modal data points and use them as the first modal data and the second modal data.

[0070] S1202: Obtain the first semantic feature vector and the first temporal feature vector corresponding to each time frame in the first modal data, and the second semantic feature vector and the second temporal feature vector corresponding to each time frame in the second modal data.

[0071] In this context, a time frame refers to a data unit obtained by discretizing continuous time-series data at preset fixed time intervals (such as 100ms, 500ms, etc.). A semantic feature vector is a feature representation of the semantics or meaning reflected in modal data, that is, what it describes or what it expresses; for example, in video or text modal data, the semantic feature vector can represent the object, scene, or event category in the modal data. A temporal feature vector refers to the dynamic change feature representation of modal data in the time dimension, which can also be understood as the pattern or trend of data evolution over time.

[0072] S1203. Determine multiple time frame pairs consisting of each time frame of the first modality data and each time frame of the second modality data, calculate the semantic similarity between the first semantic feature vector and the second semantic feature vector corresponding to each time frame pair, and the Euclidean distance between the first temporal feature vector and the second temporal feature vector corresponding to each time frame pair.

[0073] Here, a time frame pair refers to a pair consisting of each time frame of the first modality data and each time frame of the second modality data. For example, suppose each time frame pair is represented as... ,in It is the time frame index of the first modality data ( =1...,M), It is the time frame index of the second modality data ( =1,...,N), for example, the first modality data has two time frames ( =1,2), the second modality data has two time frames ( If (1,2), then we can get two time frame pairs, namely (1,1) and (1,2).

[0074] In this embodiment, the semantic similarity of each time frame pair can be calculated using cosine similarity, and the formula is as follows:

[0075]

[0076] in, The first semantic feature vector of the first modality data. This is the second semantic feature vector of the second modality data.

[0077] In this embodiment, the Euclidean distance measures the distance between two vectors in the feature space. A smaller distance value indicates greater similarity between the two temporal feature vectors, while a larger distance value indicates greater difference. The formula for the Euclidean distance is:

[0078]

[0079] in, It is the second time-series feature vector The kth element; It is the first time-series eigenvector The kth element.

[0080] S1204. The Euclidean distance is weighted based on the similarity to obtain the weighted distance value of each time frame pair. The weighted distance values ​​of multiple time frame pairs are arranged in rows and columns to obtain a weighted distance matrix.

[0081] In this embodiment, weighting refers to adjusting the contribution of Euclidean distance using semantic similarity. Specifically, the higher the semantic similarity, the lower the weight of the Euclidean distance, thereby reducing the weighted distance value. This embodiment uses the following formula to calculate the weight of each time frame pair. Weighted distance values:

[0082] = ×(1- )

[0083] The weighted distance value in this application refers to the comprehensive distance value obtained by weighting the Euclidean distance using semantic similarity. It reflects the overall degree of difference between two time frames in terms of semantic and temporal features. The smaller the weighted distance value, the more matched the two time frames are (i.e., high semantic similarity and small difference in temporal features); conversely, the larger the weighted distance value, the less matched the two time frames are.

[0084] S1205. Determine the optimal regularization path based on the weighted distance matrix, map the time frame index of the modal data corresponding to each node on the optimal regularization path to a corresponding relationship, and generate a time offset mapping matrix based on the corresponding relationship.

[0085] The optimal regular path is the path from the starting point to the ending point in the weighted distance matrix, which has the minimum cumulative weighted distance.

[0086] Optional, Figure 3 This is a flowchart illustrating the determination of the optimal regularization path provided in an embodiment of this application. For example... Figure 3 As shown, the steps for determining the optimal regularization path specifically include S12051-S12053:

[0087] S12051. Limit the path search range to a preset range along the main diagonal of the weighted distance matrix.

[0088] To reduce computational complexity and improve pathfinding efficiency, this embodiment first determines the pathfinding range. Specifically, the pathfinding range is limited to a preset range near the main diagonal of the weighted distance matrix. For example, for each element (i,j) in the weighted distance matrix, a path is allowed to pass through that position only if |ij| ≤ the preset range. This step helps to exclude combinations of time frames that differ too much in time, thereby improving the efficiency and accuracy of pathfinding.

[0089] S12052. When the semantic similarity of a time frame pair is lower than a preset threshold, the path is prohibited from passing through the matrix position corresponding to the time frame pair in the weighted distance matrix.

[0090] In this embodiment, by introducing a semantic similarity threshold, the path is prohibited from passing through matrix positions with semantic similarity below the threshold. This effectively filters out semantically mismatched time frame combinations. When faced with complex multimodal data, it can avoid incorrect alignment caused by simply relying on temporal features, thus enhancing the robustness of time alignment.

[0091] S12053. Search for the path with the minimum cumulative cost within the path search range as the optimal regularization path.

[0092] Within the path search range, dynamic programming can be used to find the path with the minimum cumulative cost as the optimal regularized path. Dynamic programming recursively calculates the minimum cumulative cost from the starting point to each node, ultimately finding the optimal path from the starting point to the destination. The specific steps include: initializing a cumulative cost matrix C of the same size as the weighted distance matrix, and setting the cumulative cost of the starting point to the value of the corresponding position in the weighted distance matrix; for each node in the matrix, selecting the path with the minimum cumulative cost from its possible predecessor nodes, and adding the corresponding cumulative cost to the weighted distance value of the current node as the cumulative cost of the current node; finally, backtracking from the destination to find the path with the minimum cumulative cost, which is the optimal regularized path.

[0093] In the optimal normalization path of this application embodiment, the degree of matching between two modal data is comprehensively measured by a weighted distance matrix. This can better adapt to various temporal change patterns of different modal data, making the time alignment results more reliable and able to more realistically reflect the correlation between different modal data in the time dimension.

[0094] In the embodiments of this application, spatial alignment refers to the process of mapping data of different modalities to the same space, so that the information of different modal data in the spatial dimension remains consistent. This process can accurately reflect the spatial state of the same scene or the same object, thereby eliminating spatial deviation.

[0095] Optional, Figure 4This is a flowchart illustrating spatial alignment of each modal data provided in an embodiment of this application. For example... Figure 4 As shown, the step of spatially aligning each modal data specifically includes S1206-S1209:

[0096] S1206. Select a modal data with global spatial reference capability from the multimodal data as the reference modal data, and obtain the first spatial feature vector of the reference modal data.

[0097] Among them, spatial feature vectors refer to the feature representations extracted from modal data that represent its spatial characteristics, reflecting the distribution, layout, or relationships of the data in the spatial dimension.

[0098] S1207. Obtain the second spatial feature vector corresponding to each non-benchmark modal data.

[0099] S1208. Determine the spatial transformation matrix based on the first spatial feature vector and each of the second spatial feature vectors.

[0100] The spatial transformation matrix can be understood as the correspondence between different modal feature spaces, thereby eliminating spatial representation differences caused by different data sources and types.

[0101] S1209. Based on the spatial transformation matrix, perform spatial transformation on each of the multimodal data to obtain spatially aligned multimodal data.

[0102] S130. Obtain the feature information corresponding to each modality in the spatially aligned multimodal data, and identify the associated feature information between every two modal feature information to obtain the associated feature information identification result between the multimodal data.

[0103] Feature information refers to information extracted from data that represents the essential characteristics of the data.

[0104] Specifically, the identification of related feature information aims to uncover the inherent connections and interrelationships between different modalities of data at the semantic, content, or feature levels.

[0105] Optional, Figure 5 This is a flowchart illustrating the process of identifying associated feature information provided in an embodiment of this application. For example... Figure 5 As shown, the steps for identifying associated feature information specifically include S1301-S1303:

[0106] S1301. Obtain the temporal feature sequence of the feature information of each modality to obtain a temporal feature sequence of multiple feature information.

[0107] S1302. The temporal feature sequences of the multiple feature information are combined in pairs to obtain multiple cross-modal joint features, wherein each cross-modal joint feature corresponds to a modality combination.

[0108] Cross-modal joint features refer to features obtained by combining the temporal feature sequences of two different modalities at each time frame. This fusion of feature information from both modalities at the same time frame reflects the joint characteristics of the two modalities at that point in time. Specifically, for each modal combination (e.g., modality A and modality B), the temporal feature sequences of the two modalities are combined frame by frame along the time dimension. For example, for each modal combination (e.g., modality A and modality B), its temporal feature sequence is obtained. Since the data is time-aligned, the two sequences are synchronized at each time frame. Therefore, for each time frame, the features of modality A and the features of modality B are combined (this combination operation can be performed using feature concatenation) to generate the cross-modal joint features for that time frame. By traversing all time frames, the cross-modal joint features of modality A and modality B are obtained. This process is repeated, combining all modalities pairwise, ultimately yielding multiple cross-modal joint features.

[0109] Modal combination refers to the method of selecting two modalities from multiple modalities and combining them. For example, for the three modalities of image, text and audio, modal combination includes image-text, image-audio and text-audio.

[0110] S1303. Based on the multiple cross-modal joint features, determine the identification result of the correlation feature information between the multimodal data.

[0111] Among them, the results of association feature information recognition refer to the association relationships between different modal data at the feature level, such as co-occurrence association, causal association, complementary association, etc.

[0112] Optional, Figure 6 This is a flowchart illustrating the identification results of correlation feature information between multimodal data provided in an embodiment of this application. For example... Figure 6 As shown, the steps for determining the identification results of the correlation feature information between multimodal data specifically include S13031-S13034:

[0113] S13031. Based on a preset sliding window size, each of the cross-modal joint features is segmented in the time dimension to obtain multiple local joint features corresponding to each modal combination.

[0114] The sliding window size refers to the size of the sliding window used when processing time series data; that is, the number of time points contained in the window. It determines the length of the time range covered by each local joint feature when performing data segmentation. For example, suppose there is a time series data containing 100 time points. If the sliding window size is set to 10, then each local joint feature will contain data from 10 consecutive time points.

[0115] Local joint features refer to a subset of features segmented from cross-modal joint features according to a preset sliding window size. Each local joint feature contains combined feature information of different modalities at corresponding time points within that time window. By segmenting cross-modal joint features in the time dimension, the joint features of long sequences can be decomposed into multiple smaller and easier-to-process local feature fragments, so as to capture the dynamic correlation between different modalities within a short time scale.

[0116] S13032. Perform cross-modal feature interaction analysis on each local joint feature of each modal combination to determine the local correlation features of the interaction between the two modes in each modal combination.

[0117] Among them, local correlation features refer to the feature subsets that show local correlation between two modalities of data.

[0118] S13033. Aggregate all the local association features of each modality combination to obtain the global association features of each modality combination.

[0119] Among them, global association features refer to the complete association set formed by integrating, classifying and aggregating all local association features under the same modality combination, which comprehensively includes the local association features of both modalities.

[0120] S13034. Determine the identification result of the association feature information between the multimodal data based on the global association features of each modality combination.

[0121] S140. Based on the identification results of the associated feature information, the feature information of each modality is filtered to obtain the filtered feature information of each modality. The filtered feature information of each modality is weighted and fused to obtain the fusion result. Based on the fusion result, data analysis is performed to obtain the analysis result corresponding to the data analysis target.

[0122] In multimodal data fusion, the feature information contained in each modality may include redundant information, or even information irrelevant to other modalities or the data analysis objective. Therefore, it is necessary to filter the feature information of each modality based on the results of the correlation feature information identification. By filtering, irrelevant or redundant features can be removed, which helps to improve the efficiency and accuracy of subsequent fusion processing and reduce computational complexity. For example, the performance of each feature in the correlation feature information identification process can be used as the filtering criterion to remove feature information that is irrelevant to other modalities or the data analysis objective.

[0123] Optional, Figure 7 This is a flowchart illustrating the determination of fusion results provided in an embodiment of this application. For example... Figure 7 As shown, the steps for determining the fusion result specifically include S1401-S1403:

[0124] S1401. Determine the weight corresponding to the feature information after each modality is filtered.

[0125] For example, the feature correlation weight of each modality feature can be determined based on the degree of correlation between modality features in the correlation feature information identification results. For instance, if a feature of modality A is closely related to features of other modalities such as modality B and modality C, and this correlation is crucial to the data analysis objective, then the feature correlation weight of that feature is relatively high.

[0126] S1402. Based on the feature information after each modality is filtered and the weight corresponding to the feature information after each modality is filtered, the weighted feature information of each modality is obtained.

[0127] S1403. The weighted feature information of each modality is fused to obtain the fusion result.

[0128] In this embodiment, feature information of each modality is filtered based on the identification results of associated feature information, and the filtered feature information of each modality is weighted and fused. When performing weighted fusion processing on the filtered feature information of each modality, redundant or irrelevant features have been removed, reducing the number of features that need to be fused, thereby greatly improving the efficiency of data fusion. This not only saves computing resources but also speeds up the data analysis process, thereby improving the data analysis efficiency of cross-modal data fusion.

[0129] In summary, the data analysis method based on cross-modal data fusion provided in this application involves: acquiring multimodal data associated with the data analysis target when the data analysis target is detected; acquiring the temporal feature sequence of each modality in the multimodal data; determining the dynamic time warping distance based on the temporal feature sequence; determining the time offset mapping matrix based on the dynamic time warping distance; performing time alignment on each modality based on the time offset mapping matrix; performing spatial alignment on each time-aligned modality to obtain spatially aligned multimodal data; acquiring the feature information corresponding to each modality in the spatially aligned multimodal data; identifying the correlation feature information between the feature information of every two modalities to obtain the correlation feature information identification result between multimodal data; filtering the feature information of each modality based on the correlation feature information identification result to obtain the filtered feature information of each modality; performing weighted fusion processing on the filtered feature information of each modality to obtain the fusion result; and performing data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target. Among the above technical means, by performing time and spatial alignment on multimodal data, the reliability of data analysis results is avoided due to the misalignment of time and space. Then, based on the aligned multimodal data, the correlation information between the feature information of different modal data is further mined. Finally, based on the identification results of the correlation feature information, the feature information of each modality is filtered, and the filtered feature information is weighted and fused to improve the efficiency of data fusion and the comprehensiveness and accuracy of data analysis.

[0130] Figure 8 This is a schematic diagram of a data analysis device based on cross-modal data fusion, provided as an embodiment of this application. (Reference) Figure 8 The data analysis device based on cross-modal data fusion provided in this embodiment specifically includes: a data acquisition module 21, a data alignment module 22, a correlation information identification module 23, and a data fusion module 24;

[0131] Data acquisition module 21 is used to acquire multimodal data associated with the data analysis target when the data analysis target is detected;

[0132] The data alignment module 22 is used to acquire the temporal feature sequence of each modality data in the multimodal data, determine the dynamic time warping distance based on the temporal feature sequence, determine the time offset mapping matrix based on the dynamic time warping distance, perform time alignment on each modality data based on the time offset mapping matrix, and perform spatial alignment on each modality data after time alignment to obtain spatially aligned multimodal data.

[0133] The association information recognition module 23 is used to acquire the feature information corresponding to each modality in the spatially aligned multimodal data, and to perform association feature information recognition between the feature information of every two modalities to obtain the association feature information recognition result between the multimodal data;

[0134] The data fusion module 24 is used to filter the feature information of each modality based on the recognition result of the associated feature information, obtain the filtered feature information of each modality, perform weighted fusion processing on the filtered feature information of each modality to obtain the fusion result, and perform data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target.

[0135] Based on the above embodiments, the data alignment module 22 includes a time alignment unit, which includes:

[0136] The modal data acquisition subunit is used to acquire any two modal data points as the first modal data and the second modal data.

[0137] The feature vector acquisition subunit is used to acquire the first semantic feature vector and the first temporal feature vector corresponding to each time frame in the first modal data, and the second semantic feature vector and the second temporal feature vector corresponding to each time frame in the second modal data;

[0138] The computational subunit is used to determine multiple time frame pairs consisting of each time frame of the first modality data and each time frame of the second modality data, calculate the semantic similarity between the first semantic feature vector and the second semantic feature vector corresponding to each time frame pair, and the Euclidean distance between the first temporal feature vector and the second temporal feature vector corresponding to each time frame pair;

[0139] The distance matrix determination subunit is used to perform weighted processing on the Euclidean distance based on the similarity to obtain the weighted distance value of each time frame pair. The weighted distance values ​​of multiple time frame pairs are arranged in rows and columns to obtain the weighted distance matrix.

[0140] The mapping matrix generation sub-unit is used to determine the optimal normalization path based on the weighted distance matrix, map the time frame index of the modal data corresponding to each node on the optimal normalization path to a corresponding relationship, and generate a time offset mapping matrix based on the corresponding relationship.

[0141] Based on the above embodiments, the mapping matrix generation subunit further includes:

[0142] The path search range is limited to a preset range along the main diagonal of the weighted distance matrix;

[0143] When the semantic similarity of a time frame pair is lower than a preset threshold, the path is prohibited from passing through the matrix position corresponding to the time frame pair in the weighted distance matrix;

[0144] Within the path search range, the path with the minimum cumulative cost is searched as the optimal regularization path.

[0145] Based on the above embodiments, the data alignment module 22 further includes a spatial alignment unit, which includes:

[0146] The first spatial feature vector acquisition subunit is used to select a certain modality data with global spatial reference capability from the multimodal data as the reference modality data, and acquire the first spatial feature vector of the reference modality data.

[0147] The second spatial feature vector acquisition subunit is used to acquire the second spatial feature vector corresponding to each non-benchmark modal data.

[0148] The transformation matrix determines the sub-unit, which is used to determine the spatial transformation matrix based on the first spatial feature vector and each of the second spatial feature vectors;

[0149] The spatial alignment subunit is used to perform spatial transformation on each of the multimodal data based on the spatial transformation matrix to obtain spatially aligned multimodal data.

[0150] Based on the above embodiments, the associated information identification module 23 includes:

[0151] A feature sequence acquisition unit is used to acquire a temporal feature sequence of feature information for each modality, thereby obtaining a temporal feature sequence of multiple feature information.

[0152] The temporal feature combination unit is used to perform pairwise temporal feature combination between the temporal feature sequences of multiple feature information to obtain multiple cross-modal joint features, wherein each cross-modal joint feature corresponds to a modality combination;

[0153] The identification result determination unit is used to determine the identification result of the correlation feature information between the multimodal data based on the multiple cross-modal joint features.

[0154] Based on the above embodiments, the identification result determination unit includes:

[0155] The joint feature segmentation subunit is used to segment each of the cross-modal joint features in the time dimension based on a preset sliding window size, so as to obtain multiple local joint features corresponding to each modality combination;

[0156] The feature interaction analysis subunit is used to perform cross-modal feature interaction analysis on each local joint feature of each modality combination to determine the local correlation features of the interaction between the two modes in each modality combination.

[0157] The associated feature aggregation subunit is used to aggregate all the local associated features of each modality combination to obtain the global associated features of each modality combination;

[0158] The identification result determination subunit is used to determine the identification result of the association feature information between the multimodal data based on the global association features of each modality combination.

[0159] Based on the above embodiments, the data fusion module 24 includes:

[0160] The weight determination unit is used to determine the weight corresponding to the feature information after each modality screening.

[0161] The feature information determination unit is used to obtain the weighted feature information of each modality based on the feature information after each modality is filtered and the weight corresponding to the feature information after each modality is filtered;

[0162] The data fusion unit is used to fuse the weighted feature information of each modality to obtain the fusion result.

[0163] In summary, the data analysis apparatus based on cross-modal data fusion provided in this application, upon detecting a data analysis target, acquires multimodal data associated with the data analysis target; acquires the temporal feature sequence of each modality in the multimodal data, determines the dynamic time warping distance based on the temporal feature sequence, determines the time offset mapping matrix based on the dynamic time warping distance, performs time alignment on each modality based on the time offset mapping matrix, and performs spatial alignment on each time-aligned modality to obtain spatially aligned multimodal data; acquires the feature information corresponding to each modality in the spatially aligned multimodal data, and identifies the correlation feature information between the feature information of every two modalities to obtain the correlation feature information identification result between multimodal data; filters the feature information of each modality based on the correlation feature information identification result to obtain the filtered feature information of each modality, performs weighted fusion processing on the filtered feature information of each modality to obtain the fusion result, and performs data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target. Among the above technical means, by performing time and spatial alignment on multimodal data, the reliability of data analysis results is avoided due to the misalignment of time and space. Then, based on the aligned multimodal data, the correlation information between the feature information of different modal data is further mined. Finally, based on the identification results of the correlation feature information, the feature information of each modality is filtered, and the filtered feature information is weighted and fused to improve the efficiency of data fusion and the comprehensiveness and accuracy of data analysis.

[0164] The data analysis device based on cross-modal data fusion provided in this application embodiment can be used to execute the data analysis method based on cross-modal data fusion provided in the above embodiment, and has corresponding functions and beneficial effects.

[0165] Figure 9 This is a schematic diagram of the structure of a data analysis device based on cross-modal data fusion provided in an embodiment of this application, with reference to... Figure 9 The data analysis device based on cross-modal data fusion includes a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 and the number of memories 32 in the data analysis device based on cross-modal data fusion can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the data analysis device based on cross-modal data fusion can be connected via a bus or other means.

[0166] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the data analysis method based on cross-modal data fusion in any embodiment of this application (e.g., the data acquisition module 21, data alignment module 22, association information identification module 23, and data fusion module 24 in the data analysis device based on cross-modal data fusion). The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] The communication device 33 is used for data transmission.

[0168] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned data analysis method based on cross-modal data fusion.

[0169] Input device 34 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 35 may include display devices such as a display screen.

[0170] The data analysis device based on cross-modal data fusion provided above can be used to execute the data analysis method based on cross-modal data fusion provided in the above embodiments, and has corresponding functions and beneficial effects.

[0171] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute a data analysis method based on cross-modal data fusion. The data analysis method based on cross-modal data fusion includes: upon detecting a data analysis target, acquiring multimodal data associated with the data analysis target; acquiring the temporal feature sequence of each modality in the multimodal data; determining a dynamic time warping distance based on the temporal feature sequence; determining a time offset mapping matrix based on the dynamic time warping distance; performing time alignment on each modality based on the time offset mapping matrix; performing spatial alignment on each time-aligned modality to obtain spatially aligned multimodal data; acquiring feature information corresponding to each modality in the spatially aligned multimodal data; identifying correlation feature information between feature information of every two modalities to obtain correlation feature information identification results between multimodal data; filtering the feature information of each modality based on the correlation feature information identification results to obtain filtered feature information for each modality; performing weighted fusion processing on the filtered feature information for each modality to obtain a fusion result; and performing data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target.

[0172] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0173] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the data analysis method based on cross-modal data fusion as described above, but can also execute related operations in the data analysis method based on cross-modal data fusion provided in any embodiment of this application.

[0174] The data analysis apparatus, storage medium, and data analysis device based on cross-modal data fusion provided in the above embodiments can execute the data analysis method based on cross-modal data fusion provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the data analysis method based on cross-modal data fusion provided in any embodiment of this application.

[0175] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A data analysis method based on cross-modal data fusion, characterized in that, include: Upon detecting a data analysis target, acquire multimodal data that is correlated with the data analysis target; The temporal feature sequence of each modality in the multimodal data is obtained, the dynamic time warping distance is determined based on the temporal feature sequence, the time offset mapping matrix is ​​determined based on the dynamic time warping distance, the temporal alignment of each modality is performed based on the time offset mapping matrix, and the spatial alignment of each temporally aligned modality is performed to obtain spatially aligned multimodal data. The feature information corresponding to each modality in the spatially aligned multimodal data is obtained, and the correlation feature information between the feature information of every two modalities is identified to obtain the correlation feature information identification result between the multimodal data. Based on the identification results of the associated feature information, the feature information of each modality is filtered to obtain the filtered feature information of each modality. The filtered feature information of each modality is then weighted and fused to obtain a fusion result. Based on the fusion result, data analysis is performed to obtain the analysis result corresponding to the data analysis target. The step of determining the time offset mapping matrix based on the dynamic time warping distance includes: Obtain any two modal data points and use them as the first modal data and the second modal data; Obtain the first semantic feature vector and the first temporal feature vector corresponding to each time frame in the first modal data, and the second semantic feature vector and the second temporal feature vector corresponding to each time frame in the second modal data; Determine multiple time frame pairs consisting of each time frame of the first modality data and each time frame of the second modality data, calculate the semantic similarity between the first semantic feature vector and the second semantic feature vector corresponding to each time frame pair, and the Euclidean distance between the first temporal feature vector and the second temporal feature vector corresponding to each time frame pair; The Euclidean distance is weighted based on the similarity to obtain the weighted distance value for each time frame pair. The weighted distance values ​​of multiple time frame pairs are arranged in rows and columns to obtain a weighted distance matrix. The optimal regularization path is determined based on the weighted distance matrix. The time frame index of the modal data corresponding to each node on the optimal regularization path is mapped to a corresponding relationship. A time offset mapping matrix is ​​generated based on the corresponding relationship.

2. The data analysis method based on cross-modal data fusion according to claim 1, characterized in that, The process of determining the optimal normalized path based on the weighted distance matrix includes: The path search range is limited to a preset range along the main diagonal of the weighted distance matrix; When the semantic similarity of a time frame pair is lower than a preset threshold, the path is prohibited from passing through the matrix position corresponding to the time frame pair in the weighted distance matrix; Within the path search range, the path with the minimum cumulative cost is searched as the optimal regularization path.

3. The data analysis method based on cross-modal data fusion according to claim 1, characterized in that, The spatial alignment based on each of the time-aligned modal data includes: Select a modality with global spatial reference capability from the multimodal data as the reference modality data, and obtain the first spatial feature vector of the reference modality data; Obtain the second spatial feature vector corresponding to each non-benchmark modality data; The spatial transformation matrix is ​​determined based on the first spatial feature vector and each of the second spatial feature vectors; Based on the spatial transformation matrix, a spatial transformation is performed on each of the multimodal data to obtain spatially aligned multimodal data.

4. The data analysis method based on cross-modal data fusion according to claim 1, characterized in that, The step of identifying the correlation feature information between the feature information of each pair of modalities includes: Obtain the temporal feature sequence of the feature information of each modality to obtain a plurality of temporal feature sequences of the feature information; The temporal feature sequences of multiple feature information are combined pairwise to obtain multiple cross-modal joint features, wherein each cross-modal joint feature corresponds to a modality combination; The association feature information recognition result between the multimodal data is determined based on the joint features of the multiple cross-modal data.

5. The data analysis method based on cross-modal data fusion according to claim 4, characterized in that, The determination of the association feature information recognition result between the multimodal data based on the joint features of the multiple cross-modal data includes: Based on a preset sliding window size, each of the cross-modal joint features is segmented in the time dimension to obtain multiple local joint features corresponding to each modality combination; Cross-modal feature interaction analysis is performed on each local joint feature of each modal combination to determine the local correlation features of the interaction between the two modes in each modal combination; Aggregate all the local association features of each modality combination to obtain the global association features of each modality combination; The identification result of the association feature information between the multimodal data is determined based on the global association features of each modality combination.

6. The data analysis method based on cross-modal data fusion according to claim 1, characterized in that, The step of weighted fusion processing of the feature information after each modality screening to obtain the fusion result includes: Determine the weights corresponding to the feature information after each modality screening; Based on the feature information after each modality is filtered and the weight corresponding to the feature information after each modality is filtered, the weighted feature information of each modality is obtained; The weighted feature information of each modality is fused to obtain the fusion result.

7. A data analysis device based on cross-modal data fusion, characterized in that, include: The data acquisition module is used to acquire multimodal data that is associated with the data analysis target when the data analysis target is detected; The data alignment module is used to acquire the temporal feature sequence of each modality in the multimodal data, determine the dynamic time warping distance based on the temporal feature sequence, determine the time offset mapping matrix based on the dynamic time warping distance, perform time alignment on each modality based on the time offset mapping matrix, and perform spatial alignment on each time-aligned modality to obtain spatially aligned multimodal data. The association information recognition module is used to acquire the feature information corresponding to each modality in the spatially aligned multimodal data, and to perform association feature information recognition between the feature information of every two modalities to obtain the association feature information recognition result between the multimodal data; The data fusion module is used to filter the feature information of each modality based on the identification result of the associated feature information, obtain the filtered feature information of each modality, perform weighted fusion processing on the filtered feature information of each modality to obtain the fusion result, and perform data analysis based on the fusion result to obtain the analysis result corresponding to the data analysis target. Specifically, the data alignment module is used to acquire any two modal data sets as the first modal data set and the second modal data set; acquire the first semantic feature vector and the first temporal feature vector corresponding to each time frame in the first modal data set, and the second semantic feature vector and the second temporal feature vector corresponding to each time frame in the second modal data set; determine multiple time frame pairs composed of each time frame of the first modal data set and each time frame of the second modal data set; calculate the semantic similarity between the first semantic feature vector and the second semantic feature vector corresponding to each time frame pair, and the Euclidean distance between the first temporal feature vector and the second temporal feature vector corresponding to each time frame pair; perform weighted processing on the Euclidean distance based on the similarity to obtain the weighted distance value of each time frame pair; arrange the weighted distance values ​​of multiple time frame pairs by rows and columns to obtain a weighted distance matrix; determine the optimal normalization path based on the weighted distance matrix; map the time frame index of the modal data corresponding to each node on the optimal normalization path to a corresponding relationship; and generate a time offset mapping matrix based on the corresponding relationship.

8. A data analysis device based on cross-modal data fusion, characterized in that, include: One or more processors; A memory that stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the data analysis method based on cross-modal data fusion as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the data analysis method based on cross-modal data fusion as described in any one of claims 1-6.

Images