Heterogeneous data dynamic conversion method and system based on integration algorithm

Abstract

The embodiment of the invention provides a heterogeneous data dynamic conversion method and system based on an integration algorithm, and belongs to the technical field of data processing. The method comprises the following steps: acquiring an original modal data set of a plurality of modals; constructing a modal dependency graph according to the original modal data set, and calculating an edge weight of the modal dependency graph; and obtaining a target modal set of the current task, and screening out a target path set from the modal dependency graph according to the target modal set. The target path is converted according to a preset strategy set, a candidate conversion result set is obtained, and the candidate conversion result set comprises a plurality of candidate feature vectors. And performing confidence coefficient calculation on each candidate feature vector according to the edge weight and the modal activeness of the target path to obtain a plurality of fusion confidence coefficients. And performing weighted averaging according to the fusion confidence and the candidate feature vectors to obtain a target conversion result. And when the data is dynamically changed, the modal combination is frequently changed or the task environment is adjusted, enough flexibility and self-adaptability are also realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for dynamic conversion of heterogeneous data based on an integrated algorithm. Background Art

[0002] Heterogeneous data originates from different types of sensors (such as cameras, temperature and humidity sensors, and vibrometers), has different data formats (such as images, text, tables, and time series signals), and has different sampling frequencies and spatiotemporal distributions. This heterogeneity results in significant differences in data dimensions, representation, time scale, and even semantics, which seriously affects the effectiveness of data fusion, unified modeling, and collaborative analysis. Therefore, heterogeneous data needs to be converted to achieve efficient data utilization. In related technologies, heterogeneous data is converted through static structure fusion methods. This method often lacks sufficient flexibility and adaptability when faced with dynamic data changes, frequent changes in modal combinations, or adjustments to the task environment. Therefore, how to improve the flexibility and adaptability of dynamic heterogeneous data conversion has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a method and system for dynamic conversion of heterogeneous data based on an integrated algorithm, aiming to improve the flexibility and adaptability of dynamic heterogeneous data conversion.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for dynamic conversion of heterogeneous data based on an integrated algorithm, the method comprising:

[0005] Acquire data of multiple modalities and obtain the original modal data set;

[0006] Constructing a modal dependency graph according to the original modal data set, and calculating edge weights of the modal dependency graph;

[0007] Get the target mode set of the current task;

[0008] Filtering a target path set from the modality dependency graph according to the target modality set; wherein the target path set includes multiple target paths;

[0009] The target path is converted according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors;

[0010] Calculating the confidence of each candidate feature vector according to the edge weight and modal activity of the target path to obtain multiple fusion confidences;

[0011] A weighted average is performed based on the fusion confidence and the candidate feature vector to obtain a target conversion result.

[0012] In some embodiments, constructing a modality dependency graph based on the original modality data set and calculating edge weights of the modality dependency graph includes:

[0013] Performing a standardized representation on the original modal data set to obtain a standardized feature vector set;

[0014] Performing activity evaluation based on the standardized feature vector set to obtain a modal activity vector;

[0015] constructing the modality dependency graph according to the set of standardized feature vectors;

[0016] The edge weights of the modality dependency graph are calculated based on the set of standardized feature vectors and the modality activity vector.

[0017] In some embodiments, the modal activity vector includes multiple modal activities, and performing activity evaluation based on the standardized feature vector set to obtain the modal activity vector includes:

[0018] The activity evaluation is:

[0019]

[0020] Among them, a i (t) represents the modal activity of the i-th modality at time t, represents the normalized eigenvector of the i-th mode in the unified eigenspace, represents the normalized value of the kth feature dimension of the i-th modality at time τ, represents the normalized value of the jth feature dimension of the i-th modality at time τ, Δ represents the time span of the sliding window, represents the normalized amplitude ratio of the kth dimension of the modal vector at time τ, and ∈ represents a very small constant to prevent the denominator from being zero.

[0021] In some embodiments, calculating the edge weights of the modality dependency graph based on the set of standardized feature vectors and the modality activity vector includes:

[0022] The initial edge weights are calculated as:

[0023]

[0024]

[0025] Among them, G ij (t) represents the initial edge weight, represents the standard cosine similarity, a i (t), a j (t) represents the modal activity of the i-th and j-th modal at the current moment, λ represents the current similarity and activity weight coefficient, β represents the activity difference penalty factor, and σ(·) represents the Sigmoid function;

[0026] The target edge weight is calculated as:

[0027]

[0028] in, represents the target edge weight, α∈(0,1) represents the fusion ratio of the current observation value and the historical value, G ij (t) represents the initial edge weight, Represents the target edge weight between node i and node j at the previous moment.

[0029] In some embodiments, filtering out a target path set from the modality dependency graph according to the target modality set includes:

[0030] Screening out, from the modality dependency graph, preliminary screening candidate paths pointing to the target modality set and having edge weights greater than a weight threshold;

[0031] Scoring the pre-screened candidate paths according to a preset path scoring function to obtain path scoring data;

[0032] The pre-screened candidate paths are screened according to the path scoring data to obtain the target path set; wherein the target path set includes multiple target paths.

[0033] In some embodiments, scoring the pre-screened candidate paths according to a preset path scoring function to obtain path scoring data includes:

[0034] The path scoring function is:

[0035]

[0036] Among them, S ij (t) represents the path score data, represents the target edge weight, γ is the weighted effect of the control variance information on the score, and δ is the intensity of the penalty for controlling the activity difference. represents the variance of the normalized eigenvector of modality j in the current window.

[0037] In some embodiments, the confidence calculation is performed on each candidate feature vector according to the edge weight and modal activity of the target path to obtain multiple fusion confidences, including:

[0038] The fusion confidence is:

[0039]

[0040] Among them, α k (t) represents the fusion confidence of the candidate feature vector corresponding to strategy k, represents the edge weight of the target path to which strategy k belongs, represents the modality activity of the source modality of strategy k at the current moment, G jk′i (t) and a jk′ (t) represents the edge weight and modal activity corresponding to traversing all candidate paths k' during the softmax normalization process, and λ and μ are adjustment coefficients.

[0041] To achieve the above objectives, a second aspect of the embodiments of the present application proposes a heterogeneous data dynamic conversion system based on an integrated algorithm, the system comprising:

[0042] A first acquisition module is used to acquire data of multiple modalities to obtain an original modal data set;

[0043] A construction module is used to construct a modal dependency graph based on the original modal data set and calculate the edge weights of the modal dependency graph;

[0044] The second acquisition module is used to obtain the target modality set of the current task;

[0045] A screening module, configured to screen out a target path set from the modal dependency graph according to the target modal set; wherein the target path set includes multiple target paths;

[0046] A conversion module, configured to convert the target path according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors;

[0047] The calculation module is used to calculate the confidence of each candidate feature vector based on the edge weight and modal activity of the target path to obtain multiple fusion confidences;

[0048] The weighting module is used to perform weighted averaging based on the fusion confidence and candidate feature vectors to obtain the target conversion result.

[0049] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0050] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0051] The present application proposes a method and system for dynamic conversion of heterogeneous data based on an integrated algorithm, which obtains a set of original modal data by acquiring data from multiple modalities. A modal dependency graph is constructed based on the original modal data set, and the edge weights of the modal dependency graph are calculated. The target modal set of the current task is obtained, and a target path set is filtered out from the modal dependency graph based on the target modal set; wherein, the target path set includes multiple target paths. The target path is converted according to a preset strategy set to obtain a set of candidate conversion results; wherein, the candidate conversion result set includes multiple candidate feature vectors. The confidence of each candidate feature vector is calculated based on the edge weight and modal activity of the target path to obtain multiple fusion confidences. The target conversion result is obtained by weighted averaging the fusion confidence and the candidate feature vectors. Sufficient flexibility and adaptability are also achieved when the data changes dynamically, the modal combination changes frequently, or the task environment is adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a method for dynamic conversion of heterogeneous data based on an integrated algorithm provided in an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0054] Figure 3 yes Figure 1 Flowchart of step S104 in FIG.

[0055] Figure 4 This is a schematic diagram of the structure of a heterogeneous data dynamic conversion system based on an integrated algorithm provided in an embodiment of the present application;

[0056] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0060] Heterogeneous data refers to data that comes from different types of sensors (such as cameras, temperature and humidity sensors, and vibrometers), has different data formats (such as images, text, tables, and time series signals), and has different sampling frequencies and spatiotemporal distributions. This heterogeneity causes significant differences in data dimensions, representation methods, time scales, and even semantic levels, which seriously affects the effectiveness of data fusion, unified modeling, and collaborative analysis. Therefore, heterogeneous data needs to be converted to achieve efficient data utilization. In related technologies, heterogeneous data is converted through static structure fusion methods. This method often lacks sufficient flexibility and adaptability when faced with dynamic data changes, frequent changes in modal combinations, or adjustments to the task environment.

[0061] Based on this, an embodiment of the present application provides a method and system for dynamic conversion of heterogeneous data based on an integrated algorithm, which aims to construct a modal dependency graph based on a set of original modal data of multiple modalities, and obtain a target modal set for the current task. The target path pointing to the target modal set in the modal dependency graph is screened out, and the target path is converted according to a plurality of preset strategies to obtain a plurality of candidate feature vectors. The fusion confidence of each candidate feature vector is calculated, and a weighted average is performed based on the fusion confidence and the candidate feature vector to obtain the target conversion result. Sufficient flexibility and adaptability are also achieved when the data changes dynamically, the modal combination changes frequently, or the task environment is adjusted.

[0062] The method and system for dynamic conversion of heterogeneous data based on an integrated algorithm provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the method for dynamic conversion of heterogeneous data based on an integrated algorithm in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0064] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0065] The method for dynamic conversion of heterogeneous data based on an integrated algorithm provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for dynamic conversion of heterogeneous data based on an integrated algorithm, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0067] Please refer to Figure 1 , Figure 1This is a flowchart of a method for dynamic conversion of heterogeneous data based on an integrated algorithm provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0068] Step S101, acquiring data of multiple modalities to obtain an original modal data set;

[0069] Step S102: constructing a modal dependency graph based on the original modal data set and calculating the edge weights of the modal dependency graph;

[0070] Step S103, obtaining the target modality set of the current task;

[0071] Step S104: filtering out a target path set from the modality dependency graph according to the target modality set; wherein the target path set includes multiple target paths;

[0072] Step S105: converting the target path according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors;

[0073] Step S106: Calculate the confidence of each candidate feature vector based on the edge weight and modal activity of the target path to obtain multiple fusion confidences;

[0074] Step S107: performing weighted averaging based on the fusion confidence and the candidate feature vectors to obtain the target conversion result.

[0075] In step S101 of some embodiments, data from multiple modalities (such as images, text, time series sensor data, etc.) may not be synchronized in time and may be completely different in format. Each X i (t) represents the observation data of modality i at time t or within the most recent time window [t-Δ,t]. The data type can be image frames (such as cameras), text fragments (such as system logs), or continuous time series (such as sensor values).

[0076] For each modality X i (t), we first define a sliding time window of length Δ to collect data from the modality over a period of time. We use sliding windows instead of single-point data because some modalities (such as text or sensors) lack sufficient single-point information and must leverage temporal features. Modalities like images, while densely packed with single-frame information, can also experience short-term frame loss in dynamic scenarios (such as vehicle movement). Therefore, using a sliding window is more robust.

[0077] See also Figure 2In some embodiments, step S102 may include but is not limited to steps S201 to S204:

[0078] Step S201, performing normalized representation on the original modal data set to obtain a normalized feature vector set;

[0079] Step S202: performing activity evaluation based on the standardized feature vector set to obtain a modal activity vector;

[0080] Step S203, constructing a modal dependency graph based on the standardized feature vector set;

[0081] Step S204: Calculate the edge weights of the modality dependency graph based on the normalized feature vector set and the modality activity vector.

[0082] In step S201 of some embodiments, after the sliding window is determined, the modality adaptive normalization phase is entered. Take three typical modalities as an example:

[0083] Image modality (e.g. road camera images):

[0084] Assume that three frames of images are collected in the window, each frame is in RGB format and has a size of 224×224. Each frame is input into a pre-trained ResNet18 network (removing the last layer of classifier and retaining only the convolution part + global pooling), and the output is a 256-dimensional vector. Then the average of the three frames is obtained. This not only preserves the image features but also smoothes the changes caused by frame jitter.

[0085] Text mode (such as device status log):

[0086] All system log entries are collected in the same time window, concatenated into a paragraph and then fed into the BERT-base model. The output [CLS] vector is taken as the overall semantic representation with a dimension of 768. Then a linear transformation layer is used. Dimensionality reduction For example, three logs, "Machine A high temperature," "Sensor abnormality," and "Control logic restart," are concatenated to form a paragraph and represented using BERT.

[0087] Sensor modality (such as temperature, humidity, vibration value):

[0088] The sensor values ​​at each time point in the window are spliced ​​in sequence (assuming the sampling frequency is 1Hz and the window is 10 seconds, the splicing is 10 dimensions), and features are extracted through a two-layer feedforward neural network (layer width is 128→256), and the output is

[0089] After the above processing, the normalized feature vector set of the current window of all modalities is obtained, as shown in the following formula (1):

[0090]

[0091] In step S202 of some embodiments, an activity evaluation mechanism is designed to evaluate the information density of each modality within the window. This mechanism is based on the intensity of change in the modal representation within the window and approximates the "information fluidity" of the modality by calculating the information entropy. The modal activity vector includes multiple modal activities. The activity evaluation is performed based on the standardized feature vector set to obtain the modal activity vector. The specific calculation of the activity evaluation is shown in the following formulas (2) and (3):

[0092]

[0093] Among them, a i (t) represents the modal activity of the i-th modality at time t, reflecting the degree of its information fluctuation. Modal activity vector Represents the contribution of each modality to the task information at the current moment. Represents the normalized feature vector of the i-th mode in the unified feature space, with dimension d = 256. represents the normalized value of the kth feature dimension of the i-th modality at time τ, represents the normalized value of the jth feature dimension of the i-th modality at time τ, and Δ represents the time span of the sliding window, which is used to capture the short-term dynamics of the modality. represents the normalized amplitude ratio of the kth dimension of the modal vector at time τ, and ∈ represents a very small constant to prevent the denominator from being zero.

[0094] Through the above steps, the problem of unifying heterogeneous data in terms of format, time series, feature space, etc. was solved, and a dynamic activity evaluation mechanism with engineering practicality was introduced, enabling the system to have the ability to perceive the density of modal information.

[0095] In some embodiments, in steps S203 to S204, the M modes in the normalized feature vector set are defined as M nodes in a graph structure, and a modality dependency graph (directed weighted graph) is constructed to represent the information flow and strength between different modes. For each pair of modes (i, j), the edge weight G is defined by a function with a time evolution mechanism. ij (t), the edge weight indicates whether modality j should be allowed to transmit information to modality i, and the importance of the path.

[0096] To achieve this adaptability, we propose a "weighted relevance-activity product graph modeling formula" and introduce a structural sparsity regularization term to encourage the graph structure to remove redundant or invalid paths while retaining critical paths, thereby improving the stability and interpretability of the system. That is, the initial edge weight calculation formula is shown in the following formulas (4) and (5):

[0097]

[0098]

[0099] Among them, G ij (t) represents the initial edge weight, Represents the standard cosine similarity, which measures the similarity between two modalities in the current feature space. i (t), a j (t) represents the modal activity of the i-th and j-th modal at the current moment, respectively, |a i (t)-a j (t)| is the activity difference between the modalities. If the information fluctuations of the two modalities differ, an increase in this term will weaken the connection strength. λ represents the current similarity and activity weight coefficient (recommended to be 1). β represents the activity difference penalty factor, which is used to suppress unreliable transition paths between high-activity and low-activity modalities. A recommended setting is 0.5. σ(·) represents the sigmoid function, which constrains edge weights to the range (0, 1), making them more suitable for normalization in subsequent path selection.

[0100] By introducing |a i (t)-a j (t)| penalty term, formula (4) can effectively suppress the risk of misconnection between two types of modes: one is the false correlation between "overactive vs. extremely stable" modes due to noise, and the other is the misjudgment of "essentially different modes" as convertible modes at certain moments. This regularization control not only enhances the robustness of the structure but also provides a clearer guidance signal for the path selection mechanism.

[0101] For example, in a smart medical scenario, if the CT image modality activity is very high during a certain period of time (possibly during the disease progression period), while the patient text report modality information is basically not updated (such as during non-doctor operation periods), the activity difference is large at this time. Although the image and text are semantically related, one side updates slowly, so it is hoped that the system will not consider the "conversion" between the two for the time being to avoid information confusion or misinterpretation.

[0102] Furthermore, in order to make the graph structure have the time memory characteristic, the exponential smoothing mechanism is used to perform historical fusion update on the edge weights to obtain the target edge weights, as shown in the following formula (6):

[0103]

[0104] in, Represents the target edge weight, α∈(0,1) represents the fusion ratio of the current observation value and the historical value, and is recommended to be set to 0.8. ij (t) represents the initial edge weight, Represents the target edge weight between node i and node j at the previous moment.

[0105] Final weights of all target edges Organized into a modal dependency graph matrix Each row represents the strength of the flow from other modes to the i-th mode, which is used to determine which modes should be considered for conversion operations in subsequent path selection.

[0106] Through the above steps S201 to S204, a modal dependency graph structure for the dynamic conversion task of heterogeneous data is constructed, and a mapping and updating mechanism that can adaptively evolve over time is designed to provide a structural basis for subsequent path selection and conversion strategy scheduling between modalities. The conversion path between modalities is not preset and fixed, but should change dynamically with the state of the input data, the modal information strength (activity) and the actual correlation between modalities. Therefore, static modal fusion structures (such as multi-branch networks) will severely limit the generalization and adaptability of the system. The graph structure constructed in this step can realize dynamic "connection strength" modeling between modalities, simulating which information between modalities can be effectively converted in the current task context, and which end should the conversion be carried out from which end first.

[0107] In step S103 of some embodiments, the system provides a target modality set for the current task at each moment. For example, in the "video understanding" task, only the image modality may be predicted, and in the "device warning" task, the target is the sensor modality. This target is generated by the system calling the task instruction set.

[0108] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S303:

[0109] Step S301: Screening out preliminary candidate paths pointing to the target modality set and with edge weights greater than a weight threshold from the modality dependency graph;

[0110] Step S302: scoring the pre-screened candidate paths according to a preset path scoring function to obtain path scoring data;

[0111] Step S303 , screening the pre-screened candidate paths according to the path scoring data to obtain a target path set; wherein the target path set includes multiple target paths.

[0112] In step S301 of some embodiments, the first step of path selection is to screen out all candidate paths pointing to the target modality set. That is, to extract all paths that meet the following conditions from the modality dependency graph: And the edge weight G in the graph ij (t) exceeds the preset weight threshold θ0. The weight threshold is usually set to 0.3, which can be adaptively set by the median of the edge weight distribution in the statistical graph structure. The form is shown in the following formula (7):

[0113]

[0114] in, represents the set of candidate paths for initial screening, i, j represent nodes in the modal dependency graph, represents the target modality set, represents the target edge weight, and θ0 represents the weight threshold.

[0115] In step S302 of some embodiments, since the judgment based solely on edge weights is not reliable enough, because the modality may be in a "casually related" state rather than a true task-related transition candidate, a path scoring mechanism is further introduced to evaluate path reliability by combining edge weights, modality activity, and modality stability (approximated by feature representation variance). Each candidate path (j→i) corresponds to a scoring function, and the path scoring function is shown in the following formula (8):

[0116]

[0117] Among them, S ij (t) represents the path score data, represents the target edge weight, γ controls the weighted effect of variance information on the score, and the recommended initial value is 0.3. δ controls the penalty intensity of activity difference, and the recommended initial value is 0.8. The variance of the normalized feature vector of mode j in the current window reflects its internal variation. This variance is calculated using several historical vectors in the sliding window, and the sliding window length is usually set to w = 5. j (t)-a i (t)| serves as a penalty term for unstable connections, preventing high-fluctuation modes from forcibly connecting to stable modes, or vice versa.

[0118] In step S303 of some embodiments, all the candidate paths are pre-screened according to S ij After sorting the (t) values, the system selects the top K paths (or all paths with scores higher than the threshold θ1) to form the target path set If there are multiple target modalities (e.g., image and sensor dual-target tasks), the above process is performed independently for each target modality, and then all selected paths are merged.

[0119] The final output path set A set of directed edges (j→i) indicates which modalities will be converted in this round. These paths are determined by combining the edge weights of the modal dependency graph, modal state information, and task objectives, making them highly targeted and interpretable.

[0120] In one example, if the current task target is modality i = "video image", and modalities j1 = "vehicle speed sensor" and j2 = "air quality index" both meet If the current activity of j1 is high and the feature variance is large, it means that the vehicle speed signal fluctuates greatly and has rich content. The score may be much higher than j2, even though the latter has a high edge weight. Ultimately, the system will only select the path from j1 to i as the conversion execution path (target path) and eliminate j2 to i, improving system efficiency and effectiveness.

[0121] Through the above steps S301 to S303, a set of optimal target paths is dynamically selected from the modal dependency graph G(t) Determine between which modalities the information conversion operation needs to be performed at the current moment. Instead of using statically set modal pair combinations, nor limited to simple edge weight sorting, it combines the task target modality, the current information status of the modality (activity and feature variance), and the degree of structural dependence between modalities (graph edge weight) to build a path selection strategy with selectivity, adjustability, and context awareness. In practical applications, such as the smart city multi-source fusion platform, the key modalities of focus at different times are different: during the morning rush hour, the focus is on traffic image modalities, while on rainy days, the focus may be on meteorological and road surface sensing modalities. Therefore, the design goal of this step is to enable the system to have the ability to independently judge, dynamically switch, and actively avoid redundant paths during operation.

[0122] In step S105 of some embodiments, for each target path Instead of directly mapping a single feature from modality j to modality i, a set of strategies for this path is designed. The preset strategy set not only includes traditional feature mapping methods (such as linear and nonlinear), but also includes structural adaptive transformation modules designed in combination with modal structure, dependency graph adjacency characteristics, task direction, etc.

[0123] For example, the policy set includes the following three typical policy construction methods:

[0124] Strategy A (basic linear transformation): maintains feature space alignment through a d→d fully connected network, and is used when the structures between modalities are similar;

[0125] Strategy B (modal semantic mapping): for input features First, a low-rank projection is performed, followed by nonlinear activation and BatchNorm. The structure is [d→r→d], where r=64, which is suitable for modal pairs with significant dimensionality differences.

[0126] Strategy C (structural graph adjustment transformation): Design an attention layer with a graph adjustment factor. Its core calculation formula is shown in the following formula (9):

[0127]

[0128] in, represents the candidate feature vector, W k , U k is the trainable weight matrix, represents the target edge weight. α, β are adjustment coefficients (e.g. α = 0.5, β = 0.1), represents the normalized eigenvector of the jth mode in the unified feature space. The structure of formula (9) takes the path structure strength as the gating control factor, making the conversion operation have the ability of structural self-awareness.

[0129] In one example, in addition to the direct conversion operation, a structural alignment regularization term is introduced to penalize the structural inconsistency between the generated candidate feature vector and the target modality.

[0130] Then, define the following formula (10):

[0131]

[0132] in, Represents the difference measure between the candidate feature vector generated by the k-th candidate path and the standardized feature vector in the covariance structure, which is used to constrain the structural alignment. represents the candidate feature vector, Represents the normalized eigenvector of the i-th mode in the unified feature space.

[0133] This measure calculates the difference between the covariance matrix of the policy output and the standard representation of the true target modality, using the Frobenius norm as the distance metric. This loss, added to the training of the attention layer of policy C, has the following advantages: it brings the policy output closer to the target modality in terms of distribution structure, enhancing cross-modal fusion; it reduces the risk of low-quality conversion policy outputs occupying fusion resources; and it provides structural adjustments for situations where modal dimensions differ or distributions are distorted.

[0134] For example, in industrial fault detection scenarios, when converting from an acoustic mode (microphone) to a temperature mode for anomaly detection, if the distribution match between the output and the target mode is not considered, the converted features may fall into abnormal areas, leading to misjudgment. By using alignment regularization, the system can automatically eliminate strategies with inconsistent structures.

[0135] After the transformation of the above strategy set, each path (j→i) will generate K candidate feature vectors All strategy outputs are aggregated to form a set of candidate transformation results for modality i Among them, each candidate feature vector is generated by different strategies, has semantic differences and structural stability, and is also optimized through edge weight adjustment and structural regularization.

[0136] In step S106 of some embodiments, from the candidate feature vectors generated by multiple strategies, the parts with high credibility, high information density and high structural credibility are selected for fusion. and modal activity Calculate the confidence of each candidate feature vector to obtain multiple fusion confidences. The specific calculation is shown in the following formula (11):

[0137]

[0138] Among them, α k (t) represents the fusion confidence of the candidate feature vector corresponding to strategy k, Represents the source mode j in the candidate path corresponding to strategy k k The edge weight to the target mode i, represents the modality activity of the source modality of strategy k at the current moment, and Indicates the edge weight and modal activity corresponding to traversing all candidate paths k' during the softmax normalization process, λ Gand μ are adjustment coefficients. Softmax ensures weight normalization and has a certain degree of "strategy competitiveness," making the fusion result more selective. Formula (11) does not rely on the black-box performance of the strategy itself (such as training error), but instead constructs a lightweight, interpretable dynamic fusion mechanism based on structure and state information, with the advantages of simple deployment and clear regulation.

[0139] In step S107 of some embodiments, a weighted average is performed based on the fusion confidence and the candidate feature vectors to obtain the target conversion result, as shown in the following formula (12):

[0140]

[0141] in, represents the target conversion result of mode i, α k represents the fusion confidence of the candidate feature vector corresponding to strategy k, Represents a candidate feature vector. The target conversion result can be directly used in the system as input for classification tasks, state input for control strategies, input for equipment warning models, or uploaded to the data platform as a standard modal data stream.

[0142] For example, in the smart industrial scenario, mode i represents the “equipment vibration state”, and strategy T k From the "sound mode" and "temperature mode" paths, after the fusion mechanism, As an input to the equipment operation assessment model, no further modeling is required, only execution is required.

[0143] This step can be run in a stream processing mode in the deployment environment: in each round of sliding window, when G(t) and After the update, the system directly calculates the fusion results and pushes them to the downstream modules.

[0144] In steps S101 to S107 shown in the embodiment of the present application, the original modal data set is obtained by acquiring data of multiple modalities. A modal dependency graph is constructed based on the original modal data set, and the edge weights of the modal dependency graph are calculated. The target modal set of the current task is obtained, and the target path set is filtered out from the modal dependency graph based on the target modal set; wherein, the target path set includes multiple target paths. The target path is converted according to the preset strategy set to obtain a candidate conversion result set; wherein, the candidate conversion result set includes multiple candidate feature vectors. The confidence of each candidate feature vector is calculated based on the edge weight and modal activity of the target path to obtain multiple fusion confidences. The target conversion result is obtained by weighted averaging the fusion confidence and the candidate feature vectors. Sufficient flexibility and adaptability are also achieved when the data changes dynamically, the modal combination changes frequently, or the task environment is adjusted.

[0145] See also Figure 4 The present application also provides a system for dynamic conversion of heterogeneous data based on an integrated algorithm, which can implement the above-mentioned method for dynamic conversion of heterogeneous data based on an integrated algorithm. The system includes:

[0146] A first acquisition module 401 is configured to acquire data of multiple modalities to obtain an original modal data set;

[0147] A construction module 402 is used to construct a modal dependency graph based on the original modal data set and calculate the edge weights of the modal dependency graph;

[0148] The second acquisition module 403 is used to obtain the target modality set of the current task;

[0149] A screening module 404 is configured to screen a target path set from the modality dependency graph according to the target modality set; wherein the target path set includes multiple target paths;

[0150] The conversion module 405 is configured to convert the target path according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors;

[0151] A calculation module 406 is used to calculate the confidence of each candidate feature vector based on the edge weight and modal activity of the target path to obtain multiple fusion confidences;

[0152] The weighting module 407 is used to perform weighted averaging based on the fusion confidence and the candidate feature vectors to obtain the target conversion result.

[0153] The specific implementation of the heterogeneous data dynamic conversion system based on the integrated algorithm is basically the same as the specific embodiment of the heterogeneous data dynamic conversion method based on the integrated algorithm described above, and will not be repeated here.

[0154] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for dynamic conversion of heterogeneous data based on an integrated algorithm. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0155] See also Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0156] The processor 501 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0157] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called by the processor 501 to execute the heterogeneous data dynamic conversion method based on the integrated algorithm of the embodiment of the present application.

[0158] Input / output interface 503, used to implement information input and output;

[0159] Communication interface 504, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0160] Bus 505 , which transmits information between various components of the device (e.g., processor 501 , memory 502 , input / output interface 503 , and communication interface 504 );

[0161] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0162] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for dynamic conversion of heterogeneous data based on the integrated algorithm.

[0163] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0165] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0166] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0167] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0168] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0169] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above-mentioned units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0171] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for dynamic conversion of heterogeneous data based on an integrated algorithm, characterized in that: The method comprises: Acquire data of multiple modalities and obtain the original modal data set; Constructing a modal dependency graph according to the original modal data set, and calculating edge weights of the modal dependency graph; Get the target mode set of the current task; Filtering a target path set from the modality dependency graph according to the target modality set; wherein the target path set includes multiple target paths; The target path is converted according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors; Calculating the confidence of each candidate feature vector according to the edge weight and modal activity of the target path to obtain multiple fusion confidences; A weighted average is performed based on the fusion confidence and the candidate feature vector to obtain a target conversion result.

2. The method according to claim 1, characterized in that The constructing a modal dependency graph according to the original modal data set and calculating edge weights of the modal dependency graph includes: Performing a standardized representation on the original modal data set to obtain a standardized feature vector set; Performing activity evaluation based on the standardized feature vector set to obtain a modal activity vector; constructing the modality dependency graph according to the set of standardized feature vectors; The edge weights of the modality dependency graph are calculated based on the set of standardized feature vectors and the modality activity vector.

3. The method according to claim 2, characterized in that The modal activity vector includes multiple modal activities, and the activity evaluation is performed according to the standardized feature vector set to obtain the modal activity vector, including: The activity evaluation is: Among them, a i (t) represents the modal activity of the i-th modality at time t, represents the normalized eigenvector of the i-th mode in the unified eigenspace, represents the normalized value of the kth feature dimension of the i-th modality at time τ, represents the normalized value of the jth feature dimension of the i-th modality at time τ, Δ represents the time span of the sliding window, represents the normalized amplitude ratio of the kth dimension of the modal vector at time τ, and ∈ represents a very small constant to prevent the denominator from being zero.

4. The method according to claim 2, characterized in that The calculating the edge weight of the modality dependency graph according to the standardized feature vector set and the modality activity vector includes: The initial edge weights are calculated as: Among them, G ij (t) represents the initial edge weight, represents the standard cosine similarity, a i (t), a j (t) represents the modal activity of the i-th and j-th modal at the current moment, λ represents the current similarity and activity weight coefficient, β represents the activity difference penalty factor, and σ(·) represents the Sigmoid function; The target edge weight is calculated as: in, represents the target edge weight, α∈(0,1) represents the fusion ratio of the current observation value and the historical value, G ij (t) represents the initial edge weight, Represents the target edge weight between node i and node j at the previous moment.

5. The method according to claim 1, wherein The step of filtering out a target path set from the modality dependency graph according to the target modality set includes: Screening out, from the modality dependency graph, preliminary screening candidate paths pointing to the target modality set and having edge weights greater than a weight threshold; Scoring the pre-screened candidate paths according to a preset path scoring function to obtain path scoring data; The pre-screened candidate paths are screened according to the path scoring data to obtain the target path set; wherein the target path set includes multiple target paths.

6. The method according to claim 5, characterized in that Scoring the pre-screened candidate paths according to a preset path scoring function to obtain path scoring data includes: The path scoring function is: Among them, S ij (t) represents the path score data, represents the target edge weight, γ is the weighted effect of the control variance information on the score, and δ is the intensity of the penalty for controlling the activity difference. represents the variance of the normalized eigenvector of modality j in the current window.

7. The method according to claim 1, characterized in that The confidence calculation is performed on each candidate feature vector according to the edge weight and modal activity of the target path to obtain multiple fusion confidences, including: The fusion confidence is: Among them, α k (t) represents the fusion confidence of the candidate feature vector corresponding to strategy k, represents the edge weight of the target path to which strategy k belongs, represents the modality activity of the source modality of strategy k at the current moment, and It represents the edge weight and modal activity corresponding to traversing all candidate paths k' during the softmax normalization process, and λ and μ are adjustment coefficients.

8. A heterogeneous data dynamic conversion system based on an integrated algorithm, characterized in that: The system comprises: A first acquisition module is used to acquire data of multiple modalities to obtain an original modal data set; A construction module is used to construct a modal dependency graph based on the original modal data set and calculate the edge weights of the modal dependency graph; The second acquisition module is used to obtain the target modality set of the current task; A screening module, configured to screen out a target path set from the modal dependency graph according to the target modal set; wherein the target path set includes multiple target paths; A conversion module, configured to convert the target path according to a preset strategy set to obtain a candidate conversion result set; wherein the candidate conversion result set includes a plurality of candidate feature vectors; The calculation module is used to calculate the confidence of each candidate feature vector based on the edge weight and modal activity of the target path to obtain multiple fusion confidences; The weighting module is used to perform weighted averaging based on the fusion confidence and candidate feature vectors to obtain the target conversion result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the heterogeneous data dynamic conversion method based on the integrated algorithm as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for dynamic conversion of heterogeneous data based on an integrated algorithm according to any one of claims 1 to 7 is implemented.

Images