Visual large model decision tracking method and system based on cross-modal semantic alignment

Abstract

The invention discloses a visual large model decision tracking method and system based on cross-modal semantic alignment, and relates to the technical field of multi-modal decision tracking, and the method comprises the steps: carrying out the knowledge sub-graph retrieval of a weighted text semantic relation graph, executing the graph convolution reasoning, generating a knowledge graph embedded set, and carrying out the segmentation of the knowledge graph embedded set; the relation weight of the weighted text semantic relation graph is updated according to the edge weight fusion coefficient, and an enhanced text semantic relation graph is generated; generating a spatial adjacent edge set by calculating a radius adjacent graph of the multi-scale visual features, and performing relation mapping and fusion on the spatial adjacent edge set and the enhanced text semantic relation graph to generate a visual relation graph; and feature decomposition and statistical analysis are carried out on the visual relation graph and the enhanced text semantic relation graph, a structure enhanced alignment feature vector and an alignment consistency index are obtained and spliced, and an alignment driving decision input vector is generated. According to the method, the accuracy and adaptability of semantic association are improved through knowledge sub-graph retrieval and graph convolution reasoning.

Description

A Visual Large Model Decision Tracking Method and System Based on Cross-Modal Semantic Alignment Technical Field

[0001] This invention relates to the field of multimodal decision tracking technology, and in particular to a visual large model decision tracking method and system based on cross-modal semantic alignment. Background Technology

[0002] In the field of visual large-scale modeling and multimodal data analysis, existing technologies typically employ methods based on single-modal feature extraction or simple feature fusion for decision tracking. Conventional methods primarily extract visual features using convolutional neural networks and combine them with text keyword matching or statistical co-occurrence relationships to construct semantic association models. These methods have made some progress in collaborative multimodal data analysis, particularly demonstrating good application potential in tasks such as visual question answering and cross-modal retrieval. Furthermore, relational reasoning techniques based on graph convolutional networks have also been used to enhance semantic association modeling capabilities, achieving structured representation of multimodal data by constructing weighted relation graphs.

[0003] Existing methods still have limitations in cross-modal semantic alignment and decision tracking. Traditional methods are insufficient in their ability to align multi-scale visual features with textual semantics, relying solely on static relation weight calculations, making it difficult to dynamically adapt to complex relationships between different modalities. Conventional techniques typically process spatial adjacency and semantic associations independently, resulting in low efficiency in collaborative reasoning between visual and textual modalities, affecting the accuracy and interpretability of decision tracking. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a visual large model decision tracking method based on cross-modal semantic alignment to solve the problems of insufficient dynamic alignment of cross-modal data and low efficiency of multimodal collaborative reasoning.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: Firstly, this invention provides a visual large-scale model decision tracking method based on cross-modal semantic alignment, comprising: collecting multimodal data; performing time alignment and normalization on the multimodal data to generate a multimodal dataset; extracting multi-scale visual features; determining relation weights and constructing a weighted text semantic relation graph; performing knowledge subgraph retrieval and graph convolutional inference on the weighted text semantic relation graph to generate a knowledge graph embedding set; updating the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient to generate an enhanced text semantic relation graph; generating a spatial adjacency edge set by calculating the radius adjacency graph of the multi-scale visual features; performing relation mapping and fusion between the spatial adjacency edge set and the enhanced text semantic relation graph to generate a visual relation graph; performing feature decomposition and statistical analysis on the visual relation graph and the enhanced text semantic relation graph to obtain structurally enhanced alignment feature vectors and alignment consistency indices, and concatenating them to generate an alignment-driven decision input vector; performing decision inference on the alignment-driven decision input vector to identify the decision category and decision confidence, and generating a decision tracking log record.

[0007] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the steps for generating the multimodal dataset are as follows: collecting multimodal data, performing time alignment on the multimodal data, generating a synchronized text log set; performing basic normalization processing on the synchronized text log set, aligning and merging the multimodal data and the synchronized text log set in chronological order, and generating the multimodal dataset.

[0008] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the extraction of multi-scale visual features includes the following steps: unifying the spatial resolution and regularizing the input channels of the multi-modal dataset to generate a regularized image sequence; performing backbone convolution and stepwise downsampling on the regularized image sequence to generate a basic feature layer; performing lateral point-by-point projection channel alignment on the basic feature layer to generate basic aligned features; and adding the basic aligned features element by element and performing smooth fusion to generate multi-scale visual features.

[0009] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the steps for determining relation weights and constructing a weighted text semantic relation graph are as follows: scan the synchronous text log set one by one, perform longest match to identify candidate entities; unify the candidate entity names and remove duplicate candidate entities to generate synchronous text log entities; scan the synchronous text log set, generate entity pairs within the logs according to the identified synchronous text log entities and count the number of co-occurrences; obtain the relation weights of entity pairs within the logs and construct a weighted text semantic relation graph.

[0010] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the steps for generating the enhanced text semantic relationship graph are as follows: Entity pairs within the logs are arranged in descending order of relation weights, and a set of initial edges is constructed; the synchronous text log entities related to the initial edge set are summarized, and a knowledge subgraph is generated by combining the weighted text semantic relationship graph; graph convolution inference is performed on the knowledge subgraph to generate a knowledge graph embedding set; the relation weights of the weighted text semantic relationship graph and the knowledge graph embedding set are weighted and fused to generate enhanced relation weights; the enhanced relation weights replace the relation weights of the weighted text semantic relationship graph to generate the enhanced text semantic relationship graph.

[0011] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the steps for generating the spatial adjacency edge set are as follows: Multi-scale visual features are divided into scales to generate a single visual scale, and the spatial distance between the multi-scale visual features is calculated; the spatial distance is relaxed, and the relaxed spatial distance is set as a radius threshold; multi-scale visual features whose spatial distance does not exceed the radius threshold are recorded as visual position pairs; all visual position pairs are aggregated into a graph to generate a radius adjacency graph; the visual position pairs in the radius adjacency graph are merged and deduplicated according to a single visual scale to generate the spatial adjacency edge set.

[0012] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the step of mapping and fusing the spatial adjacency edge set with the enhanced text semantic relationship graph to generate a visual relationship graph involves the following steps: Positioning the visual position pairs of the spatial adjacency edge set in the enhanced text semantic relationship graph, establishing a relationship mapping with entity pairs within the log, and reading the enhanced relationship weights; calculating the enhanced relationship weights according to the edge weight fusion coefficient to generate visual relationship weights; weighting and fusing the visual relationship weights with the spatial distance of the visual position pairs within a single visual scale, and solidifying them into a set of visual position pairs with visual relationship weights; and constructing a visual relationship graph using the set of visual position pairs with visual relationship weights.

[0013] As a preferred embodiment of the visual large-scale model decision tracking method based on cross-modal semantic alignment described in this invention, the steps of performing feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph to obtain the structural enhancement alignment feature vector and the alignment consistency index, and concatenating them to generate the alignment-driven decision input vector are as follows: 1) Statistically calculate the number of visual position pairs, the visual weight statistics for calculating visual relationship weights, and the distance statistics for spatial distances; and combine them to generate the visual part of the structural enhancement alignment feature vector; 2) Statistically calculate the number of entity pairs in the log at each time point, the enhanced weight statistics for calculating enhanced relationship weights, and the number of entity pairs in the log of synchronized text log entities; and combine them to generate the text part of the structural enhancement alignment feature vector; 3) Combine the visual part of the structural enhancement alignment feature vector with the text part of the structural enhancement alignment feature vector to generate the structural enhancement alignment feature vector; 4) Establish the relationship mapping between visual positions and entity pairs in the log, calculate the average spatial distance of the visual relationship graph and the average number of entity pairs in the log of the enhanced text semantic relationship graph, and merge and average them to generate the alignment consistency index, which is then concatenated with the structural enhancement alignment feature vector to generate the alignment-driven decision input vector.

[0014] As a preferred embodiment of the visual large model decision tracking method based on cross-modal semantic alignment described in this invention, the steps for generating decision tracking log records are as follows: compliance determination is performed on the alignment-driven decision input vector to generate decision categories, and the decision categories are jointly quantified with the alignment consistency index to generate decision confidence; the relationship mapping between decision categories, decision confidence, alignment consistency index, visual relationship graph, and enhanced text semantic relationship graph is recorded to generate decision tracking log records.

[0015] Secondly, this invention provides a visual large-scale model decision tracking system based on cross-modal semantic alignment, comprising: a data acquisition and alignment module for acquiring multimodal data, performing time alignment and normalization on the multimodal data to generate a multimodal dataset, extracting multi-scale visual features, determining relation weights, and constructing a weighted text semantic relation graph; a graph enhancement module for performing knowledge subgraph retrieval on the weighted text semantic relation graph and performing graph convolutional inference to generate a knowledge graph embedding set, updating the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient, and generating an enhanced text semantic relation graph; and spatial mapping. The module is used to generate a set of spatial adjacency edges by calculating the radius adjacency graph of multi-scale visual features, and to map and fuse the set of spatial adjacency edges with the enhanced text semantic relationship graph to generate a visual relationship graph; the feature evaluation module is used to perform feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph, to obtain the structure-enhanced alignment feature vector and alignment consistency index, and to concatenate them to generate an alignment-driven decision input vector; the decision tracking module is used to perform decision reasoning on the alignment-driven decision input vector, to identify the decision category and decision confidence, and to generate decision tracking log records.

[0016] The beneficial effects of this invention are as follows: By using knowledge subgraph retrieval and graph convolutional reasoning, relation weights are dynamically updated and an enhanced text semantic relation graph is generated, realizing the dynamic optimization of text semantic features and the fusion of knowledge embedding, thereby improving the accuracy and adaptability of semantic association; by mapping and fusing the spatial adjacency edge set with the enhanced text semantic relation graph, a visual relation graph is constructed, realizing the deep alignment between visual spatial structure and text semantics, thereby enhancing the collaborative reasoning capability of cross-modal data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 is a flowchart of a visual large model decision tracking method based on cross-modal semantic alignment.

[0019] Figure 2 is a schematic diagram of a visual large model decision tracking system based on cross-modal semantic alignment.

[0020] Figure 3 is a flowchart of the spatial adjacency relationship construction.

[0021] Figure 4 is a flowchart of feature alignment and decision input vector generation. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Referring to Figures 1-4, an embodiment of the present invention is provided, which offers a visual large-scale model decision tracking method based on cross-modal semantic alignment, including the following steps: S1, collecting multimodal data, performing time alignment and normalization on the multimodal data to generate a multimodal dataset, extracting multi-scale visual features, determining relation weights, and constructing a weighted text semantic relation graph; collecting multimodal data, performing time alignment on the multimodal data using a time drift compensation method, and generating a synchronized text log set; further, collecting multimodal data, calling a time unification function to solidify the time of the multimodal data into a unified time recording format, unifying the time unit, generating a time series of the multimodal data, and establishing a time alignment order reference axis based on the time series of the multimodal data; aligning the time order according to the time unit. The reference axis is divided into equal-incrementing distance intervals to generate a candidate interval set. The maximum distance interval value is fixed as the drift search range threshold (the lower limit of the drift search range threshold is one time unit, and the upper limit is one-fifth of the time-order reference axis). The cross-correlation function is called to search for the time shift within the drift search range threshold to generate the time drift amount. Time shift compensation is performed on the multimodal data according to the time drift amount, and the data is sorted and numbered according to the time alignment reference axis to generate the time alignment reference axis sequence number and the synchronization text log set. It should be noted that equal-incrementing distance means dividing the time alignment reference axis by one time unit, two time units, three time units, etc., until it exceeds the range of the time alignment reference axis, and stops at the previous division point.

[0026] The synchronized text log set undergoes basic normalization, is aligned chronologically, and then merged with the multimodal data to generate a multimodal dataset. Further, the synchronized text log set undergoes basic normalization, is aligned chronologically, and then merged with the multimodal data using a nearest neighbor matching merging function to generate another multimodal dataset. It should be noted that basic normalization refers to standardizing the encoding, time format and time zone, units of measurement, and field names of the synchronized text log set.

[0027] Spatial resolution unification and input channel normalization are performed on the multimodal dataset to generate a normalized image sequence. Further, the spatial resolution of the multimodal dataset is statistically analyzed, and the spatial resolution with the highest frequency of occurrence is selected as the spatial resolution unification benchmark. A unified grid is established at each position of the unified benchmark, and positional correspondences are generated (the correspondence between each position under the unified benchmark and its four adjacent positions in the current multimodal data, including the upper left, upper right, lower left, and lower right, as well as their horizontal and vertical offsets). Values ​​are written into the corresponding positions of the unified grid using four-neighbor weighted bilinear interpolation to generate a multimodal dataset with unified spatial resolution. The time-aligned sequence is then referenced... The field names of each axis number are registered and fixed, and deduplication and consistency processing are performed to generate a candidate channel list. The candidate channel list and the time alignment reference axis are uniformly registered and fixed to generate a channel setting list. The multimodal dataset with uniform spatial resolution is regularized according to the order of the channel setting list and output sequentially to generate a regularized image sequence. It should be noted that four-neighbor weighting refers to using the values ​​of four adjacent (upper left, upper right, lower left, lower right) at the position of the uniform spatial resolution reference of the uniform grid, assigning weights according to the decimal offsets in the horizontal and vertical directions, and then summing the four values ​​to generate a new value for the position of the uniform spatial resolution reference.

[0028] The regularized image sequence is subjected to backbone convolution and progressive downsampling to generate a basic feature layer. Further, the first sequence of the regularized image sequence is selected for backbone convolution and progressive downsampling. First, the first sequence of the regularized image sequence undergoes a first convolution and downsampling to generate the first layer of valid results. Then, the first layer of valid results is used as the input for the second convolution process until no more valid outputs are generated, generating candidate layers. The candidate layers are truncated to a predetermined number of layers (lower limit three, upper limit five). The regularized image sequence is subjected to backbone convolution and progressive downsampling. The regularized image sequence undergoes a first convolution and downsampling... The process involves a second convolutional process followed by downsampling to generate the first layer's sampling result, which is then saved as the first layer's basic features. This first layer's sampling result is then used as the input for the second convolutional process, and the process of "convolutional processing, downsampling, saving by layer number, and using as input for the next layer" is repeated until the predetermined number of layers has been processed. Finally, the basic features of each layer are sorted according to the processing order to generate the basic feature layer. It should be noted that backbone convolution refers to using the same set of convolutional operations as a backbone, stacked continuously. It should also be noted that truncation with a lower limit of three and an upper limit of five means taking three values ​​for values ​​less than three, five values ​​for values ​​greater than five, and the value in between.

[0029] Lateral point-by-point projection channel alignment is performed on the basic feature layer to generate basic alignment features. Further, the first layer of the basic feature layer is aligned by channel using lateral point-by-point projection, and the first layer is saved to generate the first layer alignment features. The process is repeated on the second layer, following the sequence of "lateral point-by-point projection channel alignment, layer-by-layer saving, and switching to the next layer," until all alignments are complete. The alignment features from each layer are then aggregated to generate basic alignment features. The basic alignment features are then bilinearly upsampled from top to bottom and added element-by-element, followed by neighborhood convolutional smoothing to generate multi-scale visual features. Further, starting from the top layer, the alignment features from the previous layer are bilinearly upsampled and moved to the next layer, and then added element-by-elementally with the alignment features of the next layer to generate element-by-element sums. These sums are then smoothed using neighborhood convolution, and saved layer by layer, named as high-level visual feature tensor, mid-level visual feature tensor, and low-level visual feature tensor, and aggregated into multi-scale visual features. The results are then aligned according to the time alignment reference axis. The ontology dictionary is generated by collecting and deduplicating frequently occurring nouns and fixed phrases from the synchronized text log set, selecting the most frequent noun or fixed phrase as the main item of the ontology dictionary, listing synonyms and abbreviations of the main item in the synchronized text log set, and standardizing the character form and numerical units. The ontology dictionary main item is used as the unified name, establishing a correspondence between ontology dictionary main item, synonym, and abbreviation. Word boundary rules and numerical unit rules are configured for the ontology dictionary main item, and a rule is set that the ontology dictionary main item is used for longest match identification. The ontology dictionary includes the ontology dictionary main item, i.e., the standard name of each concept category, used as the unified unique name; an alias and abbreviation table, pointing multiple possible spellings to the corresponding ontology dictionary main item; word boundary rules and numerical unit rules, used to perform longest match identification of candidate entities, mark start and end positions, and standardize units in the synchronized text log set; and a main item order and conflict resolution strategy, where multiple matching entries with the same coverage area at the same position are determined by the ontology dictionary main item order.

[0030] The process involves scanning the synchronized text log set line by line, identifying candidate entities based on the longest match in the synchronized text log set according to the ontology dictionary. Further, the synchronized text logs are read sequentially, and segments are truncated according to the shortest entry in the ontology dictionary. Complete consistency comparisons are performed, and matching segments are added to the matching set. These segments are then extended according to the length of the ontology dictionary entry, and complete consistency comparisons are continued until no further matches are found. The synchronized text log segment with the largest coverage area in the matching set is selected as a candidate entity, and its start and end positions are recorded. The start and end positions covered by the candidate entity are truncated, and candidate entities are reselected for the remaining synchronized text logs until processing is complete. Based on the ontology dictionary, candidate entity names are unified, and duplicate candidate entities are removed to generate synchronized text log entities. Finally, the candidate entity names are unified and duplicate candidate entities are removed according to the ontology dictionary. After removing duplicates, the entities are organized chronologically to generate synchronized text log entities. The synchronized text log set is scanned chronologically, and for each synchronized text log, entity pairs are generated by combining the identified synchronized text log entities, and the co-occurrence frequency is counted. Further, the synchronized text log set is scanned chronologically, and for each synchronized text log, entity pairs are combined and deduplicated to generate entity pairs, and the co-occurrence frequency of each entity pair within the synchronized text log set is counted. Based on the co-occurrence frequency, the relation weight of each entity pair within the log is obtained, and a weighted text semantic relation graph is constructed. Further, the maximum co-occurrence frequency is determined, and the co-occurrence frequency of the entity pair within the log is compared with the maximum co-occurrence frequency to generate the relation weight of the entity pair within the log, and this weight is integrated with the entity pair within the log to generate a relation record set. The entity part of the weighted text semantic relation graph is built based on the synchronized text log entities, and the relation record set is written into the relation part of the weighted text semantic relation graph to generate the weighted text semantic relation graph.

[0031] S2. Perform knowledge subgraph retrieval and graph convolutional inference on the weighted text semantic relation graph to generate a knowledge graph embedding set. Update the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient to generate an enhanced text semantic relation graph. Sort the log entity pairs in the weighted text semantic relation graph in descending order of relation weights and construct the initial edge set. Further, sort the log entity pairs in the weighted text semantic relation graph in descending order of relation weights. When the relation weights of the log entity pairs are still the same after sorting by time alignment, sort them by the order of the main items in the ontology dictionary to generate a descending sequence of log entity pairs. Use each log entity pair in the descending sequence of log entity pairs as... The system first identifies the log start edges and marks the synchronous text log entities of entity pairs within the log as covered. It then integrates all log start edges to generate a start edge set. Next, it summarizes the synchronous text log entities related to the start edge set and combines this with a weighted text semantic relationship graph to generate a knowledge subgraph. Further, it reads each log entity pair within the start edge set, summarizes the synchronous text log entities, removes duplicates, and generates start edge entities. Finally, it organizes the log entity pairs and their relation weights in the weighted text semantic relationship graph according to time alignment, records the start edge entities, log entity pairs, and their relation weights to generate a knowledge subgraph. Finally, it performs graph convolution inference on the knowledge subgraph and sorts them according to time order to generate a knowledge graph. The process involves embedding the knowledge graph into a set; furthermore, it reads the log entity pairs within the knowledge subgraph one by one, generates updated log entities through graph convolutional inference, uses the synchronous text log entities of the knowledge subgraph as the first round of input updated log entities for the first round of graph convolutional inference, selects log entity pairs containing the first round of input updated log entities, weights and summarizes the synchronous text logs according to the relational weights of the log entity pairs, generates the first round of updated log entities, and uses them as the input for the second round of graph convolutional inference. This graph convolutional inference process is repeated until the generated updated log entities in the current round are the same as those in the previous round, at which point the graph convolutional inference process is complete; all updated log entities are integrated to generate a knowledge graph embedding set; and weighted text log entities are then embedded into the knowledge graph. The relation weights of the semantic relation graph are weighted and fused with the knowledge graph embedding set to generate enhanced relation weights. These enhanced relation weights replace the relation weights of the weighted text semantic relation graph, generating an enhanced text semantic relation graph. Further, the similarity between two synchronous text log entities of the log entity pair within the knowledge graph embedding set is calculated using a similarity function and then equally weighted and synthesized with the relation weights of the weighted text semantic relation graph to generate enhanced relation weights. These enhanced relation weights replace the relation weights of the weighted text semantic relation graph, generating an enhanced text semantic relation graph. It should be noted that the edges of the knowledge subgraph refer to the log entity pairs processed sequentially according to time alignment, whose relation weights are confirmed and retained in the weighted text semantic relation graph.

[0032] S3. By calculating the radius adjacency graph of multi-scale visual features, a spatial adjacency edge set is generated. Then, using a rule-based matching alignment method, the spatial adjacency edge set is mapped and fused with the enhanced text semantic relationship graph to generate a visual relationship graph. The multi-scale visual features are scaled to generate a single visual scale, and the spatial distance between the multi-scale visual features is calculated. Further, the visual feature tensors of the multi-scale visual features are arranged in a column, with the top-down hierarchy being high-level visual feature tensors, mid-level visual feature tensors, and low-level visual feature tensors. The mid-level visual feature tensors are used as the dividing point to scale the multi-scale visual features, and the high-level visual feature tensors are set to a single visual scale. The spatial distance between the mid-level visual feature tensors is calculated at each scale. In a unified grid, the spatial distances between the grid positions of multi-scale visual features within a single visual scale and their adjacent grid positions on the right, below, and to the lower right are calculated. These spatial distances are then relaxed and set as a radius threshold (ranging from the 20th to 40th percentile of spatial distances within a single visual scale). Multi-scale visual features whose spatial distances do not exceed this radius threshold are recorded as visual position pairs. All visual position pairs are then aggregated to generate a radius adjacency graph. Furthermore, the spatial distances to the right, below, and lower right adjacent grids are each summarized to generate three sets of spatial distance lists, which are then averaged to generate the average right adjacent spatial distance and the average lower adjacent spatial distance. The distance between the upper right and lower right adjacent average spatial distances are compared, and the average spatial distance with the largest value is selected as the broadened spatial distance. The broadened spatial distance is set as the radius threshold. All spatial distances are traversed at a single visual scale. Multi-scale visual features that do not exceed the spatial distance and the radius threshold are recorded as visual position pairs. All visual position pairs are aggregated into a radius adjacency graph. The visual position pairs in the radius adjacency graph are merged and deduplicated at a single visual scale to generate a set of spatial adjacency edges. The visual position pairs in the set of spatial adjacency edges are located in the enhanced text semantic relationship graph, a relationship mapping with entity pairs in the log is established, and the enhanced relationship weights are read. Furthermore, at a single visual scale, the visual positions in the radius adjacency graph are... The visual position pairs are uniformly organized, duplicate visual position pairs are deleted, and a set of spatial adjacency edges is generated. The temporal alignment reference axis number of the visual position pairs is determined based on the temporal alignment reference axis, and entity pairs within the log are located in the enhanced text semantic relationship graph according to the main item of the ontology dictionary. Simultaneously, a relationship mapping between visual position pairs and entity pairs within the log is established, and the enhanced relationship weights of the enhanced text semantic relationship graph are read. The enhanced relationship weights are calculated using the edge weight fusion coefficient to generate visual relationship weights. Further, the enhanced relationship weights and co-occurrence counts are read one by one along the temporal alignment reference axis in the enhanced text semantic relationship graph, and averaged separately to generate the average enhanced relationship weight and average co-occurrence count. The edge weight fusion coefficient is then calculated, with the expression: ;in, The edge weight fusion coefficient, To enhance the average value of relation weights, The average number of co-occurrences is used; the enhanced relationship weight is multiplied by the edge weight fusion coefficient to generate visual relationship weight; the visual relationship weight is weighted and fused with the spatial distance of the visual position pair within a single visual scale, and solidified into a set of visual position pairs with visual relationship weight; further, the spatial distance between the visual relationship weight and the visual position pair is read, and the ratio of the spatial distance to the radius threshold is calculated to generate a spatial ratio. When the spatial ratio is less than or equal to one, the spatial ratio is complemented to generate a distance affinity; when the spatial ratio is greater than one, the distance affinity is set to 0; the visual relationship weight and distance affinity are weighted to generate updated visual relationship weight, and written into the visual position pair; the visual position pairs with updated visual relationship weights are summarized and deduplicated to generate a set of visual position pairs with visual relationship weight.

[0033] A visual relationship graph is constructed using a set of visual position pairs with visual relationship weights. Further, the visual position pairs with visual relationship weights are deduplicated to generate a list of visual positions. The visual relationship weights of the visual position pairs are registered as visual relationship records and deduplicated. The list of visual positions is used as the visual position part of the visual relationship graph, and the list of visual relationship records is used as the relationship part of the visual relationship graph. These are then integrated to generate the visual relationship graph.

[0034] S4. Perform feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph to obtain the structural enhancement alignment feature vector and alignment consistency index, and concatenate them to generate the alignment-driven decision input vector; based on the visual relationship graph, count the number of visual position pairs at a single visual scale, calculate the visual weight statistics of visual relationship weights and the distance statistics of spatial distances, and combine them to generate the visual part of the structural enhancement alignment feature vector; further, at a single visual scale, read the visual position pairs and count them one by one to generate the number of visual position pairs; calculate the visual weight statistics of visual relationship weights, including the mean of visual relationship weights and the standard deviation of visual relationship weights. The expression for the mean of visual relationship weights is: ;in, The average weight of visual relationships. For the set of visual position pairs, For visual position pairs, For visual relationship weights, The first endpoint of the visual position pair. Let be the second endpoint of the visual position pair; the expression for the standard deviation of the visual relation weights is: ;in, The standard deviation of visual relationship weights; calculate the distance statistics for spatial distances, including the mean of spatial distances, expressed as: ;in, The mean of spatial distance For the set of visual position pairs, For visual position pairs, For spatial distance, The first endpoint of the visual position pair. The second endpoint of the visual location pair is defined. The number of visual location pairs, the mean of visual relationship weights, the standard deviation of visual relationship weights, and the mean of spatial distance are standardized to generate a list of visual statistical fields. These fields are then arranged in the order of the number of visual location pairs, the mean of visual relationship weights, the standard deviation of visual relationship weights, and the mean of spatial distance, and labeled with serial numbers to generate a fixed-order visual statistical sequence, which is then solidified as the visual part of the structure-enhanced alignment feature vector. It should be noted that the standardization process includes name unification, unit unification, and decimal place unification. Based on the enhanced text semantic relationship graph, the number of entity pairs in the log at each time point is statistically analyzed according to the time alignment order, the enhanced weight statistics for calculating enhanced relationship weights are calculated, and the daily statistics of synchronized text log entities are statistically analyzed. The system first counts the number of entity pairs within the log and combines them to generate the text portion of the structure-enhanced alignment feature vector. Then, it iterates through the enhanced text semantic relationship graph according to the time alignment reference axis number, merging, deduplicating, and counting the entity pairs within the log for each time alignment reference axis number to obtain the number of entity pairs within the log. Next, it iterates through the enhanced text semantic relationship graph according to the time alignment reference axis number, extracting the enhanced relationship weights for all entity pairs within the log for each time alignment reference axis number, generating an enhanced relationship weight list. Finally, it calculates the mean and standard deviation of the enhanced relationship weights based on the enhanced relationship weight list, and integrates the mean and standard deviation of the enhanced relationship weights to generate the enhanced relationship weights. The weighted augmentation weight statistics are used to traverse the augmented text semantic relationship graph sequentially according to the time alignment reference axis number. Within the augmented text semantic relationship graph, log entity pairs corresponding to each time alignment reference axis number are collected, and synchronous text log entities are extracted. The number of log entity pairs formed by each synchronous text log entity is counted. Under the same time alignment reference axis number, the log entity pairs are organized in the order of the number of log entity pairs, the augmentation weight statistics of the augmented relationship weights, and the number of log entity pairs of synchronous text log entities, and then concatenated sequentially according to the time alignment reference axis number to generate the text portion of the structural augmented alignment feature vector. The visual portion of the structural augmented alignment feature vector is then combined with the structural augmented alignment feature vector... The text portion is combined to generate a structure-enhanced alignment feature vector; further, the visual portion and text portion of the structure-enhanced alignment feature vector are made consistent item by item, first placing the visual portion of the structure-enhanced alignment feature vector, then placing the text portion of the structure-enhanced alignment feature vector to generate the structure-enhanced alignment feature vector; a relational mapping between visual positions and entity pairs within the log is established, and the average spatial distance of the visual relational graph and the average number of entity pairs within the log in the enhanced text semantic relational graph are calculated for each time point; further, a relational mapping is established between each visual position pair in the visual relational graph and the corresponding entity pair within the log in the enhanced text semantic relational graph under the same time alignment order reference axis number;The spatial distances of mapped visual position pairs are averaged to generate an average spatial distance for each set. The number of entity pairs within a synchronized text log entity is averaged to generate an average number of entity pairs within that synchronized text log entity. The average spatial distance of the visual relationship graph and the average number of entity pairs within a log entity in the enhanced text semantic relationship graph are combined and averaged to generate an alignment consistency index. This index is then concatenated with the structural enhancement alignment feature vector to generate an alignment-driven decision input vector. Further, the average spatial distance of the visual relationship graph and the average number of entity pairs within a log entity in the enhanced text semantic relationship graph are normalized and then averaged with equal weights to generate an alignment consistency index. This alignment consistency index is then concatenated to the end of the structural enhancement alignment feature vector and solidified to generate the alignment-driven decision input vector.

[0035] S5. Perform decision reasoning on the alignment-driven decision input vector, identify the decision category and decision confidence, and generate decision tracking log records.

[0036] The alignment-driven decision input vector is subjected to compliance assessment to generate decision categories. These categories are then jointly quantified with the alignment consistency index to generate decision confidence. Further, the time alignment order reference axis number is extracted from the alignment-driven decision input vector. The time alignment order reference axis number is compared with the time alignment order reference axis itself. If the time alignment order reference axis number corresponds to the time alignment order reference axis, the input vector is considered compliant; otherwise, it is considered non-compliant. The relationship mapping between visual position pairs and entity pairs within the log is checked along the time alignment order reference axis number. A one-to-one correspondence is considered compliant, while a non-one-to-one correspondence is considered non-compliant. Two judgment items are read sequentially along the time alignment order reference axis number. As a result, if both time alignment order reference axes are compliant, the number is marked as passed; otherwise, it is marked as failed. If all the time alignment order reference axis numbers are passed, a pass decision category is generated. If not all the time alignment order reference axis numbers are passed, a fail decision category is generated, and the reasons for failure are checked. If the time alignment order reference axis numbers of the alignment-driven decision input vector are missing, they are filled in. If the mapping relationship is non-compliant, the visual position pairs and entity pairs in the log are repositioned according to a single visual scale and the time alignment order reference axis, and the weights of the enhanced relationships are updated. The alignment-driven decision input vector is then regenerated. The decision categories are converted into category values, with the pass decision category marked as 1 and the fail decision category marked as 0. The category values ​​and the alignment consistency index are then combined with equal weights to generate the decision confidence score.

[0037] Record decision categories, decision confidence levels, alignment consistency indicators, and the mapping between visual relationship diagrams and enhanced text semantic relationship diagrams to generate decision tracking logs.

[0038] Furthermore, the decision categories, decision confidence levels, alignment consistency indicators, and relationship mappings between visual relationship diagrams and enhanced text semantic relationship diagrams are summarized by time-aligned sequence numbering. The mapped visual location pairs, entity pairs within the log, and enhanced relationship weights are recorded in each time-aligned sequence number, thus solidifying and generating decision tracking log records.

[0039] This embodiment also provides a visual large-scale model decision tracking system based on cross-modal semantic alignment, including: a data acquisition and alignment module for acquiring multimodal data, performing time alignment and normalization on the multimodal data, and generating a multimodal dataset; a feature relation module for extracting multi-scale visual features based on the multimodal dataset, and determining relation weights and constructing a weighted text semantic relation graph through ontology dictionary-driven and co-occurrence statistics; and a graph enhancement module for performing knowledge subgraph retrieval on the weighted text semantic relation graph and performing graph convolutional inference to generate a knowledge graph embedding set, and updating the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient. The system comprises the following modules: an enhanced text semantic relationship graph; a spatial mapping module, which generates a set of spatial adjacency edges by calculating the radius adjacency graph of multi-scale visual features, and maps and fuses the spatial adjacency edge set with the enhanced text semantic relationship graph to generate a visual relationship graph; a feature evaluation module, which performs feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph, obtains the structure-enhanced alignment feature vector and alignment consistency index, and concatenates them to generate an alignment-driven decision input vector; and a decision tracking module, which performs decision reasoning on the alignment-driven decision input vector, identifies the decision category and decision confidence, and generates decision tracking log records.

[0040] This embodiment also provides a computer device applicable to the visual large model decision tracking method based on cross-modal semantic alignment, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the visual large model decision tracking method based on cross-modal semantic alignment as proposed in the above embodiment.

[0041] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0042] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the visual large model decision tracking method based on cross-modal semantic alignment as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0043] In summary, this invention achieves dynamic optimization and knowledge embedding fusion of textual semantic features by dynamically updating relation weights and generating an enhanced textual semantic relation graph through knowledge subgraph retrieval and graph convolutional inference, thereby improving the accuracy and adaptability of semantic association. Furthermore, by mapping and fusing the spatial adjacency edge set with the enhanced textual semantic relation graph, a visual relation graph is constructed, achieving deep alignment between visual spatial structure and textual semantics, and enhancing the collaborative reasoning capability across modal data.

[0044] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual large model decision tracking method based on cross-modal semantic alignment, characterized by: The process includes: collecting multimodal data; performing time alignment and normalization on the multimodal data to generate a multimodal dataset; extracting multi-scale visual features; determining relation weights and constructing a weighted text semantic relation graph; performing knowledge subgraph retrieval and graph convolutional inference on the weighted text semantic relation graph to generate a knowledge graph embedding set; updating the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient to generate an enhanced text semantic relation graph; generating a spatial adjacency edge set by calculating the radius adjacency graph of multi-scale visual features; mapping and fusing the spatial adjacency edge set with the enhanced text semantic relation graph to generate a visual relation graph; performing feature decomposition and statistical analysis on the visual relation graph and the enhanced text semantic relation graph to obtain structurally enhanced alignment feature vectors and alignment consistency indices, and concatenating them to generate an alignment-driven decision input vector; performing decision inference on the alignment-driven decision input vector, identifying decision categories and decision confidence levels, and generating decision tracking log records.

2. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 1, characterized in that: The steps for generating the multimodal dataset are as follows: collect multimodal data, perform time alignment on the multimodal data, and generate a synchronized text log set; perform basic normalization on the synchronized text log set, align and merge the multimodal data and the synchronized text log set according to time order, and generate the multimodal dataset.

3. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 2, characterized in that: The steps for extracting multi-scale visual features are as follows: spatial resolution is unified and input channel is regularized on the multimodal dataset to generate a regularized image sequence; the regularized image sequence is subjected to backbone convolution and stepwise downsampling to generate a basic feature layer; the basic feature layer is subjected to lateral point-by-point projection channel alignment to generate basic aligned features. The basic alignment features are added element by element and then smoothly blended to generate multi-scale visual features.

4. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 3, characterized in that: The steps for determining relation weights and constructing a weighted text semantic relation graph are as follows: scan the synchronized text log set one by one, perform the longest match to identify candidate entities; unify the candidate entity names and remove duplicate candidate entities to generate synchronized text log entities; scan the synchronized text log set, generate entity pairs in the logs according to the identified synchronized text log entities and count the number of co-occurrences; obtain the relation weights of entity pairs in the logs and construct a weighted text semantic relation graph.

5. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 4, characterized in that: The steps for generating the enhanced text semantic relationship graph are as follows: Entity pairs within the logs are sorted in descending order of relation weights, and a set of initial edges is constructed; the synchronous text log entities related to the initial edge set are summarized, and a knowledge subgraph is generated by combining the weighted text semantic relationship graph; graph convolution inference is performed on the knowledge subgraph to generate a knowledge graph embedding set; the relation weights of the weighted text semantic relationship graph and the knowledge graph embedding set are weighted and fused to generate enhanced relation weights; the enhanced relation weights replace the relation weights of the weighted text semantic relationship graph to generate the enhanced text semantic relationship graph.

6. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 5, characterized in that: The steps for generating the spatial adjacency edge set are as follows: Multi-scale visual features are divided into scales to generate a single visual scale, and the spatial distance between the multi-scale visual features is calculated; the spatial distance is relaxed and set as a radius threshold; multi-scale visual features whose spatial distance does not exceed the radius threshold are recorded as visual position pairs; all visual position pairs are aggregated into a graph to generate a radius adjacency graph; the visual position pairs in the radius adjacency graph are merged and deduplicated according to a single visual scale to generate the spatial adjacency edge set.

7. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 6, characterized in that: The steps for mapping and fusing the spatial adjacency edge set with the enhanced text semantic relationship graph to generate a visual relationship graph are as follows: locate the visual position pairs of the spatial adjacency edge set in the enhanced text semantic relationship graph, establish a relationship mapping with the entity pairs in the log, and read the enhanced relationship weights. Visual relation weights are generated by calculating the enhanced relation weights based on the edge weight fusion coefficient. The spatial distance between visual relationship weights and visual position pairs within a single visual scale is weighted and fused, and solidified into a set of visual position pairs with visual relationship weights; a visual relationship graph is constructed using the set of visual position pairs with visual relationship weights.

8. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 7, characterized in that: The steps for performing feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph to obtain the structure-enhanced alignment feature vector and alignment consistency index and concatenate them to generate the alignment-driven decision input vector are as follows: count the number of visual position pairs, calculate the visual weight statistics of visual relationship weights and the distance statistics of spatial distances, and combine them to generate the visual part of the structure-enhanced alignment feature vector. For each time point, the number of entity pairs in the log is counted, the enhanced weight statistic of the enhanced relationship weight is calculated, and the number of entity pairs in the log of the synchronized text log entity is counted. These are then combined to generate the text part of the structure-enhanced alignment feature vector. The visual part of the structure-enhanced alignment feature vector is combined with the text part of the structure-enhanced alignment feature vector to generate the structure-enhanced alignment feature vector; the relationship mapping between visual position and entity pairs in the log is established, the average spatial distance of the visual relationship graph and the average number of entity pairs in the log of the enhanced text semantic relationship graph are calculated, and the average is taken to generate the alignment consistency index, which is then concatenated with the structure-enhanced alignment feature vector to generate the alignment-driven decision input vector.

9. The visual large model decision tracking method based on cross-modal semantic alignment as described in claim 8, characterized in that: The steps for generating decision tracking log records are as follows: perform compliance judgment on the alignment-driven decision input vector, generate decision categories, and jointly quantify the decision categories with the alignment consistency index to generate decision confidence. Record decision categories, decision confidence levels, alignment consistency indicators, and the mapping between visual relationship diagrams and enhanced text semantic relationship diagrams to generate decision tracking logs.

10. A visual large model decision tracking system based on cross-modal semantic alignment, based on the visual large model decision tracking method based on cross-modal semantic alignment as described in any one of claims 1 to 9, characterized in that: The system includes: a data acquisition and alignment module, used to acquire multimodal data, perform time alignment and normalization on the multimodal data, generate a multimodal dataset, extract multi-scale visual features, determine relation weights, and construct a weighted text semantic relation graph; a graph enhancement module, used to perform knowledge subgraph retrieval on the weighted text semantic relation graph and perform graph convolutional inference, generate a knowledge graph embedding set, and update the relation weights of the weighted text semantic relation graph according to the edge weight fusion coefficient, generating an enhanced text semantic relation graph; and a spatial mapping module, used to generate a spatial adjacency edge set by calculating the radius adjacency graph of multi-scale visual features, and perform relation mapping and fusion between the spatial adjacency edge set and the enhanced text semantic relation graph to generate a visual relation graph. The feature evaluation module is used to perform feature decomposition and statistical analysis on the visual relationship graph and the enhanced text semantic relationship graph, obtain the structure-enhanced alignment feature vector and alignment consistency index, and concatenate them to generate the alignment-driven decision input vector; the decision tracking module is used to perform decision reasoning on the alignment-driven decision input vector, identify the decision category and decision confidence, and generate decision tracking log records.

Images