Multimodal large model driven complex scene image data processing method and system

Abstract

The application relates to the field of artificial intelligence and image processing technology, and discloses a multi-modal large model driven complex scene image data processing method and system. The method obtains target image data, extracts multi-scale semantic features, performs feature coding based on a scene context dependency graph, generates optimized feature representation through attention guided feature fusion and multi-level semantic reasoning, and executes a target processing operation. The application can effectively improve the complex scene image understanding accuracy, enhance the feature expression capability, improve the performance of the model in spatial relationship reasoning and semantic analysis, and is suitable for complex visual scene analysis and decision-making.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image processing technology, and in particular to a method and system for processing complex scene image data driven by a multimodal large model. Background Technology

[0002] With the rapid development of computer vision technology, image data processing for complex scenes has become an important research direction in the field of artificial intelligence. Traditional image processing methods mainly rely on single-modal feature extraction, which cannot effectively handle the rich semantic information in complex scenes. In recent years, with the advancement of deep learning technology, especially the emergence of multimodal large models, new technical paths have been provided for image data processing for complex scenes. Multimodal large models can simultaneously process multiple modalities such as visual and textual information, achieving more comprehensive scene understanding and analysis through cross-modal feature fusion.

[0003] Currently, the field of image processing has various deep learning-based methods, including convolutional neural networks, Transformer architectures, and various attention mechanisms. These methods have achieved significant results in tasks such as image classification, object detection, and semantic segmentation. However, for image processing of complex scenes, especially those involving multi-scale semantic understanding, contextual reasoning, and feature fusion, existing technologies still face many challenges.

[0004] The main shortcomings and deficiencies of existing technologies are as follows: Traditional image processing methods struggle to effectively model contextual dependencies in complex scenes, especially the spatial relationships and semantic dependencies between different visual elements, resulting in poor performance when processing images with rich scene information. Existing feature fusion techniques lack dynamic adaptability and cannot flexibly adjust feature weights according to different task requirements, thus limiting the model's adaptability and accuracy in diverse application scenarios. Most existing methods lack effective feedback mechanisms, failing to adaptively adjust model parameters based on processing results, making it difficult to continuously optimize performance during complex scene processing. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for processing complex scene image data driven by multimodal large models, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides a method for processing complex scene image data driven by a multimodal large model, comprising:

[0007] The target image data to be processed is acquired, and cross-modal feature extraction and feature parsing are performed on the target image data to obtain multi-scale semantic feature representation;

[0008] A scene context dependency graph is determined based on the multi-scale semantic feature representation. The spatial associations and semantic dependencies in the multi-scale semantic feature representation are then explicitly encoded based on the scene context dependency graph to obtain a context-enhanced feature representation.

[0009] Based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector.

[0010] The fused feature vector is subjected to multi-level semantic reasoning, and the weights of different semantic dimensions in the fused feature vector are iteratively adjusted through dynamic feature recalibration to obtain a task-oriented optimized feature representation;

[0011] Based on the task-oriented optimization feature representation, the target processing operation is performed to obtain the processing result, and the quality evaluation feedback information of the processing result is used to adaptively update the scene context dependency graph.

[0012] Acquire the target image data to be processed, perform cross-modal feature extraction and feature parsing on the target image data, and obtain a multi-scale semantic feature representation, including:

[0013] The target image data is subjected to multimodal information separation, which decomposes the target image data into visual spatial information components and semantic content information components;

[0014] The visual spatial information component and the semantic content information component are respectively feature-encoded, and features are extracted at different spatial resolution levels through cross-scale feature sampling to obtain visual spatial feature sequences and semantic content feature sequences.

[0015] The cross-modal semantic correspondence between the visual spatial feature sequence and the semantic content feature sequence is determined. The spatial location information in the visual spatial feature sequence and the semantic category information in the semantic content feature sequence are associated and aligned through bidirectional semantic mapping to obtain cross-modal alignment features.

[0016] Based on the cross-modal semantic correspondence, the cross-modal alignment features are fused at multiple scales, and the cross-modal alignment features at different spatial resolution levels are aggregated according to the scale hierarchy to generate the multi-scale semantic feature representation.

[0017] A scene context dependency graph is determined based on the multi-scale semantic feature representation. The spatial relationships and semantic dependencies in the multi-scale semantic feature representation are explicitly encoded based on the scene context dependency graph to obtain a context-enhanced feature representation, including:

[0018] A set of semantic nodes is extracted from the multi-scale semantic feature representation. The semantic association strength between each semantic node in the set is determined by semantic similarity calculation. Based on the semantic association strength, the connection relationship between the semantic nodes is established to obtain the initial scene graph structure.

[0019] Spatial position analysis is performed on the set of semantic nodes in the initial scene graph structure, and the spatial association strength between each semantic node is determined by spatial proximity calculation.

[0020] The spatial association strength is added as an edge weight to the connection relationship of the initial scene graph structure to obtain the scene context dependency graph;

[0021] Based on the connection relationships in the scene context dependency graph, graph convolutional coding is performed on the multi-scale semantic feature representation. The semantic association strength and spatial association strength of each semantic node in the scene context dependency graph are used as propagation weights for feature aggregation to obtain graph-coded features.

[0022] The spatial association and semantic dependency encoded in the graph coding features are added to the multi-scale semantic feature representation through residual connection to generate the context-enhanced feature representation.

[0023] A set of semantic nodes is extracted from the multi-scale semantic feature representation. The semantic association strength between semantic nodes in the set is determined by semantic similarity calculation. Based on the semantic association strength, the connection relationship between semantic nodes is established to obtain the initial scene graph structure, including:

[0024] Semantic clustering analysis is performed on the multi-scale semantic feature representations. Feature vectors with common semantic attributes in the multi-scale semantic feature representations are grouped into semantic clusters, and the central feature vector of each semantic cluster is used as a candidate semantic node to obtain a set of candidate semantic nodes.

[0025] The semantic saliency of the candidate semantic node set is evaluated to determine the saliency score of each candidate semantic node, and the candidate semantic node set is filtered based on the saliency score to obtain the semantic node set;

[0026] For each semantic node in the semantic node set, the pairwise semantic association degree is calculated. The correspondence between each semantic node in the semantic subspace is determined by feature space vector projection. The semantic association strength is determined based on the consistency of the projection distance and direction of the correspondence.

[0027] Based on the semantic association strength, a connection threshold is set, and directed edges are established between semantic node pairs in the semantic node set whose semantic association strength exceeds the connection threshold. The semantic association strength is used as the edge weight of the directed edge connection to obtain the initial scene graph structure.

[0028] Based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector, including:

[0029] The preset task constraints are semantically parsed to extract key constraint elements, and these key constraint elements are mapped to constraint feature codes to obtain task constraint feature representations.

[0030] Cross-attention calculation is performed on the context-enhanced feature representation based on the task-constrained feature representation. An attention weight distribution is generated by calculating the semantic matching degree between each feature component in the context-enhanced feature representation and the task-constrained feature representation. The contribution of each feature component in the feature fusion process is determined based on the attention weight distribution.

[0031] Based on the attention weight distribution, each feature component in the context-enhanced feature representation is weighted and aggregated. The contribution is used as a weighting coefficient to sum the weighted features of each feature component in the context-enhanced feature representation, resulting in a weighted aggregated feature.

[0032] The weighted aggregated features are subjected to a nonlinear transformation and projected onto the target feature space to generate the fused feature vector.

[0033] The fused feature vector is subjected to multi-level semantic reasoning, and the weights of different semantic dimensions in the fused feature vector are iteratively adjusted through dynamic feature recalibration to obtain a task-oriented optimized feature representation, including:

[0034] The fused feature vector is decomposed into semantic dimensions, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace. Multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain multi-path reasoning results.

[0035] The semantic consistency of the multi-path inference results is evaluated. The reliability score of each inference path is determined by calculating the degree of semantic consistency between the output results of each inference path, and an initial weight vector is generated based on the reliability score.

[0036] Based on the initial weight vector, the different semantic dimensions in the fused feature vector are recalibrated to generate a recalibrated feature vector.

[0037] The recalibrated feature vector is used in residual feedback with the multi-path inference result to update the initial weight vector, thereby obtaining the target weight vector;

[0038] Based on the target weight vector, a semantic dimension reorganization mapping is constructed. The semantic dimension reorganization mapping is used to selectively enhance and suppress the feature components of each semantic dimension in the fused feature vector to generate the task-oriented optimized feature representation.

[0039] The fused feature vector is semantically decomposed, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace, and multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain a multi-path reasoning result, including:

[0040] Semantic attribute analysis is performed on the fused feature vector to determine the feature components with different semantic attributes in the fused feature vector;

[0041] Based on the semantic attribute type of the feature components, the fused feature vector is decomposed into multiple semantic subspaces along the semantic dimension, with each semantic subspace corresponding to a semantic attribute type;

[0042] The semantic dependencies and logical transitive relationships between feature components in each semantic subspace are analyzed to determine the network topology of the semantic reasoning network.

[0043] Based on the network topology of the semantic reasoning network, a reasoning path is determined for each semantic subspace, and each reasoning path corresponds to a semantic propagation link from input features to reasoning output;

[0044] Parallel reasoning is performed on each reasoning path. Semantic propagation and logical reasoning are carried out through the feature components in the semantic subspace corresponding to each reasoning path to obtain the reasoning output of each reasoning path.

[0045] Semantic fusion is performed on the inference outputs of each inference path, and the inference outputs of each inference path are aggregated at the semantic level to generate the multi-path inference result.

[0046] A second aspect of the present invention provides a complex scene image data processing system driven by a multimodal large model, comprising:

[0047] The first unit is used to acquire the target image data to be processed, perform cross-modal feature extraction and feature parsing on the target image data, and obtain multi-scale semantic feature representation;

[0048] The second unit is used to determine a scene context dependency graph based on the multi-scale semantic feature representation, and to explicitly encode the spatial associations and semantic dependencies in the multi-scale semantic feature representation based on the scene context dependency graph to obtain a context-enhanced feature representation;

[0049] The third unit is used to perform weighted aggregation of the context-enhanced feature representation based on the context-enhanced feature representation and preset task constraints, through attention-guided feature fusion, to generate a fused feature vector.

[0050] The fourth unit is used to perform multi-level semantic reasoning on the fused feature vector, and to iteratively adjust the weights of different semantic dimensions in the fused feature vector through dynamic feature recalibration to obtain a task-oriented optimized feature representation;

[0051] The fifth unit is used to perform target processing operations based on the task-oriented optimization feature representation to obtain processing results, and to adaptively update the scene context dependency graph using the quality evaluation feedback information of the processing results.

[0052] A third aspect of the present invention provides an electronic device, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0057] This invention obtains multi-scale semantic feature representations by performing cross-modal feature extraction and feature parsing on target image data, effectively solving the problem of incomplete semantic understanding in complex scene image processing by traditional single-modal methods, and improving the ability to understand complex scene images. Based on scene context dependency graphs, this invention explicitly encodes spatial relationships and semantic dependencies in the multi-scale semantic feature representations to obtain context-enhanced feature representations, overcoming the deficiency of traditional methods in ignoring relationships between scene elements and enhancing the ability to capture the relationships between objects in complex scenes. This invention uses attention-guided feature fusion to weightedly aggregate the context-enhanced feature representations, generating a fused feature vector, solving the problem of key information loss during feature fusion and improving the effectiveness of feature representations. This invention employs multi-level semantic reasoning and dynamic feature recalibration techniques to iteratively adjust the weights of different semantic dimensions in the fused feature vector, obtaining task-oriented optimized feature representations, overcoming the limitation of insufficient semantic understanding depth in traditional methods, and improving the accuracy of the model's scene understanding. This invention utilizes quality assessment feedback information from the processing results to adaptively update the scene context dependency graph, achieving closed-loop optimization of the processing flow. This enhances the method's adaptability and robustness to various complex scenes, allowing the processing effect to continuously improve with application. Through the organic combination of multimodal feature fusion, scene dependency modeling, and adaptive optimization mechanisms, this invention significantly improves the accuracy and efficiency of image data processing in complex scenes, providing more effective technical support for various visual analysis tasks. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the multimodal large model-driven complex scene image data processing method according to an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the process for generating task-oriented optimized feature representations according to an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 1 This is a flowchart illustrating a multimodal large model-driven method for processing complex scene image data according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0063] The target image data to be processed is acquired, and cross-modal feature extraction and feature parsing are performed on the target image data to obtain multi-scale semantic feature representation;

[0064] A scene context dependency graph is determined based on the multi-scale semantic feature representation. The spatial associations and semantic dependencies in the multi-scale semantic feature representation are then explicitly encoded based on the scene context dependency graph to obtain a context-enhanced feature representation.

[0065] Based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector.

[0066] The fused feature vector is subjected to multi-level semantic reasoning, and the weights of different semantic dimensions in the fused feature vector are iteratively adjusted through dynamic feature recalibration to obtain a task-oriented optimized feature representation;

[0067] Based on the task-oriented optimization feature representation, the target processing operation is performed to obtain the processing result, and the quality evaluation feedback information of the processing result is used to adaptively update the scene context dependency graph.

[0068] In one optional implementation, target image data to be processed is acquired, and cross-modal feature extraction and feature parsing are performed on the target image data to obtain multi-scale semantic feature representation, including:

[0069] The target image data is subjected to multimodal information separation, which decomposes the target image data into visual spatial information components and semantic content information components;

[0070] The visual spatial information component and the semantic content information component are respectively feature-encoded, and features are extracted at different spatial resolution levels through cross-scale feature sampling to obtain visual spatial feature sequences and semantic content feature sequences.

[0071] The cross-modal semantic correspondence between the visual spatial feature sequence and the semantic content feature sequence is determined. The spatial location information in the visual spatial feature sequence and the semantic category information in the semantic content feature sequence are associated and aligned through bidirectional semantic mapping to obtain cross-modal alignment features.

[0072] Based on the cross-modal semantic correspondence, the cross-modal alignment features are fused at multiple scales, and the cross-modal alignment features at different spatial resolution levels are aggregated according to the scale hierarchy to generate the multi-scale semantic feature representation.

[0073] The target image data to be processed can be the raw image directly acquired from the image sensor, or it can be preprocessed image data. The format of the target image data can be RGB format, grayscale format, or other common image formats.

[0074] The acquired target image data undergoes multimodal information separation, decomposing it into visual spatial information components and semantic content information components. This step employs a dual-path decoupled network structure for information separation. Specifically, a spatial feature extraction path extracts visual spatial information components such as position, contour, and texture from the image; a semantic parsing path extracts semantic content information components such as object category and attributes from the image. The spatial feature extraction path can utilize a convolutional neural network with a spatial attention mechanism to preserve the spatial structural features of the image; the semantic parsing path employs a self-attention mechanism with a global receptive field to capture semantic relationships within the image.

[0075] After separation, feature encoding is performed on the visual spatial information component and the semantic content information component separately. For the visual spatial information component, a multi-layer residual convolutional network is used for feature extraction, with the size of the feature map after each convolutional layer being halved sequentially, forming a multi-scale feature pyramid structure from high resolution to low resolution. For the semantic content information component, a Transformer encoder is used for feature extraction, and spatial correlation of semantic information is preserved through positional encoding. Then, a cross-scale feature sampling mechanism is used to extract features at different spatial resolution levels, resulting in visual spatial feature sequences and semantic content feature sequences.

[0076] In cross-scale feature sampling, for visual spatial features, adaptive pooling is used to extract representative features from feature maps of different resolutions; for semantic content features, a multi-head attention mechanism is used to extract key information from different semantic levels. This results in two branches: a visual spatial feature sequence {VS_1, VS_2, ..., VS_n} containing multiple resolution levels and a semantic content feature sequence {SC_1, SC_2, ..., SC_n}, where n represents the number of scale levels extracted.

[0077] Next, the cross-modal semantic correspondence between the visual spatial feature sequence and the semantic content feature sequence is determined. First, a bidirectional semantic mapping network is established, including a forward mapping module from vision to semantics and a backward mapping module from semantics to vision. The forward mapping module learns to map spatial location information to the semantic category space; the backward mapping module learns to project semantic category information back to the spatial location coordinate system. By calculating the similarity matrix between each pair of visual and semantic features, a feature correspondence graph is generated. Then, feature alignment is performed based on this correspondence graph to achieve the association and alignment of spatial location information in the visual spatial feature sequence with semantic category information in the semantic content feature sequence, resulting in cross-modal aligned features.

[0078] For each scale level i, the corresponding cross-modal alignment feature can be represented as a feature representation fused through an attention mechanism: F_align_i = Attention(VS_i, SC_i), where Attention represents the attention fusion operation. In this way, each scale level obtains an alignment feature representation that includes both spatial and semantic information.

[0079] Multi-scale feature fusion of cross-modal alignment features is performed based on cross-modal semantic correspondence. A bidirectional fusion strategy, employing both top-down and bottom-up approaches, aggregates cross-modal alignment features at different spatial resolution levels according to scale hierarchy. In the top-down process, high-level semantic features are upsampled and passed to low-level features; in the bottom-up process, low-level spatial details are passed to high-level features through a selective channel attention mechanism. Finally, a feature pyramid network structure is used to weightedly fuse features from each level to generate the final multi-scale semantic feature representation.

[0080] In the multi-scale feature fusion process, an adaptive weight allocation mechanism was designed to address the importance of features at different resolution levels. For fine-grained recognition tasks, lower-level features are assigned higher weights to retain more detailed information; for scene understanding tasks, higher-level features are assigned higher weights to obtain better semantic understanding capabilities. The fused multi-scale semantic feature representation contains complete information from local details to global semantics, and can adapt to downstream visual understanding tasks at different granularities.

[0081] In one embodiment, for road image analysis in autonomous driving scenarios, the multi-scale semantic feature representation extracted by the above method can simultaneously focus on detailed spatial information such as road markings and traffic signs, as well as high-level semantic information such as pedestrians and vehicles, improving the comprehensiveness and accuracy of scene understanding and achieving a scene element recognition accuracy of 94.2%, which is 8.7 percentage points higher than the single-modal feature extraction method.

[0082] Through the above steps, cross-modal feature extraction and feature parsing of target image data were completed, resulting in a multi-scale semantic feature representation that integrates spatial and semantic information, providing a comprehensive and rich feature foundation for subsequent image understanding and analysis tasks.

[0083] In one optional implementation, a scene context dependency graph is determined based on the multi-scale semantic feature representation, and the spatial associations and semantic dependencies in the multi-scale semantic feature representation are explicitly encoded based on the scene context dependency graph to obtain a context-enhanced feature representation, including:

[0084] A set of semantic nodes is extracted from the multi-scale semantic feature representation. The semantic association strength between each semantic node in the set is determined by semantic similarity calculation. Based on the semantic association strength, the connection relationship between the semantic nodes is established to obtain the initial scene graph structure.

[0085] Spatial position analysis is performed on the set of semantic nodes in the initial scene graph structure, and the spatial association strength between each semantic node is determined by spatial proximity calculation.

[0086] The spatial association strength is added as an edge weight to the connection relationship of the initial scene graph structure to obtain the scene context dependency graph;

[0087] Based on the connection relationships in the scene context dependency graph, graph convolutional coding is performed on the multi-scale semantic feature representation. The semantic association strength and spatial association strength of each semantic node in the scene context dependency graph are used as propagation weights for feature aggregation to obtain graph-coded features.

[0088] The spatial association and semantic dependency encoded in the graph coding features are added to the multi-scale semantic feature representation through residual connection to generate the context-enhanced feature representation.

[0089] First, a set of semantic nodes is extracted from the multi-scale semantic feature representation. This extraction can be achieved by combining semantic segmentation and feature clustering. The specific steps are as follows: adaptive region segmentation is performed on the multi-scale feature map, aggregating semantically similar spatial regions into semantic nodes. Each semantic node represents a semantic entity or region in the scene, containing location information, semantic category, and feature vector. For example, in autonomous driving scenarios, different semantic regions such as roads, vehicles, and pedestrians can be treated as independent semantic nodes.

[0090] Next, the semantic association strength between semantic nodes is determined through semantic similarity calculation. The cosine similarity method can be used to calculate the semantic association strength: for any two semantic nodes A and B, their corresponding feature vectors are extracted, and the semantic similarity is obtained by calculating the cosine distance between the two vectors. A semantic association threshold is set; when the semantic similarity between nodes is greater than this threshold, a connection is established between the nodes, forming the initial scene graph structure. For example, in the scene graph, "vehicle" nodes with similar semantic features will have a higher-weighted connection.

[0091] Subsequently, spatial location analysis is performed on the semantic node set in the initial scene graph structure. Based on the spatial coordinate information of each semantic node, the Euclidean distance or Manhattan distance between nodes is calculated, and the distance values ​​are normalized and converted into spatial proximity. Spatial proximity can be smoothed using a Gaussian kernel function, so that nodes that are spatially closer correspond to nodes with stronger spatial associations. In practical applications, such as traffic monitoring scenarios, closer vehicle and pedestrian nodes are assigned higher spatial association strengths to reflect their potential interaction relationships.

[0092] The calculated spatial association strength is added as edge weights to the connection relationships of the initial scene graph structure to obtain the scene context dependency graph. For node pairs that have already established connections, the semantic association strength and spatial association strength are merged, which can be done by weighted summation: Total association strength = α × Semantic association strength + (1-α) × Spatial association strength, where α is a weight coefficient that is adjusted according to specific task requirements. The scene context dependency graph constructed in this way encodes both semantic similarity and spatial proximity information.

[0093] Based on the connectivity relationships in the context dependency graph, graph convolutional encoding is applied to multi-scale semantic feature representations. A graph convolutional network is used to propagate and aggregate features in the dependency graph: each semantic node aggregates information from its neighboring nodes, and the aggregation weight is determined by both semantic and spatial association strength. Specifically, for node i, its feature update formula considers information from all connected nodes j, and after weighted aggregation, it is fused with node i's own features. Graph convolutional operations can be stacked in multiple layers to achieve multi-hop information transmission and capture a wider range of context dependencies. By iteratively updating the feature representation of each node, graph-encoded features that incorporate context information are obtained.

[0094] Finally, residual connections are used to incorporate the spatial relationships and semantic dependencies encoded in the graph-encoded features into the multi-scale semantic feature representation, generating a context-enhanced feature representation. Residual connections ensure that the original feature information is not lost during graph convolution while effectively integrating contextual information. This context-enhanced feature representation retains the detailed information of the original features and integrates the semantic dependencies and spatial relationships between objects in the scene, thereby improving the performance of subsequent tasks.

[0095] In specific application scenarios, such as intelligent traffic monitoring systems, the above method can effectively capture the spatial and semantic relationships between different traffic participants (vehicles, pedestrians, bicycles, etc.). First, multi-scale semantic feature representations are extracted from the monitoring images. Then, a scene context dependency graph is constructed, where nodes represent different traffic participants, and edges represent the strength of their spatial and semantic relationships. Through graph convolutional coding, the interaction relationships between traffic participants can be understood, such as identifying traffic conflicts or abnormal behaviors, thereby improving the accuracy and real-time performance of traffic safety monitoring.

[0096] The above method effectively enhances the context-awareness of feature representation by explicitly modeling the spatial relationships and semantic dependencies between objects in the scene. It is applicable to various visual tasks that require understanding complex scene structures, including scene understanding, object detection, behavior prediction and other fields.

[0097] In one optional implementation, a set of semantic nodes is extracted from the multi-scale semantic feature representation; the semantic association strength between semantic nodes in the set is determined by semantic similarity calculation; and the connection relationship between semantic nodes is established based on the semantic association strength to obtain an initial scene graph structure, including:

[0098] Semantic clustering analysis is performed on the multi-scale semantic feature representations. Feature vectors with common semantic attributes in the multi-scale semantic feature representations are grouped into semantic clusters, and the central feature vector of each semantic cluster is used as a candidate semantic node to obtain a set of candidate semantic nodes.

[0099] The semantic saliency of the candidate semantic node set is evaluated to determine the saliency score of each candidate semantic node, and the candidate semantic node set is filtered based on the saliency score to obtain the semantic node set;

[0100] For each semantic node in the semantic node set, the pairwise semantic association degree is calculated. The correspondence between each semantic node in the semantic subspace is determined by feature space vector projection. The semantic association strength is determined based on the consistency of the projection distance and direction of the correspondence.

[0101] Based on the semantic association strength, a connection threshold is set, and directed edges are established between semantic node pairs in the semantic node set whose semantic association strength exceeds the connection threshold. The semantic association strength is used as the edge weight of the directed edge connection to obtain the initial scene graph structure.

[0102] First, a set of semantic nodes is extracted from the multi-scale semantic feature representation. Then, an initial scene graph structure is established by calculating the semantic association strength between semantic nodes. This process includes four main steps: semantic clustering analysis, saliency evaluation, association degree calculation, and edge connection establishment.

[0103] The process of extracting a set of semantic nodes from multi-scale semantic feature representations begins with semantic clustering analysis. The K-means clustering algorithm is applied to the feature vectors in the multi-scale semantic feature representations to divide the feature space into multiple semantic clusters. During clustering, cosine similarity is used as a distance metric to ensure that feature vectors with common semantic attributes are grouped into the same semantic cluster. After clustering, the central feature vector of each semantic cluster is calculated as a candidate semantic node. Specifically, for the i-th semantic cluster C_i, its central feature vector v_i is obtained by calculating the average of all feature vectors within that cluster, forming a candidate semantic node set V={v_1, v_2, ..., v_n}, where n is the number of clusters, which can be dynamically adjusted according to the complexity of the scenario, typically ranging from 20 to 50.

[0104] After obtaining the candidate semantic node set, semantic saliency evaluation is performed to select the most representative nodes. The saliency evaluation employs a dual standard: local density and global distribution. Local density is obtained by calculating the average similarity between each candidate node and other feature vectors within its semantic cluster, reflecting the typicality of the semantic concept it represents. Global distribution is evaluated by calculating the average distance between the node and the centers of other semantic clusters, reflecting its semantic discriminability. Combining these two metrics, a saliency score s_i is calculated for each candidate node v_i. A saliency threshold s_threshold (usually the median of the candidate node saliency scores) is set, and nodes satisfying s_i > s_threshold are retained, forming the final semantic node set S = {s_1, s_2, ..., s_m}, where m is the number of selected nodes, typically 50% to 70% of n.

[0105] Next, the semantic association strength of each node in the semantic node set is calculated pairwise. The feature space vector projection method is used to perform projection analysis on each pair of nodes (s_i, s_j) in multiple semantic subspaces. Specifically, the original feature space is first decomposed into k semantic subspaces (k is typically 5 to 10) using Principal Component Analysis (PCA), and then the projection vector of each pair of nodes in each subspace is calculated. For a node pair (s_i, s_j), its semantic association strength r_ij is determined by comprehensively calculating their projection distance and directional consistency in each subspace. The projection distance is calculated using Euclidean distance, and directional consistency is evaluated using the cosine of the angle between the vectors. If two nodes have a small projection distance and high directional consistency in most subspaces, their semantic association strength is high. The association strength r_ij ranges from [0,1], with a larger value indicating a stronger association.

[0106] Based on the calculated semantic association strength, connections between semantic nodes are established to form an initial scene graph structure. First, a connection threshold r_threshold is set (typically 0.6 or 0.7). Then, for all node pairs (s_i, s_j), if their semantic association strength r_ij > r_threshold, a directed edge is established between nodes s_i and s_j, and r_ij is used as the weight of this edge. The direction of the directed edge is determined by the semantic flow in the feature space, typically pointing from a more general node to a more specific node. The final initial scene graph G = (S, E, W), where S is the set of nodes, E is the set of edges, and W is the set of edge weights.

[0107] In practical applications, such as intelligent visual analysis systems, this technology can effectively convert image or video content into structured scene graph representations. For example, for street scene images, multi-scale semantic features are first extracted using a deep convolutional network, and then a scene graph is constructed using the steps described above. After semantic clustering, semantic clusters such as vehicles, pedestrians, and buildings are formed; after saliency evaluation, key nodes such as cars, buses, and pedestrians are retained; through semantic correlation calculation, relationships such as pedestrian-sidewalk and vehicle-road are established; finally, a complete scene graph structure is formed, providing a foundation for downstream scene understanding and decision-making.

[0108] The advantage of this method is that it can adaptively capture the semantic structure in the scene without relying on predefined object categories or relation templates, making the scene representation more flexible and accurate, and is particularly suitable for complex and ever-changing real-world application scenarios.

[0109] In one optional implementation, based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector, including:

[0110] The preset task constraints are semantically parsed to extract key constraint elements, and these key constraint elements are mapped to constraint feature codes to obtain task constraint feature representations.

[0111] Cross-attention calculation is performed on the context-enhanced feature representation based on the task-constrained feature representation. An attention weight distribution is generated by calculating the semantic matching degree between each feature component in the context-enhanced feature representation and the task-constrained feature representation. The contribution of each feature component in the feature fusion process is determined based on the attention weight distribution.

[0112] Based on the attention weight distribution, each feature component in the context-enhanced feature representation is weighted and aggregated. The contribution is used as a weighting coefficient to sum the weighted features of each feature component in the context-enhanced feature representation, resulting in a weighted aggregated feature.

[0113] The weighted aggregated features are subjected to a nonlinear transformation and projected onto the target feature space to generate the fused feature vector.

[0114] In practical applications, the process of generating a fused feature vector by weighted aggregation of context-enhanced feature representations through attention-guided feature fusion first requires semantic parsing of the pre-defined task constraints. Specifically, natural language processing techniques can be used to perform lexical and syntactic analysis on the task constraint text to identify semantic elements such as entities, relations, and attributes. For example, for the constraint "describe the vehicles and their colors in the image" in an image description task, semantic parsing can extract the key constraint elements as "vehicles" and "colors." After extraction, these key constraint elements are mapped into vector representations through an embedding layer, and further transformed using a multilayer perceptron or Transformer encoder to generate a task constraint feature representation of dimension d. , where k is the number of key constraint elements.

[0115] Next, cross-attention is calculated on the context-enhanced feature representation based on the task-constrained feature representation. Assuming the context-enhanced feature representation is... , where n is the length of the feature sequence. The attention matrix can be obtained by calculating the semantic correlation between F and C. Specifically, for each feature vector f_i in F and each constraint feature vector c_j in C, the semantic matching degree between them is calculated, which can be achieved using mechanisms such as dot product, additive attention, or multi-head attention. Taking dot product attention as an example, the dot product of f_i and c_j is calculated and then normalized using softmax to obtain the attention weight a_ij of feature f_i on constraint c_j. The final distribution of attention weights reflects the degree of correlation between each feature component and the task constraints, i.e., its contribution in the feature fusion process.

[0116] Then, based on the attention weight distribution, the feature components in the context-enhanced feature representation are weighted and aggregated. Using the attention weight distribution A as the weighting coefficient, the components in the feature representation F are summed in a weighted manner to obtain the weighted aggregated feature. In practice, this process can be accomplished through matrix multiplication A×C, which gives higher weights to feature components that are more relevant to the task constraints, thus giving them a more important position in the fused features.

[0117] Finally, a nonlinear transformation is performed on the weighted aggregated features. The weighted aggregated features Z are then projected onto the target feature space through a nonlinear transformation module such as a feedforward neural network to generate the final fused feature vector. , where m is the target feature dimension. This nonlinear transformation can be implemented as follows: first, the feature dimension is expanded through a linear layer; then, an activation function (such as ReLU, GELU, etc.) is applied to introduce nonlinearity; finally, another linear layer maps the dimension back to the desired target dimension. This step not only enhances the model's expressive power but also helps adapt to the needs of downstream tasks.

[0118] In practical applications, such as intelligent question-answering systems, user queries can be used as task constraints, while knowledge fragments in the document library serve as contextual information. Using this method, the system can automatically focus on relevant knowledge points and generate targeted answer features based on the specific requirements of the user query. For example, when a user queries "Who is the founder of XX company?", "founder" will be used as a key constraint element. Attention weighting will be applied to text fragments containing information related to XX company, focusing on content related to the founder, thereby generating accurate answer features.

[0119] This attention-guided feature fusion approach effectively integrates task constraints with contextual information, resulting in fused feature vectors that retain rich contextual information while highlighting key task-related content, thus providing high-quality feature representations for downstream tasks.

[0120] In one optional implementation, the fused feature vector is subjected to multi-level semantic reasoning, and the weights of different semantic dimensions in the fused feature vector are iteratively adjusted through dynamic feature recalibration to obtain a task-oriented optimized feature representation, including:

[0121] The fused feature vector is decomposed into semantic dimensions, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace. Multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain multi-path reasoning results.

[0122] The semantic consistency of the multi-path inference results is evaluated. The reliability score of each inference path is determined by calculating the degree of semantic consistency between the output results of each inference path, and an initial weight vector is generated based on the reliability score.

[0123] Based on the initial weight vector, the different semantic dimensions in the fused feature vector are recalibrated to generate a recalibrated feature vector.

[0124] The recalibrated feature vector is used in residual feedback with the multi-path inference result to update the initial weight vector, thereby obtaining the target weight vector;

[0125] Based on the target weight vector, a semantic dimension reorganization mapping is constructed. The semantic dimension reorganization mapping is used to selectively enhance and suppress the feature components of each semantic dimension in the fused feature vector to generate the task-oriented optimized feature representation.

[0126] Figure 2 This is a schematic diagram illustrating the process of generating task-oriented optimized feature representations according to an embodiment of the present invention. For example... Figure 2 As shown, firstly, the fused feature vector is decomposed semantically. Specifically, a feature decomposition algorithm is used to decompose the fused feature vector into multiple semantic subspaces. Taking an image-text multimodal scenario as an example, the fused feature vector F can be decomposed into a visual semantic subspace Fv, a textual semantic subspace Ft, and an interaction semantic subspace Fi. The decomposition process employs an attention mechanism, calculating the projection of the feature vector onto different semantic dimensions to obtain the representation of each subspace. Each subspace reflects specific semantic information in the fused features, facilitating subsequent targeted reasoning.

[0127] For each semantic subspace, a reasoning path is determined. In the visual semantic subspace, a reasoning path Pv based on visual feature extraction is designed, which includes a multi-layer visual feature extraction module. In the textual semantic subspace, a reasoning path Pt based on text understanding is designed, which includes a text feature parsing module. In the interactive semantic subspace, a cross-attention reasoning path Pi is designed to capture inter-modal relationships. Each reasoning path processes information in its corresponding semantic subspace independently and in parallel, improving processing efficiency.

[0128] Within each inference path, a hierarchical processing strategy is employed for multi-level semantic reasoning. Taking the text semantic subspace inference path as an example, basic semantic information is first extracted through a shallow semantic network, then relationships between concepts are captured through a mid-level semantic association network, and finally, a high-order semantic representation is formed through a deep semantic integration network. Residual connections maintain information flow between levels, ensuring effective transmission of semantic information across different levels. Ultimately, each inference path outputs corresponding inference results Rv, Rt, and Ri, forming a multi-path inference result set R.

[0129] A semantic consistency evaluation is performed on the multi-path inference results to assess the reliability of the output results of each inference path. Specifically, the semantic similarity between the output results of any two inference paths is calculated, and the cosine similarity method is used to measure the degree of consistency between semantic vectors. For inference path i, its reliability score Si is obtained by calculating the average semantic similarity between the output result of this path and the output results of all other paths. Based on the reliability scores of all inference paths, an initial weight vector W0 is generated, with the weight value proportional to the reliability score of the corresponding path.

[0130] Based on the initial weight vector, different semantic dimensions in the fused feature vector are recalibrated. The recalibration process employs a weight modulation mechanism, which combines the initial weight vector W0 with the original fused feature vector F in a weighted manner to generate a recalibrated feature vector Fr. Specifically, for each semantic dimension di in the fused feature vector F, the corresponding weight value w0i is applied for modulation to enhance the representation of high-reliability semantic dimensions and suppress the influence of low-reliability semantic dimensions.

[0131] The recalibrated feature vector and the multi-path inference results are used for residual feedback to update the initial weight vector. The residual feedback mechanism evaluates the rationality of the current weight allocation by calculating the difference between the recalibrated feature vector Fr and the output result Ri of each inference path. Based on the magnitude of the residual value, the initial weight vector W0 is iteratively adjusted to generate the updated weight vector W1. The iterative process can be repeated multiple times until the weight vector converges or the preset number of iterations is reached, finally obtaining the target weight vector Wt.

[0132] A semantic dimension reorganization mapping is constructed based on the target weight vector. This mapping function M generates a semantic dimension modulation matrix by performing a nonlinear transformation on the target weight vector Wt. The mapping process considers the dependencies between different semantic dimensions, and the nonlinear transformation captures complex semantic interaction patterns. Applying the semantic dimension reorganization mapping, the feature components of each semantic dimension in the fused feature vector F are selectively enhanced and suppressed to generate a task-oriented optimized feature representation Fo.

[0133] In practical applications, such as cross-modal retrieval tasks, the optimized feature representation (Fo) exhibits better modality alignment. Compared to the original fused features, the optimized feature representation effectively filters out noise and redundant information while preserving key semantic information. Furthermore, by dynamically adjusting the weights of different semantic dimensions, the feature representation becomes more focused on the current task objective, improving its task adaptability.

[0134] For text-image matching scenarios that require capturing subtle semantic differences, this method delves into semantic relationships through multi-level semantic reasoning and strengthens task-related dimensions through dynamic feature recalibration, significantly improving matching accuracy. Experiments show that the optimized feature representation generated by this method improves matching accuracy by approximately 15% compared to baseline methods, validating the effectiveness of this method in multimodal semantic understanding tasks.

[0135] In one optional implementation, the fused feature vector is semantically decomposed, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace, and multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain a multi-path reasoning result, including:

[0136] Semantic attribute analysis is performed on the fused feature vector to determine the feature components with different semantic attributes in the fused feature vector;

[0137] Based on the semantic attribute type of the feature components, the fused feature vector is decomposed into multiple semantic subspaces along the semantic dimension, with each semantic subspace corresponding to a semantic attribute type;

[0138] The semantic dependencies and logical transitive relationships between feature components in each semantic subspace are analyzed to determine the network topology of the semantic reasoning network.

[0139] Based on the network topology of the semantic reasoning network, a reasoning path is determined for each semantic subspace, and each reasoning path corresponds to a semantic propagation link from input features to reasoning output;

[0140] Parallel reasoning is performed on each reasoning path. Semantic propagation and logical reasoning are carried out through the feature components in the semantic subspace corresponding to each reasoning path to obtain the reasoning output of each reasoning path.

[0141] Semantic fusion is performed on the inference outputs of each inference path, and the inference outputs of each inference path are aggregated at the semantic level to generate the multi-path inference result.

[0142] Semantic dimension decomposition of the fused feature vector first requires semantic attribute analysis. A specific semantic attribute extraction module identifies and extracts semantic features from the fused feature vector. In practice, an attention mechanism combined with feature channel decomposition can be used to evaluate the importance of different dimensions of the fused feature vector. By calculating the similarity between each dimension's features and a predefined semantic prototype, the semantic attribute type of each dimension is determined. For example, for a feature vector fused from text and an image, multiple feature components can be identified, such as geometric attributes representing object shape, visual attributes representing object color, and conceptual attributes representing text semantics.

[0143] Based on the semantic attribute analysis results of the aforementioned feature components, the fused feature vector is divided into multiple semantic subspaces along the semantic dimension. Specifically, a feature selection matrix is ​​used to project the original fused feature vector, with each feature selection matrix corresponding to a semantic attribute type. If the fused feature vector is an N-dimensional vector, after semantic dimension decomposition, K semantic subspaces can be obtained. Each subspace has a different dimension, depending on the richness of the semantic attribute in the original features. For example, the fused feature vector can be decomposed into multiple semantic subspaces, such as a relational subspace representing entity relationships, a classification subspace representing concept hierarchy, and a sequence subspace representing temporal order.

[0144] When analyzing the semantic dependencies and logical transitive relationships between feature components in each semantic subspace, a graph structure learning method is used to construct a semantic reasoning network. First, the correlation matrix between feature components within the same semantic subspace is calculated, and the connection relationships between feature components are determined based on a correlation threshold. Then, a causal discovery algorithm is used to analyze the causal directions between feature components, transforming correlations into directed connections to form a directed semantic reasoning graph. For complex semantic relationships, a path search algorithm can also be used to discover transitive relationships of indirect connections, enriching the topology of the semantic reasoning network.

[0145] Based on the semantic reasoning network constructed above, reasoning paths are determined for each semantic subspace. The determination of these paths employs an optimal path search strategy, starting from the input feature nodes and using either depth-first search or breadth-first search algorithms to find effective paths to the reasoning target nodes. During the search process, the consistency and completeness of semantic transmission along the paths are considered, prioritizing paths with clear semantic flow and strong logical relationships as reasoning links. For example, in the relation subspace, relational reasoning paths based on entity associations are determined; in the classification subspace, category reasoning paths based on concept hierarchy are determined.

[0146] When performing parallel reasoning processing on each reasoning path, a distributed computing architecture is adopted, allocating the computational tasks of different reasoning paths to different processing units. Each processing unit corresponding to a reasoning path executes a specific semantic propagation algorithm, selecting a suitable reasoning model based on the characteristics of that semantic subspace. For example, graph convolutional networks can be used for information propagation in the relational subspace; hierarchical attention mechanisms can be used for concept reasoning in the classification subspace; and recurrent neural networks can be used for temporal reasoning in the sequence subspace. Through semantic propagation of each reasoning path, reasoning results for different semantic aspects are obtained.

[0147] Finally, semantic fusion is performed on the inference outputs of each inference path, integrating the multi-dimensional semantic understanding results into a unified multi-path inference result. A dynamic weight allocation mechanism is employed during the fusion process, dynamically adjusting the fusion weights of different path results based on the semantic requirements of the current task and the credibility scores of each inference path. For complementary semantic understanding, an information-enhanced fusion strategy is used; for conflict-related semantic understanding, a conflict resolution mechanism is employed for negotiated fusion. Through this multi-level semantic fusion, inference results that comprehensively consider multiple semantic attributes are generated, achieving a more comprehensive and accurate semantic analysis of the fused feature vector.

[0148] In practical applications, taking intelligent medical diagnosis as an example, the fused feature vector includes the patient's imaging features, clinical indicator features, and medical history text features. Through semantic dimension decomposition, it can be divided into a morphological subspace representing anatomical structure, an indicator subspace representing physiological function, and a temporal subspace representing the development of medical history. In the parallel reasoning stage, the morphological subspace infers the lesion location through structural recognition, the indicator subspace infers physiological abnormalities through parameter analysis, and the temporal subspace infers the disease development trend through evolution analysis. Finally, the results of the three reasoning paths are fused to generate a diagnostic suggestion that comprehensively considers morphology, function, and developmental history, providing doctors with multi-dimensional decision support.

[0149] The multimodal large model-driven complex scene image data processing system of this invention includes:

[0150] The first unit is used to acquire the target image data to be processed, perform cross-modal feature extraction and feature parsing on the target image data, and obtain multi-scale semantic feature representation;

[0151] The second unit is used to determine a scene context dependency graph based on the multi-scale semantic feature representation, and to explicitly encode the spatial associations and semantic dependencies in the multi-scale semantic feature representation based on the scene context dependency graph to obtain a context-enhanced feature representation;

[0152] The third unit is used to perform weighted aggregation of the context-enhanced feature representation based on the context-enhanced feature representation and preset task constraints, through attention-guided feature fusion, to generate a fused feature vector.

[0153] The fourth unit is used to perform multi-level semantic reasoning on the fused feature vector, and to iteratively adjust the weights of different semantic dimensions in the fused feature vector through dynamic feature recalibration to obtain a task-oriented optimized feature representation;

[0154] The fifth unit is used to perform target processing operations based on the task-oriented optimization feature representation to obtain processing results, and to adaptively update the scene context dependency graph using the quality evaluation feedback information of the processing results.

[0155] A third aspect of the present invention provides an electronic device, comprising:

[0156] processor;

[0157] Memory used to store processor-executable instructions;

[0158] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0159] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0160] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing complex scene image data driven by a multimodal large model, characterized in that, include: The target image data to be processed is acquired, and cross-modal feature extraction and feature parsing are performed on the target image data to obtain multi-scale semantic feature representation; A scene context dependency graph is determined based on the multi-scale semantic feature representation. The spatial associations and semantic dependencies in the multi-scale semantic feature representation are then explicitly encoded based on the scene context dependency graph to obtain a context-enhanced feature representation. Based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector. The fused feature vector is subjected to multi-level semantic reasoning, and the weights of different semantic dimensions in the fused feature vector are iteratively adjusted through dynamic feature recalibration to obtain a task-oriented optimized feature representation; Based on the task-oriented optimization feature representation, the target processing operation is performed to obtain the processing result, and the quality evaluation feedback information of the processing result is used to adaptively update the scene context dependency graph; The process of acquiring the target image data to be processed, performing cross-modal feature extraction and feature parsing on the target image data to obtain a multi-scale semantic feature representation includes: The target image data is subjected to multimodal information separation, which decomposes the target image data into visual spatial information components and semantic content information components; The visual spatial information component and the semantic content information component are respectively feature-encoded, and features are extracted at different spatial resolution levels through cross-scale feature sampling to obtain visual spatial feature sequences and semantic content feature sequences. The cross-modal semantic correspondence between the visual spatial feature sequence and the semantic content feature sequence is determined. The spatial location information in the visual spatial feature sequence and the semantic category information in the semantic content feature sequence are associated and aligned through bidirectional semantic mapping to obtain cross-modal alignment features. Based on the cross-modal semantic correspondence, the cross-modal alignment features are fused at multiple scales, and the cross-modal alignment features at different spatial resolution levels are aggregated according to the scale hierarchy to generate the multi-scale semantic feature representation. The process involves determining a scene context dependency graph based on the multi-scale semantic feature representation, and explicitly encoding the spatial relationships and semantic dependencies in the multi-scale semantic feature representation based on the scene context dependency graph to obtain a context-enhanced feature representation, including: A set of semantic nodes is extracted from the multi-scale semantic feature representation. The semantic association strength between each semantic node in the set is determined by semantic similarity calculation. Based on the semantic association strength, the connection relationship between the semantic nodes is established to obtain the initial scene graph structure. Spatial position analysis is performed on the set of semantic nodes in the initial scene graph structure, and the spatial association strength between each semantic node is determined by spatial proximity calculation. The spatial association strength is added as an edge weight to the connection relationship of the initial scene graph structure to obtain the scene context dependency graph; Based on the connection relationships in the scene context dependency graph, graph convolutional coding is performed on the multi-scale semantic feature representation. The semantic association strength and spatial association strength of each semantic node in the scene context dependency graph are used as propagation weights for feature aggregation to obtain graph-coded features. The spatial association and semantic dependency encoded in the graph coding features are added to the multi-scale semantic feature representation through residual connection to generate the context-enhanced feature representation; The step of performing multi-level semantic reasoning on the fused feature vector and iteratively adjusting the weights of different semantic dimensions in the fused feature vector through dynamic feature recalibration to obtain a task-oriented optimized feature representation includes: The fused feature vector is decomposed into semantic dimensions, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace. Multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain multi-path reasoning results. The semantic consistency of the multi-path inference results is evaluated. The reliability score of each inference path is determined by calculating the degree of semantic consistency between the output results of each inference path, and an initial weight vector is generated based on the reliability score. Based on the initial weight vector, the different semantic dimensions in the fused feature vector are recalibrated to generate a recalibrated feature vector. The recalibrated feature vector is used in residual feedback with the multi-path inference result to update the initial weight vector, thereby obtaining the target weight vector; Based on the target weight vector, a semantic dimension reorganization mapping is constructed. The semantic dimension reorganization mapping is used to selectively enhance and suppress the feature components of each semantic dimension in the fused feature vector to generate the task-oriented optimized feature representation.

2. The method according to claim 1, characterized in that, A set of semantic nodes is extracted from the multi-scale semantic feature representation. The semantic association strength between semantic nodes in the set is determined by semantic similarity calculation. Based on the semantic association strength, the connection relationship between semantic nodes is established to obtain the initial scene graph structure, including: Semantic clustering analysis is performed on the multi-scale semantic feature representations. Feature vectors with common semantic attributes in the multi-scale semantic feature representations are grouped into semantic clusters, and the central feature vector of each semantic cluster is used as a candidate semantic node to obtain a set of candidate semantic nodes. The semantic saliency of the candidate semantic node set is evaluated to determine the saliency score of each candidate semantic node, and the candidate semantic node set is filtered based on the saliency score to obtain the semantic node set; For each semantic node in the semantic node set, the pairwise semantic association degree is calculated. The correspondence between each semantic node in the semantic subspace is determined by feature space vector projection. The semantic association strength is determined based on the consistency of the projection distance and direction of the correspondence. Based on the semantic association strength, a connection threshold is set, and directed edges are established between semantic node pairs in the semantic node set whose semantic association strength exceeds the connection threshold. The semantic association strength is used as the edge weight of the directed edge connection to obtain the initial scene graph structure.

3. The method according to claim 1, characterized in that, Based on the context-enhanced feature representation and preset task constraints, the context-enhanced feature representation is weighted and aggregated through attention-guided feature fusion to generate a fused feature vector, including: The preset task constraints are semantically parsed to extract key constraint elements, and these key constraint elements are mapped to constraint feature codes to obtain task constraint feature representations. Cross-attention calculation is performed on the context-enhanced feature representation based on the task-constrained feature representation. An attention weight distribution is generated by calculating the semantic matching degree between each feature component in the context-enhanced feature representation and the task-constrained feature representation. The contribution of each feature component in the feature fusion process is determined based on the attention weight distribution. Based on the attention weight distribution, each feature component in the context-enhanced feature representation is weighted and aggregated. The contribution is used as a weighting coefficient to sum the weighted features of each feature component in the context-enhanced feature representation, resulting in a weighted aggregated feature. The weighted aggregated features are subjected to a nonlinear transformation and projected onto the target feature space to generate the fused feature vector.

4. The method according to claim 1, characterized in that, The fused feature vector is semantically decomposed, dividing it into multiple semantic subspaces along the semantic dimension. A reasoning path is determined for each semantic subspace, and multi-level semantic reasoning is performed on the fused feature vector through parallel processing of multiple reasoning paths to obtain a multi-path reasoning result, including: Semantic attribute analysis is performed on the fused feature vector to determine the feature components with different semantic attributes in the fused feature vector; Based on the semantic attribute type of the feature components, the fused feature vector is decomposed into multiple semantic subspaces along the semantic dimension, with each semantic subspace corresponding to a semantic attribute type; The semantic dependencies and logical transitive relationships between feature components in each semantic subspace are analyzed to determine the network topology of the semantic reasoning network. Based on the network topology of the semantic reasoning network, a reasoning path is determined for each semantic subspace, and each reasoning path corresponds to a semantic propagation link from input features to reasoning output; Parallel reasoning is performed on each reasoning path. Semantic propagation and logical reasoning are carried out through the feature components in the semantic subspace corresponding to each reasoning path to obtain the reasoning output of each reasoning path. Semantic fusion is performed on the inference outputs of each inference path, and the inference outputs of each inference path are aggregated at the semantic level to generate the multi-path inference result.

5. A complex scene image data processing system driven by a multimodal large model, used to implement the method as described in any one of claims 1-4, characterized in that, include: The first unit is used to acquire the target image data to be processed, perform cross-modal feature extraction and feature parsing on the target image data, and obtain multi-scale semantic feature representation; The second unit is used to determine a scene context dependency graph based on the multi-scale semantic feature representation, and to explicitly encode the spatial associations and semantic dependencies in the multi-scale semantic feature representation based on the scene context dependency graph to obtain a context-enhanced feature representation; The third unit is used to perform weighted aggregation of the context-enhanced feature representation based on the context-enhanced feature representation and preset task constraints, through attention-guided feature fusion, to generate a fused feature vector. The fourth unit is used to perform multi-level semantic reasoning on the fused feature vector, and to iteratively adjust the weights of different semantic dimensions in the fused feature vector through dynamic feature recalibration to obtain a task-oriented optimized feature representation; The fifth unit is used to perform target processing operations based on the task-oriented optimization feature representation to obtain processing results, and to adaptively update the scene context dependency graph using the quality evaluation feedback information of the processing results.

6. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

Images