Multi-source information collaborative fusion perception method and device based on cognitive agent driving

Abstract

The application relates to a multi-source information collaborative fusion perception method and device based on a cognitive agent driving, which comprises the following steps: performing cross-modal feature alignment and semantic modeling on multi-source heterogeneous input data to generate multi-modal features and global semantic embedding; inputting the multi-modal features and task instructions into a scheduling agent to perform cross-modal cognitive analysis, identify a scene state, and perform reliability evaluation, so as to generate a cognitive state vector based on the scene state and the evaluation result, and simultaneously perform semantic analysis on the task instructions to obtain a task target; establishing an association relationship among the multi-modal features based on the scene state and the global semantic embedding, generating a fusion control strategy based on the corrected multi-modal features, the global semantic embedding, the task target and the cognitive state vector, then correspondingly calling expert agents to perform fusion processing on the multi-modal features to obtain multi-modal fusion features, and performing cognitive guidance modulation on the multi-modal fusion features by the task target to generate a visual representation result.

Description

Technical Field

[0001] This application relates to the field of multi-source information fusion and intelligent agent perception, and in particular to a multi-source information collaborative fusion perception method and device driven by cognitive intelligent agents. Background Technology

[0002] Multimodal sensing data fusion refers to the joint modeling and collaborative processing of data from different sensors or information sources to obtain more complete, stable, and accurate environmental perception results than single-modal data. Common multimodal data include text descriptions, visible light RGB (Red, Green, Blue) images, infrared images, event camera data, multispectral images, and LiDAR point cloud data. Each modality differs in imaging mechanism, information representation, and applicable scenarios: visible light images possess rich texture, color, and semantic information, but are prone to degradation under low light, strong backlight, and complex weather conditions; infrared images can reflect the thermal radiation distribution of targets, but are insufficient in fine-grained semantic representation; event camera data is suitable for high-speed motion scenarios, but its data is sparse and lacks complete appearance information; multispectral images can reflect material properties and spectral response differences, but their spatial resolution and real-time processing capabilities are limited; LiDAR point cloud data can provide geometric structure and distance information, but is limited in semantic categories and texture representation; text information can describe task requirements and prior knowledge, but cannot replace actual sensing observations. Therefore, the key to multimodal fusion lies in establishing effective collaborative relationships between different modalities to achieve information complementarity and unified expression.

[0003] In related technologies, multimodal fusion methods mainly include data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion typically involves concatenating or mapping multi-source data during the input stage. Feature-level fusion achieves joint encoding of multimodal features through convolutional neural networks, Transformer structures, or cross-modal attention mechanisms. Decision-level fusion obtains the prediction results of each modality separately and obtains the final output through weighting or post-processing.

[0004] However, the aforementioned methods are essentially non-agent fusion paradigms based on fixed structures. Their fusion process relies on preset network structures or static parameter configurations, making it difficult to adaptively adjust to changes in the actual environment. Specifically, on the one hand, existing methods lack the ability to dynamically perceive and schedule modal states, and cannot adjust the order of modal participation and weight allocation based on factors such as lighting conditions, occlusion levels, noise distribution, and sensor effectiveness. This easily leads to low-quality modal information interfering with the fusion results. On the other hand, existing methods lack explicit cross-modal cognitive reasoning and conflict resolution mechanisms. For spatiotemporal misalignments, resolution differences, and semantic conflicts between modalities, they usually rely on implicit learning methods, making it difficult to achieve interpretability of the fusion process and traceability of the results. These issues urgently need to be addressed. Summary of the Invention

[0005] This application provides a multi-source information collaborative fusion perception method and device driven by cognitive intelligent agents to solve the problem of dynamic decision-making and collaborative scheduling of multimodal information in complex scenarios.

[0006] The first aspect of this application provides a multi-source information collaborative fusion perception method driven by a cognitive intelligent agent, including the following steps: Acquire multi-source heterogeneous input data and task instructions, and perform cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings; The multimodal features and task instructions are input into the scheduling agent, which performs cross-modal cognitive parsing to identify the scene state and conducts multimodal data reliability assessment based on the scene state. A cognitive state vector is generated based on the scene state and the multimodal reliability assessment results. At the same time, the task instructions are semantically parsed to obtain the task objective. Based on the scheduling agent, the association between multimodal features is established according to the scene state and the global semantic embedding inference, and the multimodal features are collaboratively corrected to generate a fusion control strategy based on the corrected multimodal features, the global semantic embedding, the task objective and the cognitive state vector. Based on the fusion control strategy, an expert agent is invoked, and the expert agent performs fusion processing on the multimodal features according to the fusion control strategy to obtain multimodal fusion features. Based on the task objective, the multimodal fusion features are cognitively guided and modulated to generate visual representation results.

[0007] According to one embodiment of this application, the step of having the scheduling agent perform cross-modal cognitive parsing to identify scene states includes: Based on the scheduling agent, a semantic association between the multimodal features and the task instructions is established through a cross-modal attention mechanism; Based on the semantic association, the multimodal features are analyzed for scene cognition, and the scene state is autonomously identified.

[0008] According to one embodiment of this application, the step of invoking an expert agent based on the fusion control strategy and using the expert agent to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fused features includes: Based on the fusion weights in the fusion control strategy, the target modality set and the corresponding expert agent invocation strategy are determined; Based on the fusion weights, computing resources are allocated to the first target weight modal data and the corresponding expert agent is invoked, while suppression processing is performed on the second target weight modal data; The multimodal features are fused using the expert agent according to the target order and fusion weights corresponding to the fusion control strategy to obtain the multimodal fused features.

[0009] According to one embodiment of this application, acquiring multi-source heterogeneous input data includes: Acquire multi-source heterogeneous sensor data; The format of the multi-source heterogeneous sensing data is parsed, and time stamps, spatial stamps, and modality type stamps are added to the parsed multi-source heterogeneous sensing data to obtain the multi-source heterogeneous input data.

[0010] According to one embodiment of this application, the step of performing cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings includes: The multi-source heterogeneous input data is mapped to a unified representation space, and a semantic correspondence between the multi-source heterogeneous input data is established in the unified representation space through a cross-modal attention mechanism to obtain multimodal features; Global contextual information is extracted based on the multimodal features, and global semantic embedding is generated based on the global contextual information and semantic aggregation.

[0011] According to one embodiment of this application, the step of cognitively guiding and modulating the multimodal fusion features based on the task objective to generate a visual representation result includes: Based on the task objective, the modulation parameters of the multimodal fusion feature are determined, and the multimodal fusion feature is semantically modulated using the modulation parameters. The modulated multimodal fusion features are reconstructed at multiple scales and integrated with contextual information. The integrated multimodal fusion features are then mapped to a unified representation space to generate the visual representation result corresponding to the task objective.

[0012] According to the multi-source information collaborative fusion perception method based on cognitive agent-driven embodiments of this application, cross-modal feature alignment and semantic modeling are performed on multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings. The multi-modal features and task instructions are input to a scheduling agent to perform cross-modal cognitive parsing, identify scene states, and perform reliability assessments. A cognitive state vector is generated based on the scene states and assessment results. Simultaneously, semantic parsing is performed on the task instructions to obtain the task objective. The correlation between multimodal features is established based on the scene states and global semantic embeddings. A fusion control strategy is generated based on the corrected multi-modal features, global semantic embeddings, task objective, and cognitive state vector. Then, an expert agent is invoked to perform fusion processing on the multi-modal features to obtain multi-modal fused features. The task objective cognitively guides and modulates the multi-modal fused features to generate visual representation results. This solves the problem of dynamic decision-making and collaborative scheduling of multi-modal information in complex scenarios. By introducing a multi-modal large model and a cognitive-driven agent mechanism, the multi-source heterogeneous fusion process is transformed from traditional static feature fusion into a dynamic scheduling and collaborative decision-making process, achieving unified semantic modeling and adaptive fusion of multi-modal information.

[0013] A second aspect of this application provides a multi-source information collaborative fusion sensing device driven by a cognitive intelligent agent, comprising: The acquisition module is used to acquire multi-source heterogeneous input data and task instructions, and to perform cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings. The evaluation module is used to input the multimodal features and task instructions into the scheduling agent, and the scheduling agent performs cross-modal cognitive parsing to identify the scene state and perform multimodal data reliability evaluation based on the scene state, so as to generate a cognitive state vector based on the scene state and the multimodal reliability evaluation results, and simultaneously perform semantic parsing on the task instructions to obtain the task objective; The first generation module is used to establish the association relationship between multimodalities based on the scheduling agent, the scene state, and the global semantic embedding inference, and to perform collaborative correction on the multimodal features, so as to generate a fusion control strategy based on the corrected multimodal features, the global semantic embedding, the task objective, and the cognitive state vector. The second generation module is used to invoke an expert agent based on the fusion control strategy, and use the expert agent to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fusion features. Based on the task objective, the multimodal fusion features are cognitively guided and modulated to generate visual representation results.

[0014] According to one embodiment of this application, the evaluation module includes: The construction unit is used to establish a semantic association between the multimodal features and the task instructions based on the scheduling agent through a cross-modal attention mechanism; The first parsing unit is used to perform scene recognition parsing on the multimodal features based on the semantic association, and autonomously identify the scene state.

[0015] According to one embodiment of this application, the second generation module includes: The determining unit is used to determine the target modality set and the corresponding expert agent invocation strategy based on the fusion weights in the fusion control strategy. The calling unit is used to allocate computing resources to the first target weight modal data and call the corresponding expert agent based on the fusion weight, and to perform suppression processing on the second target weight modal data. The feature fusion unit is used to perform fusion processing on the multimodal features according to the target order and fusion weights corresponding to the fusion control strategy by the expert agent to obtain the multimodal fused features.

[0016] According to one embodiment of this application, the acquisition module includes: The first acquisition unit is used to acquire multi-source heterogeneous sensing data; The second parsing unit is used to parse the format of the multi-source heterogeneous sensing data and add time markers, spatial markers and modality type markers to the parsed multi-source heterogeneous sensing data to obtain the multi-source heterogeneous input data.

[0017] According to one embodiment of this application, the acquisition module includes: The second acquisition unit is used to map the multi-source heterogeneous input data to a unified representation space, and to establish a semantic correspondence between the multi-source heterogeneous input data in the unified representation space through a cross-modal attention mechanism to obtain multimodal features; The first generation unit is used to extract global context information based on the multimodal features, and generate the global semantic embedding based on the global context information and semantic aggregation.

[0018] According to one embodiment of this application, the second generation module includes: A modulation unit is configured to determine the modulation parameters of the multimodal fusion feature based on the task objective, and to perform semantic modulation on the multimodal fusion feature using the modulation parameters; The second generation unit is used to perform multi-scale feature reconstruction and context information integration on the modulated multimodal fusion features, and to map the integrated multimodal fusion features to a unified representation space to generate the visual representation result corresponding to the task objective.

[0019] According to the embodiments of this application, a multi-source information collaborative fusion perception device driven by a cognitive agent performs cross-modal feature alignment and semantic modeling on multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings. The multi-modal features and task instructions are input to a scheduling agent to perform cross-modal cognitive parsing, identify the scene state, and perform reliability assessment. A cognitive state vector is generated based on the scene state and assessment results. Simultaneously, the task instructions are semantically parsed to obtain the task objective. The device establishes the correlation between multimodal features based on the scene state and global semantic embeddings, and generates a fusion control strategy based on the corrected multi-modal features, global semantic embeddings, task objective, and cognitive state vector. Then, an expert agent is invoked to perform fusion processing on the multi-modal features to obtain multi-modal fused features. The task objective cognitively guides and modulates the multi-modal fused features to generate visual representation results. This solves the problem of dynamic decision-making and collaborative scheduling of multi-modal information in complex scenarios. By introducing a large multi-modal model and a cognitive-driven agent mechanism, the multi-source heterogeneous fusion process is transformed from traditional static feature fusion into a dynamic scheduling and collaborative decision-making process, achieving unified semantic modeling and adaptive fusion of multi-modal information.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the multi-source information collaborative fusion perception method driven by cognitive intelligent agents as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multi-source information collaborative fusion perception method driven by a cognitive agent as described in the above embodiments.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This application provides a multi-source information collaborative fusion perception method based on cognitive intelligent agents, according to an embodiment of the present application. Figure 2 This is a flowchart of a multi-source heterogeneous fusion sensing method based on a cognitive-driven intelligent agent according to an embodiment of this application; Figure 3 This is a block diagram of a multi-source information collaborative fusion sensing device driven by a cognitive agent according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] The following describes a multi-source information collaborative fusion perception method and apparatus based on cognitive agent-driven embodiments of this application with reference to the accompanying drawings. Addressing the dynamic decision-making and collaborative scheduling problem of multimodal information in complex scenarios mentioned in the background art, this application provides a multi-source information collaborative fusion perception method based on cognitive agent-driven approaches. In this method, multi-source heterogeneous input data undergoes cross-modal feature alignment and semantic modeling to generate multimodal features and global semantic embeddings. The multimodal features and task instructions are input to a scheduling agent to perform cross-modal cognitive parsing, identify the scene state, and perform reliability assessment. A cognitive state vector is generated based on the scene state and assessment results. Simultaneously, semantic parsing of the task instructions yields the task objective. The association between multimodal features is established based on the scene state and global semantic embeddings. A fusion control strategy is generated based on the corrected multimodal features, global semantic embeddings, task objective, and cognitive state vector. Then, an expert agent is invoked to perform fusion processing on the multimodal features to obtain multimodal fusion features. The task objective cognitively guides and modulates the multimodal fusion features to generate visual representation results. This solves the problem of dynamic decision-making and collaborative scheduling of multimodal information in complex scenarios. By introducing a multimodal large model and a cognitive-driven intelligent agent mechanism, the multi-source heterogeneous fusion process is transformed from traditional static feature fusion into a dynamic scheduling and collaborative decision-making process, realizing unified semantic modeling and adaptive fusion of multimodal information.

[0026] Specifically, Figure 1 This is a flowchart illustrating a multi-source information collaborative fusion perception method driven by a cognitive agent, as provided in an embodiment of this application.

[0027] like Figure 1 As shown, the multi-source information collaborative fusion perception method driven by cognitive intelligent agents includes the following steps: In step S101, multi-source heterogeneous input data and task instructions are acquired, and cross-modal feature alignment and semantic modeling are performed on the multi-source heterogeneous input data to generate multimodal features and global semantic embedding.

[0028] According to one embodiment of this application, obtaining multi-source heterogeneous input data includes: obtaining multi-source heterogeneous sensing data; parsing the format of the multi-source heterogeneous sensing data; and adding time stamps, spatial stamps, and modality type stamps to the parsed multi-source heterogeneous sensing data to obtain multi-source heterogeneous input data.

[0029] According to one embodiment of this application, cross-modal feature alignment and semantic modeling are performed on multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings, including: mapping multi-source heterogeneous input data to a unified representation space, and establishing semantic correspondences between multi-source heterogeneous input data in the unified representation space through a cross-modal attention mechanism to obtain multi-modal features; extracting global contextual information based on multi-modal features, and generating global semantic embeddings based on global contextual information and semantic aggregation.

[0030] Specifically, in this embodiment, multi-source heterogeneous sensing data and task instructions are first acquired. The task instructions are user inputs or issued by the upper-layer task planning module to clarify the target semantic description of the current fusion task, such as "detect pedestrians ahead" or "identify lane lines". The multi-source heterogeneous sensing data includes, but is not limited to, visible light images, infrared images, event camera data, multispectral images, and LiDAR point cloud data. The above data come from different sensors or data sources and differ in imaging mechanism, data format, spatiotemporal resolution, and information expression method. Therefore, it is necessary to perform format parsing and structured preprocessing on the acquired multi-source heterogeneous sensing data.

[0031] Specifically, such as Figure 2 As shown, firstly, the sensor data for each modality is parsed separately. For example, visible light and infrared images are decoded and their sizes are normalized; LiDAR point cloud data is voxelized or rasterized; event camera data is aggregated by time windows; and multispectral images are band-selected and radiometrically corrected. Through the above format parsing, the multi-source heterogeneous sensor data is converted into a unified internal representation format. Specifically, for the i-th modality data... Its unified representation is defined as:

[0032] in, Let be the i-th modal data, t be the time stamp corresponding to the modal data, used to describe the data acquisition time or time series relationship, s be the spatial coordinates or sensor pose information, m be the modal category label, and N be the number of modes.

[0033] Secondly, three types of tags are added to the parsed multi-source heterogeneous sensing data: time tags, spatial tags, and modality type tags, forming a unified data representation structure. Among them, the time tag is the timestamp or time sequence number of data acquisition, used for time alignment and dynamic scene modeling of multi-source heterogeneous sensing data; the spatial tag is the spatial coordinates of the sensor, the extrinsic parameter matrix, or the spatial region corresponding to the data, used for cross-modal spatial alignment and coordinate transformation; the modality type tag identifies the modality category to which the data belongs, used for subsequent modality recognition and scheduling decisions. Through the above format parsing and structured preprocessing, standardized multi-source heterogeneous input data is obtained. This multi-source heterogeneous input data retains the original sensing information and adds unified spatiotemporal and modal metadata, thus providing a standardized input basis for subsequent cross-modal feature alignment and semantic modeling, reducing the modeling complexity caused by intermodal heterogeneity.

[0034] Furthermore, the standardized multi-source heterogeneous input data X and the task instructions are input together into the multimodal large model to perform cross-modal feature alignment and semantic modeling.

[0035] Specifically, firstly, using the encoder of a multimodal large model, the multi-source heterogeneous input data are mapped to a unified representation space, establishing a correspondence between visual, geometric, and semantic information in the unified representation space, thereby obtaining a unified multimodal feature representation. For example, the visual features of a "red car" in a visible light image and the geometric features of a "car point cloud cluster" in a LiDAR point cloud should have similar vector representations in this space. The mapping relationship of multi-source heterogeneous input data in the unified representation space is expressed as follows:

[0036] in, For the feature mapping function of the i-th mode, For multimodal large model MLLM mapping function, To unify multimodal features, and thus establish a semantic correspondence between different modal data through the aforementioned unified semantic embedding space, the visual, geometric and semantic information are jointly expressed.

[0037] Secondly, within the unified representation space, a cross-modal attention mechanism is used to establish semantic correspondences between different modalities. Attention weights are calculated between multi-source heterogeneous input data, enabling semantically related features to enhance each other and semantically unrelated features to suppress each other, thus ensuring consistency constraints across modal features within the shared semantic space. For example, edge information in visible light images is associated with depth information in LiDAR point clouds, and thermal radiation features in infrared images are associated with texture features in visible light images. Through cross-modal attention alignment, multimodal features that preserve local details of each modality are generated.

[0038] Finally, based on the aligned multimodal features, global contextual information and semantic aggregation are extracted to generate a global semantic embedding that compresses and expresses the overall scene semantics. The global semantic embedding represents the high-level semantic information of the overall scene, such as "city intersection on a rainy night", "densely populated area", or "structured road environment". The global semantic embedding does not retain local spatial details, but focuses on scene-level semantic abstraction.

[0039] In step S102, multimodal features and task instructions are input to the scheduling agent, which then performs cross-modal cognitive parsing to identify the scene state and conducts multimodal data reliability assessment based on the scene state. A cognitive state vector is generated based on the scene state and the multimodal reliability assessment results. At the same time, the task instructions are semantically parsed to obtain the task objective.

[0040] According to one embodiment of this application, a scheduling agent performs cross-modal cognitive parsing to identify scene states, including: establishing semantic associations between multimodal features and task instructions based on the scheduling agent through a cross-modal attention mechanism; performing scene cognitive parsing on the multimodal features based on the semantic associations to autonomously identify scene states.

[0041] Specifically, such as Figure 2 As shown, the aforementioned generated multimodal features The task instruction T and the task instruction T are input into the scheduling agent to perform cross-modal joint analysis, cognitive reasoning, and decision modeling. First, the scheduling agent establishes a semantic association between multimodal features and task instructions through a cross-modal attention mechanism. Through this mechanism, the scheduling agent can identify the modal features most relevant to the current task and their corresponding spatial regions. For example, when the task instruction is "detect pedestrians ahead", the attention mechanism will enhance the feature response of the pedestrian region in the visible light image, and at the same time enhance the feature response corresponding to the thermal radiation of the pedestrian in the infrared image.

[0042] Secondly, based on the above semantic association, the scheduling agent performs scene cognition analysis on multimodal features to obtain the current scene state description. The scene state includes, but is not limited to, environmental degradation features, target distribution information, spatial structure information, modal reliability assessment, etc., thereby providing a decision basis for subsequent modal association and policy generation.

[0043] Furthermore, based on the identified scene states, a multimodal data reliability assessment is performed. For example, based on the dynamic state information of the scene in which the agent is located, the reliability of the acquired visual, linguistic, spatiotemporal, and other multimodal data is assessed separately. The confidence and validity of each modality data in the current scene are quantified. Then, the above scene information is encoded, and the scene state encoding is deeply coupled with the multimodal reliability assessment results to generate a cognitive state vector for downstream decision-making and reasoning.

[0044] Finally, the scheduling agent simultaneously performs semantic parsing of the task instructions to extract a structured task objective C. The task objective includes, but is not limited to, task category, object of interest, output requirements, and task priority. This enables the subsequent fusion and modulation process to be task-oriented, achieving cognitive-driven fusion that "enhances what is needed." The expression for task objective C is:

[0045] in, The cognitive reasoning function of the scheduling agent enables the scheduling agent to cognitively assess the importance of different modal information based on task instructions and multimodal feature states, and to form the decision basis for subsequent cross-modal alignment, fusion scheduling strategy generation and expert agent invocation, so that the multi-source heterogeneous information fusion process has task perception and dynamic scheduling capabilities.

[0046] In step S103, based on the scheduling agent, the correlation between multimodal features is established according to the scene state and global semantic embedding reasoning, and the multimodal features are collaboratively corrected to generate a fusion control strategy based on the corrected multimodal features, global semantic embedding, task objectives and cognitive state vectors.

[0047] Specifically, based on the scene state obtained above and combined with global semantic embedding, the scheduling agent further establishes intermodal relationships and performs collaborative correction on multimodal features. By constructing a feature difference measurement relationship between modalities and mapping and transforming each modal feature based on the cognitive state, the features of each modality are uniformly mapped to the same reference semantic space, so that different modalities are consistent in spatial and semantic expression. This process can be represented as:

[0048] in, Let i be the mapping function for the i-th mode. Let be the original feature representation of the i-th mode. For the calibrated feature representation, Let be the mapping function corresponding to the i-th mode. The mapping function is conditionally modulated by the task target C to map it to a unified reference space.

[0049] Specifically, firstly, the scheduling agent identifies the features corresponding to the same target in different modal data based on the target distribution and spatial structure information in the scene state. Specifically, it establishes spatial and semantic correspondences between modalities such as visible light images, infrared images, and LiDAR point clouds through feature mapping relationships, so that the expression of the same target in different modalities remains consistent, thereby reducing the deviation caused by modal differences.

[0050] Secondly, based on the established relationships, the scheduling agent performs consistency calibration on the multimodal features, mainly including spatial alignment, semantic calibration, and anomaly handling. After collaborative calibration, the multimodal features are aligned and unified at the spatial and semantic levels, providing high-quality and consistent input for subsequent fusion processing.

[0051] Finally, based on the corrected multimodal features, global semantic embedding, task objectives, and cognitive state vectors, a fusion control strategy is generated by the scheduling agent. This strategy is the core instruction guiding subsequent fusion processing and mainly includes the modality participation order, modality weight allocation, and expert agent invocation method. The importance weight of each modality is calculated based on the cognitive state, and the set of modalities participating in the fusion and the expert agent invocation order are determined based on the weight distribution. This enables the fusion process to adaptively respond to environmental changes and task requirements, avoid interference from low-quality modalities in the fusion result, and achieve true cognitive-driven collaborative fusion. The expression for the fusion control strategy is:

[0052] in, Let be the fusion weights for the i-th mode, based on the fusion weights The set of modalities participating in the fusion and the corresponding expert agent invocation strategy are determined. Computational resources are allocated preferentially to high-weight modalities and their corresponding expert agents are invoked, while low-weight modalities are suppressed. Generate a network (or function) for the fusion control strategy, which is used to output the fusion weight set {α_i}_{i=1}^N corresponding to each mode according to the task objective C.

[0053] Therefore, through the above-mentioned fusion control strategy, the system can adaptively select and dynamically combine different modal information according to the scene state, modal reliability and task requirements, realize flexible control of the fusion path, effectively solve the problem of fixed fusion structure and inability to dynamically adjust according to environmental changes in traditional multimodal fusion methods, and enable the multi-source information fusion process to have cognitive-driven scheduling capability and task self-adaptation capability.

[0054] In step S104, an expert agent is invoked based on the fusion control strategy, and the expert agent is used to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fusion features. The multimodal fusion features are then cognitively guided and modulated based on the task objective to generate visual representation results.

[0055] According to one embodiment of this application, an expert agent is invoked based on a fusion control strategy, and the expert agent is used to perform fusion processing on multimodal features based on the fusion control strategy to obtain multimodal fusion features. This includes: determining a target modality set and a corresponding expert agent invocation strategy based on the fusion weights in the fusion control strategy; allocating computing resources to the first target weight modality data and invoking the corresponding expert agent based on the fusion weights, and performing suppression processing on the second target weight modality data; and using the expert agent to perform fusion processing on the multimodal features according to the target order and fusion weights corresponding to the fusion control strategy to obtain multimodal fusion features.

[0056] Specifically, based on the above-mentioned fusion control strategy, the system performs fusion processing on the corrected multimodal features to obtain multimodal fusion features.

[0057] Specifically, firstly, based on the fusion weights in the fusion control strategy and the calling method of the expert agents specified in the fusion control strategy, a target modality set is determined, and the corresponding expert agents are dynamically selected and activated. This application sets up specialized expert agents for different modalities or combinations of modalities. Each expert agent has specific feature extraction and fusion capabilities. The fusion control strategy selects one or more expert agents from the above target modality set for activation based on the scene state and task objectives. For example, in a nighttime pedestrian detection task, the strategy may select to activate an infrared expert agent and a LiDAR expert agent.

[0058] Secondly, the activated expert agents, following the target order and fusion weights specified by the fusion control strategy, collaboratively process and fuse the multimodal features. This involves performing feature transformations on each multimodal feature separately, and then weighting and fusing the features according to the fusion scheduling strategy to obtain the multimodal fused features. The multimodal fused features are defined as follows:

[0059] in, Let be the feature transformation function of the i-th expert agent. This process realizes weighted fusion and redundancy suppression of multimodal information. Through a selective fusion mechanism, information complementarity and redundancy suppression are achieved. Information complementarity means that features from different modalities complement each other. For example, depth information provided by LiDAR completes the missing distance information in visible light images, and thermal radiation information provided by infrared completes the semantic loss of visible light under low illumination. Redundancy suppression reduces the contribution of redundant information by assigning weights to the same information expressed by multiple modalities (such as the repeated expression of the same target in different modalities), avoiding over-enhancement. Thus, through the above collaborative processing, a unified multimodal fusion feature is generated, avoiding the negative impact of low-quality modalities on the fusion result. Each expert agent performs differentiated processing for different modal characteristics during the feature transformation process. For example, texture and semantic features are extracted for visual modalities, geometric structure modeling is performed for point cloud modalities, and dynamic edge and motion information enhancement is performed for event flow modalities, thereby realizing the complementary expression of multimodal information.

[0060] According to one embodiment of this application, cognitively guided modulation of multimodal fusion features based on task objectives is performed to generate visual representation results. This includes: determining modulation parameters of multimodal fusion features based on task objectives, and using the modulation parameters to perform semantic modulation on the multimodal fusion features; performing multi-scale feature reconstruction and contextual information integration on the modulated multimodal fusion features, and mapping the integrated multimodal fusion features to a unified representation space to generate visual representation results corresponding to the task objectives.

[0061] Specifically, firstly, based on the task objective, a cognitive modulation function is introduced into the generated multimodal fusion features for cognitively guided modulation. This involves adjusting the weights and selecting channels of the multimodal fusion features to ensure consistency between the fusion features and the task semantics, achieving alignment between feature representation and task semantics, and completing adaptive optimization of the fusion result. The expression of the cognitive modulation function is as follows:

[0062] in, These are fusion features after cognitive modulation. and These represent the scaling and bias parameters generated by the task objective, respectively, and ⊙ represents element-wise multiplication. This modulation mechanism ensures that the fused features remain consistent with the task semantics, through the target task. The consistency constraint between the guided perception results of feature representation and the task objectives further enhances the discriminative ability and task adaptability of fused features.

[0063] Specifically, firstly, in the semantic modulation and decoding reconstruction of multimodal fusion features, the modulation parameters of the multimodal fusion features are determined according to the task objective, and the modulation parameters are used to perform semantic modulation on the multimodal fusion features. The modulation process can be implemented using a channel attention mechanism or a spatial attention mechanism. After semantic modulation, the information related to the task objective in the fusion features is significantly enhanced, and irrelevant information is effectively suppressed, realizing the active alignment between feature expression and task semantics and improving the stability of the fusion result.

[0064] Secondly, the modulated multimodal fusion features are decoded and reconstructed to generate a unified visual representation result suitable for downstream tasks. This mainly includes multi-scale feature reconstruction, contextual information integration, unified representation space mapping, output adaptation and transmission. The visual representation result can be expressed as follows:

[0065] in, The decoding module, where Y represents the final visual representation result, can be used for target detection, scene understanding, and 3D reconstruction tasks. The decoding module performs multi-scale feature reconstruction and contextual information integration on the fused features, mapping the semantic, geometric, and dynamic information of the multimodal fusion to a unified expression space to obtain a visual representation with complete semantic and structural consistency. Through the above decoding and reconstruction process, the mapping from multi-source heterogeneous perceptual data to a unified visual representation is realized, enabling the fusion result to not only have cross-modal consistency but also the ability to express specific tasks, thus completing the final transformation from multimodal perception to unified cognitive representation.

[0066] Specifically, the modulated fused features undergo multi-scale feature reconstruction. Through upsampling and deconvolution operations at different scales, the spatial resolution and detail information of the features are gradually restored. Multi-scale reconstruction allows the output to retain both the large-scale global structure and the small-scale local details. Then, the reconstructed multi-scale features are integrated with contextual information. Attention mechanisms or recurrent neural networks are used to capture long-distance dependencies between features, enhancing the overall semantic coherence of the visual representation. For example, in scene understanding tasks, context integration can enable distant roads and nearby vehicles to form reasonable semantic associations. Finally, the integrated features are mapped to a unified representation space to generate visual representation results adapted to the task objective. The generated visual representation results are then output to downstream task modules, such as the target recognition module, scene understanding module, and spatial perception module, for direct use in applications such as target recognition, scene understanding, or spatial perception. The output results have a unified format and semantic meaning, requiring no additional adaptation processing from downstream modules.

[0067] Therefore, this application transforms the multi-source heterogeneous fusion process from traditional static feature fusion to dynamic scheduling and collaborative decision-making by introducing a multimodal large model and a cognitively driven intelligent agent mechanism. This achieves unified semantic modeling and adaptive fusion of multimodal information. By scheduling the intelligent agent to cognitively analyze the modal state and generate fusion strategies, the interference of low-quality modalities on the fusion results is effectively avoided, improving the stability and robustness of the fusion process, thereby enhancing the accuracy and stability of multi-source heterogeneous information fusion. Through expert intelligent agents collaboratively executing the fusion, effective complementarity of different modal information is achieved, enhancing the ability to express key features. Through cognitively guided modulation, the fusion results are kept consistent with the task semantics, thereby improving the accuracy and interpretability of unified visual representation in target recognition, scene understanding, and spatial perception tasks.

[0068] According to the multi-source information collaborative fusion perception method based on cognitive agent-driven embodiments of this application, cross-modal feature alignment and semantic modeling are performed on multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings. The multi-modal features and task instructions are input to a scheduling agent to perform cross-modal cognitive parsing, identify scene states, and perform reliability assessments. A cognitive state vector is generated based on the scene states and assessment results. Simultaneously, semantic parsing is performed on the task instructions to obtain the task objective. The correlation between multimodal features is established based on the scene states and global semantic embeddings. A fusion control strategy is generated based on the corrected multi-modal features, global semantic embeddings, task objective, and cognitive state vector. Then, an expert agent is invoked to perform fusion processing on the multi-modal features to obtain multi-modal fused features. The task objective cognitively guides and modulates the multi-modal fused features to generate visual representation results. This solves the problem of dynamic decision-making and collaborative scheduling of multi-modal information in complex scenarios. By introducing a multi-modal large model and a cognitive-driven agent mechanism, the multi-source heterogeneous fusion process is transformed from traditional static feature fusion into a dynamic scheduling and collaborative decision-making process, achieving unified semantic modeling and adaptive fusion of multi-modal information.

[0069] Next, referring to the accompanying drawings, a multi-source information collaborative fusion sensing device based on cognitive intelligent agents driven by embodiments of this application is described.

[0070] Figure 3 This is a block diagram of a multi-source information collaborative fusion sensing device driven by a cognitive intelligent agent, according to an embodiment of this application.

[0071] like Figure 3 As shown, the multi-source information collaborative fusion sensing device 10 driven by cognitive intelligent agents includes: an acquisition module 100, an evaluation module 200, a first generation module 300, and a second generation module 400.

[0072] The acquisition module 100 is used to acquire multi-source heterogeneous input data and task instructions, and to perform cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings. The evaluation module 200 is used to input multimodal features and task instructions into the scheduling agent, and the scheduling agent performs cross-modal cognitive parsing to identify the scene state and perform multimodal data reliability assessment based on the scene state. Based on the scene state and the multimodal reliability assessment results, a cognitive state vector is generated. At the same time, the task instructions are semantically parsed to obtain the task objective. The first generation module 300 is used to establish the relationship between multimodalities based on the scheduling agent, the scene state and global semantic embedding reasoning, and to perform collaborative correction on the multimodal features, so as to generate a fusion control strategy based on the corrected multimodal features, global semantic embedding, task objectives and cognitive state vectors. The second generation module 400 is used to call the expert agent according to the fusion control strategy, and use the expert agent to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fusion features. Then, based on the task objective, the multimodal fusion features are cognitively guided and modulated to generate visual representation results.

[0073] According to one embodiment of this application, the evaluation module 200 includes: The building unit is used to establish semantic associations between multimodal features and task instructions based on the scheduling agent and through a cross-modal attention mechanism; The first parsing unit is used to perform scene recognition parsing on multimodal features based on semantic association, and to autonomously identify scene states.

[0074] According to one embodiment of this application, the second generation module 400 includes: The determining unit is used to determine the target modality set and the corresponding expert agent invocation strategy based on the fusion weights in the fusion control strategy; The calling unit is used to allocate computing resources to the first target weight modal data and call the corresponding expert agent based on the fusion weight, and to perform suppression processing on the second target weight modal data; The feature fusion unit is used to fuse multimodal features by utilizing an expert agent according to the target order and fusion weights corresponding to the fusion control strategy, so as to obtain multimodal fused features.

[0075] According to one embodiment of this application, the acquisition module 100 includes: The first acquisition unit is used to acquire multi-source heterogeneous sensing data; The second parsing unit is used to parse the format of the multi-source heterogeneous sensing data and add time markers, spatial markers and modality type markers to the parsed multi-source heterogeneous sensing data to obtain multi-source heterogeneous input data.

[0076] According to one embodiment of this application, the acquisition module 100 includes: The second acquisition unit is used to map multi-source heterogeneous input data to a unified representation space, and to establish semantic correspondence between multi-source heterogeneous input data in the unified representation space through a cross-modal attention mechanism to obtain multi-modal features; The first generation unit is used to extract global context information based on multimodal features, and generate global semantic embedding based on global context information and semantic aggregation.

[0077] According to one embodiment of this application, the second generation module 400 includes: The modulation unit is used to determine the modulation parameters of the multimodal fusion features based on the task objective, and to perform semantic modulation on the multimodal fusion features using the modulation parameters; The second generation unit is used to perform multi-scale feature reconstruction and contextual information integration on the modulated multimodal fusion features, and to map the integrated multimodal fusion features to a unified representation space to generate the visual representation result corresponding to the task objective.

[0078] According to the embodiments of this application, a multi-source information collaborative fusion perception device driven by a cognitive agent performs cross-modal feature alignment and semantic modeling on multi-source heterogeneous input data to generate multi-modal features and global semantic embeddings. The multi-modal features and task instructions are input to a scheduling agent to perform cross-modal cognitive parsing, identify the scene state, and perform reliability assessment. A cognitive state vector is generated based on the scene state and assessment results. Simultaneously, the task instructions are semantically parsed to obtain the task objective. The device establishes the correlation between multimodal features based on the scene state and global semantic embeddings, and generates a fusion control strategy based on the corrected multi-modal features, global semantic embeddings, task objective, and cognitive state vector. Then, an expert agent is invoked to perform fusion processing on the multi-modal features to obtain multi-modal fused features. The task objective cognitively guides and modulates the multi-modal fused features to generate visual representation results. This solves the problem of dynamic decision-making and collaborative scheduling of multi-modal information in complex scenarios. By introducing a large multi-modal model and a cognitive-driven agent mechanism, the multi-source heterogeneous fusion process is transformed from traditional static feature fusion into a dynamic scheduling and collaborative decision-making process, achieving unified semantic modeling and adaptive fusion of multi-modal information.

[0079] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0080] When the processor 402 executes the program, it implements the multi-source information collaborative fusion perception method based on cognitive intelligent agent driven in the above embodiments.

[0081] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0082] The memory 401 is used to store computer programs that can run on the processor 402.

[0083] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0084] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0085] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0086] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0087] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described multi-source information collaborative fusion perception method driven by a cognitive agent.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0090] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A multi-source information collaborative fusion perception method driven by a cognitive intelligent agent, characterized in that, Includes the following steps: Acquire multi-source heterogeneous input data and task instructions, and perform cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings; The multimodal features and task instructions are input into the scheduling agent, which performs cross-modal cognitive parsing to identify the scene state and conducts multimodal data reliability assessment based on the scene state. A cognitive state vector is generated based on the scene state and the multimodal reliability assessment results. At the same time, the task instructions are semantically parsed to obtain the task objective. Based on the scheduling agent, the association between multimodal features is established according to the scene state and the global semantic embedding inference, and the multimodal features are collaboratively corrected to generate a fusion control strategy based on the corrected multimodal features, the global semantic embedding, the task objective and the cognitive state vector. Based on the fusion control strategy, an expert agent is invoked, and the expert agent performs fusion processing on the multimodal features according to the fusion control strategy to obtain multimodal fusion features. Based on the task objective, the multimodal fusion features are cognitively guided and modulated to generate visual representation results.

2. The method according to claim 1, characterized in that, The step of performing cross-modal cognitive parsing and identifying scene states by the scheduling agent includes: Based on the scheduling agent, a semantic association between the multimodal features and the task instructions is established through a cross-modal attention mechanism; Based on the semantic association, the multimodal features are analyzed for scene cognition, and the scene state is autonomously identified.

3. The method according to claim 1, characterized in that, The step of invoking an expert agent based on the fusion control strategy, and using the expert agent to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fused features, includes: Based on the fusion weights in the fusion control strategy, the target modality set and the corresponding expert agent invocation strategy are determined; Based on the fusion weights, computing resources are allocated to the first target weight modal data and the corresponding expert agent is invoked, while suppression processing is performed on the second target weight modal data; The multimodal features are fused using the expert agent according to the target order and fusion weights corresponding to the fusion control strategy to obtain the multimodal fused features.

4. The method according to claim 1, characterized in that, The acquisition of multi-source heterogeneous input data includes: Acquire multi-source heterogeneous sensor data; The format of the multi-source heterogeneous sensing data is parsed, and time stamps, spatial stamps, and modality type stamps are added to the parsed multi-source heterogeneous sensing data to obtain the multi-source heterogeneous input data.

5. The method according to claim 1, characterized in that, The step of performing cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings includes: The multi-source heterogeneous input data is mapped to a unified representation space, and a semantic correspondence between the multi-source heterogeneous input data is established in the unified representation space through a cross-modal attention mechanism to obtain multimodal features; Global contextual information is extracted based on the multimodal features, and global semantic embedding is generated based on the global contextual information and semantic aggregation.

6. The method according to claim 1, characterized in that, The process of cognitively guiding and modulating the multimodal fusion features based on the task objective to generate visual representation results includes: Based on the task objective, the modulation parameters of the multimodal fusion feature are determined, and the multimodal fusion feature is semantically modulated using the modulation parameters. The modulated multimodal fusion features are reconstructed at multiple scales and integrated with contextual information. The integrated multimodal fusion features are then mapped to a unified representation space to generate the visual representation result corresponding to the task objective.

7. A multi-source information collaborative fusion sensing device driven by a cognitive intelligent agent, characterized in that, include: The acquisition module is used to acquire multi-source heterogeneous input data and task instructions, and to perform cross-modal feature alignment and semantic modeling on the multi-source heterogeneous input data to generate multimodal features and global semantic embeddings. The evaluation module is used to input the multimodal features and task instructions into the scheduling agent, and the scheduling agent performs cross-modal cognitive parsing to identify the scene state and perform multimodal data reliability evaluation based on the scene state, so as to generate a cognitive state vector based on the scene state and the multimodal reliability evaluation results, and simultaneously perform semantic parsing on the task instructions to obtain the task objective; The first generation module is used to establish the association relationship between multimodalities based on the scheduling agent, the scene state, and the global semantic embedding inference, and to perform collaborative correction on the multimodal features, so as to generate a fusion control strategy based on the corrected multimodal features, the global semantic embedding, the task objective, and the cognitive state vector. The second generation module is used to invoke an expert agent based on the fusion control strategy, and use the expert agent to perform fusion processing on the multimodal features based on the fusion control strategy to obtain multimodal fusion features. Based on the task objective, the multimodal fusion features are cognitively guided and modulated to generate visual representation results.

8. The apparatus according to claim 7, characterized in that, The evaluation module includes: The construction unit is used to establish a semantic association between the multimodal features and the task instructions based on the scheduling agent through a cross-modal attention mechanism; The first parsing unit is used to perform scene recognition parsing on the multimodal features based on the semantic association, and autonomously identify the scene state.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the multi-source information collaborative fusion perception method based on cognitive agent-driven as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the cognitive agent-driven multi-source information collaborative fusion perception method as described in any one of claims 1-6.

Images