Dual-path ring-based multi-modal large model embodiment perception method and system

Abstract

The present application belongs to the field of large models, and provides a multi-modal large model embodied perception method and system based on a double-path ring to solve the problem of poor task execution real-time when part of the perception modal fails. The multi-modal large model embodied perception method based on the double-path ring includes obtaining a continuous frame image sequence, which is split into a high-frame-rate video stream and a low-frame-rate video stream according to a time resolution; a double-path perception structure is used to process the high-frame-rate video stream and the low-frame-rate video stream respectively to extract dynamic information and static information; the double-path perception structure includes a high-frame-rate path and a low-frame-rate path; in the high-frame-rate path, dynamic information in the high-frame-rate video stream is extracted; in the low-frame-rate path, static information in the low-frame-rate video stream is extracted; the dynamic information and the static information are fused to form a material attribute list and converted into a structured scene graph for representation. The present application can improve the understanding ability of an embodied intelligent system for a complex environment and improve the task execution real-time.

Description

Technical Field

[0001] This invention belongs to the field of large models, and in particular relates to a method and system for embodied perception of multimodal large models based on dual-path loops. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Embodied intelligence systems typically operate in dynamic environments and need to complete a closed-loop operation of perception, reasoning, decision-making, and execution within a short period of time. When typical embodied intelligence systems process multiple sensory information such as vision, speech, and touch, they often adopt an independent modeling approach, that is, different sensory modalities are processed by separate models, lacking a unified multimodal representation. This leads to the following problems: (1) Cross-modal information cannot be effectively integrated, and there is a lack of information in environmental understanding: a single modality often cannot fully describe the environment, and each sensory modality is modeled in different data spaces, making it difficult for the system to effectively integrate information from different sensors, and it may be difficult to establish a connection between semantics and visual content, affecting the coherence of task execution. (2) In terms of computational efficiency, it is difficult to meet the real-time requirements of highly dynamic tasks: The computational complexity of image language models is high, especially on resource-constrained robot platforms, making it difficult to achieve low-latency reasoning. Furthermore, embodied intelligent systems typically contain multiple heterogeneous sensors. Due to different sensor refresh rates, data bandwidth limitations, and network communication delays, data synchronization may be difficult, thereby affecting the system's real-time decision-making capabilities. In addition, traditional perception systems usually need to operate in complex and dynamic environments. For example, under different lighting conditions, weather, or dynamic obstacles, some sensors (such as RGB cameras) may fail or be interfered with, resulting in inaccurate perception data. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal large-scale embodied perception method and system based on a dual-path loop, which can enhance the embodied intelligent system's ability to understand complex environments and improve the real-time performance of its task execution.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of the present invention provides a multimodal large model embodied perception method based on a dual-path loop.

[0007] In one or more embodiments, a multimodal large model embodied perception method based on a dual-path loop is provided, including:

[0008] Obtain a continuous sequence of frame images and split it into a high frame rate video stream and a low frame rate video stream according to the temporal resolution;

[0009] A dual-path perception structure is used to process high-frame-rate video streams and low-frame-rate video streams separately to extract dynamic and static information. The dual-path perception structure includes a high-frame-rate path and a low-frame-rate path. In the high-frame-rate path, a target detection model is used to extract dynamic information from the high-frame-rate video stream. In the low-frame-rate path, a multimodal information processing model is used to extract static information from the low-frame-rate video stream.

[0010] By integrating the dynamic and static information, a list of material properties is formed and transformed into a structured scene diagram for representation.

[0011] In one implementation, the target detection model includes a backbone network, a feature fusion network, and a detection head; the backbone network is used to extract multi-scale features from high frame rate video streams; the feature fusion network uses a path aggregation network for feature fusion and combines an image pyramid structure to further enhance the transmission capability of multi-scale information; the detection head outputs class probabilities, bounding box coordinates, and segmentation-related mask coefficients.

[0012] In one implementation, the segmentation-related mask coefficients are combined with the global prototype mask, and the final instance segmentation mask is generated through matrix multiplication to achieve accurate target detection and instance segmentation.

[0013] As one implementation method, the global prototype mask is generated once from shared features.

[0014] In one implementation, the multimodal information processing model includes a CNN module, a ViT module, and a Transformer module. The CNN module is used to extract local texture features of a single frame of an image or video stream, while the ViT module encodes the image into blocks as embedding sequences with positional information, and the Transformer module is used to extract textual semantic information.

[0015] In one implementation, the nodes in the scene graph contain multimodal features of objects, and the edges in the scene graph contain spatial relationships, functional associations, and interaction states.

[0016] A second aspect of the present invention provides a multimodal large model embodied perception system based on a dual-path loop.

[0017] In one or more embodiments, a multimodal large model embodied perception system based on a dual-path loop includes:

[0018] The video stream splitting module is used to acquire a continuous frame image sequence and split it into a high frame rate video stream and a low frame rate video stream according to the time resolution.

[0019] A dual-path perception module is used to process high-frame-rate video streams and low-frame-rate video streams separately using a dual-path perception structure to extract dynamic and static information. The dual-path perception structure includes a high-frame-rate path and a low-frame-rate path. In the high-frame-rate path, a target detection model is used to extract dynamic information from the high-frame-rate video stream. In the low-frame-rate path, a multimodal information processing model is used to extract static information from the low-frame-rate video stream.

[0020] The scene graph representation module is used to fuse the dynamic and static information, form a list of material properties, and transform it into a structured scene graph for representation.

[0021] A third aspect of the present invention provides a computer-readable storage medium.

[0022] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal large model embodied perception method based on dual-path loops as described above.

[0023] A fourth aspect of the present invention provides a computer program product.

[0024] A computer program product having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal large model embodied perception method based on dual-path loops as described above.

[0025] A fifth aspect of the present invention provides an electronic device.

[0026] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal large model embodied perception method based on dual-path loops as described above.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] (1) The present invention uses a dual-path perception structure to process high frame rate video streams and low frame rate video streams respectively. The high frame rate path processes dynamic changes and captures high temporal resolution information in the video stream. It is mainly used to track the movements of objects or people and detect their position and posture changes in real time, which can track dynamic changes with higher accuracy. The low frame rate path focuses on capturing the static attributes of the image, such as the basic information of objects and people (such as color, shape, material, etc.), ensuring the stability and accuracy of the environment.

[0029] (2) By integrating dynamic and static information, this invention achieves the fusion of information at different levels, and transforms the objects in the image and their interrelationships into a structured graph representation, thereby making the relationships between objects clearer and more systematic, enabling intelligent systems to achieve more intelligent environmental perception and decision-making in complex environments. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a schematic diagram of the principle of multimodal large model embodied perception based on dual-path loop in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the dual-path sensing structure according to an embodiment of the present invention;

[0033] Figure 3 This is a framework diagram of the target detection model according to an embodiment of the present invention;

[0034] Figure 4 This is a framework diagram of the multimodal information processing model according to an embodiment of the present invention;

[0035] Figure 5 This is a flowchart illustrating the multimodal large model embodied perception method based on dual-path loops according to an embodiment of the present invention.

[0036] Figure 6 This is a schematic diagram of the structure of a multimodal large model embodied perception system based on a dual-path loop according to an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] Terminology Explanation:

[0041] Embodied intelligence is an emerging field that integrates artificial intelligence, robotics, and environmental perception, aiming to enable intelligent agents to interact with the real world based on multimodal information (such as vision, touch, and speech). Compared to traditional methods based on pure visual perception, embodied intelligence systems emphasize the fusion of data from multiple sensors to gain a more comprehensive understanding of the environment and perform complex tasks.

[0042] In a typical architecture of embodied intelligence, tasks are usually divided into three main modules: perception, planning, and execution. (1) Perception module: responsible for collecting data from various sensors (such as RGB-D cameras, LiDAR, tactile sensors, etc.) and using technologies such as computer vision, speech recognition, and natural language processing to construct a unified environmental representation. (2) Planning module: based on perception data, reasoning about possible action strategies and generating task execution plans. Traditional methods often rely on rule-driven or reinforcement learning, but they are often difficult to generalize under high-dimensional and multimodal data. (3) Execution module: responsible for converting planned actions into robot control instructions and adjusting behavior in conjunction with real-time feedback to ensure the smooth completion of the task.

[0043] Figure 1 This is a schematic diagram illustrating the principle of multimodal large-model embodied perception based on a dual-path loop according to an embodiment of the present invention. Combined with... Figure 1 and Figure 5 The multimodal large model embodied perception method based on dual-path loop in this embodiment may include the following steps:

[0044] S501, acquire a continuous frame image sequence and split it into a high frame rate video stream and a low frame rate video stream according to the time resolution.

[0045] In this embodiment, a continuous frame sequence is obtained from a single camera and split into a high frame rate video stream (i.e., a fast stream) according to two temporal resolutions. With low frame rate video streams (i.e., slow streams) .

[0046] Assuming the original frame rate is Sampling interval ,but:

[0047] ;

[0048] in, Represents a video frame. Representative of the original video stream The set of keyframes for the sampling interval. This represents a set of high frame rate video frames.

[0049] S502 employs a dual-path sensing structure to process high-frame-rate video streams and low-frame-rate video streams separately, extracting dynamic and static information; among which, such as Figure 2 As shown, the dual-path perception structure includes a high frame rate path and a low frame rate path. In the high frame rate path, a target detection model is used to extract dynamic information from the high frame rate video stream. In the low frame rate path, a multimodal information processing model is used to extract static information from the low frame rate video stream.

[0050] High frame rate path pair The data is fed frame by frame into a lightweight instance segmentation network based on the structure "FeatureBackbone → FPN → Neck → Head". In each frame... Above, the detection head outputs bounding box of an instance , mask Matching constitute:

[0051] ;

[0052] in, This means that the frame is in The frame sequence number in the data. It is the original video stream frame rate. represent The total number of object instances identified in the data. It is the instance's identity number (a new ID is assigned if a new target is obtained through cross-frame matching); It is the bounding box of the instance. It is the mask of the instance. This indicates that in high frame rate sequences No. frame The complete set of detection results.

[0053] mask The ID is obtained by linearly combining the prototype mask and weight coefficients. The detection results of consecutive frames are kept consistent by Kalman filtering and Hungarian matching. Due to the lightweight FPN-Neck design of the network, the single-frame inference latency is controlled within 70ms, ensuring the requirements of real-time tracking and motion feedback.

[0054] The object detection model in this embodiment of the invention is responsible for extracting and analyzing dynamic information of objects and people from video streams, including object localization, tracking, and capture of motion changes. This module combines end-to-end trained object detection algorithms, motion tracking algorithms (such as BYTETracker and BoT-SORT), and multimodal information fusion to ensure high-precision identification and tracking of objects and people in the environment.

[0055] like Figure 3As shown, the object detection model includes a backbone network, a feature fusion network, and a detection head. The backbone network (e.g., using an improved CSPDarknet structure) is used to extract multi-scale features from high frame rate video streams. The feature fusion network uses a path aggregation network (PANet) for feature fusion and combines it with a feature pyramid structure to further enhance the transmission capability of multi-scale information. Figure 3 In this block, SCDown is a lightweight downsampling block, PSA is a parallel channel attention and spatial attention branch block, CIB is a lightweight convolution block, and C2fCIB is a combination of C2f's cross-stage multi-branch mechanism and CIB as a basic unit.

[0056] The target detection model in this embodiment receives raw frames from a high frame rate RGB-D video stream. First, it performs synchronous decoding at the data layer: RGB and depth frames are aligned by timestamps to obtain... Perform proportional scaling on RGB images:

[0057] ;

[0058] in, This represents the RGB camera observation frame image aligned by timestamps. This represents the height of the image after scaling. This represents the image width after scaling. This represents the scaled image due to subsequent processing.

[0059] Then use the letter-box method (pixel fill method) to fill the empty area to the target resolution. :

[0060] ;

[0061] in, This represents the target resolution height of the network input. This represents the target resolution width of the network input. This represents the scaled image. is the fill color, and Pad is the fill algorithm.

[0062] Then, to Normalization is performed, and the normalization process is as follows:

[0063] ;

[0064] in, It is the channel mean and channel standard deviation. It is the normalized image tensor.

[0065] During the training phase, Apply a set of enhancement operators (such as random scaling / flipping / cropping, affine / perspective, etc.), and using a set Determined geometric enhancement isomorphism matrix Label boxes from the dataset and example mask Synchronous transformation, respectively obtained , , Enhanced tensor , , Input the backbone CSPDarknet, output multi-scale features PANet fused the features bidirectionally to obtain the fused characteristics. Then, the detection head is located at the grid position of each feature level. Predicting Quadruples Mapped to normalized box parameters:

[0066] ;

[0067] in, , These are the two-dimensional coordinates of the center point of the network's predicted bounding box on the image plane. It is the width and height of the network prediction box on the image plane (along...). , (pixel span in direction); , , , Similarly, this is the normalized box parameter. Let the stride of this layer be... Then the corresponding pixel coordinate system (relative) The predicted bounding box coordinates are:

[0068] ;

[0069] in, , It predicts the two-dimensional coordinates of the center point of the bounding box axis-aligned box on the image plane. , It predicts the width and height of the bounding box on the image plane; This refers to the grid position at the feature level, i.e., the coordinate index; this formula converts the relative parameters of the network's predicted bounding boxes into the true bounding boxes in the input image coordinate system. .

[0070] The detection head includes classification, regression, and segmentation branches, outputting class probabilities, bounding box coordinates, and segmentation-related mask coefficients, respectively. The segmentation-related mask coefficients are combined with a global prototype mask, and matrix multiplication is used to generate the final instance segmentation mask, achieving accurate object detection and instance segmentation. The global prototype mask is generated once from shared features. This embodiment's object detection model significantly improves segmentation accuracy while maintaining high inference speed.

[0071] Head splits the branch and outputs the mask coefficient. Global prototype mask The instance segmentation mask is generated once from shared features and then multiplied by the shared features. :

[0072] ;

[0073] in, It is the output mask coefficient vector of the Head split branch, the first... Weijizuo It characterizes the instance for the first... The preference and weight of a prototype mask; the larger the value, the more important the prototype is to the instance. It is a global prototype mask tensor, whose first... Each channel is denoted as . The log-probability map of the instance segmentation mask, after activation, shows the value of each pixel. Represents pixels The probability of belonging to this instance. Number of prototypes This refers to the spatial resolution of the prototype image. This is a pixel-wise activation function that normalizes the result of a linear combination. Its pixel-level segmentation principle applies to any pixel. have:

[0074] ;

[0075] When some of the prototypes The response is high at this pixel, and the corresponding weight is... If the value of a pixel is large, then the pixel is more likely to be classified as that instance; the contributions of multiple prototypes are linearly superimposed and then converted into probabilities by an activation function.

[0076] This embodiment uses a target detection model to process images in real time and detect objects or people within them. Each target is labeled with a bounding box and a category label (e.g., "person," "object," etc.). In addition to location labeling, the system also uses pixel-level segmentation to accurately segment each target, generating a fine outline of the target. Finally, the system outputs target data with category labels, location bounding boxes, and segmentation masks.

[0077] To ensure target consistency across frames, the system assigns a unique ID to each detected target. Existing algorithms (such as BYTE Tracker or BoT-SORT) enable continuous target tracking in subsequent frames, maintaining stable tracking even when the target is occluded or moves rapidly. Target information (including location, category, and segmentation mask) in each frame is updated in real time to ensure the accuracy of the target's state.

[0078] Simultaneously, to obtain the target's three-dimensional position information, the module uses camera intrinsics and a depth map to perform a 2D-to-3D transformation by combining the target's bounding box position and depth information in the 2D image, thereby accurately calculating the target's position in 3D space. The projection equation first uses intrinsics... Restore the pixel coordinates to the camera coordinate system, and then use and These extrinsic parameters are transformed to the world coordinate system:

[0079] ;

[0080] in, It represents the unit pixel and the image coordinates of the distortion removal target point; The optical axis depth of this pixel. It's the camera's internal parameters; It is the rotation matrix in the camera's extrinsic parameters, representing the camera's pose; It is the translation vector in the extrinsic parameters, representing the coordinates of the world origin in the camera coordinate system; Represents the world coordinates of the target point.

[0081] Binocular scene Determined by parallax; in monocular + ToF scenarios The data comes from a depth camera. To reduce depth noise, voxel grid statistics and bilateral filtering are performed on the point cloud before calculating the target center. This process achieves the goal of providing the target's three-dimensional spatial coordinates in real time.

[0082] In one or more embodiments, the target's position in future frames is predicted by analyzing its motion trajectory and position change patterns. This prediction mechanism ensures the continuity and accuracy of tracking, especially when the target is moving rapidly or occluded. Simultaneously, high frame rate paths can also perform behavior recognition based on the target's behavioral patterns (such as "walking," "running," "stationary," etc.). This facilitates a comprehensive understanding of the dynamic changes of targets in the environment, supporting further decision-making and actions.

[0083] In the low frame rate path, only A multimodal information processing model runs on keyframes. Each frame first generates global features using a visual ViT encoder. :

[0084] ;

[0085] in, Representing the Frame image, Represents the time after ViT encoding of the image. Visual features.

[0086] The spatial resolution is then refined using a pixel shuffle algorithm: ;

[0087] The pixel rearrangement algorithm rearranges sub-pixels, which are packed in the channel dimension by a multiplier r, into the spatial dimension for upsampling in detection segmentation. This is then mapped by a multilayer perceptron (MLP) to a semantic space of the same dimension as the language token.

[0088] ;

[0089] Among them, the multilayer perceptron will be constantly visual features Representative conversion dimensional vector sequence , length is This aligns with the language token.

[0090] Because the low frame rate path skips a large number of redundant non-critical frames, the total computational complexity per unit time is significantly reduced, and the efficiency of the dual-path loop is much greater than that of direct detection via a single path.

[0091] In this embodiment of the invention, the core of the multimodal information processing model is to extract core information from various data inputs (images, videos, text, etc.) and perform cross-modal feature alignment. Through in-depth analysis of various types of information in the environment, the system can understand the semantic content in images and videos, infer the relationships between objects, and thus provide support for multimodal tasks.

[0092] like Figure 4 As shown, the multimodal information processing model includes a CNN module, a ViT module, and a Transformer module. The CNN module takes an image as input and extracts local texture features and spatial patterns through multi-layer convolution, pooling, and non-linear activation operations to obtain a four-dimensional feature tensor. This is used to extract local texture features from a single frame of an image or video stream, where... Represents batch size. Represents the height of the feature map. Represents the width of the feature map. This represents the number of channels, corresponding to different feature dimensions.

[0093] Simultaneously, the ViT module encodes the feature images extracted by the CNN module into blocks of embedding sequences with location information, mathematically represented as:

[0094] ;

[0095] in, It is an image patch in the image; It is an embedding matrix that projects image patches onto a vector space; It is a positional encoding; It is a sequence of vectors of length n, where the dimension of each vector is the same as the input dimension of the Transformer module.

[0096] Text input is processed by the BERT text encoder to extract semantic features, and paragraph and positional embeddings are added, along with a sequence of text modalities. The Transformer module stack progressively establishes global dependencies between the visual modal sequences from ViT and the text modal sequences from the text encoder through alternating multi-head self-attention and feedforward network layers. Finally, the detection head outputs semantic representations of object types and spatial relationships in the image. By combining image and linguistic information, more nuanced semantic features can be extracted from images and integrated with natural language descriptions. For example, the system can analyze people, objects, and scenes in an image to understand their relationships, such as "a cup on a table," or identify a person's appearance, actions, and emotions. This not only identifies objects in an image but also understands their semantic context.

[0097] S503 integrates dynamic and static information to form a list of material properties and transforms it into a structured scene diagram for representation.

[0098] Detection trajectory of high frame rate path output Semantic token stream generated with low frame rate path Backend integration: leveraging The spatiotemporal location information guides the next step of updating the relevant tokens and constructs hints. and By concatenating input language image large language model components (VLM), complementary synergy between low frame rate high semantic expression and high frame rate accurate object detection can be achieved.

[0099] After visual language analysis, the scene graph constructs a graph structure based on the extracted objects and their semantic relationships. First, it parses the entities and their spatial relationships in the image: the object detection module identifies entities with well-defined boundaries, while the semantic understanding module analyzes the spatial relationships between entities in the image. Then, these elements are organized into a scene graph structure, where nodes contain the multimodal features of objects (visual appearance, semantic category, physical attributes), and edges encode spatial relationships, functional associations, and interaction states. The following relationship prediction formula is used to obtain the relationship prediction between two objects. , There is some kind of relationship between them. Probability (such as higher-level relationships like causality, application, interaction, etc.) :

[0100] ;

[0101] in, Nodes represent the visual or semantic features of an object. Represents the spatial relationship, functional association, and interaction state (edge) between two objects; This is a multilayer perceptron function. This allows the scene graph to include not only the physical locations of objects, but also the upstream and downstream relationships, interactions, and influences between objects. For example, after recognizing "table" and "cup," the scene graph can represent "there is a cup on the table" through relationship edges. Nodes in the scene graph contain the multimodal features of objects, and edges in the scene graph contain spatial relationships, functional associations, and interaction states.

[0102] This invention, based on fundamental image features (such as optical features, edge detection, color, and texture) and preliminary information from different perceptual channels, combined with semantic understanding (e.g., object categories, scene context) and more complex behavioral information, enables the system to perform multi-level analysis to identify complex dynamics in the environment, such as human behavior patterns and object interaction states. By fusing this information at different levels, objects in the image and their relationships are ultimately transformed into structured graph representations, making the relationships between objects clearer and more systematic. This allows intelligent systems to achieve more intelligent environmental perception and decision-making in complex environments.

[0103] As shown in Table 1, the proposed multimodal large-scale model embodied perception method based on iterative feedback is compared with typical multimodal visual-language large-scale models (MiniGPT-4v, Qwen-VL-Chat, LLaVA-1.5-13B) under perception tasks of varying complexity. The evaluation benchmark is ConvBench, which is geared towards multi-turn visual dialogue scenarios and hierarchically examines the model's perception and understanding capabilities. Specifically, ConvBench-R1 is used to test the model's performance in dialogue information retrieval scenarios, ConvBench-R2 is used to evaluate multi-turn, multi-hop retrieval capabilities, and ConvBench-S is used to measure context compression capabilities, covering single-turn and two-turn dialogues. Experimental results show that the proposed method significantly outperforms the compared algorithms in perception accuracy across all three scenarios.

[0104] Table 1. Comparison of this algorithm with other algorithms on convbench;

[0105]

[0106] in:

[0107] MiniGPT-4 is an open-source multimodal chatbot with image understanding and processing capabilities. This model combines a vision and language model architecture, improving its generative capabilities through two-stage training (pre-training and alignment fine-tuning).

[0108] The Qwen-VL-Chat model is a visual AI assistant built on the Qwen-VL series of large-scale visual language models developed by Alibaba Cloud, using an alignment mechanism. This model has better Chinese command following and supports more flexible interaction methods, including multi-image, multi-turn question answering, and creation capabilities.

[0109] LLaVA-1.5-13B is an open-source multimodal chatbot based on the LLaMA / Vicuna architecture, supporting the joint understanding and generation of images and text.

[0110] As shown in Table 2, the proposed method was compared with the typical image segmentation model SAM and its upgraded version SAM2 under different tasks. Evaluation metrics included segmentation accuracy and inference time, covering both single-object and multi-object scenes. Experimental results show that the proposed method significantly outperforms the comparative models in both encoding accuracy and inference time, demonstrating higher segmentation accuracy and efficiency.

[0111] Table 2. Comparison of time and mask accuracy of this algorithm with other algorithms on different tasks;

[0112]

[0113] As shown in Table 3, the semantic segmentation accuracy of the method of this invention is compared with that of the typical object detection model (YOLO) and the open vocabulary object detection model (Grounding DINO) under different environments. The experimental results show that the method of this invention significantly outperforms the comparison algorithms in terms of segmentation accuracy for single objects, occluded objects, and complex environments, and has the highest overall average accuracy, demonstrating stronger generalization ability and robustness.

[0114] Table 3. Comparison of semantic segmentation accuracy of this algorithm with other algorithms in different environments;

[0115]

[0116] Multimodal environment perception technology can perform cross-modal information alignment and scene graph enhancement between different perception channels, thereby achieving a comprehensive and accurate understanding of the environment, greatly enhancing its adaptability to the environment and the real-time performance of task execution.

[0117] This invention's high frame rate path (fast path) processes dynamic changes, capturing high temporal resolution information in the video stream. It is primarily used to track the movements of objects or people, detecting their position and posture changes in real time. Through target recognition and motion tracking modules, the system can track dynamic changes with higher accuracy, such as a person's gait and the trajectory of an object.

[0118] The low frame rate path (slow path) focuses on capturing static attributes of images, such as basic information about objects and people (e.g., color, shape, texture). In low frame rate video streams, the system samples fewer image frames per second (e.g., 2 frames per second), allowing it to efficiently extract static features from images while reducing computational load. This static information is then used by visual language models to achieve a deep understanding and description of image content, such as associating a person's appearance, actions, and expressions with semantic tags.

[0119] Through a dual-path perception structure, the system can efficiently distinguish and process static and dynamic information, enabling more precise task execution. Static information (basic features of objects and people) is processed through a slow path, ensuring stability and accuracy in the environment; dynamic information (actions and behaviors) is processed through a fast path, ensuring that the system can respond quickly and accurately capture rapidly changing environmental states.

[0120] This invention combines cross-modal information alignment, scene graph enhancement, and multi-level coded planning methods to construct an efficient, low-latency, and real-time adaptive agent for embodied intelligent tasks. This technology not only enhances the embodied intelligent system's understanding of complex environments but also improves the real-time performance of its task execution, laying the foundation for future applications of intelligent robots in autonomous driving, intelligent manufacturing, and smart homes.

[0121] Figure 6 This is a schematic diagram of a multimodal large model embodied perception system based on a dual-path loop according to an embodiment of the present invention. This embodiment is related to... Figure 5 Corresponding to the embodied perception method of multimodal large models based on dual-path loops, such as Figure 6 As shown, the multimodal large model embodied perception system based on dual-path loops in this embodiment may include:

[0122] The video stream splitting module 601 is used to acquire a continuous frame image sequence and split it into a high frame rate video stream and a low frame rate video stream according to the time resolution.

[0123] The dual-path perception module 602 is used to process high-frame-rate video streams and low-frame-rate video streams separately using a dual-path perception structure to extract dynamic and static information. The dual-path perception structure includes a high-frame-rate path and a low-frame-rate path. In the high-frame-rate path, a target detection model is used to extract dynamic information from the high-frame-rate video stream. In the low-frame-rate path, a multimodal information processing model is used to extract static information from the low-frame-rate video stream.

[0124] Scene graph representation module 603 is used to integrate dynamic and static information to form a list of material properties and transform it into a structured scene graph for representation.

[0125] It should be noted that each module in the embodiments of the present invention corresponds one-to-one with each step in the multimodal large model embodied perception method based on dual path loops, and their specific implementation processes are the same, so they will not be repeated here.

[0126] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 5 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0127] in, Figure 5 The computer program instructions corresponding to the method shown may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dual-path loop-based multi-modal large model embodied perception method, characterized in that, The method comprises the following steps: obtaining a continuous frame image sequence, which is split into a high-frame-rate video stream and a low-frame-rate video stream according to a time resolution; processing the high-frame-rate video stream and the low-frame-rate video stream respectively by using a dual-path perception structure to extract dynamic information and static information, wherein the dual-path perception structure comprises a high-frame-rate path and a low-frame-rate path; in the high-frame-rate path, a target detection model is used to extract dynamic information in the high-frame-rate video stream; in the low-frame-rate path, a multi-modal information processing model is used to extract static information in the low-frame-rate video stream; the target detection model comprises a backbone network, a feature fusion network and a detection head; the detection head outputs category probability, bounding box coordinates and segmentation-related mask coefficients; the segmentation-related mask coefficients are combined with a global prototype mask to generate a final instance segmentation mask through matrix multiplication, so as to realize accurate target detection and instance segmentation; the multi-modal information processing model comprises a CNN module, a ViT module and a Transformer module; the CNN module is used to extract local texture features of a single frame of an image or a video stream; the ViT module encodes the feature image extracted by the CNN module into an embedded sequence with position information; the Transformer module gradually establishes global dependency relationships between a visual modal sequence from the ViT and a text modal sequence from a text encoder through alternating multi-head self-attention and feed-forward network layers; finally, the detection head outputs semantic expressions of object categories and object spatial relationships in the image; the dynamic information and the static information are fused to form a material attribute list and are converted into a structured scene graph for representation. 2.The dual-path loop based multi-modal large model embodied perception method of claim 1, wherein, the backbone network is used to extract multi-scale features of the high-frame-rate video stream; the feature fusion network uses a path aggregation network to perform feature fusion and combines an image pyramid structure to further enhance the transmission capability of multi-scale information. 3.The dual-path loop based multi-modal large model embodied perception method of claim 2, wherein, The global prototype mask is generated once by shared features. 4.The dual-path loop based multi-modal large model embodied perception method of claim 1, wherein, The nodes in the scene graph contain multi-modal features of objects, and the edges in the scene graph contain spatial relationships, functional associations and interaction states.

5. A dual-path loop based multi-modal large model embodied perception system, adopting the dual-path loop based multi-modal large model embodied perception method according to any one of claims 1-4. The method comprises the following steps: a video stream splitting module is used to obtain a continuous frame image sequence, which is split into a high-frame-rate video stream and a low-frame-rate video stream according to a time resolution; a dual-path perception module is used to process the high-frame-rate video stream and the low-frame-rate video stream respectively by using a dual-path perception structure to extract dynamic information and static information; wherein the dual-path perception structure comprises a high-frame-rate path and a low-frame-rate path; in the high-frame-rate path, a target detection model is used to extract dynamic information in the high-frame-rate video stream; in the low-frame-rate path, a multi-modal information processing model is used to extract static information in the low-frame-rate video stream; a scene graph representation module is used to fuse the dynamic information and the static information to form a material attribute list and convert it into a structured scene graph for representation.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps in the dual-path ring-based multi-modal large model embodied perception method according to any one of claims 1-4.

7. A computer program product, characterised in that, A computer program is stored thereon, which is executed by a processor to implement the steps in the dual-path ring-based multi-modal large model embodied perception method according to any one of claims 1-4.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps in the dual-path loop-based multi-modal large model embodiment perception method as claimed in any one of claims 1-4 when executing the program.

Images