Intelligent motion track planning and interaction device based on multi-modal vision

Abstract

The invention provides an intelligent motion trajectory planning and interaction device based on multi-modal vision, and relates to the technical field of trajectory planning and interaction devices. Comprising the steps of starting and task initialization, multi-modal visual information collection, data preprocessing and feature extraction, multi-modal feature fusion and environment modeling, intelligent trajectory planning generation, interaction instruction execution and state feedback, closed-loop adjustment and task completion and the like. Auxiliary features such as touch and language are fused in the processes of data preprocessing, feature extraction, multi-modal feature fusion and environment modeling, interaction feedback is output through a voice module and a touch control module, the limitation of fixed instruction interaction of a traditional device is broken through, and when the interaction process and trajectory planning are deeply fused, intention can be analyzed through multi-modal feature fusion, so that the interaction efficiency is improved. A trajectory planning strategy is dynamically adjusted, and the problems of single sensing dimension, incomplete environment understanding, insufficient trajectory planning adaptability and dynamic response lag in a complex dynamic environment are solved.

Description

Technical Field

[0001] This invention relates to the field of trajectory planning and interaction device technology, and particularly to an intelligent motion trajectory planning and interaction device based on multimodal vision. Background Technology

[0002] With the rapid iteration of fields such as intelligent transportation and service robots, higher requirements are placed on the environmental perception accuracy, trajectory planning flexibility, and human-machine / machine-object interaction intelligence of motion devices. When existing devices are used in scenarios, they are required to break through the traditional mode of "single perception-fixed planning" and have the ability to understand the environment in multiple dimensions, optimize the dynamic trajectory, and interact flexibly. Multimodal vision, as the technology path closest to human perception, is needed to meet industry needs. Therefore, an intelligent motion trajectory planning and interaction device based on multimodal vision is required.

[0003] Currently, motion trajectory planning and interactive devices still suffer from limitations in complex dynamic environments, including a single perception dimension, incomplete environmental understanding, insufficient adaptability of trajectory planning, and lagging dynamic response. These limitations make it difficult to meet the application requirements of high-precision and high-reliability scenarios. Summary of the Invention

[0004] This invention relates to an intelligent motion trajectory planning and interaction device based on multimodal vision. It incorporates auxiliary features such as touch and language into the data preprocessing, feature extraction, multimodal feature fusion, and environmental modeling processes, and outputs interactive feedback through voice and touch modules. This breaks through the limitations of fixed command interaction in traditional devices. When the interaction process is deeply integrated with trajectory planning, the intent can be analyzed through multimodal feature fusion, the trajectory planning strategy can be dynamically adjusted, and the execution status can be confirmed to the user through real-time feedback. This solves the problems of disconnect between traditional interaction and planning and insufficient flexibility, and improves the convenience and humanization of the interaction.

[0005] This invention provides an intelligent motion trajectory planning and interaction device based on multimodal vision, which adopts the following steps: S1: Startup and task initialization; S2: Multimodal visual information acquisition; S3: Data preprocessing and feature extraction; S4: Multimodal feature fusion and environment modeling; S5: Intelligent trajectory planning and generation; S6: Interactive command execution and status feedback; S7: Closed-loop adjustment and task completion.

[0006] Preferably, step S1 includes hardware activation and task loading, as follows: The hardware activation involves starting the multimodal vision sensor array, computing unit, actuator, and interaction module, and completing hardware self-test and parameter calibration. The task loading process involves inputting the task objective and loading the pre-trained multimodal model and scenario-based parameters.

[0007] Preferably, step S2 includes multi-source visual data acquisition and data synchronization processing, as follows: The multi-source visual data acquisition includes visual images, depth / temporal data, and auxiliary sensor fusion. The visual images are obtained by acquiring environmental images through a 2D camera and point cloud data through a 3D camera to obtain the coordinate information of the target object. The depth / temporal data consists of: a depth camera acquiring scene depth information, and a temporal camera recording the historical motion trajectory of dynamic targets; The auxiliary sensing fusion can optionally fuse point cloud distance data from lidar and pressure feedback from tactile sensors to enrich the perception dimensions. The data synchronization process aligns multi-sensor data through timestamps, solving the problem of device sampling delay and ensuring the consistency of image, point cloud, and depth data in the time dimension.

[0008] Preferably, step S3 includes data cleaning and format unification, and single-modal feature encoding, as follows: The data cleaning and format standardization process involves the following steps: 1. Perform Gaussian filtering on the image to remove noise, and sample the point cloud data to reduce redundancy; 2. By using affine coordinate transformation, the target position in the pixel coordinate system is converted into the global coordinate system, thereby achieving spatial dimension unification of multimodal data; 3. Smooth the discrete trajectory of the dynamic target to remove noise interference; The single-modal feature encoding includes visual features, temporal features, and tactile / linguistic features, as detailed below: The visual features are: the spatial features of the point cloud extracted from the semantic features of the image; The temporal features are: capturing the motion trend of dynamic targets using a temporal coding module; The tactile / language features are: for interactive scenarios, the spatiotemporal features of tactile signals are extracted through architecture, or task instructions are converted into language embedding features through language models.

[0009] Preferably, step S4 includes cross-modal feature alignment, environment modeling, and intent parsing, as detailed below: The cross-modal feature alignment: uses a projector to map visual, tactile, and linguistic features to a unified embedding space, and achieves feature fusion through a cross-attention mechanism; The specific steps for environment modeling and intent parsing are as follows: 1. Generate a 3D environment model based on fusion features, segmenting static obstacles, dynamic areas, and operable areas; 2. Predict the future trajectory of dynamic targets and analyze the device's own motion intentions.

[0010] Preferably, step S5 includes initial trajectory generation, multi-objective optimization, and trajectory feasibility verification, as detailed below: The initial trajectory generation: Based on the environment model and the task objective, the initial trajectory is generated through the model, such as a basic path from the starting point to the end point, or a sequence of joint movements of the robotic arm.

[0011] The multi-objective optimization takes into account safety, efficiency, and smoothness, and incorporates visual semantic constraints.

[0012] The trajectory feasibility verification involves detecting and simulating to verify whether there is a collision risk in the trajectory or whether the device can execute the trajectory actions; if the verification fails, the trajectory is re-optimized.

[0013] Preferably, step S6 includes instruction conversion and execution, and real-time status acquisition, as detailed below: The instruction conversion and execution process converts the planned trajectory into control instructions for the actuator, while simultaneously providing interactive feedback via voice and touch modules.

[0014] The real-time status acquisition involves monitoring the motion status of the device and changes in the environment through sensors, and then transmitting the feedback data back to the fusion module.

[0015] Preferably, step S7 includes dynamic trajectory update and task determination, as follows: The dynamic trajectory update involves re-optimizing the multimodal feature fusion results based on feedback data and adjusting the trajectory planning strategy. The task determination is as follows: if the trajectory execution is completed and the task objective is met, the device enters a standby / stop state; if it is not completed, the above process is repeated.

[0016] The intelligent motion trajectory planning and interaction device based on multimodal vision provided by this invention has the following beneficial effects: In this invention, by integrating multiple visual sensors such as 2D / 3D cameras, depth cameras, and LiDAR, and combining auxiliary sensor fusion with timestamp synchronization processing, the limitations of traditional single-modal perception are overcome. This enables comprehensive acquisition of multi-dimensional data such as image texture, 3D coordinates, depth information, and dynamic trajectory, effectively eliminating perception blind spots. Furthermore, the preprocessing steps, such as Gaussian filtering for noise reduction and coordinate unification, further improve data quality and ensure the accuracy of environmental perception.

[0017] Furthermore, by generating a 3D environment model based on fusion features and parsing intent, and through the full-process design of initial trajectory generation, multi-objective optimization and trajectory feasibility verification, the planned trajectory can not only avoid static obstacles and dynamic targets and meet safety constraints, but also shorten the path length and reduce action mutations through optimization algorithms, thereby improving execution efficiency and motion smoothness; especially for dynamic environments, the planning strategy can be adjusted in advance by predicting the future trajectory of dynamic targets.

[0018] Furthermore, tactile and linguistic auxiliary features are integrated into the data preprocessing, feature extraction, multimodal feature fusion, and environmental modeling processes. Interactive feedback is output through voice and touch modules, breaking through the limitations of fixed command interaction in traditional devices. It supports multiple interaction methods such as natural language, body movements, and tactile feedback, realizing full-link intelligent interaction from "command input - scene understanding - action execution - feedback output". When the interaction process is deeply integrated with trajectory planning, the intent can be analyzed through multimodal feature fusion, the trajectory planning strategy can be dynamically adjusted, and the execution status can be confirmed to the user through real-time feedback. This solves the problems of disconnect between traditional interaction and planning and insufficient flexibility, and improves the convenience and humanization of interaction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below.

[0020] The accompanying drawings described below are only related to some embodiments of the invention and are not intended to limit the invention.

[0021] In the attached diagram: Figure 1 A flowchart illustrating the operation of an apparatus according to an embodiment of the present invention is shown; Figure 2 A step-by-step differentiation diagram according to an embodiment of the present invention is shown; Figure 3 A step-two differentiation diagram according to an embodiment of the present invention is shown; Figure 4 A three-step diagram illustrating the steps according to an embodiment of the present invention is shown; Figure 5 A step-by-step diagram according to an embodiment of the present invention is shown; Figure 6 A step-five differentiation diagram according to an embodiment of the present invention is shown; Figure 7 A diagram illustrating step six according to an embodiment of the present invention is shown; Figure 8 A diagram illustrating step seven of the present invention is shown. Detailed Implementation

[0022] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0023] Example 1: Please refer to Figures 1 to 8 This invention proposes an intelligent motion trajectory planning and interaction device based on multimodal vision, which adopts the following steps: S1: Startup and task initialization; S2: Multimodal visual information acquisition; S3: Data preprocessing and feature extraction; S4: Multimodal feature fusion and environment modeling; S5: Intelligent trajectory planning and generation; S6: Interactive command execution and status feedback; S7: Closed-loop adjustment and task completion.

[0024] Example 2: Based on Example 1, as shown in the figure, step S1 includes hardware activation and task loading, as detailed below: The hardware activation involves starting the multimodal vision sensor array, computing unit, actuator, and interaction module, and completing hardware self-test and parameter calibration. The task loading process involves inputting the task objective and loading the pre-trained multimodal model and scenario-based parameters.

[0025] S2 includes multi-source visual data acquisition and data synchronization processing, as detailed below: The multi-source visual data acquisition includes visual images, depth / temporal data, and auxiliary sensor fusion. The visual images are obtained by acquiring environmental images through a 2D camera and point cloud data through a 3D camera to obtain the coordinate information of the target object. The depth / temporal data consists of: a depth camera acquiring scene depth information, and a temporal camera recording the historical motion trajectory of dynamic targets; The auxiliary sensing fusion can optionally fuse point cloud distance data from lidar and pressure feedback from tactile sensors to enrich the perception dimensions. The data synchronization process aligns multi-sensor data through timestamps, solving the problem of device sampling delay and ensuring the consistency of image, point cloud, and depth data in the time dimension.

[0026] S3 includes data cleaning and format unification, and single-modal feature encoding, as detailed below: The data cleaning and format standardization process involves the following steps: 1. Perform Gaussian filtering on the image to remove noise, and sample the point cloud data to reduce redundancy; 2. By using affine coordinate transformation, the target position in the pixel coordinate system is converted into the global coordinate system, thereby achieving spatial dimension unification of multimodal data; 3. Smooth the discrete trajectory of the dynamic target to remove noise interference; The single-modal feature encoding includes visual features, temporal features, and tactile / linguistic features, as detailed below: The visual features are: the spatial features of the point cloud extracted from the semantic features of the image; The temporal features are: capturing the motion trend of dynamic targets using a temporal coding module; The tactile / language features are: for interactive scenarios, the spatiotemporal features of tactile signals are extracted through architecture, or task instructions are converted into language embedding features through language models.

[0027] S4 includes cross-modal feature alignment, environment modeling, and intent parsing, as detailed below: The cross-modal feature alignment: uses a projector to map visual, tactile, and linguistic features to a unified embedding space, and achieves feature fusion through a cross-attention mechanism; The specific steps for environment modeling and intent parsing are as follows: 1. Generate a 3D environment model based on fusion features, segmenting static obstacles, dynamic areas, and operable areas; 2. Predict the future trajectory of dynamic targets and analyze the device's own motion intentions.

[0028] S5 includes initial trajectory generation, multi-objective optimization, and trajectory feasibility verification, as detailed below: The initial trajectory generation: Based on the environment model and the task objective, the initial trajectory is generated through the model, such as a basic path from the starting point to the end point, or a sequence of joint movements of the robotic arm.

[0029] The multi-objective optimization takes into account safety, efficiency, and smoothness, and incorporates visual semantic constraints.

[0030] The trajectory feasibility verification involves detecting and simulating to verify whether there is a collision risk in the trajectory or whether the device can execute the trajectory actions; if the verification fails, the trajectory is re-optimized.

[0031] S6 includes instruction conversion and execution, and real-time status acquisition, as detailed below: The instruction conversion and execution process converts the planned trajectory into control instructions for the actuator, while simultaneously providing interactive feedback via voice and touch modules.

[0032] The real-time status acquisition involves monitoring the motion status of the device and changes in the environment through sensors, and then transmitting the feedback data back to the fusion module.

[0033] S7 includes dynamic trajectory updates and task determination, as detailed below: The dynamic trajectory update involves re-optimizing the multimodal feature fusion results based on feedback data and adjusting the trajectory planning strategy. The task determination is as follows: if the trajectory execution is completed and the task objective is met, the device enters a standby / stop state; if it is not completed, the above process is repeated.

[0034] The specific usage and function of this embodiment are as follows: In this invention, during use, the multimodal vision sensor array, computing unit, execution mechanism, and interaction module are activated to complete hardware self-test and parameter calibration. Then, the task target is input, and the pre-trained multimodal model and scene-specific parameters are loaded. Environmental images are acquired through a 2D camera, and point cloud data is acquired through a 3D camera to obtain the coordinate information of the target object. Then, scene depth information is acquired through a depth camera, and the historical motion trajectory of the dynamic target is recorded by a time-series camera. The perception dimensions are enriched by selecting and fusing point cloud distance data from a LiDAR and pressure feedback from a tactile sensor. Multi-sensor data is aligned using timestamps to solve the device sampling delay problem, ensuring consistency of image, point cloud, and depth data in the time dimension. Finally, Gaussian filtering is applied to the image for noise reduction, and point cloud data is sampled to reduce redundancy. The target position in the pixel coordinate system is converted to the global coordinate system through affine coordinate transformation, achieving spatial dimension unification of multimodal data. The discrete trajectory of the dynamic target is smoothed to remove noise interference. The system utilizes a projector to map visual, tactile, and linguistic features onto a unified embedded space. Feature fusion is achieved through a cross-attention mechanism. A 3D environment model is then generated based on the fused features, segmenting static obstacles, dynamic areas, and operable areas. The future trajectory of dynamic targets is predicted, and the device's own motion intentions are analyzed. Based on the environment model and task objectives, an initial trajectory is generated, such as a basic path from the starting point to the end point. The trajectory is verified through detection and simulation to check for collision risks or whether the device can execute the trajectory actions. If the verification fails, the trajectory is re-optimized, and the planned trajectory is converted into control commands for the actuators. Interactive feedback is output through voice and touch modules, and the device's motion status and environmental changes are monitored by sensors. The feedback data is sent back to the fusion module. Finally, the multimodal feature fusion results are re-optimized based on the feedback data, the trajectory planning strategy is adjusted, and then task judgment is performed. If the trajectory execution is completed and the task objective is met, the device enters a standby / stop state. If not, the above process is repeated. The following points should be noted in this article: 1. The accompanying drawings of the embodiments disclosed herein only relate to the structures involved in the embodiments disclosed herein; other structures can be referred to in general design.

[0035] 2. Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0036] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A multimodal vision-based intelligent motion trajectory planning and interaction device, characterized in that: The following steps are used: S1: Startup and task initialization; S2: Multimodal visual information acquisition; S3: Data preprocessing and feature extraction; S4: Multimodal feature fusion and environment modeling; S5: Intelligent trajectory planning and generation; S6: Interactive command execution and status feedback; S7: Closed-loop adjustment and task completion.

2. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S1 includes hardware activation and task loading, as detailed below: The hardware activation involves starting the multimodal vision sensor array, computing unit, actuator, and interaction module, and completing hardware self-test and parameter calibration. The task loading process involves inputting the task objective and loading the pre-trained multimodal model and scenario-based parameters.

3. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S2 includes multi-source visual data acquisition and data synchronization processing, as detailed below: The multi-source visual data acquisition includes visual images, depth / temporal data, and auxiliary sensor fusion. The visual images are obtained by acquiring environmental images through a 2D camera and point cloud data through a 3D camera to obtain the coordinate information of the target object. The depth / temporal data consists of: a depth camera acquiring scene depth information, and a temporal camera recording the historical motion trajectory of dynamic targets; The auxiliary sensing fusion can optionally fuse point cloud distance data from lidar and pressure feedback from tactile sensors to enrich the perception dimensions. The data synchronization process aligns multi-sensor data through timestamps, solving the problem of device sampling delay and ensuring the consistency of image, point cloud, and depth data in the time dimension.

4. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S3 includes data cleaning and format unification, and single-modal feature encoding, as detailed below: The data cleaning and format standardization process involves the following steps:

1. Perform Gaussian filtering on the image to remove noise, and sample the point cloud data to reduce redundancy; 2. By using affine coordinate transformation, the target position in the pixel coordinate system is converted into the global coordinate system, thereby achieving spatial dimension unification of multimodal data; 3. Smooth the discrete trajectory of the dynamic target to remove noise interference; The single-modal feature encoding includes visual features, temporal features, and tactile / linguistic features, as detailed below: The visual features are: the spatial features of the point cloud extracted from the semantic features of the image; The temporal features are: capturing the motion trend of dynamic targets using a temporal coding module; The tactile / language features are: for interactive scenarios, the spatiotemporal features of tactile signals are extracted through architecture, or task instructions are converted into language embedding features through language models.

5. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S4 includes cross-modal feature alignment, environment modeling, and intent parsing, as detailed below: The cross-modal feature alignment: uses a projector to map visual, tactile, and linguistic features to a unified embedding space, and achieves feature fusion through a cross-attention mechanism; The specific steps for environment modeling and intent parsing are as follows:

1. Generate a 3D environment model based on fusion features, segmenting static obstacles, dynamic areas, and operable areas; 2. Predict the future trajectory of dynamic targets and analyze the device's own motion intentions.

6. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S5 includes initial trajectory generation, multi-objective optimization, and trajectory feasibility verification, as detailed below: The initial trajectory generation: Based on the environment model and the task objective, the initial trajectory is generated through the model, such as a basic path from the starting point to the end point, or a sequence of joint movements of the robotic arm; The multi-objective optimization takes into account safety, efficiency, and smoothness, and incorporates visual semantic constraints. The trajectory feasibility verification involves detecting and simulating to verify whether there is a collision risk in the trajectory or whether the device can execute the trajectory actions; if the verification fails, the trajectory is re-optimized.

7. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S6 includes instruction conversion and execution, and real-time status acquisition, as detailed below: The instruction conversion and execution process involves converting the planned trajectory into control instructions for the actuator, while simultaneously providing interactive feedback via voice and touch modules. The real-time status acquisition involves monitoring the motion status of the device and changes in the environment through sensors, and then transmitting the feedback data back to the fusion module.

8. The intelligent motion trajectory planning and interaction device based on multimodal vision according to claim 1, characterized in that: S7 includes dynamic trajectory updates and task determination, as detailed below: The dynamic trajectory update involves re-optimizing the multimodal feature fusion results based on feedback data and adjusting the trajectory planning strategy. The task determination is as follows: if the trajectory is completed and the task objective is met, the device enters a standby / stop state. If not completed, repeat the above process.

Images