Multi-modal robot cross-scene navigation method and system based on continuous learning

Abstract

The invention provides a multi-modal robot cross-scene navigation method and system based on continuous learning, and the method comprises the steps: constructing multi-modal data sets in different scenes, and training a corresponding navigation model through employing the multi-modal data set in one scene; then, training the navigation model corresponding to the next scene on the basis of the multi-modal data in the next scene, and in the training process of the navigation model corresponding to the next scene, acquiring the multi-modal data of the next scene; based on the weight stored in the navigation model trained in the previous scene and the weight updated in each step in the navigation model training process corresponding to the next scene are fused according to the proportion, the fused training weight is obtained, and then the trained navigation model corresponding to the next scene is obtained. According to the method, knowledge in a previous scene is reserved while the robot learns navigation in a new environment, and continuous cross-scene navigation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal robot cross-scene navigation, and in particular, relates to a multimodal robot cross-scene navigation method and system based on continuous learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Robot cross-scene navigation technology is one of the important branches in robotics research. Robots need to navigate and avoid obstacles in a variety of different scenarios and have the ability to cross multiple scenarios. Robots equipped with multimodal perception capabilities can better perceive and adapt to the environment. Achieving continuous navigation in different scenarios is essential to completing complex tasks and meeting actual production needs. Continuous learning is a training paradigm that enables a system to learn new knowledge while retaining previous knowledge. It provides a new solution for robot cross-scene navigation, allowing robots to smoothly transition to new scenarios while retaining learning experience from previous scenarios. Continuous learning methods are expected to improve the robot's cross-scene navigation capabilities.

[0004] Autonomous mobile robots need to perform tasks in a variety of scenarios. However, the environmental structures of these scenarios are often quite different, which places higher demands on the generalization of navigation methods. Current navigation methods are mostly focused on applications in specific scenarios, but performance often degrades when migrating across scenarios, making it difficult for robots to achieve continuous navigation.

[0005] The navigation methods of autonomous mobile robots can be divided into two categories: rule-based and learning-based. Learning-based methods are mainly divided into reinforcement learning and imitation learning. This type of method usually adopts an end-to-end navigation mode, which directly maps the perception information into speed commands by decoupling perception and control. For example, Huang et al. proposed an end-to-end navigation framework and realized robot autonomous navigation using reinforcement learning. However, although this type of method can update strategies through continuous learning, it is easy to forget the previously learned experience, leading to the problem of "catastrophic forgetting". To meet this challenge, continuous learning methods have been introduced to enable robots to maintain and enhance their continuous navigation capabilities, becoming a promising solution.

[0006] With the rapid development of continuous learning algorithms, more and more robotics research has begun to explore navigation methods based on continuous learning. However, most of the current navigation learning methods rely on single-modal perception information such as LiDAR point clouds or RGB images. For example, Yoon et al. adopted self-supervised online continuous learning and uncertainty experience replay to enable the robot to adaptively estimate traversability. However, single-modal perception is easily disturbed by environmental changes, which affects navigation performance. In contrast, multimodal methods improve the navigation performance of robots in various scenarios by integrating multiple complementary perception inputs. For example, Xie et al. combined RGB camera and LiDAR data to enable mobile robots to navigate in complex and dynamic environments. Therefore, introducing continuous learning algorithms into multimodal robot navigation can not only improve the adaptability of robots in different environments, but also enhance their continuous navigation capabilities.

[0007] Due to the limitations of end-to-end navigation methods in continuous navigation tasks and the advantages of multimodal perception, multimodal navigation based on continuous learning has become a very promising solution. However, current robot navigation research is limited by several key factors. First, there is a lack of public multimodal datasets specifically for continuous navigation tasks. Second, there is currently no dedicated continuous learning algorithm designed for multimodal navigation. Summary of the invention

[0008] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a multimodal robot cross-scene navigation method and system based on continuous learning. While the robot learns to navigate in a new environment, it retains the knowledge in previous scenes to achieve continuous cross-scene navigation.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a multimodal robot cross-scenario navigation method based on continuous learning, comprising:

[0011] Constructing a multimodal dataset of the robot in different scenarios; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data;

[0012] Training a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model;

[0013] The second navigation model is trained based on the multimodal data corresponding to the next scene of the robot to obtain a trained second navigation model; wherein the weights saved in the first navigation model trained in the previous scene are proportionally merged with the weights updated at each step in the training process of the second navigation model to obtain a trained second navigation model;

[0014] Acquire multimodal data of the robot in the current scene and perform navigation based on the trained second navigation model.

[0015] In a second aspect, the present invention provides a multimodal robot cross-scenario navigation system based on continuous learning, comprising:

[0016] A construction module, which is configured to: construct a multimodal dataset of the robot in different scenes; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data;

[0017] A first training module is configured to: train a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model;

[0018] A second training module is configured to: train the second navigation model based on the multimodal data corresponding to the next scene of the robot to obtain a trained second navigation model; wherein the weights saved in the first navigation model trained in the previous scene are proportionally merged with the weights updated in each step of the training process of the second navigation model to obtain the trained second navigation model;

[0019] The navigation module is configured to: obtain multimodal data of the robot in the current scene, and perform navigation movement based on the trained second navigation model.

[0020] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0023] One or more of the above technical solutions have the following beneficial effects:

[0024] In the present invention, by constructing multimodal data sets in different scenes, the corresponding navigation model is trained using the multimodal data set in one scene; then the navigation model corresponding to the next scene is trained based on the multimodal data in the next scene, and in the training process of the navigation model corresponding to the next scene, the weights saved based on the navigation model trained in the previous scene and the weights updated in each step of the training process of the navigation model corresponding to the next scene are fused in proportion to obtain the fused training weights, and then the trained navigation model corresponding to the next scene is obtained. The method of the present invention retains the knowledge in the previous scene while the robot learns to navigate in a new environment, and realizes continuous cross-scene navigation.

[0025] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 This is an example diagram of a modal data set in Embodiment 1 of the present invention;

[0028] Figure 2 The end-to-end navigation framework in the first embodiment of the present invention;

[0029] Figure 3 Schematic diagram of the momentum update method in the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0032] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0033] Embodiment 1

[0034] This embodiment discloses a multimodal robot cross-scenario navigation method based on continuous learning, including:

[0035] Constructing a multimodal dataset of the robot in different scenarios; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data;

[0036] Training a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model;

[0037] The second navigation model in the next scene is trained based on the multimodal data corresponding to the next scene of the robot to obtain the trained second navigation model in the next scene; wherein, the weights saved in a trained first navigation model are taken out from the previous scene and the weights updated at each step in the training process of the second navigation model are fused in proportion to obtain the trained second navigation model in the next scene, and the training of subsequent scenes is also carried out in the same way to obtain the trained navigation model corresponding to each scene; the multimodal data of the robot in the current scene is obtained, and navigation movement is performed based on the trained navigation model corresponding to the current scene.

[0038] In this embodiment, there may be more than two scenes, for example, the data set may come from five scenes in total. For the training of the third scene, a second navigation model with relatively good performance is firstly taken out from the second scene, and the weights saved in the second navigation model are proportionally fused with the weights of the model updated at each step in the training process of the third navigation model corresponding to the third scene, thereby obtaining a trained third navigation model; the subsequent scene training methods are similar.

[0039] In this embodiment, the multimodal data set in the next scene of the robot trains the corresponding navigation model, wherein the weights saved in the navigation model trained in the previous scene are proportionally fused with the weights updated at each step in the training process of the navigation model corresponding to the next scene to obtain multiple fused training weights corresponding to the next scene; the navigation model corresponding to the multiple fused training weights corresponding to the next scene is tested based on the test data set of the next scene to obtain the navigation model corresponding to the next scene with the best performance; the subsequent scenes are also trained sequentially using the same method to finally form a trained navigation model, and the robot performs navigation movements based on the trained navigation model in the actual scene. The method of this embodiment retains the knowledge in the previous scene while the robot learns to navigate in the new environment, thereby achieving continuous cross-scene navigation.

[0040] This embodiment sets five different scenarios for the mobile robot: daytime, nighttime, rainy day, highway and field road. For these scenarios, this embodiment collects multimodal data sets respectively.

[0041] Figure 1An example of a multimodal dataset is shown, where the dataset is collected in the CARLA simulator. The first image is a global view, and the remaining three images represent the three modalities of the dataset: image, point cloud, and odometer data. This embodiment uses an RGB camera to collect images. The RGB camera is mounted 2 meters in front of and 2 meters above the center of the vehicle, tilted 25 degrees downward, with an image resolution of 800×600 pixels and a field of view of 90.0 degrees. The single-line LiDAR is mounted 1.5 meters above the center of the vehicle with a maximum detection range of 100 meters. The images provide information about the passable area, while the LiDAR data provides accurate distance measurements of obstacles, and the two complement each other. The odometer is directly integrated on the robot in the simulator, recording position, direction, linear velocity, and angular velocity data, which can be used to track the vehicle status in real time.

[0042] For the above data set, this embodiment proposes an end-to-end navigation framework, such as Figure 2 As shown, the framework includes a perception module and a control module.

[0043] Specifically, in the perception module, the pre-trained ResNet50 is first used to process the RGB image and extract useful features; then the feature tensor is flattened for integration with subsequent tensors.

[0044] The LiDAR point cloud is also included in the perception module. Considering that robot navigation mainly focuses on forward movement, this embodiment filters the point cloud data, the point cloud covers 360 degrees, only uses the front part of the point cloud, and discards the rear point cloud data. The front point cloud data spans 180 degrees and contains 512 feature points, each point represents coordinate information, which is directly input into the fully connected network.

[0045] The odometer data contains the position and direction of the vehicle, providing the robot's position and speed information. The position of the vehicle is represented by three dimensions (x, y, z), and the direction is represented by four dimensions (x, y, z, w). These seven dimensions of data are input into the fully connected layer for processing, focusing on the relative position of the robot between different frames, and then combined with the processed image and point cloud data for subsequent processing.

[0046] like Figure 2 As shown in Figure 1, the processed image, point cloud, and odometer information are input to the control module, where the features extracted by the perception module are used to generate speed commands. These features are combined and processed through several fully connected layers, and the ReLU activation function is applied after each fully connected layer, except for the last fully connected layer.

[0047] The final output of the control module includes linear velocity and angular velocity, where the linear velocity range is [0,2] and the angular velocity range is [-1,1]. Therefore, the last fully connected layer uses sigmoid and tanh activation functions for these outputs respectively. The linear velocity is doubled to fit the required speed range. The odometer speed data is used as a label to calculate the mean square error (MSE) loss of the predicted speed, guiding the network to optimize the strategy during training.

[0048] like Figure 3 As shown, this embodiment proposes a new continuous learning method with momentum update, which combines the weights learned from the previous task with the weights of the current learning task, promoting the integration of prior knowledge in the current training process. Specifically, the first stage represents the learning stage, and the weights saved at this time are W t-1 .

[0049] The second stage is the learning stage, where the navigation model continuously updates the weights at each step. In this process, the weights W of the new model are t With W t-1 The fusion is performed in the ratio of a / (1-a), where the ratio factor a determines the contribution of the navigation model trained in the first stage and the navigation model being trained in the second stage to the final fused navigation model. In the second stage, weights corresponding to multiple navigation models are generated, and the weights corresponding to each navigation model in the second stage are proportionally fused with the model weights saved in the first stage.

[0050] This method blends the weight of a specific layer in the previous navigation model with the corresponding layer in the current navigation model by calculating a weighted average of the weights of each layer through a scaling factor.

[0051] The weight of the new navigation model is shown in the following formula (1):

[0052] W t ′=a·W t +(1-a)·W (t-1) (1)

[0053] The fused navigation model will be saved as the basic model. In the subsequent stages, the navigation model will continue to undergo fusion learning to maintain and integrate prior knowledge.

[0054] In the continuous learning navigation task, this embodiment designs three indicators to comprehensively evaluate the performance of the navigation system, which are described in detail below:

[0055] The navigation success degradation rate (NSDR) quantifies the relative decline in the success rate of completing navigation tasks in different environments. After training in the current scene, the system's navigation performance degradation is evaluated by comparing the success rate in this scene with the success rates in the learned and unlearned scenes. The NSDR calculation is shown in the following formula (2):

[0056]

[0057] Among them, SR current Indicates the success rate in the current scenario, SR other Represents the success rate in the other two scenarios.

[0058] The navigation time variation (NTV) is used to evaluate the change in the time required for the system to complete the navigation task under the same path conditions in different environments. By comparing the average navigation time in each scenario, NTV reflects the change in the efficiency of the system in adapting to the new environment and dealing with the previously encountered environment. A smaller time variation indicates that the robot can maintain a higher navigation efficiency in cross-scenario navigation. The calculation formula is shown in the following formula (3):

[0059] NTV=|NT current -NT other | (3)

[0060] Among them, NT current Indicates the navigation time in the current scene, NT other Indicates the navigation time in the other two scenarios.

[0061] Cumulative path deviation (CPD) is used to evaluate the deviation between the actual path and the ideal path during robot navigation. This indicator measures the accuracy of path tracking by cumulative deviation. Specifically, this embodiment divides the route into multiple sub-path points, and when the robot navigates to the position corresponding to each sub-path point, the deviation between the ideal path point and the actual position is calculated. A lower CPD value indicates a higher accuracy of path tracking. The CPD calculation formula is shown in the following formula (4):

[0062]

[0063] Where N is the number of discrete path points on the path. i and q i denote the coordinates of the i-th point on the ideal path and the actual path, respectively. ‖·‖ denotes the Euclidean distance, which is calculated based on the coordinates of the two points.

[0064] Embodiment 2

[0065] The purpose of this embodiment is to provide a multimodal robot cross-scenario navigation system based on continuous learning, including:

[0066] A construction module, which is configured to: construct a multimodal dataset of the robot in different scenes; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data;

[0067] A first training module is configured to: train a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model;

[0068] A second training module is configured to: train the second navigation model based on the multimodal data corresponding to the next scene of the robot to obtain a trained second navigation model; wherein the weights saved in the first navigation model trained in the previous scene are proportionally merged with the weights updated in each step of the training process of the second navigation model to obtain the trained second navigation model;

[0069] The navigation module is configured to: obtain multimodal data of the robot in the current scene, and perform navigation movement based on the trained second navigation model.

[0070] In further embodiments, there is also provided:

[0071] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0072] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0073] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0074] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0075] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0076] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0077] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0078] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0079] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0080] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0081] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal robot cross-scene navigation method based on continuous learning, characterized in that: include: Constructing a multimodal dataset of the robot in different scenarios; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data; Training a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model; The second navigation model is trained based on the multimodal data corresponding to the next scene of the robot to obtain a trained second navigation model; wherein the weights saved in the first navigation model trained in the previous scene are proportionally merged with the weights updated at each step in the training process of the second navigation model to obtain a trained second navigation model; Acquire multimodal data of the robot in the current scene and perform navigation based on the trained second navigation model.

2. A multimodal robot cross-scene navigation method based on continuous learning as claimed in claim 1, characterized in that: In the navigation model, ResNet50 is used to extract scene features of the scene image, and the extracted scene features, point cloud data and odometer data are processed by the control module to obtain the robot's linear velocity and angular velocity.

3. A multimodal robot cross-scene navigation method based on continuous learning as claimed in claim 1, characterized in that: The performance of the navigation model is evaluated using the navigation success drop rate, navigation time variation, and cumulative path deviation.

4. A multimodal robot cross-scene navigation method based on continuous learning as claimed in claim 1, characterized in that: The weights of the trained navigation model obtained by training the multimodal dataset corresponding to the previous scene are fused in proportion with the weights of multiple navigation models obtained by training the navigation model in the next scene with the multimodal dataset in the next scene to obtain the fused training weights of the navigation model in the next scene.

5. The multimodal robot cross-scene navigation method based on continuous learning as claimed in claim 1, characterized in that: The weights saved by the pre-trained navigation model are fused with the updated weights at each step in the retraining process in proportion to obtain the updated weights, which are: W′ t =a·W t +(1-a)·W (t-1) , Among them, W t-1 is the weight saved by the pre-trained navigation model, and a is the scaling factor.

6. A multimodal robot cross-scenario navigation system based on continuous learning, characterized in that: include: A construction module, which is configured to: construct a multimodal dataset of the robot in different scenes; wherein the scene multimodal dataset includes scene images, point cloud data and odometer data; A first training module is configured to: train a first navigation model using a multimodal data set corresponding to a constructed scene to obtain a trained first navigation model; A second training module is configured to: train the second navigation model based on the multimodal data corresponding to the next scene of the robot to obtain a trained second navigation model; wherein the weights saved in the first navigation model trained in the previous scene are proportionally merged with the weights updated in each step of the training process of the second navigation model to obtain the trained second navigation model; The navigation module is configured to: obtain multimodal data of the robot in the current scene, and perform navigation movement based on the trained second navigation model.

7. A multimodal robot cross-scenario navigation system based on continuous learning as claimed in claim 6, characterized in that: In the second training module, the weights of the trained navigation model obtained by training the multimodal data set corresponding to the previous scene are fused in proportion with the weights of multiple navigation models obtained by training the navigation model in the next scene with the multimodal data set in the next scene to obtain the fused training weights of the navigation model in the next scene.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 5 is completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 5.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images