Micro-gravity environment flying robot perception and scene understanding method and system

Abstract

This invention provides a method and system for perception and scene understanding of a flying robot in a microgravity environment, belonging to the field of robot intelligent perception and scene understanding. It addresses the problems of crowded facility layouts, complex spatial lighting, and floating object obstruction in microgravity indoor work scenarios, which lead to difficulties in image feature extraction and obstructed sensor lines of sight, affecting the accuracy of 3D environment modeling and target recognition. A lightweight convolutional neural network structure is adopted to reduce computational burden; in terms of multimodal data fusion, data from RGB images and laser rangefinders are combined, and a multimodal information fusion module is added to target detection and pose estimation, suitable for the identification and localization of target objects required for high-precision operations; in terms of task-adaptive semantic segmentation, the network is trained on a specific dataset for specific task scenarios in a microgravity environment; and transfer learning is used to enhance the model's adaptability.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent perception and scene understanding technology, and more specifically, to a method and system for perception and scene understanding of a flying robot in a microgravity environment. Background Technology

[0002] With the continuous advancement of my country's manned space program and deep space exploration missions, the tasks performed in microgravity environments are becoming increasingly diversified. Astronauts not only need to undertake technical work such as microgravity scientific experiments and materials processing, but also regularly perform routine maintenance tasks such as equipment inspections and troubleshooting. To improve astronauts' work efficiency and reduce their workload, the introduction of autonomous space-based intelligent flying robots is of great significance. These robots can independently complete tasks such as monitoring scientific experimental payloads, inspecting equipment status, and performing routine maintenance when astronauts are absent or unable to operate them, which is crucial for ensuring the long-term stable operation of the microgravity working environment and the efficient conduct of space experiments.

[0003] However, the increasing diversity and complexity of space missions in microgravity environments pose unprecedented challenges to the autonomous perception and decision-making capabilities of space-flying robots. In microgravity environments, indoor work scenarios are characterized by irregular structural heights, dense facility layouts, variable lighting conditions, and dynamic floating interference. This results in significant deficiencies in the completeness of information acquisition and the robustness of algorithms in traditional environmental perception technologies, making it difficult to meet the real-time and reliability requirements of autonomous operations in dynamic space environments.

[0004] Environmental perception and scene understanding are core technologies for robots to achieve autonomous operation. The key lies in building accurate environmental cognition models in real time, and based on this, developing capabilities such as obstacle recognition, area function understanding, target localization, and path planning. Although deep learning-based technologies such as target detection, semantic segmentation, and depth estimation have made significant progress in terrestrial environments, their direct application in microgravity environments still faces many challenges. The lighting conditions in space environments are complex, with strong reflections and interlacing shadows, easily leading to the loss of RGB camera features. Indoor work areas have dense equipment layouts, and the presence of floating objects and personnel movement interferes with the field of view of depth and laser sensors. These factors make it difficult for traditional single-sensor perception solutions to meet task requirements in terms of environmental modeling accuracy and target recognition reliability.

[0005] To address the aforementioned issues, this invention innovatively proposes an intelligent perception and scene understanding technology based on multi-sensor fusion. This technology enables robots to achieve real-time three-dimensional environmental perception in microgravity fields and to identify key objects and passable areas, thereby significantly improving the robot's autonomy and environmental adaptability, and providing technical support for future deep space exploration missions. Summary of the Invention

[0006] The technical problem to be solved by this invention is:

[0007] To address the challenges of crowded facility layouts, complex lighting, and floating object obstruction in microgravity indoor work environments, which lead to difficulties in image feature extraction and obstructed sensor lines of sight, thus affecting the accuracy of 3D environment modeling and target recognition.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0009] This invention provides a method for perception and scene understanding of a flying robot in a microgravity environment, comprising the following steps:

[0010] S100, Multi-sensor system model establishment and data fusion: By integrating the airborne forward-looking binocular camera, rear-looking binocular camera and laser distance sensor, a multi-sensor network is modeled.

[0011] S200, Deep Learning-Based Object Detection and Semantic Segmentation, including real-time object detection. It employs an improved YOLO network for rapid detection of key targets. The loss function is constructed by comprehensively considering classification error, bounding box regression error, and target confidence error, enabling accurate identification of devices, objects, and obstacles. Based on the Mask R-CNN semantic segmentation algorithm, transfer learning is introduced to classify image pixels into different semantic categories. Through data fusion, the system identifies the target location, category, and its role in the scene in real time.

[0012] S300, a high-precision positioning system based on pose estimation, utilizes the results of target detection and semantic segmentation for multimodal data fusion. It employs a PointNet++ network model to extract point cloud features for pose estimation, thereby determining the precise position and orientation of objects in real time. Through feature matching and image- and point cloud-based depth estimation algorithms, combined with depth data, and by using the iterative nearest point method, it completes pose calculation, thus obtaining the position and orientation of the target in three-dimensional space.

[0013] S400, path planning and real-time adjustment, including feasible area identification and path planning, path adjustment and task execution, enable intelligent flying robots to respond quickly when encountering dynamic obstacles or spatial changes.

[0014] Further, step S100 specifically includes,

[0015] S110, Sensor data acquisition: RGB images and depth data are acquired through a forward-looking binocular camera, a laser distance sensor, and a rear-looking binocular camera, respectively.

[0016] Among them, the depth data from the laser distance sensor Depth data estimated visually by forward-looking binocular cameras A unified depth estimate d(x,y) is generated through weighted fusion:

[0017]

[0018] Where (x,y) represents the image pixels; These are the weights of the depth data;

[0019] S120, Data Alignment and Enhancement: Synchronous calibration of RGB image and laser ranging data; assuming the camera coordinate system is... The lidar coordinate system is The transformation relationship is as follows:

[0020]

[0021] in, This represents the position of the target in the camera coordinate system at time t. This indicates the position of the target in the lidar coordinate system at time t. This represents the transformation matrix from the radar coordinate system to the camera coordinate system.

[0022] Furthermore, step S200 specifically includes,

[0023] S210, Real-time object detection, including lightweight object detection model construction and obstacle and key target identification.

[0024] S211. Lightweight Object Detection Model Construction: In the object detection stage, an optimized YOLO model is used to detect key objects and obstacles in the microgravity environment, and a lightweight convolutional neural network structure is introduced; the loss function for object detection in the model... Taking classification error into account Bounding box regression error Error with target confidence level The definition is as follows:

[0025]

[0026] in, For classification error weights, Bounding box regression error weights, Weights for the target confidence error;

[0027] S212, Obstacle and Key Target Identification: The lightweight target detection model constructed in step S211 identifies obstacles, passable areas, and operational targets, and updates the detection results in real time.

[0028] Speed ​​of dynamic targets Achieved through inter-frame motion estimation:

[0029]

[0030] in, It is the change in the position of the target object. It is the time interval between frames;

[0031] S220, Semantic Segmentation and Region Recognition, including constructing a task-adaptive semantic segmentation model and fusing segmented regions with detection results.

[0032] S221, a task-adaptive semantic segmentation model, introduces transfer learning based on the Mask R-CNN semantic segmentation algorithm, and initializes parameters pre-trained on a large-scale dataset. Subsequently, in the task-related dataset Fine-tuning training is performed on the model to obtain updated model weight parameters. :

[0033]

[0034] in, Represents the updated model weight parameters. Represents the current model weight parameters. Here, C represents the weight of the regularization term, where N is the total number of pixels and C is the number of categories. This represents the true category label of the i-th pixel. This represents the probability that the i-th pixel in the model output belongs to class c;

[0035] S222. The segmentation region and detection results are fused together. The segmentation results are combined with the output of target detection to generate an environment model, enabling the intelligent flying robot to more accurately judge the layout of its surrounding environment.

[0036] Furthermore, step S300 specifically includes,

[0037] S310, multimodal data fusion, introduces collaborative processing of point cloud and visual information to enhance the depth perception capability of pose estimation for target objects; assuming the point cloud provided by the laser sensor is... Each point Representing spatial coordinates, the set of detection boxes in the image is... The fusion strategy is as follows:

[0038]

[0039] in, Let m represent the m-th point, and k represent the total number of points. Let represent the nth detection box, and l represent the total number of detection boxes. This represents the camera intrinsic projection function that projects 3D points onto the image plane; through this operation, local point cloud regions matching the detection bounding box are extracted. , used for subsequent pose estimation;

[0040] S320, Pose Estimation and Dynamic Tracking: This section utilizes a PointNet++ network model to extract point cloud features for pose estimation, obtaining the precise spatial position and orientation of the target object in real time. This ensures the robot can adjust to the optimal pose promptly during operation. Simultaneously, it continuously updates the pose of detected target objects, enabling the robot to accurately track targets in dynamic environments. The network training objective is to minimize the predicted position t, pose q, and ground truth values. , Weighted loss function between:

[0041]

[0042] in, and These are the weighting coefficients for the translation and rotation components, respectively. This represents the dot product operation of quaternions.

[0043] Furthermore, combining steps S200 and S300, three network models are trained in a distributed manner: an object detection network, an image segmentation network, and a pose estimation network. Let the loss functions of the lightweight object detection model, the adaptive semantic segmentation model, and the pose estimation network model be respectively... , and Distributed optimization , and .

[0044] Further, step S400 specifically includes,

[0045] S410. Feasible Region Identification and Path Planning: Utilizing the environment model generated from the segmentation results, feasible regions are identified in real time. Combined with feedback from the binocular camera and laser rangefinder, the optimal path is planned, enabling the robot to efficiently avoid obstacles and quickly reach the target location. The category set is defined as follows: Let the c-th class be the passable area, then the passable area mask is... for:

[0046]

[0047] in, The (x, y) value of a pixel represents the category of the region it belongs to.

[0048] S420, Path Adjustment and Task Execution: During path execution, the robot monitors environmental changes in real time and continuously adjusts the path based on the latest peer area perception information to ensure that the robot can respond quickly when encountering dynamic obstacles or spatial changes.

[0049] A microgravity environment flight robot perception and scene understanding system is provided. The system has program modules corresponding to the above steps and executes the steps in the microgravity environment flight robot perception and scene understanding method described above when running.

[0050] A computer-readable storage medium storing a computer program configured to implement, when invoked by a processor, steps of a method for perception and scene understanding of a flying robot in a microgravity environment.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] High-precision target detection and recognition: This invention employs deep learning-based target detection and semantic segmentation technologies, enabling the robot to accurately identify various key targets (such as equipment, tools, and personnel) in complex cabin environments and effectively distinguish between operable areas and obstacles. Through an improved deep neural network model, the system can quickly identify target locations and categories, adapt to diverse object and structure distributions in microgravity environments, and meet the perception requirements of high precision and rapid response.

[0053] Reliable Pose Estimation: For complex operational tasks, this invention designs a pose estimation method based on visual and laser sensing data, enabling the robot to grasp the accurate spatial position and orientation of the target object in real time, providing support for subsequent precise operations. This pose estimation algorithm, through feature extraction and matching of multimodal data, ensures that the robot can obtain reliable target pose information in dynamic and confined spaces, and adapt to environmental changes in real time.

[0054] Comprehensive Scene Understanding and Dynamic Perception: Combining object detection, semantic segmentation, and obstacle detection technologies, this invention enables robots to possess a comprehensive understanding of their surrounding environment. A front-facing obstacle avoidance binocular camera can identify the locations of obstacles and people in the space, generating real-time risk warnings to assist the robot in flexibly adjusting in dynamic environments. A rear-facing navigation binocular camera and other sensing devices work together to give the robot a deeper understanding of the scene during tasks, enabling it to quickly respond to subtle changes in the environment and ensuring efficient and safe task execution in narrow and complex microgravity environments.

[0055] In summary, the environmental perception and scene understanding method provided by this invention significantly improves the robot's ability to identify and understand in microgravity environments. Compared with traditional methods, it has higher detection accuracy and stronger adaptability, and has important engineering application value and broad practical application prospects. Attached Figure Description

[0056] Figure 1 This is a flowchart of the intelligent flying robot perception method in an embodiment of the present invention;

[0057] Figure 2 This is a diagram of the perception network architecture in an embodiment of the present invention;

[0058] Figure 3 This is a graph showing the change in mAP during the target detection training process in an embodiment of the present invention.

[0059] Figure 4 This is a graph showing the change in loss during the training process of the target detection network in an embodiment of the present invention.

[0060] Figure 5 This is a graph showing the change in loss during the image segmentation training process in an embodiment of the present invention. Detailed Implementation

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0062] To meet the demands of intelligent flying robots for precise perception and flexible decision-making in complex microgravity environments, this invention proposes an advanced environmental perception and scene understanding method. Unlike traditional single-sensor or simple visual detection schemes, this invention significantly enhances the robot's perception capabilities in complex dynamic environments through the collaborative action of multimodal perception devices, combined with optimized deep learning algorithms and multimodal data fusion technology. This method not only demonstrates superior performance in the confined spaces of microgravity environments but also exhibits strong adaptability and robustness, enabling flexible responses to dynamic changes and unpredictable risk factors in various complex scenarios. Through precise processing of environmental information using deep learning models, the system can identify key target areas in real time, assess obstacle distribution, and efficiently execute operational tasks. Furthermore, to adapt to the special working requirements of microgravity environments, this invention achieves efficient perception software algorithms through multi-sensor network fusion, greatly reducing energy consumption and hardware load, ensuring real-time operation under limited resource conditions. Based on this perception framework, the robot can not only effectively complete routine inspections and auxiliary operations but also quickly adjust its strategy in emergency situations, ensuring efficient and safe operation in microgravity environments.

[0063] Combination Figure 1As shown, this is the composition of the omnidirectional obstacle perception hardware system. The robot body is equipped with a front-facing binocular camera, a rear-facing binocular camera, and a laser distance sensor. These perception devices work together for scene understanding and environmental modeling tasks. Through the collaborative work of multiple sensors, the system can analyze the surrounding environment in real time, assisting the robot in making autonomous decisions and operating safely in complex microgravity environments. In terms of environmental understanding, the robot constructs a comprehensive understanding of the microgravity environment through neural networks. The front-facing binocular camera and laser distance sensor can scan the surrounding area in real time, capturing the position, shape, and distance information of obstacles, thereby generating accurate 3D environmental perception information. The rear-facing binocular camera assists in localization and path planning, ensuring the robot navigates accurately in the environment. The collaborative work of these devices provides the robot with a deep understanding of the environment, enabling it to identify key objects, judge spatial layout, and dynamically adapt to environmental changes. The fusion of multiple sensors further improves the robot's environmental perception accuracy, enabling it to comprehensively assess different environmental factors.

[0064] Specific Implementation Plan 1: Combining Figures 1 to 5 As shown, this invention provides a method for perception and scene understanding of a flying robot in a microgravity environment, comprising the following steps:

[0065] S100 Multi-sensor System Model Establishment and Data Fusion: By integrating the onboard front-view binocular camera, rear-view binocular camera, and laser distance sensor, a multi-sensor network is modeled. The front-view binocular camera collects structured light point cloud and binocular image data in front of the robot, providing accurate obstacle and dynamic object detection information. The rear-view binocular camera is used to acquire stereo vision information from behind, enhancing overall environmental perception. The laser distance sensor provides additional point cloud information and depth data, which, through fusion with image data, form a high-precision multimodal perception foundation.

[0066] This omnidirectional obstacle perception hardware system comprises various sensing devices that work together to perform scene understanding and environmental modeling tasks. Through the collaborative operation of multiple sensors, the system can analyze the surrounding environment in real time, assisting the robot in making autonomous decisions and operating safely in complex microgravity environments. In terms of environmental understanding, the robot constructs a comprehensive understanding of the microgravity environment through neural networks. The forward-facing binocular camera and laser distance sensor can scan the surrounding area in real time, capturing the position, shape, and distance information of obstacles, thereby generating accurate 3D environmental perception information. The rear-facing binocular camera assists in localization and path planning, ensuring the robot navigates accurately in the environment. The collaborative work of these devices provides the robot with a deep understanding of the environment, enabling it to identify key objects, determine spatial layout, and dynamically adapt to environmental changes. The fusion of multiple sensors further improves the robot's environmental perception accuracy, enabling it to comprehensively assess different environmental factors.

[0067] Specifically, it includes the following steps:

[0068] S110, Sensor Data Acquisition: RGB images and depth data are acquired through a forward-looking binocular camera, a laser distance sensor, and a rear-looking binocular camera, respectively; these data contain key information about obstacles, target objects, and spatial layout in the surrounding environment;

[0069] For the target, the depth data from the lidar (laser distance sensor) Depth estimation with forward-looking binocular camera A unified depth estimate d(x,y) is generated through weighted fusion:

[0070]

[0071] Where (x,y) represents the image pixels; These are the weights of the depth data; here we take... ;

[0072] S120. Data Alignment and Enhancement: To ensure the consistency of multimodal data, the system synchronously calibrates RGB images and laser ranging data; in the preprocessing stage, the input data quality is improved by increasing image contrast, laying a foundation for subsequent detection and segmentation; the camera coordinate system is set as follows: The coordinate system is a right-handed coordinate system established with the camera installation position as the origin and the z-axis directly in front of the camera. The lidar coordinate system is... If we establish a right-handed coordinate system with the radar installation location as the origin and the area directly above the radar as the z-axis, then the following rigid body transformation relationships apply:

[0073]

[0074] in, This represents the position of the target in the camera coordinate system at time t. This indicates the position of the target in the lidar coordinate system at time t. This represents the transformation matrix from the radar coordinate system to the camera coordinate system;

[0075] In terms of semantic segmentation, the image is divided into different semantic categories at the pixel level to clearly identify obstacles and passable areas in the scene. For pose estimation, the PointNet++ deep learning method is used, combined with data from a laser rangefinder sensor, to estimate the pose of the target object through the fusion of point cloud and visual information. Figure 2The network structure diagram shown divides the specific perception task of robot scene understanding into three parts, which are implemented sequentially through distributed training. Specifically, firstly, specific datasets are collected for different application scenarios of different tasks. Then, different network structures are designed according to different tasks. For image feature extraction, a ResNet50 structure is uniformly used, and for pose estimation, PointNet++ is used to extract point cloud features to assist in pose estimation. Finally, three network models, including an object detection network, an image segmentation network, and a pose estimation network, are trained in a distributed manner and integrated to achieve efficient robot perception. The perception method proposed in this invention can complete specific recognition, segmentation, and pose estimation tasks. Let the loss function of each network be as follows: , and This invention employs a distributed training method to optimize the loss values ​​for different tasks.

[0076] S200. Target Detection and Semantic Segmentation Based on Deep Learning: This invention employs improved deep learning models (including improved YOLO and Mask R-CNN) for real-time target detection and semantic segmentation in environmental perception. The improved YOLO network can quickly detect key targets in complex backgrounds, achieving accurate identification of devices, objects, and obstacles. Simultaneously, the Mask R-CNN semantic segmentation algorithm divides image pixels into different semantic categories (such as passable areas, obstacles, and instruments), ensuring a comprehensive understanding of spatial layout. Through data fusion, it can identify the target's location, category, and role in the scene in real time, providing effective support for autonomous decision-making.

[0077] Specifically, it includes the following steps:

[0078] S210, Real-time object detection, including lightweight object detection model construction and obstacle and key target identification.

[0079] S211, Lightweight Object Detection Model Construction: In the object detection stage, an optimized YOLO model is used to quickly detect key objects and obstacles in the microgravity environment; we introduce a lightweight convolutional neural network structure to reduce the computational burden and ensure that the detection model runs efficiently with limited hardware resources; Figure 3 The graph shows the changes in the mAP curve during the training of the object detection model. Figure 4 The graph shows the decrease in the total loss of the object detection model during training. As the number of training epochs increases, the recognition accuracy gradually improves. The loss function of the model for object detection is also shown. Taking classification error into account Bounding box regression error Error with target confidence level The definition is as follows:

[0080]

[0081] in, For classification error weights, Bounding box regression error weights, The target confidence error weights are all set to 1 here;

[0082] S212, Obstacle and Key Target Recognition: The model can efficiently identify obstacles, passable areas and operational targets (such as handrails and equipment interfaces), and update the detection results in real time, providing basic environmental information for robot navigation and operation;

[0083] Laser distance sensors, combined with information captured by cameras, can effectively improve depth information of different types of obstacles and update environmental information in real time. Through this sensory data, the robot can not only obtain precise distance information of key obstacles but also identify the movement trends and potential paths of dynamic targets. This environmental understanding capability ensures that the robot can make autonomous decisions, avoid potential risks, and stably execute tasks in complex microgravity environments; dynamic target speed... Detection can be achieved through inter-frame motion estimation:

[0084]

[0085] in, It is the change in the position of the target object. It is the time interval between frames;

[0086] S220, Semantic Segmentation and Region Recognition, including constructing a task-adaptive semantic segmentation model and fusing segmented regions with detection results.

[0087] S221. Task-Adaptive Semantic Segmentation Model: To adapt to specific task requirements in microgravity environments (such as identifying the location of handrails, equipment, or tools), we introduce transfer learning on top of a conventional segmentation network. We further train the segmentation model using a specific dataset to improve its accuracy in identifying specific regions. Figure 5 The image segmentation network training process is illustrated with loss descent graphs; the model is initialized using parameters pre-trained on a large-scale dataset (COCO). Subsequently, in the task-related dataset Fine-tuning training was conducted on top of that.

[0088] Obtain the updated model weight parameters :

[0089]

[0090] in, Represents the updated model weight parameters. Represents the current model weight parameters. The weight of the regularization term is taken here. N is the total number of pixels, and C is the number of categories. This represents the true category label of the i-th pixel. This represents the probability that the i-th pixel in the model output belongs to class c;

[0091] S222, Fusion of Segmentation Regions and Detection Results: The semantic segmentation module performs pixel-level segmentation on the RGB image, dividing it into passable areas, obstacles, and key information regions of target objects; the segmentation results are combined with the output of object detection to generate a more complete environment model, enabling the robot to more accurately judge the layout of its surrounding environment;

[0092] S300. High-precision positioning based on pose estimation: Utilizing the results of target detection and semantic segmentation, this invention employs pose estimation technology to determine the precise position and orientation of an object in real time. Through feature matching and image- and point cloud-based depth estimation algorithms, combined with depth data, and using the Iterative Closest Point (ICP) method to complete pose calculation, the position and orientation of the target in three-dimensional space are obtained. Through this step, the robot can accurately acquire the spatial information of the object being manipulated, supporting precise positioning and manipulation of subsequent tasks.

[0093] Specifically, including,

[0094] S310, Multimodal Data Fusion: To ensure the accuracy of pose estimation, this invention fuses laser ranging data with visual detection results; by introducing the collaborative processing of point cloud and visual information, the pose estimation's depth perception capability of the target object is enhanced.

[0095] Let the point cloud provided by the laser sensor be... Each point Representing spatial coordinates, the set of detection boxes in the image is... The fusion strategy is as follows:

[0096]

[0097] in, Let m represent the m-th point, and k represent the total number of points. Let represent the nth detection box, and l represent the total number of detection boxes. This represents the camera intrinsic projection function that projects 3D points onto the image plane; through this operation, local point cloud regions matching the detection bounding box are extracted. , used for subsequent pose estimation;

[0098] S320. Pose Estimation and Dynamic Tracking: Referring to the classic PointNet++ method, a pose estimation method integrating laser distance information is designed. This method can acquire the precise spatial position and orientation of the target object in real time, ensuring the robot can adjust to the optimal pose in a timely manner during operation. Simultaneously, the system continuously updates the pose of the detected target object, enabling the robot to accurately track the target in dynamic environments. The network training objective is to minimize the predicted position t, rotation q, and ground truth annotation. , Weighted loss function between:

[0099]

[0100] in, and These are the weighting coefficients for the translation and rotation components, respectively. Represents the dot product operation for quaternions;

[0101] S400, route planning and real-time adjustment, including feasible area identification and route planning, route adjustment and task execution.

[0102] S410. Feasible Region Identification and Path Planning: Utilizing the environment model generated from the segmentation results, the system can identify feasible regions in real time. Combining feedback from the binocular camera and laser rangefinder, the system plans the optimal path, enabling the robot to efficiently avoid obstacles and quickly reach the target location. The category set is defined as follows: Let the c-th class be the passable area, then the passable area mask is... for:

[0103]

[0104] in, Represents the category of the region to which the pixel (x, y) belongs;

[0105] S420, Path Adjustment and Task Execution: During path execution, the system monitors environmental changes in real time and continuously adjusts the path based on the latest passable area perception information to ensure that the robot can respond quickly when encountering dynamic obstacles or spatial changes; the close integration of path planning and environmental perception enables the robot to be adaptive and flexibly respond to changes in the cabin environment.

[0106] Specific implementation scheme two: The present invention provides a microgravity environment flight robot perception and scene understanding system. The system has a program module corresponding to the above steps, and executes the steps in the above microgravity environment flight robot perception and scene understanding method when running.

[0107] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0108] Specific Implementation Scheme 2: The present invention provides a computer-readable storage medium storing a computer program configured to implement, when called by a processor, the steps of a method for perception and scene understanding of a flying robot in a microgravity environment.

[0109] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0110] The technical achievements of this invention were validated on publicly available datasets. The accuracy of object detection, image segmentation, and pose estimation were validated on the YCB-Video dataset, with the following metrics: Object detection performance was measured using the mean accuracy (mAP), with a higher mAP indicating higher detection accuracy. Image segmentation was evaluated using the mean intersection-over-union (mIoU), with a higher mIOU indicating higher segmentation accuracy. Pose estimation accuracy was measured using the 5cm / 5° success rate, i.e., the proportion of samples where the positional error between the predicted pose and the true pose is less than 5cm and the angle error is less than 5°.

[0111] Table 1: Multi-task performance evaluation results on the YCB-Video dataset

[0112]

[0113] The results above show that the method of the present invention exhibits excellent performance in the three key tasks of target detection, image segmentation and pose estimation, verifying its effectiveness and adaptability in multimodal perception tasks, and effectively fulfilling the intelligent perception and scene understanding task requirements of intelligent flying robots in microgravity environments.

[0114] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for perception and scene understanding of a flying robot in a microgravity environment, characterized in that, Includes the following steps: S100, Multi-sensor system model establishment and data fusion: By integrating the airborne forward-looking binocular camera, rear-looking binocular camera and laser distance sensor, a multi-sensor network is modeled. S200, Deep Learning-Based Object Detection and Semantic Segmentation, including real-time object detection. It employs an improved YOLO network for rapid detection of key targets. The loss function is constructed by comprehensively considering classification error, bounding box regression error, and target confidence error, enabling accurate identification of devices, objects, and obstacles. Based on the Mask R-CNN semantic segmentation algorithm, transfer learning is introduced to classify image pixels into different semantic categories. Through data fusion, the system identifies the target location, category, and its role in the scene in real time. Specifically, including, S210, Real-time object detection, including lightweight object detection model construction and obstacle and key target identification. S211. Lightweight target detection model construction: In the target detection stage, an optimized YOLO model is used to detect key objects and obstacles in the microgravity environment, and a lightweight convolutional neural network structure is introduced. Loss function for model object detection Taking classification error into account Bounding box regression error Error with target confidence level The definition is as follows: in, For classification error weights, Bounding box regression error weights, Weights for the target confidence error; S212, Obstacle and Key Target Identification: The lightweight target detection model constructed in step S211 identifies obstacles, passable areas, and operational targets, and updates the detection results in real time. Speed ​​of dynamic targets Achieved through inter-frame motion estimation: in, It is the change in position of the target object. It is the time interval between frames; S220, Semantic Segmentation and Region Recognition, including constructing a task-adaptive semantic segmentation model and fusing segmented regions with detection results. S221, a task-adaptive semantic segmentation model, introduces transfer learning based on the Mask R-CNN semantic segmentation algorithm, and initializes parameters pre-trained on a large-scale dataset. Subsequently, in the task-related dataset Fine-tuning training is performed on the model to obtain updated model weight parameters. : in, Represents the updated model weight parameters. Represents the current model weight parameters. Here, C represents the weight of the regularization term, where N is the total number of pixels and C is the number of categories. This represents the true category label of the i-th pixel. This represents the probability that the i-th pixel in the model output belongs to class c; S222. The segmentation region and detection results are fused together. The segmentation results are combined with the output of target detection to generate an environment model, enabling the intelligent flying robot to more accurately judge the layout of its surrounding environment. S300, a high-precision positioning system based on pose estimation, utilizes the results of target detection and semantic segmentation for multimodal data fusion. It employs a PointNet++ network model to extract point cloud features for pose estimation, thereby determining the precise position and orientation of objects in real time. Through feature matching and image- and point cloud-based depth estimation algorithms, combined with depth data, and by using the iterative nearest point method, it completes pose calculation, thus obtaining the position and orientation of the target in three-dimensional space. Specifically, including, S310, multimodal data fusion, introduces collaborative processing of point cloud and visual information to enhance the depth perception capability of pose estimation for target objects; assuming the point cloud provided by the laser sensor is... Each point Representing spatial coordinates, the set of detection boxes in the image is... The fusion strategy is as follows: in, Let m represent the m-th point, and k represent the total number of points. Let represent the nth detection box, and l represent the total number of detection boxes. This represents the camera intrinsic projection function that projects 3D points onto the image plane; through this operation, local point cloud regions matching the detection bounding box are extracted. , used for subsequent pose estimation; S320, Pose Estimation and Dynamic Tracking: This section utilizes a PointNet++ network model to extract point cloud features for pose estimation, obtaining the precise spatial position and orientation of the target object in real time. This ensures the robot can adjust to the optimal pose promptly during operation. Simultaneously, it continuously updates the pose of detected target objects, enabling the robot to accurately track targets in dynamic environments. The network training objective is to minimize the predicted position t, pose q, and ground truth values. , Weighted loss function between: in, and These are the weighting coefficients for the translation and rotation components, respectively. Represents the dot product operation of quaternions; S400, path planning and real-time adjustment, including feasible area identification and path planning, path adjustment and task execution, enable intelligent flying robots to respond quickly when encountering dynamic obstacles or spatial changes.

2. The method for perception and scene understanding of a microgravity environment flying robot according to claim 1, characterized in that: Step S100 specifically includes, S110, Sensor data acquisition: RGB images and depth data are acquired through a forward-looking binocular camera, a laser distance sensor, and a rear-looking binocular camera, respectively. Among them, the depth data from the laser distance sensor Depth data estimated visually by forward-looking binocular cameras A unified depth estimate d(x,y) is generated through weighted fusion: Where (x,y) represents the image pixels; These are the weights of the depth data; S120, Data Alignment and Enhancement: Synchronous calibration of RGB image and laser ranging data; assuming the camera coordinate system is... The lidar coordinate system is The transformation relationship is as follows: in, This represents the position of the target in the camera coordinate system at time t. This indicates the position of the target in the lidar coordinate system at time t. This represents the transformation matrix from the radar coordinate system to the camera coordinate system.

3. The method for perception and scene understanding of a microgravity environment flying robot according to claim 2, characterized in that: Combining steps S200 and S300, three network models are trained in a distributed manner: an object detection network, an image segmentation network, and a pose estimation network. Let the loss functions of the lightweight object detection model, the adaptive semantic segmentation model, and the pose estimation network model be respectively... , and Distributed optimization , and .

4. The method for perception and scene understanding of a microgravity environment flying robot according to claim 3, characterized in that: Step S400 specifically includes, S410. Feasible Region Identification and Path Planning: Utilizing the environment model generated from the segmentation results, feasible regions are identified in real time. Combining feedback from the binocular camera and laser rangefinder, the optimal path is planned, enabling the robot to efficiently avoid obstacles and quickly reach the target location. Let class c be the traversable region, then the traversable region mask... for: in, The (x, y) value of a pixel represents the category of the region it belongs to. S420, Path Adjustment and Task Execution: During path execution, the robot monitors environmental changes in real time and continuously adjusts the path based on the latest peer area perception information to ensure that the robot can respond quickly when encountering dynamic obstacles or spatial changes.

5. A perception and scene understanding system for a microgravity environment flying robot, characterized in that: The system has a program module corresponding to the steps described in any one of claims 1-4 above, and executes the steps in the above-described microgravity environment flight robot perception and scene understanding method when running.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the microgravity environment flight robot perception and scene understanding method according to any one of claims 1-4 when invoked by a processor.

Images