Microgravity environment flying robot sensing and scene understanding method and system

Abstract

The invention provides a microgravity environment flying robot sensing and scene understanding method and system, and belongs to the field of robot intelligent sensing and scene understanding. The problems that in an indoor working scene under the microgravity environment, facility layout is crowded, space illumination is complex, and floating objects are blocked, so that image feature extraction is difficult, the sight line of a sensor is blocked, and the three-dimensional environment modeling precision and the target recognition accuracy are affected are solved. A lightweight convolutional neural network structure is adopted, so that the calculation burden is reduced; rGB images and data of a laser ranging sensor are combined in the aspect of multi-modal data fusion, a multi-modal information fusion module is added in target detection and pose estimation, and the method is suitable for recognition and positioning of a target object needed by high-precision operation; in the aspect of task adaptive semantic segmentation, aiming at a specific task scene in a microgravity environment, performing network training based on a specific data set; and through a transfer learning mode, the model has higher adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot intelligent perception and scene understanding, in particular to a microgravity environment flying robot perception and scene understanding method and system. BACKGROUND

[0002] With the continuous advancement of China's manned space engineering and deep space exploration missions, the working tasks in the microgravity environment are showing a diversified development trend. Astronauts not only need to undertake technical work such as microgravity scientific experiments and material processing, but also need to regularly carry out equipment inspection, fault diagnosis and other daily maintenance work. In order to improve the work efficiency of astronauts and reduce the work load, it is of great significance to introduce space intelligent flying robots with autonomous operation capability. Such robots can independently complete tasks such as scientific experiment load monitoring, equipment state inspection and daily maintenance in the absence of astronauts or when it is inconvenient for astronauts to operate, which is of great significance to ensure the long-term stable operation of the microgravity working environment and the efficient performance of space test tasks.

[0003] However, the increasing diversity and complexity of space tasks in the microgravity environment pose unprecedented challenges to the autonomous perception and decision-making capabilities of space flying robots. In the microgravity environment, indoor working scenes have the characteristics of irregular structure height, dense facility layout, variable lighting conditions and dynamic floating interference, which leads to significant shortcomings in the completeness of information acquisition and the robustness of algorithms of traditional environment perception technology, making it difficult to meet the real-time and reliability requirements of autonomous operation in a dynamic space environment.

[0004] Environment perception and scene understanding are the core technologies for robots to achieve autonomous operation, and the key lies in constructing an accurate environment cognition model in real time and forming the ability to identify obstacles, understand regional functions, locate operation targets and plan paths based on it. Although target detection, semantic segmentation and depth estimation based on deep learning have made significant progress in ground environments, their direct application in microgravity environments still faces many challenges. The lighting conditions in space environments are complex, with strong reflections and interlaced shadows, which can easily lead to feature loss of RGB cameras; the indoor working area has dense equipment layout, and there are floating objects and equipment personnel movements, which interfere with the field of view of depth sensors and laser sensors. These factors make it difficult for traditional single-sensor perception schemes to meet the task requirements in terms of environment modeling accuracy and target recognition reliability.

[0005] To solve the above problems, the present application innovatively proposes an intelligent perception and scene understanding technology based on multi-sensor fusion, which can enable robots to realize real-time three-dimensional environment perception in microgravity and identify key objects and passable areas, thereby greatly improving the autonomy and environmental adaptability of robots and providing technical support for future deep space exploration missions. SUMMARY

[0006] The technical problem solved by the present application is:

[0007] In order to solve the problems of crowded facility layout, complex space lighting and floating object shielding in indoor working scene under microgravity environment, which lead to difficult image feature extraction and blocked sensor line of sight, affecting the accuracy of three-dimensional environment modeling and target recognition.

[0008] The technical scheme adopted by the present application to solve the above technical problem is:

[0009] The present application provides a microgravity environment flying robot perception and scene understanding method, comprising the following steps:

[0010] S100, multi-sensor system model establishment and data fusion, by integrating the on-board forward-looking binocular camera, rear-view binocular camera and laser distance sensor, modeling a multi-sensor network;

[0011] S200, target detection and semantic segmentation based on deep learning, including real-time target detection, using the improved YOLO network to quickly detect key targets, using the loss function of the model target detection, considering the classification error, boundary box regression error and target confidence error to construct the loss function, realizing accurate identification of equipment, objects and obstacles; introducing transfer learning based on MaskRCNN semantic segmentation algorithm, dividing the image pixels into different semantic categories; through data fusion, real-time identification of target position, category and role in the scene;

[0012] S300, high-precision positioning based on pose estimation, using the results of target detection and semantic segmentation for multi-modal data fusion, using the PointNet++ network model to extract point cloud features for pose estimation, real-time determination of the accurate position and orientation of the object; through feature matching and depth estimation algorithm based on image and point cloud, combining depth data, and completing pose calculation through iterative closest point method, so as to obtain the position and attitude of the target in three-dimensional space;

[0013] S400, path planning and real-time adjustment, including feasible area discrimination and path planning, path adjustment and task execution, so that the intelligent flying robot can quickly respond when encountering dynamic obstacles or space changes.

[0014] Further, in step S100, specifically comprising,

[0015] S110, sensor data acquisition, acquiring RGB images and depth data through forward-looking binocular camera, laser distance sensor and rear-view binocular camera respectively;

[0016] Wherein, the depth data of the laser distance sensor Depth data estimated from forward-looking binocular camera vision A unified depth estimate d(x,y) is generated by weighted fusion:

[0017]

[0018] where (x,y) represents an image pixel point; is the weight of the depth data;

[0019] S120, data alignment and enhancement, synchronously calibrate the RGB image and the laser ranging data; let the camera coordinate system be , the laser radar coordinate system be , and the transformation relationship be:

[0020]

[0021] wherein, represents the position of the target in the camera coordinate system at time t, represents the position of the target in the laser radar coordinate system at t, represents the transformation matrix from the radar coordinate system to the camera coordinate system.

[0022] Further, in step S200, specifically comprising,

[0023] S210, real-time target detection, including lightweight target detection model construction and obstacle and key target identification,

[0024] S211, lightweight target detection model construction, in the target detection stage, an optimized YOLO model is used to detect key objects and obstacles in the microgravity environment, and a lightweight convolutional neural network structure is introduced; the loss function of the model target detection is considering the classification error , the bounding box regression error and the target confidence error , which are defined as follows:

[0025]

[0026] wherein, is the classification error weight, is the bounding box regression error weight, is the target confidence error weight;

[0027] S212, obstacle and key target identification, the lightweight target detection model constructed through step S211 identifies obstacles, passable areas and operation targets, and updates the detection results in real time;

[0028] the speed of the dynamic target is Realized by inter-frame motion estimation:

[0029]

[0030] wherein, is the position change value of the target object, is the time interval between frames;

[0031] S220, semantic segmentation and region recognition, including constructing a task-adaptive semantic segmentation model and fusing segmented regions and detection results,

[0032] S221, task-adaptive semantic segmentation model, introducing transfer learning based on MaskRCNN semantic segmentation algorithm, initializing with pre-trained parameters on a large-scale dataset , then fine-tuning training on task-related dataset to obtain updated model weight parameters :

[0033]

[0034] wherein, represents the updated model weight parameters, represents the current model weight parameters, is the regularization weight, N is the total number of pixels, and C is the number of categories, indicates the true class label of the i-th pixel, indicates the probability that the i-th pixel output by the model belongs to category c;

[0035] S222, fusion of segmented regions and detection results, combining the segmentation results with the output of target detection to generate an environment model, enabling the intelligent flying robot to more accurately judge the layout of its surrounding environment.

[0036] Further, in step S300, specifically including,

[0037] S310, multi-modal data fusion, introducing collaborative processing of point cloud and visual information to enhance the depth perception ability of pose estimation for target objects; let the point cloud provided by the laser sensor be , each point represents the spatial coordinates, and the set of detection boxes in the image is ; the fusion strategy is as follows:

[0038]

[0039] wherein, represents the m-th point, and k represents the total number of points, represents the n-th detection box, and l represents the total number of detection boxes, a camera intrinsic projection function representing projecting a three-dimensional point to an image plane; by this operation, a local point cloud region matching the detection box is extracted for subsequent pose estimation;

[0040] S320, pose estimation and dynamic tracking, using the PointNet++ network model to extract point cloud features for pose estimation, real-time acquisition of the accurate spatial position and orientation of the target object, to ensure that the robot can adjust to the optimal pose in time when operating; at the same time, continuously update the pose of the detected target object, so that the robot can accurately track the target in a dynamic environment; the network training target is to minimize the weighted loss function between the predicted position t and the real label 、

[0041]

[0042] wherein, and are the weighting coefficients of the translation and rotation parts, denotes the dot product operation of the quaternion.

[0043] Further, in combination with steps S200 and S300, three network models including target detection network, image segmentation network and pose estimation network are trained in a distributed manner, and the loss functions of the lightweight target detection model, adaptive semantic segmentation model and pose estimation network model are respectively , and , and .

[0044] Further, in step S400, specifically comprising,

[0045] S410, feasible region discrimination and path planning, using the environment model generated by the segmentation result to discriminate the feasible region in real time; combined with the feedback of the binocular camera and the laser ranging sensor, the best path is planned to make the robot can efficiently avoid obstacles and quickly reach the target position; define the category set as Let the c-th category be a passable region, then the passable region mask is:

[0046]

[0047] wherein, represents the category of the region to which the pixel point (x, y) belongs;

[0048] ​​S420, path adjustment and task execution, in the path execution process, real-time monitoring of environmental changes and based on the latest peer area perception information constantly adjust the path, ensure that the robot can quickly respond when encountering dynamic obstacles or space changes.

[0049] A microgravity environment flying robot perception and scene understanding system has a program module corresponding to the above steps, which executes the steps of the microgravity environment flying robot perception and scene understanding method described above.

[0050] A computer readable storage medium stores a computer program, the computer program is configured to be called by the processor to realize the steps of the microgravity environment flying robot perception and scene understanding method.

[0051] Compared with the prior art, the beneficial effects of the present application are:

[0052] High-precision target detection and recognition: the present application adopts deep learning-based target detection and semantic segmentation technology, enabling the robot to accurately identify a variety of key targets (such as equipment, tools and personnel) in a complex cabin environment, and effectively distinguish between operable areas and obstacles. Through the improved deep neural network model, the system can quickly identify the target position and category, adapt to the diversified object and structure distribution in the microgravity environment, and meet the high-precision and fast-response perception requirements.

[0053] Reliable pose estimation: for complex operation tasks, the present application designs a pose estimation method based on visual and laser sensing data, enabling the robot to accurately grasp the spatial position and orientation of the target object in real time, providing support for subsequent precise operations. The pose estimation algorithm extracts and matches the features of multi-modal data, ensuring that the robot can obtain reliable target pose information in dynamic and narrow spaces, and adapt to environmental changes in real time.

[0054] Comprehensive scene understanding and dynamic perception: combining target detection, semantic segmentation and obstacle detection technology, the present application enables the robot to have a comprehensive understanding of the surrounding environment. The front obstacle avoidance binocular camera can identify the positions of obstacles and personnel in the space, generate real-time risk prompts, and assist the robot in adjusting flexibly in dynamic environments. The rear navigation binocular camera and other perception devices cooperate with each other, enabling the robot to have a deeper understanding of the scene during the task, quickly respond to subtle changes in the environment, and ensure that it can efficiently and safely execute tasks in narrow and complex microgravity environments.

[0055] In summary, the environmental perception and scene understanding method provided by the present application significantly improves the recognition and understanding ability of the robot in the microgravity environment, has higher detection accuracy and stronger adaptability compared with traditional methods, and has important engineering application value and broad practical application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the intelligent flying robot perception method according to an embodiment of the present invention;

[0057] Figure 2 This is a diagram of the perception network architecture in an embodiment of the present invention;

[0058] Figure 3 This is a graph showing changes in mAP during the target detection training process in an embodiment of the present invention;

[0059] Figure 4 A curve diagram showing the change in loss during the target detection network training process in an embodiment of the present invention;

[0060] Figure 5 2 is a graph showing the change in loss during the image segmentation training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] To meet the demands for precise perception and flexible decision-making in complex microgravity environments for intelligent flying robots, this paper proposes an advanced environmental perception and scene understanding method. Unlike traditional single-sensor or simple visual detection solutions, this method significantly enhances the robot's perception of complex and dynamic environments by leveraging the synergy of multimodal sensing devices, combined with optimized deep learning algorithms and multimodal data fusion technology. This method not only demonstrates excellent performance in the confined spaces of microgravity but also exhibits strong adaptability and robustness, enabling it to flexibly respond to dynamic changes and unpredictable risk factors in a variety of complex scenarios. Through the precise processing of environmental information by a deep learning model, the system can identify key target areas in real time, assess obstacle distribution, and efficiently execute operational tasks. Furthermore, to meet the unique operational requirements of microgravity, this paper implements efficient perception software algorithms through multi-sensor network fusion, significantly reducing energy consumption and hardware load, ensuring real-time operation under limited resource conditions. Based on this perception framework, the robot can not only effectively complete routine inspections and auxiliary operations, but also rapidly adjust its strategy in emergency situations, ensuring efficient and safe operations in microgravity.

[0063] Combine Figure 1As shown, the omnidirectional obstacle perception hardware system is composed of a front-looking binocular camera, a rear-looking binocular camera, and a laser distance sensor. These perception devices work together for scene understanding and environment modeling tasks. Through the cooperative work of multiple sensors, the system can analyze the surrounding environment in real time, assist the robot in making autonomous decisions and safely operating in a complex microgravity environment. In terms of environment understanding, the robot builds a comprehensive understanding of the microgravity environment through a neural network. The front-looking binocular camera and laser distance sensor can scan the surrounding area in real time, capture the position, shape, and distance information of obstacles, and generate accurate three-dimensional environment perception information. The rear-looking binocular camera assists in positioning and path planning to ensure accurate navigation of the robot in the environment. The cooperative work of these devices provides the robot with a deep understanding of the environment, enabling it to recognize key objects, judge spatial layout, and dynamically adapt to environmental changes. The fusion of multiple sensors further improves the environmental perception accuracy of the robot, enabling it to comprehensively evaluate different environmental factors

[0064] Specific implementation scheme one: combined Figures 1 to 5 As shown, the present application provides a microgravity environment flying robot perception and scene understanding method, comprising the following steps:

[0065] S100, multi-sensor system model establishment and data fusion, through the integration of on-board front-looking binocular camera, rear-looking binocular camera and laser distance sensor, modeling as a multi-sensor network; the front-looking binocular camera collects the structured light point cloud and binocular image data in front of the robot, providing accurate obstacle and dynamic object detection information; the rear-looking binocular camera is used to obtain the stereo vision information behind, enhancing the overall environmental perception; the laser distance sensor provides additional point cloud information and depth data, through the fusion with image data, forming a high-precision multi-modal perception basis;

[0066] The omnidirectional obstacle perception hardware system is composed of these perception devices working together for scene understanding and environment modeling tasks. Through the cooperative work of multiple sensors, the system can analyze the surrounding environment in real time, assist the robot in making autonomous decisions and safely operating in a complex microgravity environment. In terms of environment understanding, the robot builds a comprehensive understanding of the microgravity environment through a neural network. The front-looking binocular camera and laser distance sensor can scan the surrounding area in real time, capture the position, shape, and distance information of obstacles, and generate accurate three-dimensional environment perception information. The rear-looking binocular camera assists in positioning and path planning to ensure accurate navigation of the robot in the environment. The cooperative work of these devices provides the robot with a deep understanding of the environment, enabling it to recognize key objects, judge spatial layout, and dynamically adapt to environmental changes. The fusion of multiple sensors further improves the environmental perception accuracy of the robot, enabling it to comprehensively evaluate different environmental factors;

[0067] Specifically includes the following steps,

[0068] S110, sensor data acquisition: RGB images and depth data are acquired by front-facing binocular camera, laser distance sensor and rear-facing binocular camera respectively; these data contain key information of obstacles, target objects and spatial layout in the surrounding environment;

[0069] For the target, the depth data of the laser radar (laser distance sensor) and the depth estimated by the front-facing binocular camera vision Generate a unified depth estimate d(x, y) by weighted fusion:

[0070]

[0071] Where (x, y) represents the image pixel point; is the weight of depth data, here ;

[0072] S120, data alignment and enhancement: in order to ensure the consistency of multi-modal data, the system synchronously calibrates the RGB image and laser ranging data; in the preprocessing stage, the input data quality is improved by increasing the picture contrast, laying a good foundation for subsequent detection and segmentation; let the camera coordinate system be , which is a right-handed coordinate system with the camera installation position as the origin and the front of the camera as the z-axis, and the laser radar coordinate system be , which is a right-handed coordinate system with the radar installation position as the origin and the top of the radar as the z-axis, then there is the following rigid body transformation relationship:

[0073]

[0074] Where, represents the position of the target in the camera coordinate system at time t, represents the position of the target in the laser radar coordinate system at time t, represents the transformation matrix from the radar coordinate system to the camera coordinate system;

[0075] In terms of semantic segmentation, the image is divided into different semantic categories at the pixel level to clearly define the obstacles and passable areas in the scene; for pose estimation, based on the PointNet++ deep learning method, combined with the data of the laser ranging sensor, the pose of the target object is estimated by fusing point cloud and visual information; according to Figure 2The network structure diagram shown divides the specific perception tasks of robot scene understanding into three parts, and realizes the three tasks in turn through distributed training; specifically, first, specific data sets are collected for different application scenarios of the tasks, then different network structures are designed according to different tasks, ResNet50 structure is uniformly adopted for image feature extraction, and PointNet++ is adopted for extracting point cloud features to help pose estimation; finally, the three network models including the target detection network, the image segmentation network and the pose estimation network are trained in a distributed manner, and are comprehensively used to realize efficient perception of the robot; the perception method proposed in the application can complete specific recognition, segmentation and pose estimation tasks; the loss functions of each network are respectively , and , the distributed training method is adopted to optimize the loss values of different tasks respectively;

[0076] S200, target detection and semantic segmentation based on deep learning, in the environment perception, the improved deep learning model (including improved YOLO and MaskRCNN) is used for real-time target detection and semantic segmentation, the improved YOLO network can quickly detect key targets in a complex background, and realizes accurate identification of equipment, objects and obstacles; at the same time, the MaskRCNN semantic segmentation algorithm divides the image pixels into different semantic categories (such as passable area, obstacle, instrument), and ensures comprehensive understanding of the spatial layout; through data fusion, the target position, category and role in the scene can be identified in real time, which provides effective support for autonomous decision-making;

[0077] Specifically, the following steps are included,

[0078] S210, real-time target detection, including lightweight target detection model construction and obstacle and key target identification,

[0079] S211, lightweight target detection model construction: in the target detection stage, the optimized YOLO model is adopted to quickly detect key objects and obstacles in the microgravity environment; we introduce a lightweight convolutional neural network structure to reduce the computational burden and ensure that the detection model runs efficiently under limited hardware resources; Figure 3 The mAP curve change graph in the target detection model training process is shown, Figure 4 The total loss drop graph of the target detection model in the training process is shown, and the recognition accuracy gradually improves with the increase of the training round; the loss function of the model target detection comprehensively considers the classification error , the boundary box regression error and the target confidence error , which are defined as follows:

[0080]

[0081] wherein, is the classification error weight, is the bounding box regression error weight, is the target confidence error weight, which is 1 here;

[0082] S212, obstacle and key target recognition: the model can efficiently identify obstacles, passable areas and operation targets (such as handrails and device interfaces), and update the detection results in real time, providing basic environmental information for robot navigation and operation;

[0083] The laser distance sensor combined with the information captured by the camera can effectively improve the depth information of different types of obstacles and update the environmental information in real time; through these perception data, the robot can not only obtain accurate distance information of key obstacles, but also identify the motion trend and potential path of dynamic targets; this environmental understanding ability ensures that the robot can make autonomous decisions, avoid potential risks and stably execute tasks in a complex microgravity environment; the speed of dynamic targets The detection can be achieved through inter-frame motion estimation:

[0084]

[0085] wherein, is the position change value of the target object, is the time interval between frames;

[0086] S220, semantic segmentation and region recognition, including constructing a task-adaptive semantic segmentation model and fusing segmented regions with detection results,

[0087] S221, task-adaptive semantic segmentation model: in order to adapt to the specific task requirements in the microgravity environment (such as identifying handrails, equipment or tool positions), we introduce transfer learning based on the conventional segmentation network, further train the segmentation model through specific data sets, to improve its recognition accuracy for specific regions, Figure 5 The figure shows the loss reduction graph in the image segmentation network training process; the model initialization uses the pre-trained parameters on a large-scale data set (COCO) , then fine-tune the training on the task-related data set ,

[0088] get the updated model weight parameters :

[0089]

[0090] wherein, represent the updated model weight parameters, represent the current model weight parameters, is the regularization term weight, here , N is the total number of pixels, and C is the number of classes, represents the true class label of the i-th pixel, represents the probability that the i-th pixel output by the model belongs to class c;

[0091] S222, fusion of segmentation region and detection result: the semantic segmentation module performs pixel-level segmentation on the RGB image, and divides the passable region, obstacle, and target object key information region; the segmentation result is combined with the output of target detection to generate a more complete environment model, so that the robot can more accurately judge the surrounding environment layout;

[0092] S300, high-precision positioning based on pose estimation, using the results of target detection and semantic segmentation, the application uses pose estimation technology to determine the accurate position and orientation of the object in real time; through feature matching and depth estimation algorithm based on image and point cloud, combined with depth data, and through the Iterative Closest Point (ICP) method, the pose calculation is completed, so as to obtain the position and attitude of the target in three-dimensional space; through this step, the robot can accurately obtain the spatial information of the operation object, supporting the precise positioning and operation of the subsequent tasks;

[0093] Specifically, it includes,

[0094] S310, multi-modal data fusion: in order to ensure the accuracy of pose estimation, the application fuses laser ranging data and visual detection results; by introducing the cooperative processing of point cloud and visual information, the depth perception ability of pose estimation for target objects is enhanced;

[0095] Let the point cloud provided by the laser sensor be , and each point represents the spatial coordinates, and the set of detection boxes in the image is ; the fusion strategy is as follows:

[0096]

[0097] Among them, represents the m-th point, and k represents the total number of points, represents the n-th detection box, and l represents the total number of detection boxes, represents the camera intrinsic projection function for projecting three-dimensional points to the image plane; through this operation, the local point cloud region matching the detection box is extracted, which is used for subsequent pose estimation;

[0098] S320, pose estimation and dynamic tracking: referring to the classic method of PointNet++, a pose estimation method integrating laser distance information is designed, which can obtain the accurate spatial position and orientation of the target object in real time, ensuring that the robot can adjust to the optimal pose in time during operation; at the same time, the system continuously updates the pose of the detected target object, so that the robot can accurately track the target in a dynamic environment; the network training target is to minimize the weighted loss function between the predicted position t and the real label , :

[0099]

[0100] wherein, and are the weighting coefficients of the translation and rotation parts respectively, denotes the dot product operation of the quaternion;

[0101] S400, path planning and real-time adjustment, including feasible region discrimination and path planning, and path adjustment and task execution,

[0102] S410, feasible region discrimination and path planning: using the environment model generated by the segmentation result, the system can discriminate the feasible region in real time; combined with the feedback of the binocular camera and the laser ranging sensor, the system plans the best path, so that the robot can efficiently avoid obstacles and quickly reach the target position; define the category set as Let the c-th category be the passable region, then the passable region mask is:

[0103]

[0104] wherein, represents the category of the region to which the pixel point (x, y) belongs;

[0105] S420, path adjustment and task execution: during path execution, the system monitors the environmental changes in real time and continuously adjusts the path based on the latest passable region perception information, ensuring that the robot can quickly respond when encountering dynamic obstacles or spatial changes; path planning and environment perception are closely combined, so that the robot has adaptability and can flexibly cope with changes in the cabin environment.

[0106] Specific implementation scheme two: the microgravity environment flying robot perception and scene understanding system of the application has a program module corresponding to the above steps, and executes the steps in the microgravity environment flying robot perception and scene understanding method described above when running.

[0107] The other combinations and connection relationships of the present embodiment are the same as those of specific implementation scheme one.

[0108] Specific embodiment two: a computer readable storage medium of the application, the computer readable storage medium stores a computer program, the computer program is configured to realize the steps of the microgravity environment flying robot perception and scene understanding method when called by the processor.

[0109] Other combinations and connection relationships of the present embodiment are the same as specific embodiment one.

[0110] The technical achievements researched by the application are verified on public data sets. The target detection accuracy, image segmentation accuracy and pose estimation accuracy are verified on the YCB-Video data set respectively, and the related indexes are as follows: the target detection effect is measured by the average precision mean mAP, the larger the mAP is, the higher the detection precision is. The image segmentation task evaluation is evaluated by using the mean Intersection over Union (mIoU), the larger the mIOU is, the higher the segmentation precision is. The pose estimation accuracy is measured by the 5cm / 5° success rate, that is, the proportion of samples whose position error between the predicted pose and the real pose is less than 5cm and the angle error is less than 5°.

[0111] Table 1: Multi-task performance evaluation results on YCB-Video dataset

[0112]

[0113] From the above results, it can be seen that the method of the application shows excellent performance in the three key tasks of target detection, image segmentation and pose estimation, verifies the effectiveness and adaptability of the method in the multi-modal perception task, and better completes the intelligent perception and scene understanding task requirements of the intelligent flying robot in the microgravity environment.

[0114] Although the application discloses as above, the protection scope of the application disclosure is not limited to this. The person skilled in the art of the application can make various changes and modifications without departing from the spirit and scope of the application disclosure, and these changes and modifications will fall within the protection scope of the application.

Claims

1. A method for sensing and understanding scenes of a flying robot in a microgravity environment, characterized in that: The following steps are involved: S100, multi-sensor system model establishment and data fusion, by integrating the airborne forward-looking binocular camera, rear-looking binocular camera and laser range sensor to model a multi-sensor network; S200, deep learning-based object detection and semantic segmentation, including real-time object detection, using an improved YOLO network to quickly detect key targets. The model's target detection loss function is constructed by comprehensively considering classification error, bounding box regression error, and target confidence error to achieve accurate recognition of devices, objects, and obstacles. Transfer learning is introduced based on the MaskRCNN semantic segmentation algorithm to classify image pixels into different semantic categories. Through data fusion, the target's location, category, and role in the scene can be identified in real time. S300, high-precision positioning based on pose estimation, uses the results of target detection and semantic segmentation for multimodal data fusion, and uses the PointNet++ network model to extract point cloud features for pose estimation, determining the precise position and orientation of objects in real time; By combining feature matching and depth estimation algorithms based on images and point clouds, depth data is combined and pose calculation is completed using the iterative closest point method to obtain the position and pose of the target in three-dimensional space. S400, path planning and real-time adjustment, including feasible area identification and path planning, path adjustment and task execution, enables the intelligent flying robot to respond quickly when encountering dynamic obstacles or spatial changes.

2. The microgravity environment flying robot perception and scene understanding method according to claim 1, characterized in that: In step S100, specifically including: S110, sensor data acquisition, acquiring RGB images and depth data through the front-view binocular camera, laser distance sensor, and rear-view binocular camera respectively; Among them, the depth data of the laser distance sensor Depth data estimated by forward-looking stereo camera vision Generate a unified depth estimate d(x,y) through weighted fusion: ； Among them, (x, y) represents the image pixel; is the weight of the depth data; S120, data alignment and enhancement, synchronous calibration of RGB image and laser ranging data; set the camera coordinate system to , the laser radar coordinate system is , the transformation relationship is: ； in, represents the position of the target in the camera coordinate system at time t, represents the position of the target in the laser radar coordinate system at time t, Represents the transformation matrix from the radar coordinate system to the camera coordinate system.

3. The microgravity environment flying robot perception and scene understanding method according to claim 2, characterized in that: In step S200, specifically including: S210, real-time target detection, including lightweight target detection model construction and obstacle and key target identification, S211. Constructing a lightweight target detection model. In the target detection phase, an optimized YOLO model is used to detect key objects and obstacles in a microgravity environment, and a lightweight convolutional neural network structure is introduced. Loss function for model target detection Comprehensive consideration of classification error , bounding box regression error Confidence error with target , defined as follows: ； in, is the classification error weight, Bounding box regression error weight, is the target confidence error weight; S212: Identify obstacles and key targets. The lightweight target detection model constructed in step S211 is used to identify obstacles, traversable areas, and operational targets, and update the detection results in real time. Speed ​​of dynamic targets Through inter-frame motion estimation: ； in, is the position change value of the target object, is the time interval between frames; S220, semantic segmentation and region recognition, including building a task-adaptive semantic segmentation model and fusing segmentation regions with detection results. S221, task-adaptive semantic segmentation model, introduces transfer learning based on the MaskRCNN semantic segmentation algorithm, and initializes the parameters pre-trained on a large-scale dataset , then in the task-related dataset Fine-tune the training to get the updated model weight parameters : ； in, represents the updated model weight parameters, Represents the current model weight parameter, is the regular term weight, N is the total number of pixels, C is the number of categories, represents the true category label of the i-th pixel, Represents the probability that the i-th pixel output by the model belongs to category c; S222. The segmented area is integrated with the detection results. The segmentation results are combined with the output of the target detection to generate an environmental model, so that the intelligent flying robot can more accurately judge the layout of the environment around it.

4. The microgravity environment flying robot perception and scene understanding method according to claim 2, characterized in that: In step S300, specifically including: S310, multimodal data fusion, introduces the collaborative processing of point cloud and visual information to enhance the depth perception ability of pose estimation for target objects; suppose the point cloud provided by the laser sensor is , each point Represents the spatial coordinates, and the set of detection boxes in the image is ; The fusion strategy is as follows: ； in, represents the mth point, k represents the total number of points, represents the nth detection frame, l represents the total number of detection frames, Represents the camera intrinsic parameter projection function that projects a 3D point onto the image plane; through this operation, the local point cloud area that matches the detection box is extracted , used for subsequent pose estimation; S320, pose estimation and dynamic tracking, uses the PointNet++ network model to extract point cloud features for pose estimation, obtains the precise spatial position and orientation of the target object in real time, and ensures that the robot can adjust to the optimal pose in time during operation; at the same time, continuously updates the detected pose of the target object, enabling the robot to accurately track the target in a dynamic environment; the network training goal is to minimize the difference between the predicted position t, pose q and the actual annotation 、 The weighted loss function between: ； in, and are the weighted coefficients of the translation and rotation parts respectively, Represents a dot product operation on a quaternion.

5. A microgravity environment flying robot perception and scene understanding method according to claim 3 or 4, characterized in that: Combined with step S200 and step S300, three network models are trained in a distributed manner, including the target detection network, the image segmentation network and the pose estimation network. The loss functions of the lightweight target detection model, the adaptive semantic segmentation model and the pose estimation network model are respectively , and , distributed optimization , and .

6. The method for sensing and understanding a flying robot in a microgravity environment according to claim 5, characterized in that: In step S400, specifically including: S410, feasible area identification and path planning, uses the environmental model generated by the segmentation results to identify the feasible area in real time; combines the feedback from the binocular camera and the laser ranging sensor to plan the best path so that the robot can efficiently avoid obstacles and quickly reach the target location; define the category set as Assume that the c-th category is the passable area, then the passable area mask for: ； in, The representative pixel (x, y) represents the category of the area to which it belongs; S420, path adjustment and task execution. During the path execution process, the robot monitors environmental changes in real time and continuously adjusts the path based on the latest peer area perception information to ensure that the robot can respond quickly when encountering dynamic obstacles or spatial changes.

7. A microgravity environment flying robot perception and scene understanding system, characterized by: The system has a program module corresponding to the steps described in any one of claims 1 to 6 above, and executes the steps in the above-mentioned microgravity environment flying robot perception and scene understanding method when running.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the microgravity environment flying robot perception and scene understanding method according to any one of claims 1 to 6 when called by a processor.

Images