An autonomous exploration method for unmanned systems based on multi-view scene understanding

Abstract

The present invention relates to an autonomous exploration method for an unmanned system based on multi-view visual scene understanding. The method integrates the advantages of visible light and infrared images, and improves the adaptability of the unmanned system in adverse weather, illumination changes and communication-restricted scenarios. Pixel-level semantic segmentation is achieved through multimodal image fusion, lightweight target detection and background filtering based on depth thresholds, and the pose of the target of interest is accurately estimated. The exploration boundary is constructed using voxel maps and target detection results, and based on this, exploration decisions that take into account both exploration space and scene understanding are made to guide the unmanned system to perform autonomous exploration tasks. The present invention provides strong support for visual positioning and scene perception in complex mission scenarios, significantly improving the robustness and environmental adaptability of the unmanned system, and is suitable for complex mission scenarios such as field rescue, tunnel inspection and mine exploration.

Description

Technical Field

[0001] The present invention relates to the field of autonomous exploration technology for unmanned aerial vehicles (UAVs), and in particular to an autonomous exploration method for an unmanned aerial vehicle (UAV) system based on multi-view scene understanding. Background Art

[0002] Unmanned systems, including vehicles and drones, possess significant application value in areas such as national defense security, industrial inspection, and environmental monitoring, due to their high degree of autonomy and flexibility. However, their widespread adoption is limited by the lack of prior maps in unknown scenarios and the inability to use GPS positioning due to scene restrictions or interference. With the rapid development of artificial intelligence (AI) technology and improvements in sensor manufacturing, autonomous exploration and mission completion by vision-based unmanned systems is becoming a future trend. Visual SLAM technology provides the unmanned system with its position and pose in the mission scenario, along with the scene structure information required for navigation. Furthermore, the use of visual images and deep learning algorithms enables a high-level understanding of the mission scenario, further enhancing the unmanned system's perception, exploration, and decision-making capabilities.

[0003] During autonomous exploration, unmanned systems are affected by complex mission scenarios, particularly weather and lighting conditions, which can lead to the loss of some image information, affecting positioning and mapping accuracy, and even causing the autonomous exploration mission to fail. In complex autonomous exploration mission scenarios, due to the hardware limitations of visual imaging equipment, single-modal visual sensors are often unable to fully describe the imaging scene. Image fusion, on the other hand, integrates the basic information of multi-modal images, providing more accurate and reliable information. In published patents / literature related to autonomous exploration of unmanned systems, it is usually assumed that the exploration scenario has good weather and lighting conditions or that multi-modal sensors are used to improve positioning quality in relatively complex scenarios. However, there are currently no reports on the use of multi-view vision in different modalities to improve image quality, achieve more accurate positioning and scene understanding, and thus enable autonomous exploration in complex mission scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an autonomous exploration method for an unmanned system based on multi-view scene understanding, which can realize autonomous exploration under conditions of severe weather and drastic changes in lighting.

[0005] The technical solution adopted by the present invention to solve the technical problem is to provide an unmanned system autonomous exploration method based on multi-view scene understanding, comprising the following steps:

[0006] Acquire a left infrared image, a right infrared image, and a visible light image, and fuse the left infrared image with the visible light image, and the right infrared image with the visible light image, through a multi-eye image fusion network to obtain a left fused image and a right fused image;

[0007] Calculate the odometer data of the unmanned system according to the left fused image, the right fused image and the IMU data;

[0008] Using the left fused image to complete the detection of the target of interest in the task scene through a lightweight target detection network, and obtain a detection bounding box of the target of interest;

[0009] Depth is calculated using the parallax of the left fused image and the right fused image, and combined with the odometry data of the unmanned system to obtain three-dimensional point cloud data in a world coordinate system, and a voxel map is established through voxelization processing;

[0010] Establishing an exploration boundary using semantic information of the object of interest detection bounding box and occupancy information of the voxel map;

[0011] Based on the exploration boundary, an exploration gain function is established that integrates the geometric information gain of the exploration space and the semantic information gain of target detection. The exploration boundary that maximizes the gain function value is obtained and used as the exploration destination.

[0012] Taking the exploration destination as the end point of global path planning and the current posture of the unmanned system as the starting point, a feasible path in free space is planned on the voxel map using a global path planning algorithm to obtain a global path;

[0013] Based on the global path, the local path is planned, and the local path points are used as control points for trajectory optimization. The optimized trajectory is then transmitted to the unmanned system controller to achieve autonomous exploration movement.

[0014] The multi-view image fusion network is integrated with an adaptive guidance module, which selects a higher-quality image from the SeAFusion fusion image and the optimal image obtained in the previous training process as a guidance image to guide the training process of the multi-view image fusion network.

[0015] The multi-view image fusion network includes a first convolution block, a second convolution block, a first ResNeXt block, a second ResNeXt block, a third convolution block and a fourth convolution block connected in sequence; the output end of the first convolution block is also connected to the input end of the tenth convolution block; the first convolution block includes a 7×7 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the second convolution block and the third convolution block have the same structure, both including a 3×3 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the fourth convolution block includes a 3×3 convolution layer and a Sigmoid activation function layer connected in sequence; the first ResNeXt block and the second ResNeXt block have the same structure, both consisting of two 1×1 convolution layers and one 3×3 grouped convolution layer.

[0016] The loss of the multi-image fusion network includes content loss and guidance loss, which is expressed as: L f =L cont +wL guid , where L f is the loss of the multi-image fusion network, L cont is the content loss, L guid is the guided loss, w is the adaptive weight, and the adaptive weight w is set to zero in the first training cycle.

[0017] After using the left fused image to complete the detection of the target of interest in the task scene through a lightweight target detection network and obtain the target of interest detection bounding box, it also includes: using the target area provided by the target of interest detection bounding box, combining it with the aligned depth image, filtering out the background area according to the depth threshold of the target object, and completing the pose calculation of the target object.

[0018] The depth threshold of the target object is determined by Calculated, where d th is the depth threshold of the target object, d obj is the depth value of the center of the detection bounding box of the object of interest, and Δ is the difference between the maximum depth value of the corners of the detection bounding box of the object of interest and the depth value of the center of the detection bounding box of the object of interest.

[0019] The calculation of the pose of the target object is completed, specifically including:

[0020] Using the camera intrinsic parameter matrix and the depth value of the target area, the pixel coordinates of the target area are converted to the camera coordinate system;

[0021] Calculate the center point coordinates of the target object in the target area based on the camera coordinate system of the target area;

[0022] Based on the target point cloud composed of the positions of target pixels in the camera coordinate system, the target orientation is calculated using the principal component analysis method;

[0023] According to the external parameters and odometer of the camera relative to the unmanned system, the target pose in the camera coordinate system is converted to the world coordinate system.

[0024] The semantic information of the detection bounding box of the target of interest and the occupancy information of the voxel map are used to establish the exploration boundary, specifically: the voxels are detected and the voxels that meet the requirements are detected. The voxels of the boundary voxels are taken as the boundary voxels, and the center of the boundary voxels is calculated using the K-means clustering algorithm, and the center of the boundary voxels is taken as the boundary point. All boundary points constitute the exploration boundary set; where F represents the voxel set in the exploration boundary, v f represents a free voxel, neihbor(v f) represents the neighboring voxels of the free voxel, V free represents the free voxel set, V unknown Represents a set of unknown voxels.

[0025] The exploration gain function is expressed as: G(v i )=α·I GG (v i )+β·I SG (v i ), where I GG (v i ) is the candidate exploration destination v i The geometric information gain is expressed as: Indicates the current position of the unmanned system p c and candidate destinations The Euclidean distance between the locations, I un (v i ) is the number of candidate exploration destinations v i The measure of the potentially visible unexplored area, λ is a positive constant; I SG (v i ) is the candidate exploration destination v i The semantic information gain is expressed as: Among them I obj (v i ) is the semantic information gain of a single target, expressed as: L(p obj ,p vi ) is the target p obj The location and candidate destination p vi The Euclidean distance between the locations, I range represents the effective range of semantic perception; α and β represent the geometric information gain exploration weight and semantic information gain exploration weight, respectively.

[0026] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned unmanned system autonomous exploration method based on multi-view scene understanding are implemented.

[0027] Beneficial effects

[0028] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention proposes a method for fusing visible light and long-wave infrared images, which provides favorable support for visual positioning and scene perception, and helps to improve the adaptability of unmanned systems in harsh environments. Subsequently, the fused image is used to obtain a rough area of ​​the target of interest through a lightweight Yolo algorithm, and the accuracy of the target area is further improved by combining the method of filtering out the background with a depth threshold, achieving an effect similar to pixel-level semantic segmentation. An exploration boundary is established based on the detection information and voxel map information of the target of interest, and the exploration destination is determined through the exploration strategy of mission scenario understanding, guiding the unmanned system to detect the target of interest in detail while exploring the unknown space. The present invention is particularly suitable for mission scenarios such as field rescue, tunnel inspection, mine exploration, etc. under conditions of severe weather, darkness or lighting changes, and will further promote the widespread application of unmanned systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a framework diagram of an autonomous exploration method for an unmanned system based on multi-view scene understanding according to a first embodiment of the present invention;

[0030] Figure 2 2. It is a schematic diagram of the coordinate definition and configuration of the multi-eye vision system in the first embodiment of the present invention;

[0031] Figure 3 1 is a diagram of the training process of the multi-image fusion network in the first embodiment of the present invention;

[0032] Figure 4 It is a structural diagram of the multi-image fusion network in the first embodiment of the present invention. DETAILED DESCRIPTION

[0033] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0034] The first embodiment of the present invention relates to an autonomous exploration method for an unmanned system based on multi-view scene understanding. This method is designed to cope with mission scenarios such as severe weather or drastic changes in lighting. It combines the advantages of visible light cameras that can capture rich textures and infrared cameras that can capture sensitive pixel intensities, and uses multimodal fusion images to improve adaptability in complex mission scenarios. In order to obtain depth information in the scene, a stereoscopic vision system is formed by two groups of left and right multimodal cameras (similar to binocular cameras), which also expands the field of view.

[0035] like Figure 1As shown, the unmanned system autonomous exploration method based on multi-view scene understanding of this embodiment includes the following steps:

[0036] S1: To obtain accurate measurement information, the multi-viewing sensor is calibrated. Specifically, the two camera groups of the multi-viewing system are calibrated separately. Each camera group includes a visible light camera and two long-wave infrared cameras. The coordinates are defined as follows: Figure 2 As shown in (a). The calibration of two sets of cameras is achieved by designing a self-heating chessboard that can be used for calibration in the visible light and long-wave infrared bands. The configuration of the multi-viewing system is shown in Figure 2 As shown in (b).

[0037] S2: Acquire a left infrared image, a right infrared image, and a visible light image, and fuse the left infrared image with the visible light image, and the right infrared image with the visible light image, through a multi-eye image fusion network to obtain a left fused image and a right fused image.

[0038] This step aims to obtain images with good pixel intensity and texture details, and realizes the fusion of infrared images and visible light images through a real-time end-to-end multi-view image fusion network based on an adaptive guidance module.

[0039] The multimodal image fusion network in this implementation introduces an adaptive guidance module. This module integrates state-of-the-art image fusion methods and the best fusion results during training to achieve collaborative guidance of network training. Furthermore, based on the adaptive guidance module, this implementation designs a loss function that includes content loss and guidance loss, and uses adaptive weights to balance the two loss functions to improve framework performance.

[0040] In order to better train the fusion network and produce better fusion results, this implementation method is designed as follows Figure 3 The adaptive guidance module shown in the figure is used. The fusion image of the SOTA method, SeAFusion, is selected as the I SeA The image generated by the previous training cycle (epochs) contains gradient information and structural information, which is conducive to image fusion. By "save", the adaptive guidance module saves the fusion results of all previous epochs and selects the best result from the previous output, which is recorded as I pre Finally, the adaptive guidance module is SeA and I pre Select higher quality images as guide images to guide the training process of the multimodal image fusion network.

[0041] like Figure 4As shown, the multimodal image fusion network in this embodiment is a lightweight network, which includes a first convolution block, a second convolution block, a first ResNeXt block, a second ResNeXt block, a third convolution block and a fourth convolution block connected in sequence, which is used to retain more feature information; the output end of the first convolution block is also connected to the input end of the tenth convolution block; the first convolution block includes a 7×7 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the second convolution block and the third convolution block have the same structure, both including a 3×3 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the fourth convolution block includes a 3×3 convolution layer and a Sigmoid activation function layer connected in sequence; the first ResNeXt block and the second ResNeXt block have the same structure, both consisting of two 1×1 convolution layers and one 3×3 grouped convolution layer.

[0042] Most convolutional blocks in this multimodal image fusion network use 3×3 convolutions and ReLU as the activation function. Padding is used at each layer to maintain the size of the fusion output, ensuring that the fusion result is the same size as the source image. Input images are directly concatenated and passed to the network, enabling the framework to capture common features while also learning their unique characteristics. To expand the receptive field and obtain more local features without significantly increasing computational complexity, the kernel size of the first convolutional block is set to 7×7, and the output channels are set to 16. Two ResNeXt blocks are used to prevent gradient vanishing and preserve detailed information from the source image, enhancing the network's expressive power. The ResNeXt block consists of two 1×1 convolutional layers and a 3×3 grouped convolutional layer. This convolution operation performs independent computations on each channel, fully mining and utilizing feature information. Finally, a sigmoid activation function is used to generate the fused image. To preserve information from the previous convolutional layers and enhance texture details and salient features, the output of the third convolutional block is added to the output of the first convolutional block.

[0043] This implementation scheme designs a content loss to fully fuse the meaningful information of the source images, including the salient objects in the infrared image and the texture details in the visible light image, while ensuring the visual fidelity of the fusion result. Secondly, a guidance loss based on an adaptive guidance module is designed to enable the fusion framework to learn from the results of other methods and previous outputs. The loss L of the multi-view image fusion network is f The expression is:

[0044] L f =L cont +wL guid ;

[0045] Where w is the adaptive weight used to balance the content loss and the guidance loss. In the first epoch, set w = 0. L cont and Lguid denote content loss and guidance loss respectively.

[0046] S3: To obtain the odometry information of the unmanned system, the left and right fused images and IMU data obtained in S2 are used as input, and the OpenVINS algorithm that takes into account both positioning accuracy and real-time performance is used to calculate the odometry data of the unmanned system itself.

[0047] S4: Using the fused image set on the left acquired in S2, the lightweight object detection network YoloV5 is used to detect the target of interest in the task scenario. After target detection, the rough target region provided by the target detection bounding box is combined with the aligned depth image to further filter out the background area based on the target object's depth threshold. The pose of the target object is then calculated, achieving an effect similar to pixel-level semantic segmentation and improving the accuracy of target object pose calculation.

[0048] The depth threshold determination process for the target object is as follows: Most target object detection bounding boxes correspond to background points, and the target object usually has a large depth difference from the background. The maximum background depth of the target object is defined as the maximum depth value of the four corners of its detection bounding box. The depth threshold position of the bounding box is determined by the difference between the depth value of the four corners and the center of the bounding box. The expression is as follows:

[0049]

[0050] Among them, d th is the depth threshold of the target object, d obj is the depth value of the center of the bounding box of the target of interest, and Δ is the difference between the maximum depth value of the corner depth values ​​of the bounding box of the target of interest and the depth value of the center of the bounding box of the target of interest. When the depth value of the target area is less than the depth threshold, it is the target area.

[0051] When calculating the pose of the target object, it specifically includes:

[0052] (1) Using the camera intrinsic parameter matrix and the depth value of the target area, the pixel coordinates of the target area are converted to the camera coordinate system. The expression is:

[0053]

[0054] Among them, (f x ,f y ) is the pixel focal length, c x and c y is the intersection of the optical axis and the imaging plane, (u i ,v i ) is a pixel point in the target area, Z dep-i is a certain depth value of the target area, (Xi ,Y i ,Z i ) is the coordinate of the pixel in the camera coordinate system.

[0055] (2) Calculate the coordinates of the center point of the target object. Assuming that there are N pixels in the target area, the center position of the target in the camera coordinate system (X c ,Y c ,Z c ) is:

[0056]

[0057] (3) The position of the target pixel in the camera coordinate system (X i ,Y i ,Z i ) to form the target point cloud, and the principal component analysis method is used to calculate the target orientation. The covariance matrix expression of the point cloud is:

[0058]

[0059] Among them, p i =[X i ,Y i ,Z i ] T is a point in the target point cloud, p c =[X c ,Y c ,Z c ] T The target centroid is the target. The eigenvalue of the covariance is solved and the eigenvector with the minimum eigenvalue, i.e., the normal vector, is extracted as the target orientation.

[0060] (4) According to the external parameters of the camera relative to the unmanned system and the odometer, the target pose in the camera coordinate system is converted to the world coordinate system. The expression is:

[0061]

[0062] in, is the target’s pose in the camera coordinate system, is the transformation matrix from the camera to the unmanned system body, The transformation from the unmanned system to the world coordinate system is the odometer (here, for simplicity, the odometer coordinate origin is used as the world coordinate system origin).

[0063] S5: To achieve a sparse representation of the exploration space, discretize it into a fixed-size voxel grid. Use the parallax of the left and right fused images to calculate the depth to update the occupancy state of each voxel. Combined with the odometry data of the unmanned system, obtain three-dimensional point cloud data in the world coordinate system, and establish a voxel map through voxelization processing.

[0064] S6: To guide the unmanned system to conduct autonomous exploration, the exploration boundary is established using the semantic information of target detection in S4 and the occupancy information of the voxel map in S5.

[0065] In this step, the target of interest is directly established as the boundary of the semantic label, and the area between the known free voxels and their neighboring unknown voxels is established as a common boundary through the clustering algorithm. Specifically, the voxels are detected and the voxels that meet The voxels of the boundary voxels are taken as the boundary voxels, and the center of the boundary voxels is calculated using the K-means clustering algorithm, and the center of the boundary voxels is taken as the boundary point. All boundary points constitute the exploration boundary set; where F represents the voxel set in the exploration boundary, v f represents a free voxel, neihbor(v f ) represents the neighboring voxels of the free voxel, V free represents the free voxel set, V unknown Represents a set of unknown voxels.

[0066] During the exploration process, the depth information changes with the movement of the unmanned system, and the voxel map is updated accordingly. At this time, voxel detection is performed regularly, the explored boundaries are removed from the boundary set, and the newly detected boundaries are added to the boundary set to ensure that the boundary set is constantly updated.

[0067] To avoid an exploration boundary that is too large and therefore cannot be covered in one go, principal component analysis is used to analyze each boundary cluster. If the maximum eigenvalue exceeds the threshold, it is split into two even clusters along the first principal axis. Boundary segmentation is performed recursively until the exploration boundary meets the threshold requirement.

[0068] S7: Leveraging the exploration boundary established in S6, we gradually explore unknown mission scenarios and perform detailed detection of objects of interest to enhance scene understanding. We establish an exploration gain function that combines the geometric information gain of the exploration space with the semantic information gain of object detection, and find the exploration boundary that maximizes this gain, which we then use as the exploration destination.

[0069] The exploration gain function in this step is expressed as:

[0070] G(v i )=α·I GG (v i )+β·I SG (vi );

[0071] Among them, I GG (v i ) is the candidate exploration destination v i The geometric information gain, I SG (v i ) is the candidate exploration destination v i , α and β represent the geometric information gain exploration weight and semantic information gain exploration weight, respectively. Their values ​​are determined by experiments and depend on the task environment layout.

[0072] Candidate exploration destinationsv i The geometric information gain I GG (v i ) is expressed as:

[0073]

[0074] in, Indicates the current position of the unmanned system p c and candidate destination p vi The Euclidean distance between the locations, I un (v i ) is the number of candidate exploration destinations v i The measure of the potentially visible unexplored area is defined as i The proportion of unknown voxels in the field of view, λ is a positive constant, which measures the importance of the motion cost of the unmanned system relative to the expected information gain.

[0075] Candidate exploration destinationsv i Semantic information gain I SG (v i ) is expressed as:

[0076]

[0077] Among them, I obj (v i ) is the semantic information gain of a single target, is the target p obj Location and candidate destinations The Euclidean distance between the locations, I range Indicates the effective scope of semantic perception.

[0078] Each candidate exploration destination v i The semantic information gain of the candidate exploration destination v i The sum of all visible targets at , which is inversely proportional to the target distance. Finally, the optimal exploration destination is the one that makes the total information gain G(v i ) maximizes the viewpoint, that is:

[0079]

[0080] S8: To obtain a safe and feasible global path on the map, the exploration destination in S7 is used as the end point of the global path planning. The current position of the unmanned system is used as the starting point. The A* global path planning algorithm is used to plan a feasible path in free space on the voxel map to obtain the global path.

[0081] S9: To avoid dynamic or unmapped obstacles in real time during exploration, the D*Lite algorithm is used to plan local paths based on the global path. To obtain a smooth trajectory, the B-spline curve optimization method is used to optimize the trajectory using local path points as control points. The optimized trajectory is then transmitted to the unmanned system controller to achieve autonomous exploration.

[0082] It is not difficult to find that the present invention proposes a method for fusing visible light and long-wave infrared images, which provides favorable support for visual positioning and scene perception, and helps to improve the adaptability of unmanned systems in harsh environments. Subsequently, the fused image is used to obtain a rough area of ​​the target of interest through a lightweight Yolo algorithm, and the method of filtering out the background with a depth threshold is combined to further improve the accuracy of the target area, achieving an effect similar to pixel-level semantic segmentation. The exploration boundary is established based on the detection information and voxel map information of the target of interest, and the exploration destination is determined through the exploration strategy of mission scenario understanding, guiding the unmanned system to detect the target of interest in detail while exploring the unknown space. The present invention is particularly suitable for mission scenarios such as field rescue, tunnel inspection, mine exploration, etc. under conditions of severe weather, darkness or changes in lighting, and will further promote the widespread application of unmanned systems.

[0083] A second embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the unmanned system autonomous exploration method based on multi-view scene understanding of the first embodiment are implemented.

[0084] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) that contain computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction method, which is implemented in the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An autonomous exploration method for unmanned systems based on multi-view scene understanding, characterized in that: The following steps are involved: Acquire a left infrared image, a right infrared image, and a visible light image, and fuse the left infrared image with the visible light image, and the right infrared image with the visible light image, through a multi-eye image fusion network to obtain a left fused image and a right fused image; Calculate the odometer data of the unmanned system according to the left fused image, the right fused image and the IMU data; Using the left fused image to complete the detection of the target of interest in the task scene through a lightweight target detection network, and obtain a detection bounding box of the target of interest; Depth is calculated using the parallax of the left fused image and the right fused image, and combined with the odometry data of the unmanned system to obtain three-dimensional point cloud data in a world coordinate system, and a voxel map is established through voxelization processing; Establishing an exploration boundary using semantic information of the object of interest detection bounding box and occupancy information of the voxel map; Based on the exploration boundary, an exploration gain function is established that integrates the geometric information gain of the exploration space and the semantic information gain of target detection. The exploration boundary that maximizes the gain function value is obtained and used as the exploration destination. Taking the exploration destination as the end point of global path planning and the current posture of the unmanned system as the starting point, a feasible path in free space is planned on the voxel map using a global path planning algorithm to obtain a global path; Based on the global path, the local path is planned, and the local path points are used as control points for trajectory optimization. The optimized trajectory is then transmitted to the unmanned system controller to achieve autonomous exploration movement.

2. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 1 is characterized in that: The multi-view image fusion network is integrated with an adaptive guidance module, which selects a higher-quality image from the SeAFusion fusion image and the optimal image obtained in the previous training process as a guidance image to guide the training process of the multi-view image fusion network.

3. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 1, characterized in that: The multi-view image fusion network includes a first convolution block, a second convolution block, a first ResNeXt block, a second ResNeXt block, a third convolution block and a fourth convolution block connected in sequence; the output end of the first convolution block is also connected to the input end of the tenth convolution block; the first convolution block includes a 7×7 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the second convolution block and the third convolution block have the same structure, both including a 3×3 convolution layer, a BN layer and a ReLU activation function layer connected in sequence; the fourth convolution block includes a 3×3 convolution layer and a Sigmoid activation function layer connected in sequence; the first ResNeXt block and the second ResNeXt block have the same structure, both consisting of two 1×1 convolution layers and one 3×3 grouped convolution layer.

4. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 2 is characterized in that: The loss of the multi-image fusion network includes content loss and guidance loss, which is expressed as: L f =L cont +wL guid , where L f is the loss of the multi-image fusion network, L cont is the content loss, L guid is the guided loss, w is the adaptive weight, and the adaptive weight w is set to zero in the first training cycle.

5. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 1 is characterized in that: After using the left fused image to complete the detection of the target of interest in the task scene through a lightweight target detection network and obtain the target of interest detection bounding box, it also includes: using the target area provided by the target of interest detection bounding box, combining it with the aligned depth image, filtering out the background area according to the depth threshold of the target object, and completing the pose calculation of the target object.

6. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 5 is characterized in that: The depth threshold of the target object is determined by Calculated, where d th is the depth threshold of the target object, d obj is the depth value of the center of the detection bounding box of the object of interest, and Δ is the difference between the maximum depth value of the corners of the detection bounding box of the object of interest and the depth value of the center of the detection bounding box of the object of interest.

7. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 5, characterized in that: The calculation of the pose of the target object is completed, specifically including: Using the camera intrinsic parameter matrix and the depth value of the target area, the pixel coordinates of the target area are converted to the camera coordinate system; Calculate the center point coordinates of the target object in the target area based on the camera coordinate system of the target area; Based on the target point cloud composed of the positions of target pixels in the camera coordinate system, the target orientation is calculated using the principal component analysis method; According to the external parameters and odometer of the camera relative to the unmanned system, the target pose in the camera coordinate system is converted to the world coordinate system.

8. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 1, characterized in that: The semantic information of the detection bounding box of the target of interest and the occupancy information of the voxel map are used to establish the exploration boundary, specifically: the voxels are detected and the voxels that meet the requirements are detected. The voxels of the boundary voxels are taken as the boundary voxels, and the center of the boundary voxels is calculated using the K-means clustering algorithm, and the center of the boundary voxels is taken as the boundary point. All boundary points constitute the exploration boundary set; where F represents the voxel set in the exploration boundary, v f represents a free voxel, neihbor(v f ) represents the neighboring voxels of the free voxel, V free represents the free voxel set, V unknown Represents a set of unknown voxels.

9. The unmanned system autonomous exploration method based on multi-view scene understanding according to claim 1, characterized in that: The exploration gain function is expressed as: G(v i )=α·I GG (v i )+β·I SG (v i ), where I GG (v i ) is the candidate exploration destination v i The geometric information gain is expressed as: L(p c ,p vi ) represents the current position of the unmanned system p c and candidate destination p vi The Euclidean distance between the locations, I un (v i ) is the number of candidate exploration destinations v i The measure of the potentially visible unexplored area, λ is a positive constant; I SG (v i ) is the candidate exploration destination v i The semantic information gain is expressed as: Among them I obj (v i ) is the semantic information gain of a single target, expressed as: L(p obj ,p vi ) is the target p obj The location and candidate destination p vi The Euclidean distance between the locations, I range represents the effective range of semantic perception; α and β represent the geometric information gain exploration weight and semantic information gain exploration weight, respectively.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the unmanned system autonomous exploration method based on multi-view scene understanding as described in any one of claims 1 to 9 are implemented.

Images