Semantic understanding and learning-based unstructured environment cognition method
By constructing a tiered body matching model and semantic split backbone network model, obtaining the geometric characteristics and semantic attribute description of the moon facial landforms, solving the accuracy and real -time problems of environmental cognition in the non -structured and complex environment of the moon surface, realized Efficient environmental characteristics and semantic classification.
Patent Information
- Application Number
- CN202411308724.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art is difficult to achieve accurate, reliable and real-time environmental cognition in unstructured complex environments on the lunar surface, especially in complex scenarios where there is a lack of barrier detection and understanding of semantic information.
The method of understanding and learning based on semantic understanding and learning is adopted to build a level -level matching model and semantic split bone network model, obtain the geometric characteristics and semantic attribute descriptions of the moonlight landform, and form a parameterized descriptor to achieve environmental cognition.
It improves the accuracy, adaptability and real-time performance of lunar surface environment recognition, and can effectively represent and semantically classify all terrains.
Smart Images
Figure CN119339306A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an unstructured environment recognition method based on semantic understanding and learning, and belongs to the field of computer vision technology such as image recognition and image segmentation. Background Art
[0002] The lunar environment is characterized by extreme lighting, poor texture, and unstructured terrain. Lunar robots need to fully perceive and assess the environment within a certain neighborhood in order to reliably perform detection tasks. Traditional environmental cognition methods mostly identify the geometric characteristics of the target at the visual layer and perform feature clustering based on the typical geometric configuration of the terrain. However, terrain clustering or feature engineering methods are difficult to obtain semantic information at the target object layer and concept layer in complex scenarios, and there are limitations in obstacle detection and understanding.
[0003] In recent years, image perception methods based on deep neural networks have performed well in tasks such as scene recognition and semantic segmentation, and the Top-5 error rate of classic image classification tasks has been reduced to below 5%. Long et al. from the University of California, Berkeley proposed a fully convolutional neural network (FCN), which replaced the fully connected layer at the end of the typical convolutional neural network (CNN) structure with a 1×1 convolutional layer to obtain a low-resolution prediction result, and then used upsampling to restore it to the original resolution, achieving end-to-end semantic segmentation for the first time. On this basis, Yu et al. proposed the theory of dilated convolution to extract semantic information on a larger scale. However, the FCN network structure also has certain limitations and shortcomings. It is easy to lose detailed information in complex environments, the contextual information between pixels is not fully considered, and the segmentation results lack spatial consistency.
[0004] The lack of obvious structural landmark features and the difficulty in modeling the features of diverse and heterogeneous objects are key technical bottlenecks that restrict the recognition of unstructured complex environments. How to conduct environmental recognition with high accuracy, high reliability and good real-time performance in the unstructured complex environment on the lunar surface has become an important research topic. The Apollo series of lunar rovers did not carry active sensors such as visual cameras and lidars, and could not classify and identify the lunar environment; the Yutu series of lunar rovers carried active sensors such as visual cameras, but could only identify some typical obstacles and could not semantically classify and represent the surrounding terrain. This unstructured lunar environmental recognition capability cannot meet the requirements of future manned lunar exploration, large-scale unmanned lunar exploration, lunar human-machine / multi-machine collaborative exploration, and lunar scientific research station construction. Summary of the invention
[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide an unstructured environment cognition method based on semantic understanding and learning. By constructing a cascaded stereo matching model and a semantic segmentation backbone network model, the lunar environment is classified, identified and semantically represented, thereby improving the accuracy, adaptability and real-time performance of lunar environment cognition.
[0006] The technical solution of the present invention is: an unstructured environment cognition method based on semantic understanding and learning, comprising:
[0007] Build a waterfall semantic segmentation backbone network model and deploy it in the lunar environment cognition unit;
[0008] When the lunar surface environment recognition unit is located on the lunar surface, the geomorphic geometric characteristic descriptor for describing the lunar surface topography at the lunar surface position where the lunar surface environment recognition unit is located and the geomorphic semantic attribute descriptor for describing the entire lunar surface topography are obtained in real time through the waterfall semantic segmentation backbone network model;
[0009] The lunar surface topography parameterized descriptor is obtained according to the topography geometric feature descriptor and the topography semantic attribute descriptor, and the lunar surface environment in which the lunar surface environment cognition unit is located is parsed through the lunar surface topography parameterized descriptor.
[0010] Furthermore, the terrain geometric feature descriptor is G=[h,w,s,d]; wherein h is the height of the object, w is the width of the obstacle target, s is the slope of the slope terrain, and d is the distance between the obstacle object and the lunar environment cognition unit.
[0011] Furthermore, the landform semantic attribute descriptor is e i , representing five different landforms: hillside, terrace, rock, gully and plain, i=1,…,5.
[0012] Furthermore, the lunar surface topography parameter descriptor is F = [G, e i ]; where G is the geomorphic geometric feature descriptor; e i It is a landform semantic feature descriptor.
[0013] Furthermore, obtaining a landform semantic attribute descriptor includes:
[0014] Acquire binocular camera images within the neighborhood of the lunar environment recognition unit;
[0015] The binocular camera images are respectively input into two feature extraction network models with shared weights for convolution, pooling, and feature fusion calculations. The two output graphic features are then subjected to cascaded cost convolution calculations to obtain the field of view calculation layer features and implement depth and disparity calculations.
[0016] The field of view calculation layer features output by any feature extraction network model are fused with the graphic features output by another feature extraction network model, and then convolved and upsampled to obtain the landform semantic attribute descriptor through the activation function.
[0017] Furthermore, obtaining the topographic geometric characteristic descriptor includes: obtaining the topographic geometric characteristic descriptor from the field of view calculation layer features through a regression function.
[0018] Furthermore, the lunar topography parameterized descriptor F is obtained through the topography geometric feature descriptor and the topography semantic attribute descriptor, and the F value is assigned to all the pixels of the object to form an environmental cognition and information feature cube, which is used as the environmental prior information for reconstructing the optimization model.
[0019] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of an unstructured environment cognition method based on semantic understanding and learning are implemented.
[0020] A device for unstructured environment cognition based on semantic understanding and learning comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for unstructured environment cognition based on semantic understanding and learning are implemented.
[0021] The advantages of the present invention compared with the prior art are:
[0022] (1) The present invention constructs a parametric description of environmental information from two aspects: geometric feature extraction and topographic semantic information acquisition, and uses it as the environmental cognition basis for robot configuration optimization and human-machine collaboration;
[0023] (2) Considering the real-time requirements of the algorithm, the present invention will adopt a cascaded stereo matching cost function to enhance the robustness of the matching algorithm and improve the matching efficiency;
[0024] (3) The present invention uses a concise and efficient waterfall segmentation network framework as the semantic segmentation backbone network to enhance the real-time matching performance;
[0025] (4) The segmentation types of the present invention for unstructured complex environments are better than 5 types and can effectively characterize the entire terrain. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0027] Figure 1 It is a schematic diagram of the process of the present invention;
[0028] Figure 2 Schematic diagram of the network structure of depth information estimation based on stereo vision in the present invention. DETAILED DESCRIPTION
[0029] In order to better understand the above technical scheme, the technical scheme of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0030] The following is a further detailed description of an unstructured environment recognition method based on semantic understanding and learning provided by an embodiment of the present invention in conjunction with the accompanying drawings. Figure 1 , the specific implementation methods may include:
[0031] Step S1, defining a terrain geometry feature descriptor, where the terrain geometry is described by four parameters: object height (h), width (w), slope (s), and distance from the robot (d);
[0032] Step S2: define the landform semantic attribute descriptor. The lunar terrain is described by a combination of five basic landforms: hillside, steps, rocks, gullies, and flat land. The limited semantic attribute variable is e i ;
[0033] Step S3: define the lunar topography parameterized descriptor F = [G, e i ];
[0034] Step S4, constructing a cascade stereo matching network model, obtaining a geomorphic geometric characteristic descriptor G, and improving matching efficiency;
[0035] Step S5: construct a waterfall semantic segmentation backbone network model to obtain the landform semantic attribute descriptor e i , enhance the real-time performance of matching.
[0036] Preferably, the step S1 further includes:
[0037] Unstructured complex environments lack man-made objects such as roads, buildings, and signs similar to those in urban environments, resulting in a lack of geometric and texture features with high local recognition. The false detection rate and missed detection rate of targets are very high, making it difficult to extract environmental landmarks as input for decision-making tasks. For the geometric morphology of the terrain, it is important to use parameters that can be both concise and locally recognizable. Since the field of view of the sensors carried by the robot is limited, especially the visual sensors, the terrain morphology that can be detected is within the neighborhood space around the robot, and the size of the terrain morphology is relatively fixed. Therefore, the description of the geometric morphology of the terrain can only focus on the four parameters of the height (h), width (w), slope (s) and robot distance (d) of the object in its domain space to describe the geometric characteristics of the object. The concise representation of geometric features is helpful in subsequent semantic understanding.
[0038] Preferably, the step S2 further includes:
[0039] As an upstream technology, deep learning-based landform segmentation plays an important role in downstream autonomous navigation tasks such as robot feasibility analysis, path planning, and slip prediction. Many landform classifications can be used to characterize landform semantic attributes. However, the basic types of typical lunar landforms are relatively fixed and can be described by different combinations of several common types of landforms. Common landform types include hillsides, steps, rocks, gullies, flat land, etc. The complex terrain of the moon can be described by different combinations of these five basic landform types, and the semantic description of all landforms can be achieved using limited semantic attribute variables.
[0040] Preferably, the S4 further includes:
[0041] Step S41, using a binocular vision camera as the robot's active perception sensor to obtain images within the robot's neighborhood, which are respectively recorded as left eye image I and left eye image II. l , right eye image I r . Preprocess the camera image, including overexposure, noise reduction and other processing. Considering the star catalog computing power, the data source and algorithm are lightweight, the image is windowed and cropped, and the sky and other areas are cropped, with a focus on the lunar surface landform environment area;
[0042] Step S42: l with I r The same convolutional neural network module (CNN) with shared weights is input for convolution calculation to prevent overfitting and high dimensionality, and then down-sampled through the spatial pyramid pooling module (SPP) to further improve the efficiency and robustness of the model. The fully connected layer combines and integrates the features to achieve multi-scale, multi-space, and multi-channel feature extraction, and obtains two feature maps for the left and right eyes;
[0043] Step S43, performing feature fusion through multiple convolutional layers to obtain a 4D representation vector;
[0044] Step S44, input the fused features into the cascade cost convolution module (C-Cost Volume) to realize the depth / disparity calculation of the left eye image and the right eye image. The cascade cost convolution module intends to adopt a stage encoding method. The subsequent stages use the prediction of the previous stage to continuously narrow the depth and disparity range to achieve the depth information prediction from coarse to detailed. This multi-stage encoding method is significantly more efficient than the encoding method directly based on difference regression, so real-time performance can be guaranteed;
[0045] Step S45, outputting depth information and scale information through a regression function.
[0046] Preferably, step S5 further includes:
[0047] Step S51, based on stereoscopic vision matching, extract I l The features after convolution of the image feature extraction layer and the features output by the field of view calculation layer C-Cost Volume enter the segmentation subnet;
[0048] Step S52, performing fusion, convolution and upsampling operations on the features after convolution of the image feature extraction layer and the features output by the field of view calculation layer;
[0049] Step S53, outputting pixel-level semantic segmentation results through activation function f to achieve extraction of semantic attributes;
[0050] Step S54, calculate the parameterized descriptor F of each object in the semantic map, and assign the F value to all pixels of the object to form an environmental cognition and information feature cube, which serves as environmental prior information for reconstructing the optimization model.
[0051] The technical solution provided in the embodiment of the present application specifically includes the following steps:
[0052] Build a waterfall semantic segmentation backbone network model and deploy it in the lunar environment cognition unit;
[0053] When the lunar surface environment recognition unit is located on the lunar surface, the geomorphic geometric characteristic descriptor for describing the lunar surface topography at the lunar surface position where the lunar surface environment recognition unit is located and the geomorphic semantic attribute descriptor for describing the entire lunar surface topography are obtained in real time through the waterfall semantic segmentation backbone network model;
[0054] The lunar surface topography parameterized descriptor is obtained according to the topography geometric feature descriptor and the topography semantic attribute descriptor, and the lunar surface environment in which the lunar surface environment cognition unit is located is parsed through the lunar surface topography parameterized descriptor.
[0055] The terrain geometric feature descriptor is G=[h,w,s,d]; wherein h is the height of the object, w is the width of the obstacle target, s is the slope of the slope terrain, and d is the distance between the obstacle object and the lunar environment cognition unit.
[0056] The landform semantic attribute descriptor is e i , representing five different landforms: hillside, terrace, rock, gully and plain, i=1,…,5.
[0057] The lunar topography parameterization descriptor is F = [G, e i ]; where G is the geomorphic geometric feature descriptor; e i It is a landform semantic feature descriptor.
[0058] Furthermore, obtaining a landform semantic attribute descriptor includes:
[0059] Acquire binocular camera images within the neighborhood of the lunar environment recognition unit;
[0060] like Figure 2 , the binocular camera images are respectively input into two shared weight feature extraction network models for convolution, pooling, and feature fusion calculations, and the two output graphic features are subjected to cascade cost convolution calculations to obtain the field of view calculation layer features and realize depth and disparity calculations;
[0061] The field of view calculation layer features output by any feature extraction network model are fused with the graphic features output by another feature extraction network model, and then convolved and upsampled to obtain the landform semantic attribute descriptor through the activation function.
[0062] Furthermore, obtaining the topographic geometric characteristic descriptor includes: obtaining the topographic geometric characteristic descriptor from the field of view calculation layer features through a regression function.
[0063] In one possible implementation, the lunar topography parameterized descriptor F is obtained through the topography geometric feature descriptor and the topography semantic attribute descriptor, and the F value is assigned to all pixels of the object to form an environmental cognition and information feature cube, which is used as the environmental prior information for reconstructing the optimization model.
[0064] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.
[0065] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0069] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
[0070] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.
Claims
1. A method for unstructured environmental cognition based on semantic understanding and learning, characterized in that: include: Build a waterfall semantic segmentation backbone network model and deploy it in the lunar environment cognition unit; When the lunar surface environment recognition unit is located on the lunar surface, the geomorphic geometric characteristic descriptor for describing the lunar surface topography at the lunar surface position where the lunar surface environment recognition unit is located and the geomorphic semantic attribute descriptor for describing the entire lunar surface topography are obtained in real time through the waterfall semantic segmentation backbone network model; The lunar surface topography parameterized descriptor is obtained according to the topography geometric feature descriptor and the topography semantic attribute descriptor, and the lunar surface environment in which the lunar surface environment cognition unit is located is parsed through the lunar surface topography parameterized descriptor.
2. The unstructured environment cognition method based on semantic understanding and learning according to claim 1 is characterized in that: The terrain geometric feature descriptor is G=[h,w,s,d]; wherein h is the height of the object, w is the width of the obstacle target, s is the slope of the slope terrain, and d is the distance between the obstacle object and the lunar environment cognition unit.
3. The unstructured environment cognition method based on semantic understanding and learning according to claim 1 is characterized in that: The landform semantic attribute descriptor is e i , representing five different landforms: hillside, terrace, rock, gully and plain, i=1,…,5.
4. The unstructured environment cognition method based on semantic understanding and learning according to claim 1 is characterized in that: The lunar topography parameterization descriptor is F = [G, e i ]; where G is the geomorphic geometric feature descriptor; e i It is a landform semantic feature descriptor.
5. The unstructured environment cognition method based on semantic understanding and learning according to claim 1 is characterized in that: Obtaining landform semantic attribute descriptors includes: Acquire binocular camera images within the neighborhood of the lunar environment recognition unit; The binocular camera images are respectively input into two feature extraction network models with shared weights for convolution, pooling, and feature fusion calculations. The two output graphic features are then subjected to cascaded cost convolution calculations to obtain the field of view calculation layer features and implement depth and disparity calculations. The field of view calculation layer features output by any feature extraction network model are fused with the graphic features output by another feature extraction network model, and then convolved and upsampled to obtain the landform semantic attribute descriptor through the activation function.
6. The unstructured environment cognition method based on semantic understanding and learning according to claim 5 is characterized in that: Acquiring the topographic geometric characteristic descriptor includes: acquiring the topographic geometric characteristic descriptor from the field of view calculation layer features through a regression function.
7. The unstructured environment cognition method based on semantic understanding and learning according to claim 6 is characterized in that: The parametric descriptor F of the lunar topography is obtained through the topography geometric feature descriptor and the topography semantic attribute descriptor, and the F value is assigned to all the pixels of the object to form an environmental cognition and information feature cube, which is used as the environmental prior information for reconstructing the optimization model.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
9. An unstructured environment recognition device based on semantic understanding and learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.