Surface object identification method based on fusion of spectral information image and three-dimensional information image

By fusing spectral and 3D information images and utilizing encoders and advanced feature space processing techniques, the problems of parallelization difficulties and high computational load in remote sensing monitoring are solved, thereby improving the efficiency of surface object identification.

CN118941909AInactive Publication Date: 2024-11-12NANJING UNIV OF INFORMATION SCI & TECH

Patent Information

Application Number
CN202411419818.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In remote sensing monitoring, the fusion and classification of hyperspectral and radar images face challenges in parallelization and require massive computational resources, resulting in low machine efficiency and difficulty in effectively identifying objects on the ground.

Method used

A method for fusing spectral information images and 3D information images is adopted. The fused high-level feature data is obtained through an encoder and a high-level feature space, and then subjected to non-temporal linear arrangement, discretization, piecewise scanning and synthesis processing to solve the problems of parallelization difficulties and huge computational load.

Benefits of technology

It improves the computing efficiency of computers and enhances the efficiency of surface object recognition, especially showing significant efficiency advantages under conditions of large sample sizes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941909A_ABST
    Figure CN118941909A_ABST
Patent Text Reader

Abstract

The invention discloses a spectral information image and three-dimensional information image fused earth surface object recognition method, which belongs to the technical field of image processing and comprises the following steps: acquiring a spectral information image and a three-dimensional information image of an earth surface object; respectively encoding the spectral information image and the three-dimensional information image to obtain low-level feature data of the spectral information image and low-level feature data of the three-dimensional information image, and projecting the low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image into a high-level feature space for fusion, obtaining the fused advanced feature data; sequentially performing non-sequential arrangement processing, discretization processing and piece-by-piece scanning processing on the fused advanced feature data to obtain processed advanced feature data; and performing two-dimensional feature synthesis on the processed advanced feature data, and performing decoding reconstruction on the synthesized advanced feature data to obtain an identification result of the earth surface object. According to the invention, the efficiency of surface object identification can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for identifying ground objects by fusing a spectral information image with a three-dimensional information image. Background Art

[0002] Hyperspectral images and radar images are two different means of obtaining spectral information images and three-dimensional structures of surface objects in remote sensing monitoring. Through the fusion and classification of hyperspectral images and radar images, surface objects can be identified more accurately. The new data set formed by the two can play a unique role in agriculture, such as observing the growth of different crops.

[0003] Hyperspectral images use the different spectral curve characteristics generated by different objects to identify objects that other spectral images cannot identify. Although radar images cannot clearly distinguish different objects, they can identify the three-dimensional structure of surface objects. This technology has a unique position in obtaining the appearance characteristics of objects.

[0004] However, under the condition of huge data sets, such as surface remote sensing observation scenarios, the parallelization difficulties and huge computational workload of recurrent neural networks (RNNs) and Transformers models have become a major obstacle to the development of related technologies.

[0005] In order to solve the problems of large computational complexity and low machine efficiency, an integrated convolution algorithm that has advantages in image processing, good fitting effect and low computational complexity is urgently needed in the current remote sensing field. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for identifying surface objects by fusing spectral information images and three-dimensional information images. The fused high-level feature data is obtained through an encoder and a high-level feature space, and then the fused high-level feature data is processed through non-temporal linear arrangement, discretization processing, piece-by-piece scanning and synthesis processing, so as to solve the problems that parallelization difficulties and huge computational complexity have become obstacles to the development of related technologies, improve the computing efficiency of computers, and thus improve the efficiency of surface object recognition.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] In one aspect, the present invention provides a method for identifying ground objects by fusing spectral information images and three-dimensional information images, comprising:

[0009] Obtain spectral information images and three-dimensional information images of surface objects;

[0010] Encoding the spectral information image and the three-dimensional information image respectively to obtain low-level feature data of the spectral information image and low-level feature data of the three-dimensional information image, projecting the low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image into a high-level feature space for fusion, and obtaining fused high-level feature data;

[0011] The fused high-level feature data is sequentially subjected to non-temporal arrangement processing, discretization processing, and piece-by-piece scanning processing to obtain processed high-level feature data;

[0012] Performing two-dimensional feature synthesis on the processed high-level feature data to obtain synthesized high-level feature data;

[0013] The synthesized high-level feature data is decoded and reconstructed to obtain the recognition results of surface objects.

[0014] Optionally, the fused high-level feature data is expressed as:

[0015] ;

[0016] ;

[0017] ;

[0018] In the formula, It is the high-level feature data after fusion; It is the image feature fusion layer; It is high-frequency detail information extracted from low-level feature data of three-dimensional information images; It is the high-frequency detail information extracted from the low-level feature data of the spectral information; It is low-level feature data of the three-dimensional information image; It is the low-level feature data of the spectral information image; To process low-level feature data using Transformer encoder; To process low-level feature data using CNN encoder.

[0019] Optionally, the fused high-level feature data is spectral information;

[0020] The non-temporal arrangement of the fused high-level feature data includes:

[0021] After the fused high-level feature data are arranged in an arbitrary order independent of time, they are then arranged in order of wavelength size.

[0022] Optionally, the high-level feature data after non-temporal arrangement processing is represented as:

[0023] ;

[0024] ;

[0025] In the formula, It is high-level feature data after non-temporal arrangement processing; It is high-level feature data arranged in an arbitrary order independent of time; Real-time status of spectral information; is the fused high-level feature data; A is the state transfer matrix; B is the input matrix; C is the output matrix; and D is the transfer matrix.

[0026] Optionally, the discretized high-level feature data is represented as:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] In the formula, It is the high-level feature data after discretization; It is high-level feature data after non-temporal arrangement processing; is a discretized model; is the discretized state transfer matrix; is the discretized input matrix; C is the output matrix; L is the sequence length of the high-level feature data after non-temporal arrangement processing; is the sampling step; I is the unit matrix.

[0032] Optionally, the high-level feature data after slice-by-slice scanning is represented as:

[0033] PSS ( y ) = [ S 1 y , ⋅⋅⋅ , S R y ] ;

[0034] S r y = [ y ( r − 1 ) C 1 + n , ⋅⋅⋅ , y r C 1 ] T ∈ R C 1 + 1 ;

[0035] In the formula, It is the high-level feature data after scanning and processing piece by piece; is the high-level feature data after discretization; C1 is the length; R is the number of segments with the same length as C1; r∈[1, R]; T is the transpose of the matrix; For the The pixels of the high-level feature data after each segment scan and processing; S r ∈ [ S 1 , S R ] ; For the The feature values ​​of the high-level feature data pixels after the fragments are scanned and processed piece by piece, where n is a discrete counting constant, n∈[1, C1]; It is the total number of high-level feature data after scanning and processing slice by slice.

[0036] Optionally, the processed high-level feature data is subjected to two-dimensional feature synthesis through a synthesis module, wherein the synthesis module includes a pre-layer, a main path, a first branch path, and a second branch path, specifically:

[0037] The high-level feature data after the slice-by-slice scanning process is vectorized through the pre-layer to obtain the initial feature vector;

[0038] The initial feature vector is sequentially subjected to data normalization processing, data expansion, convolution, nonlinear activation, feature dimension processing, information extraction, modeling and normalization through the main path to obtain a feature vector of the main path;

[0039] Performing data expansion and nonlinear activation on the initial feature vector through a first branch to obtain a feature vector of the first branch;

[0040] Multiplying and fusing the feature vector of the main path and the feature vector of the first branch path to obtain a multi-level feature vector;

[0041] The multi-level feature vectors are processed in feature dimension to obtain feature vectors with reduced dimension; the feature vectors with reduced dimension are additively fused with the initial feature vectors through a second branch to obtain synthesized high-level feature data.

[0042] In a second aspect, the present invention provides a computer system, comprising:

[0043] Memory, for storing computer instructions;

[0044] A processor is used to execute the computer instructions to implement the steps of the surface object recognition method by fusing the spectral information image and the three-dimensional information image as described in the first aspect.

[0045] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the surface object recognition method of fusing spectral information images and three-dimensional information images as described in the first aspect.

[0046] Beneficial Effects

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention introduces a dual-branch feature extractor to respectively extract low-level features of spectral information images and three-dimensional information images, and obtains fused high-level feature data by encoding and projecting them into a high-level feature space. The fused high-level feature data is then processed in sequence by non-temporal linear arrangement processing, discretization processing, piece-by-piece scanning processing, and synthesis processing, thereby solving the problems that parallelization difficulties and huge computational complexity have become obstacles to the development of related technologies, improving the computing efficiency of computers, and thus improving the efficiency of surface object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a schematic flow chart of a method for surface recognition by fusing spectral information images and three-dimensional information images in Example 1 of the present invention;

[0050] Figure 2 The figure is a schematic flow chart of a method for surface recognition by fusing spectral information images and three-dimensional information images in Example 2 of the present invention;

[0051] Figure 3 It is a schematic diagram of the process of synthesizing two-dimensional features through a synthesis module of the present invention;

[0052] Figure 4 Shown is a schematic diagram of the surface identification result of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0054] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment introduces a surface object recognition method by fusing spectral information images and three-dimensional information images, including:

[0057] Obtain spectral information images and three-dimensional information images of surface objects.

[0058] Encoding the spectral information image and the three-dimensional information image respectively to obtain low-level feature data of the spectral information image and low-level feature data of the three-dimensional information image, projecting the low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image into a high-level feature space for fusion, and obtaining fused high-level feature data;

[0059] The fused high-level feature data is sequentially subjected to non-temporal arrangement processing, discretization processing, and piece-by-piece scanning processing to obtain processed high-level feature data;

[0060] Performing two-dimensional feature synthesis on the processed high-level feature data to obtain synthesized high-level feature data;

[0061] The synthesized high-level feature data is decoded and reconstructed to obtain the recognition results of surface objects.

[0062] When this embodiment is applied, the fused high-level feature data is obtained through the encoder and the high-level feature space, and then the fused high-level feature data is processed through non-time-series linear arrangement processing, discretization processing, piece-by-piece scanning processing and synthesis processing, which solves the problems of parallelization difficulties and huge computational complexity that have become obstacles to the development of related technologies, and improves the computer's computing efficiency. In particular, the high-efficiency advantage shown by this embodiment is more obvious in the application context of a large number of samples such as remote sensing.

[0063] Example 2

[0064] like Figure 2 As shown, based on Example 1, this embodiment further performs the following designs, including:

[0065] Obtaining spectral information images and three-dimensional information images of surface objects. In this embodiment, hyperspectral images and radar images are two different means of obtaining spectral information images and three-dimensional information images of surface objects in remote sensing monitoring. By fusing and classifying hyperspectral images and radar images, surface objects can be identified more accurately.

[0066] The spectral information image and the three-dimensional information image are encoded respectively to obtain low-level feature data of the spectral information image and low-level feature data of the three-dimensional information image. Specifically, it includes:

[0067] In most traditional fusion of spectral information images and three-dimensional information images, a three-step fusion process is followed. First, the source image is mapped to a specific transform domain, such as wavelet transform. Then, the transform coefficients are fused according to the fusion rule, and finally the fusion coefficients are transformed back to the image domain to obtain the final fused image.

[0068] In this embodiment, the spectral information source image features are respectively and 3D information source image features Input each dual-branch Transformer-CNN encoder, then map the low-level image features to the high-level feature space for fusion, and finally synthesize in the synthesis module. The encoder mainly consists of two components, namely the shared feature encoder (Shared Feature Encoder, SFE) based on the Restormer block and the local CNN encoder (Detail CNN Encoder, DCE) based on the invertible neural network (INN) block. The encoder in this embodiment can use the Transformer-CNN encoder. The specific operations are as follows:

[0069] Respectively , Input the shared feature encoder to obtain low-level features. The shared encoder extracts low-level features of two different modal data and obtains , The low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image are expressed as:

[0070] ;

[0071] ;

[0072] In the formula, It is the low-level feature data of the three-dimensional information image; It is the low-level feature data of the spectral information image; is three-dimensional information image data; is the spectral information image data; To process low-level feature data using encoders.

[0073] The low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image are projected into the high-level feature space to obtain fused high-level feature data. Specifically, the low-level feature semantics are projected into the high-level feature space through a function, and are transformed into high-level feature data after layer normalization and nonlinear activation function transformation.

[0074] The function of the shared feature encoder is to convert the original data into a feature representation suitable for further processing and analysis. The shared feature encoder in this embodiment mainly obtains low-dimensional data after extraction, which is easier to process in the next step than the original image.

[0075] The above low-level features are input into the local encoder (Detail CNNEncoder, DCE for short) based on the invertible neural network (INN) block. The local CNN encoder extracts high-frequency detail information from the features. , . Such as the formula:

[0076] ;

[0077] .

[0078] The purpose of using a local encoder is to extract local features such as the edge and texture features of a specific target from the original image, and in the extraction process, it focuses more on a small range without considering the overall structural impact of the image.

[0079] The above low-level features are input into the Lite Transformer-based encoder (Base Transformer Encoder, BTE for short), and the global CNN encoder extracts high-frequency detail information from the features , . Such as the formula:

[0080] ;

[0081] .

[0082] The purpose of using a global encoder is to collect global features, which in this embodiment are mainly color and spectrum features, etc., for detecting and classifying targets.

[0083] The above features are combined and added separately to obtain , . Such as the formula:

[0084] ;

[0085] .

[0086] In this embodiment, the advantage of using the local feature encoder and the global feature encoder in combination is that features of different levels and sizes can be obtained, and the data dimensions of the local features and the global features are higher than the results of the shared feature encoder, so that the fused results are more consistent with the original data.

[0087] The low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image are projected into the high-level feature space for fusion to obtain fused high-level feature data. The fused high-level feature data is expressed as:

[0088] ;

[0089] In the formula, It is the high-level feature data after fusion; It is the image feature fusion layer; It is high-frequency detail information extracted from low-level feature data of three-dimensional information images; It is the high-frequency detail information extracted from the low-level feature data of the spectral information image; It is low-level feature data of three-dimensional information image; It is the low-level feature data of the spectral information image; To process low-level feature data using Transformer encoder; To process low-level feature data using CNN encoder.

[0090] The fused high-level feature data is processed in a non-temporal arrangement to obtain high-level feature data after non-temporal arrangement. The fused high-level feature data is spectral information.

[0091] The non-temporal arrangement of the fused high-level feature data comprises:

[0092] After the fused high-level feature data are arranged in an arbitrary order independent of time, they are then arranged in order of wavelength.

[0093] Since the spectral span of spectral information is very long, each input pixel contains spectral information of multiple wavelengths. Each wavelength corresponds to a separate spectral band, and these spectral information can be divided into many spectral channels of the same wavelength range as needed. The spectral information is the fused high-level feature data, which is first arranged in a time-independent order and then arranged in order according to its own wavelength size.

[0094] The high-level feature data after the non-temporal arrangement processing is expressed as:

[0095] ;

[0096] ;

[0097] In the formula, It is high-level feature data after non-temporal arrangement processing; It is high-level feature data arranged in an arbitrary order independent of time; Real-time status of spectral information; is the fused high-level feature data; A is the state transfer matrix; B is the input matrix; C is the output matrix; and D is the transfer matrix.

[0098] In deep learning and generative models, the GPU's hidden state space usually refers to the intermediate calculation results and activation states stored on the GPU during model training and inference. These hidden states are part of the model, capturing the characteristics and patterns of the input data and used to generate output.

[0099] Hidden state is a key component of the network to remember and transfer information when processing sequence data. The hidden state is updated in each time step and passed as input to the next time step. The hidden state in RNN saves and transfers information in each time step, enabling RNN to process sequence data and capture dependencies between time steps. Through the hidden state, RNN can maintain contextual information when processing long sequence data. In this embodiment, the hidden potential state is the real-time state of spectral information.

[0100] The high-level feature data after the non-time-sequential arrangement processing is discretized to obtain the high-level feature data after the discretization processing.

[0101] The high-level feature data after the non-time-sequential arrangement processing is discretized, and the continuous series is cut into a number of small blocks.

[0102] Specifically: represents the discretized state transfer matrix, represents the discretized input matrix, both of which are discretized by the A and B matrices through the zero-order hold technique (ZOH), and the time scale parameter Represents the sampling step in the zero-order hold technique (ZOH). The longer the sampling step, the more the synthesis model focuses on the current features, and the shorter the sampling step, the more the synthesis model focuses on the features in the details.

[0103] The discretized model (discretized SSM) can be expressed as:

[0104] ;

[0105] in,

[0106] ;

[0107] ;

[0108] Where L is the sequence length of the high-level feature data after non-temporal arrangement processing; is the sampling step; I is the unit matrix.

[0109] In this embodiment, in order to better adapt to GPU computing, the discretization model can also be expressed by the following recursive function:

[0110] ;

[0111] ;

[0112] Finally, we get the global convolution, that is, the discretized high-level feature data, as shown in the formula:

[0113] ;

[0114] In the formula, It is the high-level feature data after discretization; It is high-level feature data after non-temporal arrangement processing; is a discretized model.

[0115] In this embodiment, the high-level data features are processed by discretization so that the system can work more efficiently. This processing method is more efficient than directly entering the module without preprocessing. The specific discretization measure is to divide the continuous spectral data features into several small blocks, randomly discard some of them and then participate in the next step of training.

[0116] Convert the input one-dimensional feature vector along the spatial direction into a two-dimensional feature vector. Specifically include:

[0117] The discretized high-level feature data is scanned piece by piece to obtain high-level feature data after the scanned piece by piece. The high-level feature data after the scanned piece by piece is expressed as:

[0118] PSS ( y ) = [ S 1 y , ⋅⋅⋅ , S R y ] ;

[0119] S r y = [ y ( r − 1 ) C 1 + n , ⋅⋅⋅ , y r C 1 ] T ∈ R C 1 + 1 ;

[0120] In the formula, It is the high-level feature data after scanning and processing piece by piece; is the high-level feature data after discretization; C1 is the length; R is the number of segments with the same length as C1; r∈[1, R]; T is the transpose of the matrix; For the The pixels of the high-level feature data after each segment scan and processing; S r ∈ [ S 1 , S R ] ; For the The feature values ​​of the high-level feature data pixels after the fragments are scanned and processed piece by piece, where n is a discrete counting constant, n∈[1, C1]; It is the total number of high-level feature data after scanning and processing slice by slice.

[0121] This method speeds up computer operations and can reveal the local characteristics of the spectrum profile, improving the accuracy and efficiency of the system.

[0122] That is, after scanning each slice, the feature vector is output from a one-dimensional direction vector to a two-dimensional vector in multiple segments. That is, cutting the feature into small segments.

[0123] The high-level feature data after the piece-by-piece scanning process is subjected to two-dimensional feature synthesis to obtain synthesized high-level feature data.

[0124] The processed high-level feature data is synthesized into two-dimensional features by a synthesis module, wherein the synthesis module includes a main path, a first branch path, and a second branch path, and specifically includes:

[0125] The high-level feature data after the piece-by-piece scanning process is subjected to feature vectorization through a pre-layer to obtain an initial feature vector with a length of k; wherein the scale of the pre-layer is k and is added before the Softmax function.

[0126] Specific formula:

[0127] ;

[0128] In the formula, is the feature vector of the input pre-layer (PCA layer); is the feature vector of the output pre-layer (PCA layer).

[0129] In the main path, the initial feature vector is subjected to data normalization processing to obtain a feature vector with reduced variable offset; the feature vector with reduced variable offset is subjected to data expansion to obtain a feature vector with data expansion; the feature vector with data expansion is subjected to convolution to obtain a convolved feature vector; the convolved feature vector is subjected to nonlinear activation to obtain an activated feature vector; the activated feature vector is subjected to feature dimension processing to obtain a feature vector with unchanged feature dimension; the feature vector with unchanged feature dimension is subjected to information extraction and modeling to obtain a modeled feature vector; the modeled feature vector is normalized to obtain a normalized feature vector.

[0130] In the first branch, the initial feature vector is subjected to data expansion and nonlinear activation to obtain the feature vector of the first branch.

[0131] The feature vector of the main path and the feature vector of the first branch path are multiplied and fused to obtain a multi-level feature vector.

[0132] The multi-level feature vectors are processed in feature dimension to obtain feature vectors with reduced dimension; the feature vectors with reduced dimension are additively fused with the initial feature vectors through a second branch to obtain synthesized high-level feature data.

[0133] In this embodiment, the synthesized high-level feature data is further subjected to feature dimension processing, normalization, and feature dimension processing again in sequence to output the synthesized high-level feature data.

[0134] The synthesis model consists of three branches. The main branch includes two normalization layers (LayerNorm layers) for input and expansion of state space, three linear layers for expansion, retention and compression of feature matrices, a SiLU activation function and an S6 module. The other two branches use skip connections and similarly excited multiplication for feature fusion, respectively. The branches are mainly used to adaptively transform the original information across layers. The synthesis model in this embodiment can use the Mamba model. The specific framework diagram is as follows: Figure 3 shown.

[0135] The processed high-level feature data is synthesized into two-dimensional features through the synthesis module, as shown in the formula:

[0136] ;

[0137] In the formula, is the input discrete pixel; It is S-type activation; It is a composite function composed of depth convolution and point-wise convolution; PW is point-wise convolution; DW is depth convolution; To perform the Einstein summation of two one-dimensional vectors along the spatial dimension.

[0138] In this embodiment, a processing method on a two-dimensional plane of gated space and spectrum synthesis is adopted instead of simply using one dimension to model data features. This method considers the relationship between spatial and spectral information more than before, thereby improving the accuracy of modeling results.

[0139] The synthesized high-level feature data is decoded and reconstructed to obtain the recognition results of surface objects.

[0140] The recognition results of the surface objects are obtained. Compared with other models, the synthetic model of this embodiment has greater advantages.

[0141] Example 3

[0142] like Figure 4 As shown, this embodiment introduces a specific experimental design of a surface object recognition method that fuses spectral information images and three-dimensional information images.

[0143] The hyperspectral remote sensing image dataset used in this embodiment is the Houston2013Dataset hyperspectral dataset, which was captured by an aerial visible / infrared imaging spectrometer sensor in Houston in 2013. The dataset contains spectral information images and three-dimensional information images of Houston in 2013, which were collected by the National Airborne Laser Mapping Center on the campus of the University of Houston using hyperspectral images (HSI) and radar images (LiDAR), especially the laser radar data method of the digital surface model (DSM). The spatial resolution of the hyperspectral image (HSI) and radar image (LiDAR) data is 2.5m, containing a total of 349×1905 pixels. Among them, the hyperspectral image (HSI) has 144 bands ranging from 0.38nm to 1.05nm. The dataset is labeled with 15029 ground truth samples and samples selected from 15 categories.

[0144] Active learning mainly uses the Transformer-CNN model, the new segmented sequential scanning (PSS) model, and the synthesis model to perform active learning and image classification on the data. Specifically, the synthesis model consists of three branches, the main branch includes two normalization layers (LayerNorm layers) for input and expansion of the state space, three linear layers for expansion, retention and compression of the feature matrix, and one by the SiLU activation function and the S6 module. The other two branches use skip connections and similar excitation multiplication for feature fusion, respectively, and the branches are mainly used to adaptively convert the original information across layers. The use of the synthesis model, a model with outstanding sequence modeling capabilities, can greatly reduce the computational redundancy in the field of remote sensing and promote the application and development of remote sensing in the future. The new segmented sequential scanning (PSS) model used in the present invention speeds up the system operation efficiency by dividing the image features into different small blocks. Specifically, the PSS model feature vector is output from a one-dimensional direction vector in multiple segments of two-dimensional output, that is, the feature is cut into small segments, which not only speeds up the computer operation speed but also enables the model to reveal the local characteristics of the spectrum profile.

[0145] Under the same data conditions, the image results of the present invention have an accuracy improvement of 1% to 2% compared with the CNN-Transforms processing under the traditional multi-layer perception method. Compared with the traditional classification method, the calculation amount and the number of formulas of this method are reduced by 46% and 79% respectively. In terms of the overall time of the experimental process, the computing efficiency of the operating system is improved by about 40% to 50%.

[0146] The above proves the feasibility of the present invention in image recognition. The present invention greatly reduces the calculation time of multimodal image fusion in the field of remote sensing images, so that under the same data conditions, the surface object recognition method of the embodiment of the spectral information image and the three-dimensional information image fusion can play a more outstanding performance.

[0147] Example 4

[0148] This embodiment introduces a computer system, including:

[0149] Memory, for storing computer instructions;

[0150] A processor is used to execute the computer instructions to implement the steps of the surface object recognition method of fusing spectral information images and three-dimensional information images as described in Example 1, 2 or 3.

[0151] Example 5

[0152] This embodiment introduces a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the surface object recognition method of fusing spectral information images and three-dimensional information images described in Embodiments 1, 2 or 3 are implemented.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for identifying ground objects by fusing spectral information images and three-dimensional information images, characterized in that: include: Obtain spectral information images and three-dimensional information images of surface objects; Encoding the spectral information image and the three-dimensional information image respectively to obtain low-level feature data of the spectral information image and low-level feature data of the three-dimensional information image, projecting the low-level feature data of the spectral information image and the low-level feature data of the three-dimensional information image into a high-level feature space for fusion, and obtaining fused high-level feature data; The fused high-level feature data is sequentially subjected to non-temporal arrangement processing, discretization processing, and piece-by-piece scanning processing to obtain processed high-level feature data; Performing two-dimensional feature synthesis on the processed high-level feature data to obtain synthesized high-level feature data; The synthesized high-level feature data is decoded and reconstructed to obtain the recognition results of surface objects.

2. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 1, characterized in that: The fused high-level feature data is expressed as: ; ; ; In the formula, It is the high-level feature data after fusion; It is the image feature fusion layer; It is high-frequency detail information extracted from low-level feature data of three-dimensional information images; It is the high-frequency detail information extracted from the low-level feature data of the spectral information image; It is low-level feature data of the three-dimensional information image; It is the low-level feature data of the spectral information image; To process low-level feature data using Transformer encoder; To process low-level feature data using CNN encoder.

3. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 1, characterized in that: The fused high-level feature data is spectral information; The non-temporal arrangement of the fused high-level feature data includes: After the fused high-level feature data are arranged in an arbitrary order independent of time, they are then arranged in order of wavelength size.

4. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 3, characterized in that: The high-level feature data after non-temporal arrangement processing is expressed as: ; ; In the formula, It is high-level feature data after non-temporal arrangement processing; It is high-level feature data arranged in an arbitrary order independent of time; Real-time status of spectral information; is the fused high-level feature data; A is the state transfer matrix; B is the input matrix; C is the output matrix; and D is the transfer matrix.

5. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 1, characterized in that: The high-level feature data after discretization is expressed as: ; ; ; ; In the formula, It is the high-level feature data after discretization; It is high-level feature data after non-temporal arrangement processing; is a discretized model; is the discretized state transfer matrix; is the discretized input matrix; is the output matrix; L is the sequence length of the high-level feature data after non-temporal arrangement processing; is the sampling step size; I is the identity matrix.

6. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 1, characterized in that: The high-level feature data after scanning and processing piece by piece is expressed as: ; ; In the formula, It is the high-level feature data after scanning and processing piece by piece; is the high-level feature data after discretization; C1 is the length; R is the number of segments with the same length as C1; r∈[1, R]; T is the transpose of the matrix; For the The pixels of the high-level feature data after each segment scan and processing; ; For the The feature values ​​of the high-level feature data pixels after the fragments are scanned and processed piece by piece, where n is a discrete counting constant, n∈[1, C1]; It is the total number of high-level feature data after scanning and processing slice by slice.

7. The method for identifying ground objects by fusing spectral information images and three-dimensional information images according to claim 1, characterized in that: The processed high-level feature data is synthesized into two-dimensional features through a synthesis module, wherein the synthesis module includes a pre-layer, a main path, a first branch path, and a second branch path, specifically: The high-level feature data after the slice-by-slice scanning process is vectorized through the pre-layer to obtain the initial feature vector; The initial feature vector is sequentially subjected to data normalization processing, data expansion, convolution, nonlinear activation, feature dimension processing, information extraction, modeling and normalization through the main path to obtain a feature vector of the main path; Performing data expansion and nonlinear activation on the initial feature vector through a first branch to obtain a feature vector of the first branch; Multiplying and fusing the feature vector of the main path and the feature vector of the first branch path to obtain a multi-level feature vector; The multi-level feature vectors are processed in feature dimension to obtain feature vectors with reduced dimension; the feature vectors with reduced dimension are additively fused with the initial feature vectors through a second branch to obtain synthesized high-level feature data.

8. A computer system, characterized in that: include: Memory, for storing computer instructions; A processor is used to execute the computer instructions to implement the steps of the method for identifying surface objects by fusing spectral information images and three-dimensional information images as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying a ground object by fusing a spectral information image and a three-dimensional information image according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Hyperspectral and laser radar data fusion classification method based on interactive Transform-CNN (Convolutional Neural Network)

    CN117994616A

  • Single-target tracking network based on combination of Vmamba and Transform

    CN118537369A

  • Hyperspectral image and laser radar image fusion method and system for field of remote sensing

    WO2024174314A1

Cited By

  • Space-time decoupling feature fusion method and system based on space-time bidirectional Vision Mama

    CN119693751A