Engineering facility target identification method and system based on multi-source fusion data

Abstract

The invention provides an engineering facility target identification method and system based on multi-source fusion data, and the method comprises the steps: obtaining a multi-source image of a to-be-processed region, carrying out the dense matching of the multi-source image through a multi-view stereo matching technology, and generating a three-dimensional point cloud; inputting the three-dimensional point cloud into a pre-constructed target recognition model for engineering facility target recognition to obtain the category of the engineering facility target; the target recognition model is obtained by training an initial target recognition model through a pre-constructed training sample, and the initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module and an up-sampling decoding module for semantic recognition; according to the method, the mechanism of front attention feature extraction, feature integration and semantic recognition of human vision is simulated through the target recognition model, precise recognition of the engineering facility target is achieved, the multi-level feature integration module ensures that each level feature always participates in semantic construction, and the recognition result is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target recognition, and particularly relates to an engineering facility target recognition method and system based on multi-source fusion data. BACKGROUND

[0002] With the rapid development of remote sensing technology, multi-source satellite data has been widely used in the fields of city planning, disaster monitoring, etc. Automatic recognition and extraction of engineering facility targets, such as ports, airports, oil depots, bridges, etc., is one of the important tasks of remote sensing interpretation. However, due to the complexity of imaging conditions, such as light changes, cloud and fog blocking, multi-angle imaging, etc., data quality is not the same, and the target form is diverse, so it is difficult for traditional methods to directly realize high-precision automatic recognition of engineering facility targets based on multi-source satellite data. Therefore, although multi-source satellite data fusion can provide rich color and texture information, how to effectively extract engineering facility targets from multi-source data is one of the core challenges of current remote sensing intelligent interpretation.

[0003] Some existing researches try to improve the performance of target recognition through early fusion (directly splicing multi-source data to extract features) or late fusion (splicing after extracting features respectively), but there are the following problems: feature conflict, that is, the data distribution of different sensors is quite different, such as the difference in imaging mechanism between optical images and SAR, which can easily introduce noise in direct fusion; ignoring the hierarchical feature correlation in the process of target recognition using data features, that is, the low-level features (such as edges, textures) and high-level semantic features (such as target structure) cannot be effectively optimized in coordination, and the obtained recognition result is not accurate enough. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application proposes an engineering facility target recognition method based on multi-source fusion data, which comprises:

[0005] Obtaining multi-source images of a region to be processed, and performing dense matching on the multi-source images through multi-view stereo matching technology to generate a three-dimensional point cloud;

[0006] Inputting the three-dimensional point cloud into a pre-constructed target recognition model to perform engineering facility target recognition, and obtaining the category of the engineering facility target;

[0007] The target recognition model is obtained by training an initial target recognition model through pre-constructed training samples, and the initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module, and an up-sampling decoding module for semantic recognition.

[0008] Optionally, the construction process of the initial target recognition model is as follows:

[0009] The idea of ​​densely connected convolutional networks is used to construct multiple independent self-learning sub-networks, each of which is used to process one of the primary features. The idea of ​​pseudo-twin architecture is used to integrate multiple self-learning sub-networks to construct a multi-feature parallel learning module. The input of the multi-feature parallel learning module is a three-dimensional point cloud, and the output is the primary features extracted at each level in each self-learning sub-network, which are recorded as feature components.

[0010] Based on a densely connected convolutional network, a multi-feature integration module is constructed for multi-level feature accumulation and transmission and multi-feature integration. The input of the multi-feature integration module is the feature components, and the output is the comprehensive features after fusion of each level;

[0011] In the feature fusion upsampling algorithm, a local attention mechanism is introduced to construct an upsampling decoding module, which is used to splice the comprehensive features of the corresponding layers layer by layer through the upsampling algorithm to obtain the spliced ​​comprehensive features, and the local attention mechanism is used to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0012] An initial target recognition model is obtained by combining the multi-feature parallel learning module, the multi-feature integration module and the upsampling decoding module.

[0013] Optionally, the multi-feature integration module for multi-level feature accumulation, transfer and multi-feature integration based on the densely connected convolutional network includes:

[0014] Based on densely connected convolutional networks, a cross-layer connection mechanism is introduced to build a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit.

[0015] The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fuse the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer;

[0016] The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations;

[0017] The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

[0018] Optionally, the primary features include one or more of the following: point cloud geometric shape features, point cloud color features, and point cloud three-dimensional texture features.

[0019] Optionally, the training process of the target recognition model is:

[0020] Initializing parameters in the initial target recognition model;

[0021] Input the three-dimensional point cloud samples in the training samples into the initial target recognition model, and use the multi-feature parallel learning module to extract multiple feature components; use the multi-feature integration module to fuse the feature components of the same level layer by layer to obtain the initial comprehensive features, and transfer the comprehensive features of the lower level across levels to fuse them with the initial comprehensive features of the level to obtain the comprehensive features of the level; use the upsampling decoding module to splice the comprehensive features of the corresponding levels layer by layer to obtain the spliced ​​comprehensive features, and use the local attention mechanism to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0022] Calculate the cross entropy loss between the target category and the true category label in the training sample, backpropagate the error through the optimizer and update the parameters in the initial target recognition model, iteratively perform the forward propagation, loss calculation and backpropagation processes until the iteration termination condition is reached, and obtain the trained target recognition model.

[0023] Optionally, the pre-built training samples include an expert knowledge base and a training sample library;

[0024] The construction process of the training sample library is as follows:

[0025] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0026] Using 3D point cloud processing software to annotate the 3D point cloud of each engineering facility target based on the image data, and obtain a target category label and a 3D point cloud label in each image data;

[0027] constructing an initial training sample library based on target category labels and three-dimensional point cloud labels in the image data;

[0028] The three-dimensional point cloud of the engineering facility target is randomly rotated and scaled at multiple angles and axes to simulate the three-dimensional point cloud corresponding to the engineering facility target under different viewing angles and environments to obtain simulation results, and the initial training sample library is expanded based on the simulation results to obtain a training sample library.

[0029] Optionally, the expert knowledge base is constructed as follows:

[0030] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0031] Using 3D point cloud processing software combined with expert experience to label the target category label and 3D point cloud label in each piece of image data;

[0032] Format the target category label and the three-dimensional point cloud label to obtain an expert knowledge base.

[0033] Based on the same inventive concept, the application provides an engineering facility target identification system based on multi-source fusion data, comprising:

[0034] A three-dimensional point cloud generation subsystem is configured to acquire multi-source images of a region to be processed, perform dense matching on the multi-source images through multi-view stereo matching technology, and generate three-dimensional point clouds.

[0035] A target identification subsystem is configured to input the three-dimensional point clouds into a pre-constructed target identification model, perform engineering facility target identification, and obtain the category of the engineering facility target.

[0036] The target identification model is obtained by training an initial target identification model using pre-constructed training samples, and the initial target identification model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module, and an up-sampling decoding module for semantic recognition.

[0037] Optionally, the target identification subsystem is further configured to:

[0038] A plurality of independent self-learning subnetworks are constructed using the idea of dense connection convolutional networks, and each self-learning subnetwork is configured to process one primary feature; a multi-feature parallel learning module is constructed by integrating a plurality of self-learning subnetworks using the idea of pseudo-twin architecture, and the input of the multi-feature parallel learning module is three-dimensional point clouds, and the output is primary features extracted by each self-learning subnetwork at different levels, which are denoted as feature components.

[0039] A multi-feature integration module for multi-level feature accumulation and transmission and multi-feature integration is constructed based on a dense connection convolutional network, and the input of the multi-feature integration module is the feature components, and the output is integrated comprehensive features at different levels.

[0040] In the up-sampling algorithm for feature fusion, an up-sampling decoding module is constructed by introducing a local attention mechanism, which is configured to obtain spliced comprehensive features by layer-by-layer splicing of corresponding level comprehensive features through the up-sampling algorithm, and to perform adaptive dynamic weighted fusion on the spliced comprehensive features through the local attention mechanism, and finally output the target category through a classifier.

[0041] The initial target identification model is obtained in combination with the multi-feature parallel learning module, the multi-feature integration module, and the up-sampling decoding module.

[0042] Optionally, the target identification subsystem is specifically configured to:

[0043] Based on densely connected convolutional networks, a cross-layer connection mechanism is introduced to build a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit.

[0044] The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fuse the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer;

[0045] The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations;

[0046] The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

[0047] Optionally, the primary features include one or more of the following: point cloud geometric shape features, point cloud color features, and point cloud three-dimensional texture features.

[0048] Optionally, the target identification subsystem is further configured to:

[0049] Initializing parameters in the initial target recognition model;

[0050] Input the three-dimensional point cloud samples in the training samples into the initial target recognition model, and use the multi-feature parallel learning module to extract multiple feature components; use the multi-feature integration module to fuse the feature components of the same level layer by layer to obtain the initial comprehensive features, and transfer the comprehensive features of the lower level across levels to fuse them with the initial comprehensive features of the level to obtain the comprehensive features of the level; use the upsampling decoding module to splice the comprehensive features of the corresponding levels layer by layer to obtain the spliced ​​comprehensive features, and use the local attention mechanism to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0051] Calculate the cross entropy loss between the target category and the true category label in the training sample, backpropagate the error through the optimizer and update the parameters in the initial target recognition model, iteratively perform the forward propagation, loss calculation and backpropagation processes until the iteration termination condition is reached, and obtain the trained target recognition model.

[0052] Optionally, the target identification subsystem is further configured to:

[0053] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0054] Using 3D point cloud processing software to annotate the 3D point cloud of each engineering facility target based on the image data, and obtain a target category label and a 3D point cloud label in each image data;

[0055] constructing an initial training sample library based on target category labels and three-dimensional point cloud labels in the image data;

[0056] The three-dimensional point cloud of the engineering facility target is randomly rotated and scaled at multiple angles and axes to simulate the three-dimensional point cloud corresponding to the engineering facility target under different viewing angles and environments to obtain simulation results, and the initial training sample library is expanded based on the simulation results to obtain a training sample library.

[0057] Optionally, the target identification subsystem is further configured to:

[0058] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0059] Using 3D point cloud processing software combined with expert experience to label the target category label and 3D point cloud label in each piece of image data;

[0060] The target category labels and the three-dimensional point cloud labels are formatted in a unified manner to obtain an expert knowledge base.

[0061] In another aspect, the present application further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0062] The memory is used to store one or more programs;

[0063] When the one or more programs are executed by the at least one processor, the engineering facility target recognition method based on multi-source fusion data as described above is implemented.

[0064] On the other hand, the present application also provides a computer-readable storage medium having an execution program stored thereon. When the execution program is executed, the engineering facility target identification method based on multi-source fusion data as described above is implemented.

[0065] Compared with the closest prior art, the present invention has the following beneficial effects:

[0066] The present invention provides an engineering facility target recognition method and system based on multi-source fusion data, including: obtaining multi-source images of the area to be processed, densely matching the multi-source images through multi-view stereo matching technology, and generating a three-dimensional point cloud; inputting the three-dimensional point cloud into a pre-constructed target recognition model to perform engineering facility target recognition and obtain the category of the engineering facility target; the target recognition model is obtained by training an initial target recognition model through pre-constructed training samples, and the initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module, and an upsampling decoding module for semantic recognition. In the present invention, the three-dimensional point cloud directly contains key information such as coordinates and colors, reducing the model's complex alignment of the original multi-source data and improving computational efficiency; through the multi-feature parallel learning module for extracting primary features, the multi-level feature integration module, and the upsampling decoding module for semantic recognition, the "pre-attention feature extraction → feature integration → semantic recognition" mechanism of human vision is simulated to achieve accurate recognition of engineering facility targets, and the multi-level feature integration module ensures that each level feature always participates in semantic construction, and the semantic recognition is more accurate, thereby ensuring that the final recognition result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of the process of the engineering facility target recognition method based on multi-source fusion data provided by the present invention;

[0068] Figure 2 A schematic diagram of the construction process of a deep learning model for engineering facility target recognition supporting a multi-feature visual attention mechanism provided by the present invention;

[0069] Figure 3 A schematic diagram of the structure of the engineering facility target recognition system based on multi-source fusion data provided by the present invention;

[0070] Figure 4 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0071] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0072] Example 1

[0073] The invention provides an engineering facility target recognition method based on multi-source fusion data, such as Figure 1 Shown, including:

[0074] S1. Acquire multi-source images of the area to be processed, and densely match the multi-source images using multi-view stereo matching technology to generate a three-dimensional point cloud;

[0075] S2. Inputting the three-dimensional point cloud into a pre-built target recognition model to perform engineering facility target recognition and obtain the category of the engineering facility target;

[0076] The target recognition model is obtained by training an initial target recognition model through pre-constructed training samples. The initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module and an upsampling decoding module for semantic recognition.

[0077] In step S1, multi-source images of the area to be processed are obtained, including but not limited to satellite remote sensing visible light images and SAR images;

[0078] Calculating the disparity map between the multi-source images using a semi-global matching algorithm or a PatchMatch block matching algorithm;

[0079] Combining the device parameters and image poses of the multi-source images, converting the disparity map into three-dimensional point coordinates to generate an initial sparse point cloud;

[0080] A high-precision three-dimensional point cloud of the area to be processed is obtained by performing dense matching optimization and filtering on the initial sparse point cloud.

[0081] In step S2, (1) before the engineering facility target is identified, the initial target identification model is constructed according to the following method, that is, the overall framework design of the target identification model is combined with Figure 2 Provide detailed explanation.

[0082] In this scheme, the construction principle of the initial target recognition model is: combining the stereoscopic vision human eye recognition experience with the classical visual theory, namely the attention feature integration theory, starting from the data conditions of satellite multi-source remote sensing data fusion, so that the target segmentation and recognition of the three-dimensional point cloud can achieve the human eye recognition effect, and deeply analyze the human eye stereoscopic vision mechanism of point cloud target recognition, and combine expert experience to analyze the optimization effect of the visual mechanism on the point cloud target recognition method, so as to guide the construction of this model: taking into account the human eye visual attention mechanism, it can effectively optimize the processing flow of point cloud classification; combining the attention feature integration mechanism (including the pre-attention stage and the feature integration stage) with deep learning, exploring the point cloud deep learning method that supports multi-feature parallel self-learning and multi-feature integration mechanism, which can effectively optimize the model.

[0083] Among them, the pre-attention stage includes the primary feature description of point clouds and the study of self-learning deep learning sub-modules; the feature integration stage is the study of the semantic features of engineering facility targets and structures and the deep learning sub-module of feature integration, and combines the construction of the overall deep learning model of engineering facility target recognition and structure three-dimensional boundary extraction, and finally realizes the deep learning technology of engineering facility target recognition and information extraction that supports multi-feature integration visual attention mechanism.

[0084] (1-1) The design idea of ​​the overall framework of the target recognition deep learning model (hereinafter referred to as the target recognition model) is as follows:

[0085] Determination of Primary Features for Engineering Facility Target Recognition: Based on human visual experience in stereoscopic recognition, in a 3D surface scene represented by multi-source data (specifically, 3D coordinates and color), the human eye can easily and clearly identify the types of engineering facilities, such as topography, mudflat types, docks, important warehouses, and office buildings, primarily by comprehensively perceiving the 3D shape, color, and texture features of engineering facility targets. Therefore, this proposal uses three types of features: point cloud geometric shape features, point cloud color features, and point cloud 3D texture features, as primary features for pre-attentive perception. Regarding the specific description of geometric shape features, the focus is on the performance of scale-invariant feature transformation methods in perceiving the local 3D spatial relationships of feature points. By incorporating the ability to perceive point cloud normals, orientations, gradients, and point importance, a self-learning function for the 3D shape of engineering facility targets in point clouds is derived. An analysis and comparison is conducted based on the types of engineering facility targets, recognition accuracy, and model efficiency, and an optimized algorithm for calculating geometric shape features that matches the specific engineering facility target type is selected. In terms of point cloud 3D texture, given that the data mainly describes the surface of the object, in addition to color texture, point cloud roughness is also selected to describe the point cloud 3D texture.

[0086] Analysis of the visual perception mechanism for engineering facility target and structure recognition: Based on an analysis of the human visual patterns in the recognition of engineering facility targets and structures from three-dimensional point clouds, "Under the influence of the stereoscopic visual feature integration mechanism, the engineering facility target and structure recognition task can be divided into a pre-attention stage of point cloud multi-primary feature perception and a feature integration stage of forming semantic target judgments of engineering facility targets and structures." Furthermore, primary feature perception is a process of parallel perception of multiple types of features horizontally, and a process of serial cumulative perception of multiple levels of features vertically, forming semantic features of engineering facility targets and structures. Furthermore, parallel multi-type feature perception and serial multi-level feature integration are not completely isolated; they continuously carry out multi-level feature accumulation and transmission, as well as multi-feature accumulation and integration, in both the vertical and horizontal directions. That is, the pre-attention stage and feature integration stage are relatively independent, yet they closely carry out the accumulation and transmission of multiple types and levels of features and information exchange. Therefore, the key issues and innovations are to use deep learning theories and methods to simulate the multi-feature parallel self-learning mechanism that supports primary feature perception, the multi-feature accumulation and integration mechanism in the feature integration stage of engineering facility targets and structures, and the relationship between the multi-level feature accumulation transmission, multi-feature integration, and multi-level and multi-feature accumulation and integration between the pre-attention stage and the feature integration stage.

[0087] The overall framework design approach for a deep learning model for engineering facility object and structure recognition: Based on the aforementioned regularity analysis, and to better design an engineering facility object and structure recognition model that incorporates an attention-based feature integration mechanism, this project, after analyzing the architectural concepts and characteristics of existing major deep learning models, proposes to employ the concepts of pseudo-twin neural networks and DenseNet deep learning networks for the overall model framework design. The pseudo-twin neural network concept will be used to implement a parallel self-learning model mechanism for primary features of point clouds during the pre-attention phase. The DenseNet network concept, with its powerful feature extraction, feature accumulation and transfer, and feature reuse capabilities, will be applied to the point cloud single feature self-learning submodule and the overall deep learning model architecture that supports multi-feature accumulation and integration. This will enable the continuous accumulation of primary features, thereby forming semantically advanced features of engineering facility objects and structures.

[0088] (1-2) Based on the design ideas of the overall framework of the target recognition model mentioned above, the overall framework of the target recognition model is constructed. The target recognition model consists of three modules: a multi-feature multi-level parallel self-learning module that simulates the pre-attention stage, a multi-feature integration module that simulates the feature integration stage, and an upsampling decoding module that simulates the multi-feature fusion of local attention.

[0089] Specifically, the construction of the initial target recognition model, that is, the process of building the target recognition model framework is as follows:

[0090] (a) constructing multiple independent self-learning sub-networks using the concept of densely connected convolutional networks, each of which is used to process one of the primary features;

[0091] The pseudo-twin architecture idea is used to integrate multiple self-learning sub-networks to construct a multi-feature parallel learning module;

[0092] The primary features include at least one of the following: point cloud geometric shape features, point cloud color features, and point cloud three-dimensional texture features.

[0093] That is, the multi-feature parallel learning module simulating the pre-attention stage first takes advantage of the DenseNet network (densely connected convolutional network) idea in multi-level feature accumulation self-learning to establish three-dimensional geometric shape, color, and three-dimensional texture self-learning sub-networks respectively: the pseudo-twin neural network idea is used to construct a feature parallel self-learning network structure, thereby realizing multi-feature self-learning and multi-feature parallel self-learning network architecture.

[0094] Employing a pseudo-twin architecture, each sub-network shares some basic convolution kernels at shallow layers, while independently optimizing deeper layers, balancing commonalities and unique characteristics. Cross-feature interaction modules are introduced at specific layers (selectable based on needs), allowing sub-networks processing different features to exchange information at specific layers, avoiding information silos. Shape, color, and texture have distinct physical meanings and require specific representations learned through independent parameters. However, similar engineering facilities share underlying features, such as edges and brightness distribution, so shallowly shared convolution kernels can improve efficiency.

[0095] The input of the above-mentioned multi-feature parallel learning module is a three-dimensional point cloud, and the output is the extracted feature components at each level in each self-learning sub-network, including geometric shape feature components, color feature components and three-dimensional texture feature components.

[0096] (b) Based on a densely connected convolutional network, a multi-feature integration module is constructed for multi-level feature accumulation and transmission and multi-feature integration. The input of the multi-feature integration module is the feature components, and the output is the comprehensive features after fusion of each level.

[0097] Taking advantage of the technical advantages of the DenseNet network idea in multi-level feature accumulation and transmission and multi-type feature integration, the multi-type and multi-level feature self-learning components obtained in the pre-attention stage are used as model input. Through the target and structure feature integration sub-module, the sub-features extracted at a certain level are integrated into the total features at that level. For example, the first-level extracted feature components FShapeL1, FcolorL1, FTextL1 (the geometric shape feature components, color feature components and three-dimensional texture feature components extracted at the first level) are integrated into the first-level comprehensive feature description F-ALL-L1, that is, the initial comprehensive feature. Then, the first-level comprehensive feature description F-ALL-L1 and the second-level extracted components FShapeL2, FcolorL2, FTextL2 (the geometric shape feature components, color feature components and three-dimensional texture feature components extracted at the second level) are integrated as the model input of the multi-feature integration module, thereby establishing a deep learning sub-model mechanism for engineering facility target and structure feature integration with multi-type feature integration, multi-level feature transmission and integration.

[0098] Specifically, based on the densely connected convolutional network, a cross-layer connection mechanism is introduced to construct a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit.

[0099] The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fuse the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer;

[0100] The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations;

[0101] The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

[0102] (c) In the feature fusion upsampling algorithm, a local attention mechanism is introduced to construct an upsampling decoding module, which is used to splice the comprehensive features of the corresponding levels layer by layer through the upsampling algorithm to obtain the spliced ​​comprehensive features, and to adaptively and dynamically weighted fuse the spliced ​​comprehensive features through the local attention mechanism, and finally output the target category through the classifier.

[0103] Through the local attention multi-feature fusion convolution layer, while performing cumulative decoding, the comprehensive feature description of the corresponding level is continuously combined to achieve upsampling decoding; finally, the semantic recognition of engineering facility targets and structures is achieved through the SoftMax classifier to obtain the target category.

[0104] (d) Combining the multi-feature parallel learning module, the multi-feature integration module and the upsampling decoding module to obtain an initial target recognition model.

[0105] Among them, the multi-level feature integration module uses the output features of each layer as the input of all subsequent layers based on the DenseNet idea, ensuring that low-level details always participate in high-level semantic construction and the output results are more accurate; in the pseudo-twin architecture, each sub-network shares some basic convolution kernels in the shallow layer to improve data processing efficiency, while the deep layer is independently optimized, which is suitable for independent learning of each sub-network, taking into account both commonalities and characteristics; upsampling provides a semantic framework by restoring the approximate position and category of the target, and the attention mechanism extracts key details from low-level features to avoid blur and improve the accuracy of target boundaries.

[0106] The entire model is constructed as a multi-type feature parallel self-learning sub-module in the horizontal direction, and a multi-level feature serial accumulation and transmission sub-module in the vertical direction to form the semantic feature integration sub-module of engineering facility targets and structures. The two are not completely independent. Through feature transmission, a cumulative transmission relationship of multi-type and multi-level features between the two is constructed. A design is made from multi-type and multi-level primary feature self-learning to multi-type and multi-level feature integration, and a self-learning deep learning model mechanism of engineering facility targets and structural semantic features is established to achieve full support and simulation of the visual mechanism of feature integration of attention.

[0107] (2) Training of target recognition model.

[0108] The pseudo-twin neural network structure is able to perform parallel, multi-level self-perception of different angle features of the same target, as well as the ability to integrate multiple features horizontally. First, the original data is self-learned and extracted through the sub-network of the pseudo-twin neural network for three-dimensional shape, color, three-dimensional texture and other different angle features. Then, through the Densenet multi-feature accumulation transmission and overall fusion features, the same level and multiple features are fused horizontally to obtain the fully perceived overall features of different levels of continuous abstraction. Afterwards, the DenseNet model idea is used to construct the fully perceived features of different levels and different types of features of engineering facility targets and structures to obtain the classification results. Finally, the network's error cost function is constructed in combination with the labels in the training samples, and the network is iteratively solved through the error feedback mechanism. Specifically:

[0109] Initializing parameters in the initial target recognition model;

[0110] Input the three-dimensional point cloud samples in the training samples into the initial target recognition model, and use the multi-feature parallel learning module to extract multiple feature components; use the multi-feature integration module to fuse the feature components of the same level layer by layer to obtain the initial comprehensive features, and transfer the comprehensive features of the lower level across levels to fuse them with the initial comprehensive features of the level to obtain the comprehensive features of the level; use the upsampling decoding module to splice the comprehensive features of the corresponding levels layer by layer to obtain the spliced ​​comprehensive features, and use the local attention mechanism to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0111] Calculate the cross entropy loss between the target category and the true category label in the training sample, backpropagate the error through the optimizer and update the parameters in the initial target recognition model, iteratively perform the forward propagation, loss calculation and backpropagation processes until the iteration termination condition is reached, and obtain the trained target recognition model.

[0112] The construction process of the above training samples is as follows.

[0113] Due to the influence of various factors such as weather conditions, sun angle, lighting conditions, imaging tilt angle, imaging resolution, terrain undulation, ground reflection, and ground landscape type, the quality of multi-source data varies. In particular, due to the influence of different viewing angles, shooting angles, backlighting, and backlighting, as well as the different imaging conditions of multi-source satellites such as optical satellites, SAR satellites, and hyperspectral satellites, there are widespread and obvious data omissions, data occlusions and data overlaps, shape or texture distortions, shadow interference, flying spots or dirty spots, and other data anomalies, all of which increase the difficulty of identifying engineering facility targets and structures.

[0114] The deep learning is used to develop an effective automatic identification technology of engineering facility targets. In addition to introducing a stereo vision mechanism, the expert knowledge base for engineering facility target identification needs to be established under the consideration of diversity, and the deep learning training sample library for engineering facility target identification needs to be established according to the principle of covering different factors and data abnormal conditions as comprehensively as possible.

[0115] The engineering facility targets in the training samples include targets of various facility types. According to the actual situation of the project, the following engineering facility types and corresponding target elements (target facilities) are mainly determined:

[0116] Facility type: topography, beach type, port, oil depot, road and bridge, airport, communication, etc.

[0117] Target facility: wharf, harbor basin, entrance, breakwater, oil tank, signal station, main road, etc.

[0118] For the types and elements that need to be added later, the classification system established in this project can be expanded by means of expanding samples, expanding feature descriptions, and model improvement to further increase the engineering facility types and target facilities.

[0119] The training samples include an expert knowledge base and a training sample library. The establishment process of the expert knowledge base is as follows: The expert knowledge base is a summary of the expert experience of engineering facility target and boundary extraction in a three-dimensional point cloud scene, which is mainly embodied by typical examples of expert identification. The establishment of the expert knowledge base provides the most basic reference for visual interpretation and sample library establishment. In this project, under the guidance of field experts, typical examples of three-dimensional point cloud target and structure three-dimensional boundary extraction are collected in different engineering facility types and element types by means of artificial interaction, which cover different scene backgrounds, same-track multi-angle / different-track multi-angle imaging and other different data conditions.

[0120] Specifically, image data of various types of engineering facility targets in multi-angle and multi-environment are acquired;

[0121] The target class label and three-dimensional point cloud label in each of the image data are labeled by using a three-dimensional point cloud processing software combined with expert experience;

[0122] The target class label and three-dimensional point cloud label are uniformly formatted to obtain the expert knowledge base.

[0123] The construction process of the training sample library is as follows:

[0124] Image data of various types of engineering facility targets in multi-angle and multi-environment are acquired;

[0125] Using 3D point cloud processing software to annotate the 3D point cloud of each engineering facility target based on the image data, and obtain a target category label and a 3D point cloud label in each image data;

[0126] constructing an initial training sample library based on target category labels and three-dimensional point cloud labels in the image data;

[0127] The three-dimensional point cloud of the engineering facility target is randomly rotated and scaled at multiple angles and axes to simulate the three-dimensional point cloud corresponding to the engineering facility target under different viewing angles and environments to obtain simulation results, and the initial training sample library is expanded based on the simulation results to obtain a training sample library.

[0128] The sample selection must be representative. The samples of each target type need to cover different situations such as different acquisition sources, different resolutions, typical imaging conditions, typical landscape types, typical data anomalies, etc. (see Table 1 for details); the number of samples for each target and covering various typical situations should be distributed as evenly as possible; the number of samples for each target type and covering various typical situations must reach a certain number.

[0129] Table 1 Typical situations that samples need to cover

[0130]

[0131] Due to the scarcity of engineering facility sample data, it is necessary to use sample enhancement and sample simulation technology to enhance the obtained raw data samples for pre-processing, mainly including random multi-angle and multi-axis rotation, random scaling of point cloud size, random adjustment of brightness, contrast, saturation, hue, etc. in color, random sub-area interception of point cloud samples, etc., and the sample enhancement results are placed in the overall sample library. This technology is simple and practical, and can effectively increase the richness of the sample library, reduce the impact of factors such as scale, rotation, angle, etc. on the image target recognition model, and improve the robustness of the model. In addition, for some difficult-to-obtain training samples, sample simulation can also be performed through point cloud sample simulation technology. For example, point cloud samples can be simulated by adding random point cloud noise to ensure that the samples fully cover typical situations.

[0132] Specifically, before acquiring the image data, a sampling principle for image data was developed based on factors affecting the quality of 3D point cloud data fused from multi-source satellite data and typical data. Factors affecting data quality were considered to avoid acquiring image data with poor resolution, obstruction, or excessive noise, while typical data was considered to ensure sample diversity.

[0133] The three-dimensional point cloud processing software is combined with the three-dimensional point cloud labeled by expert experience to expand the sample, so that the sample type of the training library is reasonable, the type distribution is reasonable, the representation is strong, the sample imaging conditions are complex and diverse, and the sample has a certain scale.

[0134] The point cloud sample enhancement technology includes geometric enhancement and occlusion simulation. The geometric enhancement includes rotating, scaling, and translating the three-dimensional point cloud, and increasing the diversity of the view angle. The occlusion simulation includes randomly removing part of the point cloud to simulate the occlusion in the real scene. The point cloud sample simulation technology includes generating a simulated three-dimensional point cloud of a target facility based on a SketchUp modeling software, and supplementing rare samples such as power facilities and rare building structures.

[0135] Thus, the target recognition model is obtained, and is used for identifying the engineering facility target.

[0136] In step S2, the three-dimensional point cloud is input into a pre-constructed target recognition model to identify the engineering facility target, and the category of the engineering facility target is obtained.

[0137] Specifically, the three-dimensional point cloud sample is input into the target recognition model, a plurality of feature components are extracted by using a multi-feature parallel learning module, the feature components of the same level are fused by using a multi-feature integration module to obtain comprehensive features, and the comprehensive features are transmitted across levels. Low-layer high-resolution comprehensive features of corresponding levels are spliced layer by layer by using an up-sampling decoding module, and the spliced comprehensive features are adaptively and dynamically weighted and fused by using a local attention mechanism. Finally, the category of the engineering facility target is output by using a classifier.

[0138] In the scheme, the three-dimensional point cloud directly contains key information such as coordinates and colors, reduces the complex alignment of the model to the original multi-source data, improves the calculation efficiency, avoids the feature conflict caused by the difference between data sources, makes the model learning more stable, and simulates the “pre-attention feature extraction→feature integration→semantic recognition” mechanism of human vision by using the multi-feature parallel learning module, the multi-feature integration module, and the up-sampling decoding module for semantic recognition. The primary features (geometry, color, and texture) of the multi-source point cloud data are integrated into high-level semantic features, that is, specific targets, so that the engineering facility target can be accurately identified and extracted. Moreover, the model directly identifies the regional three-dimensional point cloud, has high automation, and has high identification efficiency.

[0139] The primary feature perception performs multi-type feature parallel perception in the horizontal direction, captures diversified local information such as geometry, color, and texture of the engineering facility target, performs multi-level feature serial cumulative perception and semantic feature recognition in the vertical direction, gradually abstracts global semantic features, and significantly improves the target recognition accuracy in a complex scene.

[0140] In traditional networks, the input of each layer comes only from the output of the previous layer. In densely connected convolutional networks, the input of each layer is the concatenation of the outputs of all previous layers. The underlying features in the convolutional network can be directly passed to the higher layers, improving feature utilization. Due to the densely connected structure, any layer can directly access the gradient of the loss function. The gradient path is shorter during backpropagation and the training is more stable. The resulting trained target recognition model has higher precision and more accurate target recognition results.

[0141] Example 2

[0142] Based on the same inventive concept, the present invention also provides an engineering facility target recognition system based on multi-source fusion data, such as Figure 3 Shown, including:

[0143] A 3D point cloud generation subsystem is used to obtain multi-source images of the area to be processed, and densely match the multi-source images using multi-view stereo matching technology to generate a 3D point cloud;

[0144] A target recognition subsystem is used to input the three-dimensional point cloud into a pre-built target recognition model to perform engineering facility target recognition and obtain the category of the engineering facility target;

[0145] The target recognition model is obtained by training an initial target recognition model through pre-constructed training samples. The initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module and an upsampling decoding module for semantic recognition.

[0146] In one possible implementation, the target recognition subsystem is further configured to:

[0147] The idea of ​​densely connected convolutional networks is used to construct multiple independent self-learning sub-networks, each of which is used to process one of the primary features. The idea of ​​pseudo-twin architecture is used to integrate multiple self-learning sub-networks to construct a multi-feature parallel learning module. The input of the multi-feature parallel learning module is a three-dimensional point cloud, and the output is the primary features extracted at each level in each self-learning sub-network, which are recorded as feature components.

[0148] Based on a densely connected convolutional network, a multi-feature integration module is constructed for multi-level feature accumulation and transmission and multi-feature integration. The input of the multi-feature integration module is the feature components, and the output is the comprehensive features after fusion of each level;

[0149] In the feature fusion upsampling algorithm, a local attention mechanism is introduced to construct an upsampling decoding module, which is used to splice the comprehensive features of the corresponding layers layer by layer through the upsampling algorithm to obtain the spliced ​​comprehensive features, and the local attention mechanism is used to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0150] An initial target recognition model is obtained by combining the multi-feature parallel learning module, the multi-feature integration module and the upsampling decoding module.

[0151] In one possible implementation, the target recognition subsystem is specifically configured to:

[0152] Based on densely connected convolutional networks, a cross-layer connection mechanism is introduced to build a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit.

[0153] The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fuse the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer;

[0154] The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations;

[0155] The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

[0156] In a possible implementation, the primary features include one or more of the following: point cloud geometric shape features, point cloud color features, and point cloud three-dimensional texture features.

[0157] In one possible implementation, the target recognition subsystem is further configured to:

[0158] Initializing parameters in the initial target recognition model;

[0159] Input the three-dimensional point cloud samples in the training samples into the initial target recognition model, and use the multi-feature parallel learning module to extract multiple feature components; use the multi-feature integration module to fuse the feature components of the same level layer by layer to obtain the initial comprehensive features, and transfer the comprehensive features of the lower level across levels to fuse them with the initial comprehensive features of the level to obtain the comprehensive features of the level; use the upsampling decoding module to splice the comprehensive features of the corresponding levels layer by layer to obtain the spliced ​​comprehensive features, and use the local attention mechanism to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier;

[0160] Calculate the cross entropy loss between the target category and the true category label in the training sample, backpropagate the error through the optimizer and update the parameters in the initial target recognition model, iteratively perform the forward propagation, loss calculation and backpropagation processes until the iteration termination condition is reached, and obtain the trained target recognition model.

[0161] In one possible implementation, the target recognition subsystem is further configured to:

[0162] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0163] Using 3D point cloud processing software to annotate the 3D point cloud of each engineering facility target based on the image data, and obtain a target category label and a 3D point cloud label in each image data;

[0164] constructing an initial training sample library based on target category labels and three-dimensional point cloud labels in the image data;

[0165] The three-dimensional point cloud of the engineering facility target is randomly rotated and scaled at multiple angles and axes to simulate the three-dimensional point cloud corresponding to the engineering facility target under different viewing angles and environments to obtain simulation results, and the initial training sample library is expanded based on the simulation results to obtain a training sample library.

[0166] In one possible implementation, the target recognition subsystem is further configured to:

[0167] Acquire image data of various types of engineering facilities in multiple angles and environments;

[0168] Using 3D point cloud processing software combined with expert experience to label the target category label and 3D point cloud label in each piece of image data;

[0169] The target category labels and the three-dimensional point cloud labels are formatted in a unified manner to obtain an expert knowledge base.

[0170] Example 3

[0171] like Figure 4 As shown, the present invention also provides an electronic device, which may be a computer, a single-chip microcomputer, a smart mobile device, or the like. The electronic device in this embodiment may include a processor, a memory, a transceiver component, and the like. The memory, processor, and transceiver component are connected via a bus; the memory may be used to store an execution program, which may include instructions; and the processor may be used to execute the instructions stored in the memory. The memory may also be used to store data, which may be accessed and / or modified during the execution of the instructions.

[0172] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding function, so as to realize the steps of an engineering facility target identification method based on multi-source fusion data in the above embodiment.

[0173] Example 4

[0174] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in the electronic device for storing programs and data. It can be understood that the storage medium here can include both the built-in storage medium in the electronic device and, of course, the extended storage medium supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of the engineering facility target identification method based on multi-source fusion data in the above embodiment.

[0175] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art may still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims.

Claims

1. The engineering facility target recognition method based on multi-source fusion data is characterized by: include: Acquire multi-source images of the area to be processed, and perform dense matching on the multi-source images using multi-view stereo matching technology to generate a three-dimensional point cloud; Inputting the three-dimensional point cloud into a pre-built target recognition model to perform engineering facility target recognition and obtain the category of the engineering facility target; The target recognition model is obtained by training an initial target recognition model through pre-constructed training samples. The initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module and an upsampling decoding module for semantic recognition.

2. The method according to claim 1, wherein The construction process of the initial target recognition model is as follows: Using the idea of ​​densely connected convolutional networks, multiple independent self-learning sub-networks are constructed, each of which is used to process one of the primary features; The pseudo-twin architecture is used to integrate multiple self-learning sub-networks to construct a multi-feature parallel learning module. The input of the multi-feature parallel learning module is a three-dimensional point cloud, and the output is the primary features extracted by each self-learning sub-network, which are recorded as feature components. Based on a densely connected convolutional network, a multi-feature integration module is constructed for multi-level feature accumulation and transmission and multi-feature integration. The input of the multi-feature integration module is the feature components, and the output is the comprehensive features after fusion of each level; In the feature fusion upsampling algorithm, a local attention mechanism is introduced to construct an upsampling decoding module, which is used to splice the comprehensive features of the corresponding layers layer by layer through the upsampling algorithm to obtain the spliced ​​comprehensive features, and the local attention mechanism is used to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier; An initial target recognition model is obtained by combining the multi-feature parallel learning module, the multi-feature integration module and the upsampling decoding module.

3. The method according to claim 2, wherein The multi-feature integration module for multi-level feature accumulation, transmission and multi-feature integration based on the densely connected convolutional network is constructed, including: Based on densely connected convolutional networks, a cross-layer connection mechanism is introduced to build a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit. The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fuse the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer; The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations; The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

4. The method according to claim 2, wherein The primary features include one or more of the following: point cloud geometric shape features, point cloud color features, and point cloud three-dimensional texture features.

5. The method according to claim 2 or 3, wherein: The training process of the target recognition model is as follows: Initializing parameters in the initial target recognition model; Input the three-dimensional point cloud samples in the training samples into the initial target recognition model, and use the multi-feature parallel learning module to extract multiple feature components; use the multi-feature integration module to fuse the feature components of the same level layer by layer to obtain the initial comprehensive features, and transfer the comprehensive features of the lower level across levels to fuse them with the initial comprehensive features of the level to obtain the comprehensive features of the level; use the upsampling decoding module to splice the comprehensive features of the corresponding levels layer by layer to obtain the spliced ​​comprehensive features, and use the local attention mechanism to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier; Calculate the cross entropy loss between the target category and the true category label in the training sample, backpropagate the error through the optimizer and update the parameters in the initial target recognition model, iteratively perform the forward propagation, loss calculation and backpropagation processes until the iteration termination condition is reached, and obtain the trained target recognition model.

6. The method according to claim 1, wherein The pre-built training samples include an expert knowledge base and a training sample base; The construction process of the training sample library is as follows: Acquire image data of various types of engineering facilities in multiple angles and environments; Using 3D point cloud processing software to annotate the 3D point cloud of each engineering facility target based on the image data, and obtain a target category label and a 3D point cloud label in each image data; constructing an initial training sample library based on target category labels and three-dimensional point cloud labels in the image data; The three-dimensional point cloud of the engineering facility target is randomly rotated and scaled at multiple angles and axes to simulate the three-dimensional point cloud corresponding to the engineering facility target under different viewing angles and environments to obtain simulation results, and the initial training sample library is expanded based on the simulation results to obtain a training sample library.

7. The method according to claim 6, wherein The construction process of the expert knowledge base is as follows: Acquire image data of various types of engineering facilities in multiple angles and environments; Using 3D point cloud processing software combined with expert experience to label the target category label and 3D point cloud label in each piece of image data; The target category labels and the three-dimensional point cloud labels are formatted in a unified manner to obtain an expert knowledge base.

8. The engineering facility target recognition system based on multi-source fusion data is characterized by: include: A 3D point cloud generation subsystem is used to obtain multi-source images of the area to be processed, and densely match the multi-source images using multi-view stereo matching technology to generate a 3D point cloud; The target category recognition subsystem is used to input the three-dimensional point cloud into a pre-built target recognition model to perform engineering facility target recognition and obtain the category of the engineering facility target; The target recognition model is obtained by training an initial target recognition model through pre-constructed training samples. The initial target recognition model is constructed based on a multi-feature parallel learning module for extracting primary features, a multi-feature integration module and an upsampling decoding module for semantic recognition.

9. The system according to claim 8, wherein The target category identification subsystem is further used to: Using the idea of ​​densely connected convolutional networks, multiple independent self-learning sub-networks are constructed, each of which is used to process one of the primary features; The pseudo-twin architecture is used to integrate multiple self-learning sub-networks to construct a multi-feature parallel learning module. The input of the multi-feature parallel learning module is a three-dimensional point cloud, and the output is the primary features extracted at each level in each self-learning sub-network, which are recorded as feature components. Based on a densely connected convolutional network, a multi-feature integration module is constructed for multi-level feature accumulation and transmission and multi-feature integration. The input of the multi-feature integration module is the feature components, and the output is the comprehensive features after fusion of each level; In the feature fusion upsampling algorithm, a local attention mechanism is introduced to construct an upsampling decoding module, which is used to splice the comprehensive features of the corresponding layers layer by layer through the upsampling algorithm to obtain the spliced ​​comprehensive features, and the local attention mechanism is used to adaptively and dynamically weight the spliced ​​comprehensive features, and finally output the target category through the classifier; An initial target recognition model is obtained by combining the multi-feature parallel learning module, the multi-feature integration module and the upsampling decoding module.

10. The system according to claim 9, wherein: The target category identification subsystem is specifically used to: Based on densely connected convolutional networks, a cross-layer connection mechanism is introduced to build a hierarchical multi-feature integration module. Each layer includes a feature fusion unit, a densely connected convolution block, and a cross-layer connection unit. The feature fusion unit is used to splice and fuse multiple feature components to output the initial comprehensive features of the layer; and fusing the low-level comprehensive features received by the cross-layer connection unit with the high-level semantic features obtained by the densely connected convolution block to obtain the comprehensive features of the layer; The densely connected convolution block is used to deepen the initial comprehensive features into high-level semantic features through dense connections and nonlinear transformations; The cross-layer connection unit is used to downsample the comprehensive features of the low level to match the size of the level through pooling, and then transfer the comprehensive features of the low level to the level where it is located.

Images