Method for assisting classification extraction of remote sensing images by using multi-modal data
By selecting multimodal remote sensing image data with close features in the neural network model for fusion, and using auxiliary models for correction training, monitoring the fusion process for early warning processing, the problem of difficulty and poor accuracy of multimodal data fusion is solved. Improve the accuracy of remote sensing image classification and extraction.
Patent Information
- Application Number
- CN202410808230.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the data fusion of multimodal remote sensing images is difficult, which affects the difficulty of model training and classification extraction accuracy, and the deterioration of data accuracy after fusion affects the final result.
In neural network model training, data with close features are selected for fusion, and data features with large differences are used as auxiliary models for correction training, monitoring the multimodal data fusion process, generating influence coefficients and issuing early warning prompts to ensure data accuracy.
The model training complexity and number of parameters are reduced, the classification accuracy of the backbone network model is improved, and data fusion accuracy abnormalities are discovered and processed in a timely manner, ensuring the accuracy of remote sensing image classification and extraction.
Smart Images

Figure CN118968114A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image classification and extraction, and in particular to a method for assisting remote sensing image classification and extraction using multimodal data. Background Art
[0002] In the existing deep learning field, multimodality refers to the use of multiple forms of data for training and learning in deep learning, such as text, images, videos, audio, etc. In deep learning, different types of data can work together and complement each other to improve the accuracy and efficiency of the model. In the field of remote sensing image data, multimodal data refers to: optical remote sensing image data, SAR radar data, hyperspectral image data, DEM terrain data, etc. These data are different in form, so in deep learning, multimodal technology is needed to fuse the data and train the neural network model.
[0003] The prior art has the following deficiencies:
[0004] 1. The core of the multimodal method for classifying and extracting remote sensing images in the existing technology is to use the multimodal fusion strategy in computer vision to provide data support for the training of deep neural network models by fusing data in different forms. However, the disadvantage of this method is that data fusion is difficult and the training of the fused model is difficult, which directly affects the accuracy of the final remote sensing image classification extraction.
[0005] 2. When the accuracy of multimodal remote sensing image data fusion is affected and deteriorated, the accuracy of the fused data will also be affected and deteriorated. When the remote sensing images are subsequently classified and extracted through multimodal methods, it will also directly affect the accuracy of the final remote sensing image classification extraction.
[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention
[0007] The purpose of the present invention is to provide a method for assisting remote sensing image classification and extraction using multimodal data. In the process of neural network model training, data with similar features are selected for fusion, and data with too large a difference in features are used as an auxiliary model for correction training, which can greatly reduce the difficulty of model training, and can train auxiliary modules and backbone networks separately, reducing the complexity of the model training process, reducing network model parameters, and at the same time improving the classification accuracy of the backbone network model. By monitoring the process of multimodal remote sensing image data fusion, when the accuracy of multimodal remote sensing image data fusion is greatly affected, the accuracy of the fused data may also be affected and deteriorated, which may eventually affect the accuracy of remote sensing image classification and extraction. When this happens, an early warning prompt is issued to notify relevant users of the situation, so that timely measures can be taken to timely perform operation and maintenance management on abnormal data fusion accuracy, ensure the accuracy of multimodal remote sensing image data fusion, and then ensure the accuracy of final remote sensing image classification and extraction, so as to solve the problems in the above-mentioned background technology.
[0008] In order to achieve the above object, the present invention provides the following technical solution: a method for using multimodal data to assist remote sensing image classification and extraction, comprising the following steps:
[0009] S101, taking the optical remote sensing image data as the main line, selecting multimodal data with data feature forms similar to the optical remote sensing image data, and fusing the selected data into a training data set of the backbone classification network;
[0010] S102, screening the data that is not similar to the backbone network data in the multimodal data, and fusing the screened data into a training data set for the auxiliary model;
[0011] S103, obtaining a plurality of information when the multimodal remote sensing image data is fused, including physical characteristic information and time domain information, and after obtaining, processing the physical characteristic information and time domain information when the multimodal remote sensing image data is fused;
[0012] S104, comprehensively analyzing the processed physical characteristic information and time domain information when fusing the multimodal remote sensing image data to generate an influence coefficient;
[0013] S105, comparing and analyzing the influence coefficient generated when the multimodal remote sensing image data is fused with a preset reference threshold of the influence coefficient, generating a high-impact signal or a low-impact signal, and issuing an early warning prompt for the high-impact signal;
[0014] S106, training the main classification network model and the auxiliary network model respectively to obtain a main classification network model file and an auxiliary classification network model file;
[0015] S107, deploy and test the network model. After the data passes through the backbone network model, a classification result will be obtained. The result will then be determined by the auxiliary model to finally obtain the extraction result.
[0016] Preferably, the physical characteristic information when fusing multimodal remote sensing image data includes a resolution deviation coefficient and a band spectrum deviation coefficient. After acquisition, the resolution deviation coefficient and the band spectrum deviation coefficient are calibrated as and
[0017] Preferably, the logic of obtaining the resolution deviation coefficient is as follows:
[0018] A101. Obtain the actual resolution of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual resolution as x represents the number of actual resolutions of several data to be fused within T time when multimodal remote sensing image data are fused, x=1, 2, 3, 4, ..., m, where m is a positive integer;
[0019] A102. The total resolution deviation is calculated by the actual resolution of two adjacent multimodal remote sensing image data. The calculation expression is: In the formula, FBL' represents the total resolution deviation, and m represents the total amount of actual resolution obtained within T time when multimodal remote sensing image data is fused;
[0020] A103. Calculate the resolution deviation coefficient. The calculation expression is:
[0021] Preferably, the logic of obtaining the band spectrum deviation coefficient is as follows:
[0022] B101. Obtain the actual band spectrum bandwidth of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual band spectrum bandwidth as y represents the number of the actual band spectrum bandwidth of several data to be fused within T time when the multimodal remote sensing image data is fused, y = 1, 2, 3, 4, ..., n, n is a positive integer;
[0023] B102. Calculate the standard deviation and average value of the band spectrum bandwidth by using the actual band spectrum bandwidth obtained within T time when fusing multimodal remote sensing image data, and calibrate the standard deviation and average value of the band spectrum bandwidth as and but: but:
[0024] B103. Standard deviation of band spectrum bandwidth obtained within T time when multimodal remote sensing image data is fused and band spectrum bandwidth average Calculate the band spectrum bandwidth variation index and calibrate the band spectrum bandwidth variation index as but:
[0025] B104. Calculate the band spectrum deviation coefficient. The calculation expression is:
[0026] Preferably, the time domain information when fusing the multimodal remote sensing image data includes a time synchronization deviation coefficient. After acquisition, the time synchronization deviation coefficient is calibrated as SJ ω .
[0027] Preferably, the logic of obtaining the time synchronization deviation coefficient is as follows:
[0028] C101. Obtain the initial time interval for data collection of multimodal remote sensing image data and calibrate the initial time interval as SJ ω 初始 ;
[0029] C102. Obtain several actual time intervals generated within T time when acquiring multimodal remote sensing image data, and calibrate the actual time intervals as SJ ω k , k represents the number of actual time intervals generated within T time when multimodal remote sensing image data is collected, k = 1, 2, 3, 4, ..., p, p is a positive integer;
[0030] C103. Calculate the time synchronization deviation coefficient. The calculation expression is:
[0031] Preferably, the resolution deviation coefficient after processing when obtaining multimodal remote sensing image data for fusion is Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω After that, and SJ ω Perform a formulaic analysis to generate impact coefficients The formula is: Where f1, f2, and f3 are the resolution deviation coefficients respectively. Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω The preset proportional coefficient, and f1, f2, and f3 are all greater than 0.
[0032] Preferably, the influence coefficient generated when the multimodal remote sensing image data is fused is compared and analyzed with a preset influence coefficient reference threshold, and the comparison and analysis is as follows:
[0033] If the impact coefficient is greater than or equal to the impact coefficient reference threshold, a high impact signal is generated and an early warning prompt is issued for the high impact signal;
[0034] If the impact coefficient is less than the impact coefficient reference threshold, a low impact signal is generated, and no early warning is issued for the low impact signal.
[0035] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0036] The present invention proposes an auxiliary multimodal method. In the process of neural network model training, data with similar features are selected for fusion, and data with too large a difference in features are used as an auxiliary model for correction training, which can greatly reduce the difficulty of model training, and can train the auxiliary module and the backbone network separately, thereby reducing the complexity of the model training process, reducing the network model parameters, and at the same time improving the classification accuracy of the backbone network model.
[0037] The present invention monitors the process of multimodal remote sensing image data fusion. When the accuracy of multimodal remote sensing image data fusion is greatly affected, the accuracy of the fused data may also be affected and deteriorate, which may eventually affect the accuracy of remote sensing image classification and extraction. When this happens, an early warning prompt is issued to notify relevant users of the situation, so that timely measures can be taken to timely perform operation and maintenance management on the abnormal data fusion accuracy, thereby ensuring the accuracy of multimodal remote sensing image data fusion, and further ensuring the accuracy of the final remote sensing image classification and extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0039] Figure 1 The present invention is a method flow chart of the method for utilizing multimodal data to assist remote sensing image classification and extraction. DETAILED DESCRIPTION
[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.
[0041] The present invention provides Figure 1 A method for using multimodal data to assist remote sensing image classification and extraction is shown, comprising the following steps:
[0042] S101, taking the optical remote sensing image data as the main line, selecting multimodal data with data feature forms similar to the optical remote sensing image data, and fusing the selected data into a training data set of the backbone classification network;
[0043] Multimodal data with similar features to optical remote sensing image data include multispectral data, panchromatic image data, hyperspectral data, infrared remote sensing data, radar remote sensing data, etc.
[0044] S102, screening the data that is not similar to the backbone network data in the multimodal data, and fusing the screened data into a training data set for the auxiliary model;
[0045] Data that is not similar to the backbone network data includes SAR radar data, laser point cloud data, etc.
[0046] S103, obtaining a plurality of information when the multimodal remote sensing image data is fused, including physical characteristic information and time domain information, and after obtaining, processing the physical characteristic information and time domain information when the multimodal remote sensing image data is fused;
[0047] The physical characteristic information when fusing multimodal remote sensing image data includes resolution deviation coefficient and band spectrum deviation coefficient. After acquisition, the resolution deviation coefficient and band spectrum deviation coefficient are calibrated as and
[0048] Large differences in the resolution of multimodal remote sensing image data may have a certain impact on the accuracy of multimodal remote sensing image data fusion. This is because the resolution difference reflects the degree of detail of different sensors or data sources when capturing surface information. Large differences in resolution may cause the following effects:
[0049] Information loss: High-resolution data may contain more surface details and information, while low-resolution data may lose these details. During the fusion process, low-resolution data may not be able to fully capture the features in high-resolution data, resulting in information loss;
[0050] Blurred and confused: Due to the difference in resolution, the objects captured by different sensors may appear blurred or confused in the fusion results, because the spatial positions of different sensors may be inconsistent, which will affect the accuracy of the classifier, especially when fine classification is required at high spatial resolution;
[0051] Difficulty in spatial alignment: Spatial alignment of data with different resolutions may become more complicated. Improper alignment of data may cause misalignment of objects, thus affecting the accuracy of fusion.
[0052] Information omission: Low-resolution data may not capture all objects and details in high-resolution data, which may lead to omission of objects or inaccurate feature extraction, thus affecting the accuracy of classification and object identification;
[0053] Therefore, monitoring the data resolution when fusing multimodal remote sensing image data can timely discover the hidden dangers of abnormal fusion accuracy when fusion data is caused by large differences in multimodal remote sensing image data resolution;
[0054] The logic of obtaining the resolution deviation coefficient is as follows:
[0055] A101. Obtain the actual resolution of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual resolution as x represents the number of actual resolutions of several data to be fused within T time when multimodal remote sensing image data are fused, x=1, 2, 3, 4, ..., m, where m is a positive integer;
[0056] It should be noted that remote sensing data are usually accompanied by metadata and documentation information, which contains a detailed description of the data, including resolution information. Metadata files usually include information about the data acquisition process, sensor specifications and performance, which usually includes the spatial resolution of the data;
[0057] A102. The total resolution deviation is calculated by the actual resolution of two adjacent multimodal remote sensing image data. The calculation expression is: In the formula, FBL' represents the total resolution deviation, and m represents the total amount of actual resolution obtained within T time when multimodal remote sensing image data is fused;
[0058] A103. Calculate the resolution deviation coefficient. The calculation expression is:
[0059] From the calculation expression of the resolution deviation coefficient, it can be seen that the larger the performance value of the resolution deviation coefficient generated within T time when the multimodal remote sensing image data is fused, the greater the impact on the accuracy of the multimodal remote sensing image data fusion is, and vice versa, the smaller the impact on the accuracy of the multimodal remote sensing image data fusion is;
[0060] The large differences in the band spectrum bandwidth of multimodal remote sensing image data may have a certain impact on the accuracy of data fusion, but the specific degree of impact depends on the size of the difference in the band spectrum bandwidth, the application scenario and the fusion method. The following is a detailed explanation of the reasons for this problem:
[0061] Information inconsistency: The large differences in spectrum bandwidths of different bands may cause these bands to have different sensitivities in capturing ground features and information, which will lead to information inconsistency between different bands, thus affecting the accuracy of data fusion;
[0062] Information loss and distortion: Bands with large differences in spectrum bandwidth may not be able to capture certain details and feature information present in other bands, which may lead to missing or distorted ground feature information, thus affecting the accuracy of classification and feature extraction;
[0063] Spectral mismatch: Multimodal data collected by different sensors may have different spectral bandwidths, which means that different sensors have different spectral resolutions in different bands. When fusion is performed, this difference may cause information mismatch between some bands, making data fusion complicated;
[0064] For example, a sensor with a wider spectral bandwidth may be able to capture richer spectral information, while a sensor with a narrower spectral bandwidth may lose some details;
[0065] Data consistency: Data in different bands may have different radiation correction and atmospheric correction conditions, resulting in insufficient consistency between the data. This may require further preprocessing before fusion to reduce the impact of inconsistency;
[0066] Spectral overlap: Data with large differences in band spectral bandwidth may cause overlap or loss between bands when fused, which may make the identification and classification of spectral features more difficult. Overlapping areas may cause confusion, while missing areas may cause information loss.
[0067] Therefore, by monitoring the band spectrum bandwidth when fusing multimodal remote sensing image data, it is possible to timely discover the hidden danger of abnormal fusion accuracy when data fusion occurs due to large differences in the band spectrum bandwidth of multimodal remote sensing image data;
[0068] The logic for obtaining the band spectrum deviation coefficient is as follows:
[0069] B101. Obtain the actual band spectrum bandwidth of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual band spectrum bandwidth as y represents the number of the actual band spectrum bandwidth of several data to be fused within T time when the multimodal remote sensing image data is fused, y = 1, 2, 3, 4, ..., n, n is a positive integer;
[0070] It should be noted that each remote sensing sensor is accompanied by detailed documentation and metadata, including information about the band spectrum bandwidth. These documents usually provide information such as the central wavelength, bandwidth, and spectral response curve of each band. By consulting the technical specifications and documentation of the sensor, the data of the actual band spectrum bandwidth can be obtained;
[0071] B102. Calculate the standard deviation and average value of the band spectrum bandwidth by using the actual band spectrum bandwidth obtained within T time when fusing multimodal remote sensing image data, and calibrate the standard deviation and average value of the band spectrum bandwidth as and but: but:
[0072] B103. Standard deviation of band spectrum bandwidth obtained within T time when multimodal remote sensing image data is fused and band spectrum bandwidth average Calculate the band spectrum bandwidth variation index and calibrate the band spectrum bandwidth variation index as but:
[0073] It can be seen from the band spectrum bandwidth variation index that the greater the performance value of the band spectrum bandwidth variation index generated within T time when the multimodal remote sensing image data is fused, the greater the difference in the actual band spectrum bandwidths obtained within T time when the multimodal remote sensing image data is fused. Conversely, the smaller the difference in the actual band spectrum bandwidths obtained within T time when the multimodal remote sensing image data is fused.
[0074] B104. Calculate the band spectrum deviation coefficient. The calculation expression is:
[0075] From the calculation expression of the band spectrum deviation coefficient, it can be seen that the larger the performance value of the band spectrum deviation coefficient generated within T time when the multimodal remote sensing image data is fused, the greater the impact on the accuracy of the multimodal remote sensing image data fusion is, and vice versa, the smaller the impact on the accuracy of the multimodal remote sensing image data fusion is;
[0076] The temporal information when fusing multimodal remote sensing image data includes the time synchronization deviation coefficient. After acquisition, the time synchronization deviation coefficient is calibrated as SJ ω ;
[0077] In the process of multimodal data-assisted remote sensing image classification and extraction, the time intervals during the acquisition process should generally be kept consistent. Especially when planning time series analysis, data fusion, ground feature change monitoring and other tasks that require precise time information, maintaining consistent time intervals helps ensure that data from different sensors or data sources are synchronized in time, which is very important for data fusion, time series analysis and ground feature change monitoring. When fusing multimodal data, consistent time intervals can make the fusion process simpler and more reliable, and the data can be processed and combined according to the same time step;
[0078] The deviation of data collection time interval may affect the accuracy of multimodal remote sensing image data fusion, especially in applications that require time alignment or time series analysis. The following are the reasons for this problem in detail:
[0079] Data inconsistency: If data collected by different data sources or sensors have different time collection intervals, then when data fusion is performed, these data will not be aligned in time, which may lead to time inconsistency between data and affect the accuracy of fusion;
[0080] Time series analysis: In some applications, such as monitoring climate change, vegetation growth, or changes in surface water bodies, time series data need to be analyzed. If there is a deviation in the time interval of data collection, it will be difficult to perform accurate time series analysis because the time distances between data points are inconsistent;
[0081] Temporal alignment problem: For tasks that require aligning data at different time points, such as generating synthetic seasonal images or time series synthesis, the deviation of the time interval may cause the data to be incorrectly aligned in time, thus affecting the quality of the synthesis result;
[0082] Time synchronization problem: Deviations in the data collection time interval may cause the data from different sensors or data sources to be out of sync in time, which will lead to time inconsistency during data fusion, causing the data to not match on the time axis, thereby reducing the effectiveness of fusion;
[0083] Difficulty in data fusion: Time interval deviation may complicate data fusion, especially when data from different sensors need to be interpolated or corrected to match time intervals, which may introduce additional errors and uncertainties;
[0084] Therefore, monitoring the time when multimodal remote sensing image data is fused can timely discover hidden dangers that may cause abnormal fusion accuracy in data fusion due to time asynchrony during the fusion process of multimodal remote sensing image data;
[0085] The logic for obtaining the time synchronization deviation coefficient is as follows:
[0086] C101. Obtain the initial time interval for data collection of multimodal remote sensing image data and calibrate the initial time interval as SJ ω 初始 ;
[0087] C102. Obtain several actual time intervals generated within T time when acquiring multimodal remote sensing image data, and calibrate the actual time intervals as SJ ω k , k represents the number of actual time intervals generated within T time when multimodal remote sensing image data is collected, k = 1, 2, 3, 4, ..., p, p is a positive integer;
[0088] It should be noted that by using professional remote sensing image processing software, such as ENVI, ArcGIS, QGIS, etc., metadata information, including temporal resolution, can be obtained from remote sensing images;
[0089] C103. Calculate the time synchronization deviation coefficient. The calculation expression is:
[0090] From the calculation expression of the time synchronization deviation coefficient, it can be seen that the larger the value of the time synchronization deviation coefficient generated within T time when the multimodal remote sensing image data is fused, the greater the impact on the accuracy of the multimodal remote sensing image data fusion is, and vice versa, the smaller the impact on the accuracy of the multimodal remote sensing image data fusion is;
[0091] S104, comprehensively analyzing the processed physical characteristic information and time domain information when fusing the multimodal remote sensing image data to generate an influence coefficient;
[0092] The resolution deviation coefficient after processing when obtaining multimodal remote sensing image data for fusion Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω After that, and SJ ω Perform a formulaic analysis to generate impact coefficients The formula is: Where f1, f2, and f3 are the resolution deviation coefficients respectively. Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω The preset proportional coefficient, and f1, f2, and f3 are all greater than 0;
[0093] It can be seen from the calculation formula that the larger the resolution deviation coefficient, the larger the band spectrum deviation coefficient, and the larger the time synchronization deviation coefficient generated within T time when multimodal remote sensing image data is fused, that is, the influence coefficient generated within T time when multimodal remote sensing image data is fused The larger the performance value is, the greater the impact on the accuracy of multimodal remote sensing image data fusion is; conversely, the smaller the impact on the accuracy of multimodal remote sensing image data fusion is;
[0094] It should be noted that the above T time is selected as a relatively short time period. The time within the time period is not specifically limited here and can be set according to actual conditions. Its purpose is to monitor the situation within the T time when the multimodal remote sensing image data is fused, so as to monitor the fusion accuracy of the multimodal remote sensing image data in different time periods (T time) in real time through this method;
[0095] S105, comparing and analyzing the influence coefficient generated when the multimodal remote sensing image data is fused with a preset reference threshold of the influence coefficient, generating a high-impact signal or a low-impact signal, and issuing an early warning prompt for the high-impact signal;
[0096] The influence coefficient generated when the multimodal remote sensing image data is fused is compared and analyzed with the pre-set reference threshold of the influence coefficient. The comparison and analysis are as follows:
[0097] If the influence coefficient is greater than or equal to the influence coefficient reference threshold, a high-impact signal is generated, and an early warning is issued for the high-impact signal, prompting relevant users that the accuracy of multimodal remote sensing image data fusion is greatly affected, and the accuracy of the fused data may also be affected and deteriorate, which may eventually affect the accuracy of remote sensing image classification extraction. The relevant users are notified of the situation, and timely measures are taken to timely perform operation and maintenance management on abnormal data fusion accuracy, to ensure the accuracy of multimodal remote sensing image data fusion, and then to ensure the accuracy of the final remote sensing image classification extraction;
[0098] If the impact coefficient is less than the impact coefficient reference threshold, a low impact signal is generated, and no warning prompt is issued for the low impact signal;
[0099] S106, training the main classification network model and the auxiliary network model respectively to obtain a main classification network model file and an auxiliary classification network model file;
[0100] S107, deploying and testing the network model, the data will obtain a classification result after passing through the backbone network model, and then the obtained result will be determined by the auxiliary model to finally obtain the extraction result;
[0101] The present invention proposes an auxiliary multimodal method. During the training process of the neural network model, data with similar features are selected for fusion, and data with too large a difference in features are used as an auxiliary model for correction training. This can greatly reduce the difficulty of model training, and can train the auxiliary module and the backbone network separately, thereby reducing the complexity of the model training process and the network model parameters, while also improving the classification accuracy of the backbone network model.
[0102] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.
[0103] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0104] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0106] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0109] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0110] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for using multimodal data to assist remote sensing image classification and extraction, characterized in that: The following steps are involved: S101, taking the optical remote sensing image data as the main line, selecting multimodal data with data feature forms similar to the optical remote sensing image data, and fusing the selected data into a training data set for the backbone classification network; S102, filtering data in the multimodal data that is not similar to the backbone network data, and fusing the filtered data into a training data set for the auxiliary model; S103, obtaining a plurality of information when the multimodal remote sensing image data is fused, including physical characteristic information and time domain information, and after obtaining, processing the physical characteristic information and time domain information when the multimodal remote sensing image data is fused; S104, comprehensively analyzing the processed physical characteristic information and time domain information when fusing the multimodal remote sensing image data to generate an influence coefficient; S105, comparing and analyzing the influence coefficient generated when the multimodal remote sensing image data is fused with a preset reference threshold of the influence coefficient, generating a high-impact signal or a low-impact signal, and issuing an early warning prompt for the high-impact signal; S106, training the main classification network model and the auxiliary network model respectively to obtain a main classification network model file and an auxiliary classification network model file; S107, deploy and test the network model. After the data passes through the backbone network model, a classification result will be obtained. The result will then be determined by the auxiliary model to finally obtain the extraction result.
2. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 1, characterized in that: The physical characteristic information when fusing multimodal remote sensing image data includes resolution deviation coefficient and band spectrum deviation coefficient. After acquisition, the resolution deviation coefficient and band spectrum deviation coefficient are calibrated as and 3. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 2, characterized in that: The logic of obtaining the resolution deviation coefficient is as follows: A101. Obtain the actual resolution of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual resolution as x represents the number of actual resolutions of several data to be fused within T time when multimodal remote sensing image data are fused, x=1, 2, 3, 4, ..., m, where m is a positive integer; A102. The total resolution deviation is calculated by the actual resolution of two adjacent multimodal remote sensing image data. The calculation expression is: In the formula, FBL' represents the total resolution deviation, and m represents the total amount of actual resolution obtained within T time when multimodal remote sensing image data is fused; A103. Calculate the resolution deviation coefficient. The calculation expression is:
4. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 3, characterized in that: The logic for obtaining the band spectrum deviation coefficient is as follows: B101. Obtain the actual band spectrum bandwidth of several data to be fused within T time when fusing multimodal remote sensing image data, and calibrate the actual band spectrum bandwidth as y represents the number of the actual band spectrum bandwidth of several data to be fused within T time when the multimodal remote sensing image data is fused, y = 1, 2, 3, 4, ..., n, n is a positive integer; B102. Calculate the standard deviation and average value of the band spectrum bandwidth by using the actual band spectrum bandwidth obtained within T time when fusing multimodal remote sensing image data, and calibrate the standard deviation and average value of the band spectrum bandwidth as and but: but: B103. Standard deviation of band spectrum bandwidth obtained within T time when multimodal remote sensing image data is fused and band spectrum bandwidth average Calculate the band spectrum bandwidth variation index and calibrate the band spectrum bandwidth variation index as but: B104. Calculate the band spectrum deviation coefficient. The calculation expression is:
5. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 4, characterized in that: The temporal information when fusing multimodal remote sensing image data includes the time synchronization deviation coefficient. After acquisition, the time synchronization deviation coefficient is calibrated as SJ ω .
6. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 5, characterized in that: The logic for obtaining the time synchronization deviation coefficient is as follows: C101. Obtain the initial time interval for data collection of multimodal remote sensing image data and calibrate the initial time interval as SJ ω 初始 ; C102. Obtain several actual time intervals generated within T time when acquiring multimodal remote sensing image data, and calibrate the actual time intervals as SJ ω k , k represents the number of actual time intervals generated within T time when multimodal remote sensing image data is collected, k = 1, 2, 3, 4, ..., p, p is a positive integer; C103. Calculate the time synchronization deviation coefficient. The calculation expression is:
7. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 6, characterized in that: The resolution deviation coefficient after processing when obtaining multimodal remote sensing image data for fusion Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω After that, and SJ ω Perform a formulaic analysis to generate impact coefficients The formula is: Where f1, f2, and f3 are the resolution deviation coefficients respectively. Band spectrum deviation coefficient And the time synchronization deviation coefficient SJ ω The preset proportional coefficient, and f1, f2, and f3 are all greater than 0.
8. The method for utilizing multimodal data to assist remote sensing image classification and extraction according to claim 7, characterized in that: The influence coefficient generated when the multimodal remote sensing image data is fused is compared and analyzed with the pre-set reference threshold of the influence coefficient. The comparison and analysis are as follows: If the impact coefficient is greater than or equal to the impact coefficient reference threshold, a high impact signal is generated and an early warning prompt is issued for the high impact signal; If the impact coefficient is less than the impact coefficient reference threshold, a low impact signal is generated, and no early warning is issued for the low impact signal.
