Remote sensing image fusion method, device, storage medium and electronic equipment

Abstract

This specification discloses a remote sensing image fusion method, device, storage medium and electronic device. When the remote sensing image fusion method provided in this specification is used to realize the fusion between the multispectral image and the panchromatic image collected by the satellite, the multispectral image and the panchromatic image can be first processed in blocks to obtain the multispectral image sub-image and the panchromatic image sub-image; then the multispectral image sub-image and the panchromatic image sub-image describing the same area are fused to obtain the fused image sub-image; finally, the fused image sub-images are merged to obtain the target fused image. The use of this method can realize the entire process of remote sensing image fusion on the onboard computer, fully utilizing the idle computing power of the onboard computer while greatly reducing the amount of data that needs to be transmitted during satellite-to-ground transmission, effectively improving the effect and efficiency of remote sensing image fusion, reducing transmission costs, and ensuring the data timeliness of the target fused image.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a remote sensing image fusion method, device, storage medium and electronic device. Background Art

[0002] Nowadays, remote sensing images have become an important way to obtain information about the earth's surface. Through sensors carried by satellites, aircraft and other platforms, remote sensing technology can efficiently and remotely collect a wide range of earth surface image data, providing data support for resource exploration, environmental monitoring and other fields. However, due to the limitations of sensor technology and physics, it is often difficult to simultaneously obtain remote sensing images with high spectral resolution and high spatial resolution in actual detection.

[0003] At present, remote sensing image fusion technology based on principal component analysis is used to solve the above problems. However, traditional image fusion methods and systems based on principal component analysis rely on abundant computing resources and highly reliable satellite-to-ground transmission, and often require satellite-borne remote sensing images to be downloaded to the ground and run on high-power, high-performance terminals at ground stations. This method, which requires a large amount of raw image data to be transmitted from the satellite to the ground for processing, not only occupies precious satellite-to-ground transmission bandwidth resources, but is also susceptible to interference and delays during the transmission process, resulting in damage to the accuracy and integrity of the data, which in turn affects the efficiency and timeliness of remote sensing information collection.

[0004] Therefore, how to reduce the resource requirements of remote sensing image fusion technology and improve the efficiency of remote sensing image fusion is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a remote sensing image fusion method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions: This specification provides a remote sensing image fusion method, which is applied to a satellite-borne computer and includes: Acquire multispectral and panchromatic images of the target area; The multispectral image and the panchromatic image are processed in blocks to obtain a multispectral image sub-image and a panchromatic image sub-image; For a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, determining a first principal component of the multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtaining a target feature matrix of the multispectral image sub-image according to the first principal component; A fused image sub-image is obtained according to the target feature matrix, and each fused image sub-image is merged to obtain a target fused image.

[0007] Optionally, the sizes of the multispectral image sub-images are the same, and the sizes of the panchromatic image sub-images are the same; Before determining the first principal component of the multispectral image subimage included in the subimage pair according to the panchromatic image subimage included in the subimage pair, the method further includes: Each multispectral image sub-image is scaled so that the size of each multispectral image sub-image is the same as the size of each panchromatic image sub-image.

[0008] Optionally, determining a first principal component of a multispectral image subimage included in the subimage pair according to the panchromatic image subimage included in the subimage pair, and obtaining a target feature matrix of the multispectral image subimage according to the first principal component specifically includes: Performing principal component analysis on the multispectral image sub-images contained in the sub-image pair to obtain a feature matrix of the multispectral image contained in the sub-image pair; The full-color image sub-image included in the sub-image pair is used as a new first principal component to replace the first principal component of the feature matrix, so as to obtain a target feature matrix.

[0009] Optionally, before performing principal component analysis on the multispectral image sub-images contained in the sub-image pair, the method further comprises: Determine a first mean value and a first standard deviation of each pixel included in the multispectral image sub-image; Performing a decentralization operation on the multispectral image sub-image according to the first mean and the first standard deviation; Before merging the fused image sub-images, the method further includes: Perform a decentralized inverse transform operation on each fused image sub-image.

[0010] Optionally, using the full-color image sub-image included in the sub-image pair as a new first principal component to replace the first principal component of the feature matrix specifically includes: Determine transformation coefficients according to the full-color image sub-image contained in the sub-image pair and the characteristic matrix; Using the transformation coefficient, linearly transform each pixel in the full-color image sub-image contained in the sub-image pair to obtain a transformed full-color image sub-image; The transformed full-color image sub-image is used as a new first principal component to replace the first principal component of the feature matrix.

[0011] Optionally, determining the transformation coefficient according to the full-color image sub-image included in the sub-image pair and the characteristic matrix specifically includes: Determine a second mean and a second standard deviation of the pixel values ​​of the panchromatic image sub-image included in the sub-image pair, and determine a third mean and a third standard deviation of the first principal component of the feature matrix; A transformation coefficient is determined according to the second mean, the second standard deviation, the third mean, and the third standard deviation.

[0012] Optionally, the onboard computer includes at least a field programmable gate array FPGA, a central processing unit CPU, and a graphics processing unit GPU; The FPGA is used to perform block processing operations on the multispectral image and the panchromatic image, the CPU is used to transfer each multispectral image sub-image and each panchromatic image sub-image obtained by the block processing to the GPU, and the GPU is used to perform principal component analysis operations and image fusion operations on each multispectral image sub-image.

[0013] This specification provides a remote sensing image fusion device, the device comprising: An acquisition module is used to acquire multispectral images and panchromatic images of the target area; A block division module, used for performing block processing on the multispectral image and the panchromatic image to obtain a multispectral image sub-image and a panchromatic image sub-image; A determination module, for determining, for a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, a first principal component of a multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtaining a target feature matrix of the multispectral image sub-image according to the first principal component; The fusion module is used to obtain a fused image sub-image according to the target feature matrix, and merge the fused image sub-images to obtain a target fused image.

[0014] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the remote sensing image fusion method is implemented.

[0015] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the remote sensing image fusion method when executing the program.

[0016] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the remote sensing image fusion method provided in the present specification, a multispectral image and a panchromatic image of a target area are obtained; the multispectral image and the panchromatic image are processed in blocks to obtain a multispectral image sub-image and a panchromatic image sub-image; for a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, the first principal component of the multispectral image sub-image included in the sub-image pair is determined according to the panchromatic image sub-image included in the sub-image pair, and a target feature matrix of the multispectral image sub-image is obtained according to the first principal component; a fused image sub-image is obtained according to the target feature matrix, and each fused image sub-image is merged to obtain a target fused image.

[0017] When the remote sensing image fusion method provided in this specification is used to realize the fusion between the multispectral image and the panchromatic image collected by the satellite, the multispectral image and the panchromatic image can be first processed in blocks to obtain the multispectral image sub-image and the panchromatic image sub-image; then the multispectral image sub-image and the panchromatic image sub-image describing the same area are fused to obtain the fused image sub-image; finally, the fused image sub-images are merged to obtain the target fused image. The use of this method can realize the whole process of remote sensing image fusion on the onboard computer, fully utilizing the idle computing power of the onboard computer, greatly reducing the amount of data that needs to be transmitted during satellite-to-ground transmission, effectively improving the effect and efficiency of remote sensing image fusion, reducing transmission costs, and ensuring the data timeliness of the target fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 It is a flowchart of a remote sensing image fusion method in this manual; Figure 2 A schematic diagram of using different devices of the onboard computer to complete different steps provided in this manual; Figure 3 A comparison diagram of the multispectral image and the panchromatic image before and after fusion provided in this manual; Figure 4 A schematic diagram of a remote sensing image fusion method and device provided in this specification; Figure 5 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 The following is a flow chart of a remote sensing image fusion method in this specification. The method is applied to a satellite-borne computer and specifically includes the following steps: S100: Acquire multispectral images and panchromatic images of the target area.

[0022] All steps in the remote sensing image fusion method provided in this specification can be implemented by any onboard computer with computing capabilities, such as terminals, servers and other devices.

[0023] This method is mainly used to fuse two types of remote sensing images collected by satellites, namely multispectral images and panchromatic images, to obtain a fused image with both high spectral resolution and high spatial resolution. Based on this, in this step, the multispectral image and panchromatic image of the target area to be fused can be first obtained.

[0024] Multispectral imaging is a set of black-and-white photographs containing multiple bands of spectral information obtained by multispectral photography using different color filters and photosensitive films. The basic principle of multispectral imaging is to divide the visible light and near-infrared light reflected by the ground objects into several bands through different color filters, and expose them on several screens of the same size at the same time. Due to the differences in the spectral characteristics of the ground objects in each band, the image density of the same ground object on the film of different bands will be different. This difference allows people to identify the ground objects based on the spectral characteristics of the ground objects and their measurement data.

[0025] Panchromatic images are black and white images that capture the entire visible light wavelength range. Panchromatic images are usually defined within the visible light band, but in order to avoid the influence of atmospheric scattering, the blue light band is often filtered out. Therefore, the common range of the panchromatic band is roughly between 0.4μm and 0.7μm, but the specific range may vary depending on different remote sensors and sensors. Panchromatic images only use a single band for imaging, so they appear as grayscale images on the image.

[0026] Since multispectral images usually use multiple bands, black and white images of several bands can be selected according to the interpretation requirements, and false color images can be obtained through various color synthesis techniques. This false color image can simulate real colors and help people identify objects more intuitively. Panchromatic images focus on imaging in the visible light band and usually have a higher spatial resolution, which can clearly show the details and structure of objects. Therefore, the advantages of the two can be combined through image fusion technology to obtain a fused image that meets higher requirements for both spectral resolution and spatial resolution.

[0027] The acquisition of multispectral images and panchromatic images mainly relies on the acquisition equipment carried by the onboard computer. It is a relatively mature technology and will not be elaborated in this manual.

[0028] S102: performing block processing on the multispectral image and the panchromatic image to obtain a multispectral image sub-image and a panchromatic image sub-image.

[0029] One of the key inventive concepts of the remote sensing image fusion method provided in this specification is that all steps can be executed by the satellite's onboard computer to reduce the communication requirements between the satellite and the ground station. Compared with the terminal equipment set up at the ground station, the onboard computer is smaller in size, so the maximum amount of data processed is given priority.

[0030] In the above scenario, in order to ensure that the onboard computer can complete the remote sensing image fusion with faster efficiency, in this step, the multispectral image and the panchromatic image obtained in step S100 can be further processed in blocks to obtain a single multispectral image sub-image and a panchromatic image sub-image of smaller size, so that the computing power of the onboard computer can meet the processing requirements of each sub-image.

[0031] In order to ensure that the contents of the sub-images of different images in the subsequent fusion stage can correspond to each other, when dividing the multispectral image and the panchromatic image into smaller sub-images, the division can be performed according to the area described by the sub-image. Specifically, the acquired target area can be directly divided to obtain several sub-regions that do not contain repeated areas; then, the multispectral image and the panchromatic image are cut according to each sub-region to obtain a multispectral image sub-image and a panchromatic image sub-image that describe each sub-region.

[0032] Under the above division method, the number of obtained multispectral image sub-images and panchromatic image sub-images is the same, and each sub-region has a corresponding multispectral image sub-image and panchromatic image sub-image describing the sub-region.

[0033] Furthermore, in order to ensure the overall processing efficiency, when multiple groups of sub-images can be processed in parallel and fused at the same time, it is necessary to make the processing time of each group of sub-images close. Therefore, when dividing the sub-images, the above requirements can be met by making the size of each divided multispectral image sub-image the same, and the size of each panchromatic image sub-image the same. In other words, the size of each divided sub-area needs to be the same.

[0034] In addition, in most cases, the original size of the multispectral image acquired by the satellite is larger than the original size of the panchromatic image. Therefore, in this case, the size of each multispectral image sub-image obtained can be additionally adjusted during the division. Specifically, each multispectral image sub-image can be scaled so that the size of each multispectral image sub-image is the same as the size of each panchromatic image sub-image. When the size of the multispectral image sub-image is the same as that of the panchromatic image sub-image, the subsequent fusion operation can be simpler and more efficient.

[0035] S104: For a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, determine a first principal component of the multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtain a target feature matrix of the multispectral image sub-image according to the first principal component.

[0036] After completing the block processing of the multispectral image and the panchromatic image in step S102, the obtained multispectral image sub-image and the panchromatic image sub-image can be fused in this step. In this method, the multispectral image sub-image and the panchromatic image sub-image are fused in a principal component analysis-based manner.

[0037] There is a sub-image pair under each sub-region, including a multispectral image sub-image and a panchromatic image sub-image that describe this sub-region, and the same fusion operation can be performed on the sub-image pair of each sub-region. In this method, the panchromatic image sub-image is fused into the multispectral image sub-image with the multispectral image sub-image as the base. That is, the multispectral image sub-image is first subjected to principal component analysis, and then the first principal component obtained by principal component analysis of the multispectral image sub-image is adjusted according to the corresponding panchromatic image sub-image, so as to obtain the target feature matrix of the multispectral image sub-image. Specifically, principal component analysis can be performed on the multispectral image sub-image contained in the sub-image pair to obtain the feature matrix of the multispectral image contained in the sub-image pair; the panchromatic image sub-image contained in the sub-image pair is used as the new first principal component to replace the first principal component of the feature matrix, so as to obtain the target feature matrix.

[0038] Principal component analysis is a mathematical dimensionality reduction method that converts the original image data into a set of linearly independent eigenvectors, namely principal components, through linear transformation. These principal components are arranged in descending order according to the size of the variance. The first principal component represents the projection of the data in the direction of the maximum variance, the second principal component is orthogonal to the first principal component and represents the direction with the largest residual variance, and so on. Principal component analysis can retain most of the information of the original variables, while greatly reducing the number of variables, and the principal components are independent of each other.

[0039] After the principal component analysis is performed on the multispectral image sub-image to obtain the feature matrix, since the panchromatic image sub-image has a better spatial resolution, the panchromatic image sub-image can be used as the new first principal component to replace the first principal component of the multispectral image sub-image to obtain the target feature matrix. In layman's terms, the target feature matrix is ​​a new matrix obtained by replacing the first principal component of the multispectral image sub-image in the feature matrix after principal component analysis with the panchromatic image sub-image.

[0040] Furthermore, in order to make the multispectral image sub-image data easier to analyze and process when performing principal component analysis, the multispectral image sub-image data can be first analyzed and processed before principal component analysis. Figure 1 Specifically, a first mean and a first standard deviation of each pixel included in the multispectral image sub-image may be determined; and a decentralization operation may be performed on the multispectral image sub-image according to the first mean and the first standard deviation.

[0041] The decentralization operation refers to reducing the mean of each data point in the data set, thereby obtaining a new data set in which the mean of each data point is zero. The decentralization operation can eliminate the central trend in the data, improve the algorithm performance, eliminate bias, and facilitate comparison and analysis. Applied to this method, the specific formula of the decentralization operation can be:

[0042] in, is the pixel value of the i-th pixel in the multispectral image subimage, is the pixel value of the i-th pixel in the decentralized multispectral image sub-image, mean and std are the mean and standard deviation of the pixel values ​​of the multispectral image sub-image, respectively.

[0043] It should be noted that, in general, multispectral images contain multiple bands. Therefore, when performing principal component analysis on each multispectral image sub-image, the data of the multispectral image sub-image can be transformed in shape first, and then the principal component analysis can be performed. For example, assuming that the multispectral image sub-image contains four bands, B1, B2, B3, and B4, and the size of each band of the multispectral image sub-image is (n1, n1), then the data size of the multispectral image sub-image is (4, n1, n1). The transformation operation required is to transform the (n1, n1) shape data of each band into a (1, n1*n1) shape, and the data size of the multispectral image sub-image will become (4, n1*n1) accordingly. At this time, the principal component analysis of the multispectral image sub-image can be performed.

[0044] The first half of the principal component analysis transformation process can be expressed as follows:

[0045] Among them, X is the data matrix of the multispectral image sub-image after shape transformation. In the case of the above example, the size of X can be (4, n1*n1), is the transposed matrix of X; n represents the number of columns of the data matrix of the multispectral image sub-image, which is n1*n1; T is the covariance matrix of X.

[0046] After obtaining the covariance matrix T, the eigenvalues ​​and corresponding eigenvectors of the covariance matrix T can be solved, and the eigenvectors can be arranged in sequence according to the eigenvalues ​​to construct the characteristic matrix Q of the covariance matrix T; finally, Q*X can be calculated to obtain the characteristic matrix H obtained by performing principal component analysis on the multispectral image subimage, and the first row vector of the characteristic matrix H is the first principal component of the multispectral image subimage.

[0047] After obtaining the feature matrix H of the multispectral image sub-image, the panchromatic image sub-image under the same sub-image pair can be used as the new first principal component to replace the first principal component of the feature matrix H to obtain the target feature matrix.

[0048] Furthermore, when the full-color image sub-image is used to replace the first principal component of the feature matrix, the pixels of the full-color image sub-image may be first transformed to obtain a better fusion effect. Specifically, the transformation coefficient may be determined based on the full-color image sub-image contained in the sub-image pair and the feature matrix; the transformation coefficient may be used to perform a linear transformation on each pixel in the full-color image sub-image contained in the sub-image pair to obtain a transformed full-color image sub-image; and the transformed full-color image sub-image may be used as a new first principal component to replace the first principal component of the feature matrix.

[0049] After each pixel in the full-color image sub-image is linearly transformed by the determined transformation coefficient, it is then used to replace the first principal component of the feature matrix, so that the overall fusion effect can be improved. The method for determining the transformation coefficient can be specifically to determine the second mean and second standard deviation of the pixel values ​​of the full-color image sub-image contained in the sub-image pair, and determine the third mean and third standard deviation of the first principal component of the feature matrix; determine the transformation coefficient according to the second mean, the second standard deviation, the third mean and the third standard deviation.

[0050] Since the replacement operation in the fusion process is to replace the first principal component of the multispectral image subimage with the panchromatic image subimage under a subimage pair, the transformation coefficient can be determined based on the two involved in the replacement operation, that is, the first principal component of the panchromatic image subimage and the multispectral image subimage. After determining the second mean and second standard deviation of the pixel values ​​of the panchromatic image subimage and the third mean and third standard deviation of the first principal component of the feature matrix, the transformation coefficients K1 and K2 can be determined according to the following formula:

[0051]

[0052] Among them, K1 and K2 are the determined transformation coefficients, and are the mean and standard deviation of the pixel values ​​of the panchromatic image sub-image, that is, the second mean and the second standard deviation; and They are the mean and standard deviation of the first principal component of the multispectral image subimage, that is, the third mean and third standard deviation.

[0053] After the transformation coefficients K1 and K2 are determined, the transformation coefficients can be used to perform linear transformation on each pixel of the full-color image sub-image according to the following formula:

[0054] in, is the pixel value of the i-th pixel of the original full-color image sub-image, is the pixel value of the i-th pixel of the transformed full-color image sub-image; K1 and K2 are transformation coefficients.

[0055] S106: Obtain fused image sub-images according to the target feature matrix, and merge the fused image sub-images to obtain a target fused image.

[0056] After the target feature matrix is ​​determined in step S104, the fused image sub-image of each sub-region can be determined in this step according to the target feature matrix of each sub-image pair. Specifically, the fused image sub-image can be obtained by performing an inverse transformation operation of principal component analysis on the target feature matrix.

[0057] It should be noted that if decentralized preprocessing is performed before principal component analysis is performed on the multispectral image sub-image in step S104, then after the inverse transformation operation of principal component analysis is performed on the target feature matrix in this step, an additional decentralized inverse transformation operation is required on the obtained data matrix to obtain the fused image sub-image.

[0058] Then, each fused image sub-image is reassembled and merged to obtain the final target fused image.

[0059] After obtaining the target fusion image, the target fusion image can be directly transmitted to the ground station in subsequent applications. Compared with transmitting the original multispectral image and panchromatic image, transmitting only the target fusion image can greatly reduce the amount of data to be transmitted, effectively alleviating the negative impact caused by the difficulty of transmission between the satellite and the ground.

[0060] Furthermore, in the remote sensing image fusion method provided in this specification, each step can be executed by different devices on the onboard computer, so as to realize the parallel processing pipeline of each step, maximize the use of the computing power of the onboard computer, and improve the overall efficiency. Specifically, the onboard computer may include at least a field programmable gate array (FPGA), a central processing unit (CPU), and a graphics processing unit (GPU); wherein the FPGA is used to perform block processing operations on the multispectral image and the panchromatic image, the CPU is used to transfer each multispectral image sub-image and each panchromatic image sub-image obtained by the block processing to the GPU, and the GPU is used to perform principal component analysis operations and image fusion operations on each multispectral image sub-image.

[0061] Figure 2 A schematic diagram of using different devices of the onboard computer to complete different steps is provided for this manual, such as Figure 2 As shown in the figure, according to the actual time consumption of each step, the computing resources such as FPGA, CPU, GPU can be dynamically scheduled and allocated, so that the FPGA sub-graph cutting, CPU sub-graph data handling and GPU principal component analysis image fusion calculation form a pipeline parallel. Under the above settings, the processing of each sub-graph can be realized in parallel pipeline processing, which effectively reduces the demand for onboard computer resources and speeds up processing efficiency.

[0062] When the remote sensing image fusion method provided in this specification is used to realize the fusion between the multispectral image and the panchromatic image collected by the satellite, the multispectral image and the panchromatic image can be first processed in blocks to obtain the multispectral image sub-image and the panchromatic image sub-image; then the multispectral image sub-image and the panchromatic image sub-image describing the same area are fused to obtain the fused image sub-image; finally, the fused image sub-images are merged to obtain the target fused image. The use of this method can realize the whole process of remote sensing image fusion on the onboard computer, fully utilizing the idle computing power of the onboard computer, greatly reducing the amount of data that needs to be transmitted during satellite-to-ground transmission, effectively improving the effect and efficiency of remote sensing image fusion, reducing transmission costs, and ensuring the data timeliness of the target fused image.

[0063] Figure 3 The following is a comparison diagram of the multispectral image and the panchromatic image before and after fusion provided in this manual, as shown in Figure 3 As shown in the figure, after the fusion of multispectral image and panchromatic image, a high-resolution clear image with better color and texture details can be obtained.

[0064] The above is the remote sensing image fusion method provided in this manual. Based on the same idea, this manual also provides a corresponding remote sensing image fusion device, such as Figure 4 shown.

[0065] Figure 4 A schematic diagram of a remote sensing image fusion device provided in this specification specifically includes: An acquisition module 200 is used to acquire a multispectral image and a panchromatic image of a target area; A block division module 202 is used to perform block processing on the multispectral image and the panchromatic image to obtain a multispectral image sub-image and a panchromatic image sub-image; A determination module 204 is used to determine, for a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, a first principal component of the multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtain a target feature matrix of the multispectral image sub-image according to the first principal component; The fusion module 206 is used to obtain fused image sub-images according to the target feature matrix, and merge the fused image sub-images to obtain a target fused image.

[0066] Optionally, the sizes of the multispectral image sub-images are the same, and the sizes of the panchromatic image sub-images are the same; The device further comprises a scaling module 208, which is specifically used to scale each multispectral image sub-image so that the size of each multispectral image sub-image is the same as the size of each panchromatic image sub-image.

[0067] Optionally, the determination module 204 is specifically used to perform principal component analysis on the multispectral image subimage contained in the subimage pair to obtain a feature matrix of the multispectral image contained in the subimage pair; and use the panchromatic image subimage contained in the subimage pair as a new first principal component to replace the first principal component of the feature matrix to obtain a target feature matrix.

[0068] Optionally, the device further includes a pre-processing module 210, which is specifically used to determine a first mean and a first standard deviation of each pixel included in the multispectral image sub-image; and perform a decentralization operation on the multispectral image sub-image according to the first mean and the first standard deviation; The device further comprises an inverse processing module 212, which is specifically configured to perform a decentralized inverse transformation operation on each fused image sub-image.

[0069] Optionally, the determination module 204 is specifically used to determine the transformation coefficients according to the full-color image sub-image included in the sub-image pair and the characteristic matrix; use the transformation coefficients to perform a linear transformation on each pixel in the full-color image sub-image included in the sub-image pair to obtain a transformed full-color image sub-image; and use the transformed full-color image sub-image as a new first principal component to replace the first principal component of the characteristic matrix.

[0070] Optionally, the determination module 204 is specifically used to determine the second mean and second standard deviation of the pixel values ​​of the full-color image sub-image contained in the sub-image pair, and determine the third mean and third standard deviation of the first principal component of the feature matrix; determine the transformation coefficient according to the second mean, the second standard deviation, the third mean and the third standard deviation.

[0071] Optionally, the onboard computer includes at least a field programmable gate array FPGA, a central processing unit CPU, and a graphics processing unit GPU; The FPGA is used to perform block processing operations on the multispectral image and the panchromatic image, the CPU is used to transfer each multispectral image sub-image and each panchromatic image sub-image obtained by the block processing to the GPU, and the GPU is used to perform principal component analysis operations and image fusion operations on each multispectral image sub-image.

[0072] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The remote sensing image fusion method provided.

[0073] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0074] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0075] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0076] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0077] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0084] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0086] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0088] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0089] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A remote sensing image fusion method, characterized in that: The method is applied to a satellite-borne computer, and the method comprises: Acquire multispectral and panchromatic images of the target area; The multispectral image and the panchromatic image are processed in blocks to obtain a multispectral image sub-image and a panchromatic image sub-image; For a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, determining a first principal component of the multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtaining a target feature matrix of the multispectral image sub-image according to the first principal component; A fused image sub-image is obtained according to the target feature matrix, and each fused image sub-image is merged to obtain a target fused image.

2. The method according to claim 1, characterized in that The size of each multispectral image sub-image is the same, and the size of each panchromatic image sub-image is the same; Before determining the first principal component of the multispectral image subimage included in the subimage pair according to the panchromatic image subimage included in the subimage pair, the method further includes: Each multispectral image sub-image is scaled so that the size of each multispectral image sub-image is the same as the size of each panchromatic image sub-image.

3. The method according to claim 1, characterized in that Determining a first principal component of a multispectral image subimage included in the subimage pair according to the panchromatic image subimage included in the subimage pair, and obtaining a target feature matrix of the multispectral image subimage according to the first principal component, specifically includes: Performing principal component analysis on the multispectral image sub-images contained in the sub-image pair to obtain a feature matrix of the multispectral image contained in the sub-image pair; The full-color image sub-image included in the sub-image pair is used as a new first principal component to replace the first principal component of the feature matrix, so as to obtain a target feature matrix.

4. The method according to claim 3, characterized in that Before performing principal component analysis on the multispectral image sub-images contained in the sub-image pair, the method further comprises: Determine a first mean value and a first standard deviation of each pixel included in the multispectral image sub-image; Performing a decentralization operation on the multispectral image sub-image according to the first mean and the first standard deviation; Before merging the fused image sub-images, the method further includes: Perform a decentralized inverse transform operation on each fused image sub-image.

5. The method according to claim 3, characterized in that The panchromatic image sub-image included in the sub-image pair is used as a new first principal component to replace the first principal component of the feature matrix, specifically including: Determine transformation coefficients according to the full-color image sub-image contained in the sub-image pair and the characteristic matrix; Using the transformation coefficient, linearly transform each pixel in the full-color image sub-image contained in the sub-image pair to obtain a transformed full-color image sub-image; The transformed full-color image sub-image is used as a new first principal component to replace the first principal component of the feature matrix.

6. The method according to claim 5, characterized in that Determining the transformation coefficients according to the full-color image sub-image contained in the sub-image pair and the characteristic matrix specifically includes: Determine a second mean and a second standard deviation of the pixel values ​​of the panchromatic image sub-image included in the sub-image pair, and determine a third mean and a third standard deviation of the first principal component of the feature matrix; A transformation coefficient is determined according to the second mean, the second standard deviation, the third mean, and the third standard deviation.

7. The method according to claim 1, characterized in that The onboard computer at least includes a field programmable gate array FPGA, a central processing unit CPU, and a graphics processing unit GPU; The FPGA is used to perform block processing operations on the multispectral image and the panchromatic image, the CPU is used to transfer each multispectral image sub-image and each panchromatic image sub-image obtained by the block processing to the GPU, and the GPU is used to perform principal component analysis operations and image fusion operations on each multispectral image sub-image.

8. A remote sensing image fusion device, characterized in that: include: An acquisition module is used to acquire multispectral images and panchromatic images of the target area; A block division module, used for performing block processing on the multispectral image and the panchromatic image to obtain a multispectral image sub-image and a panchromatic image sub-image; A determination module, for determining, for a sub-image pair of a multispectral image sub-image and a panchromatic image sub-image describing the same area, a first principal component of a multispectral image sub-image included in the sub-image pair according to the panchromatic image sub-image included in the sub-image pair, and obtaining a target feature matrix of the multispectral image sub-image according to the first principal component; The fusion module is used to obtain a fused image sub-image according to the target feature matrix, and merge the fused image sub-images to obtain a target fused image.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images