Image processing method and device for near-field security scene, and electronic equipment

Abstract

The invention provides an image processing method and device for a nearby security scene, and electronic equipment, and is applied to the technical field of image data processing. The method comprises the following steps: determining first modal edge enhancement features and second modal edge enhancement features of multiple levels according to a first shallow feature of a first hyperspectral image and a second shallow feature of a multispectral image facing a near-ground security scene; according to the first modal edge enhancement feature of the last level and a preset super-pixel center feature, determining a segmentation image and a super-pixel center feature, and then combining the second modal edge enhancement feature of the last level to determine a calibration feature; and determining a second hyperspectral image according to the first modal edge enhancement feature of the final level, the calibration feature and the corresponding spectral context information. By modeling edge complementary knowledge and instance-level edge context between two modalities of the first hyperspectral image and the multispectral image, the expression of edge information is effectively enhanced, so that the second hyperspectral image with relatively high accuracy is obtained.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to an image processing method, apparatus and electronic device for local security scenarios. Background Technology

[0002] Pre-territorial security refers to a diversified, cross-domain, three-dimensional, collaborative, and intelligent technical system addressing the needs of defense, protection, production, safety, and rescue within space. Specific application scenarios include, but are not limited to, low-altitude security, underwater security, and cross-domain security. The pre-territorial security technology system encompasses six major research directions: related optical detection, stable detection, multimodal cognitive computing, swarm intelligence decision-making, waterborne optics, and cross-domain remote sensing. Obtaining high-resolution imagery is a key technology for the effective implementation of typical pre-territorial security tasks such as low-altitude UAV detection, extreme disaster early warning and rescue, and target reconnaissance.

[0003] Hyperspectral images contain fine-grained spectral information, reflecting subtle differences between various substances, and are an important information carrier in cross-domain remote sensing research within near-field security technology systems. Compared to natural images, hyperspectral imaging systems have narrower spectral intervals, resulting in less photon energy entering the system. To maintain a high signal-to-noise ratio, the photosensitive units of each sensor need to collect more photons over a larger spatial area, directly leading to lower spatial resolution in hyperspectral images. Furthermore, in designing hyperspectral imagers, spatial resolution is often sacrificed to maintain sensor sensitivity in spectral resolution. The low spatial resolution caused by hardware limitations inevitably affects the detailed identification and analysis of targets in the images, severely limiting the application of hyperspectral images. Therefore, it is urgent to improve the spatial resolution of hyperspectral images through technological means to enhance the accuracy and efficiency of hyperspectral image interpretation, thereby improving the performance of downstream detailed identification tasks. Thus, enhancing the spatial resolution of hyperspectral images from a signal processing perspective without changing the hardware level of the imaging system is more practically significant—that is, hyperspectral image super-resolution technology. Unlike super-resolution tasks for natural images, hyperspectral image super-resolution techniques need to maintain the consistency of spectral information while improving spatial resolution in order to reduce the interference of spectral disturbances on downstream tasks.

[0004] To overcome the resolution bottleneck of single-modal images, utilizing deep networks to mine high-level semantic features embedded in hyperspectral images and their auxiliary high-resolution images has become the mainstream method for hyperspectral image super-resolution reconstruction. These methods combine high spatial resolution prior information from auxiliary modalities such as natural images and multispectral images, leveraging the complementarity between multimodal data to achieve spatial resolution enhancement and effective restoration of spatial details in hyperspectral images at large scales. However, existing methods often focus on the overall intensity features of the image, resulting in low resolution and consequently low accuracy of the final determined hyperspectral image. Summary of the Invention

[0005] This application provides an image processing method, apparatus, and electronic device for near-site security scenarios. It addresses the shortcomings of existing technologies that often focus on the overall intensity features of an image, resulting in low accuracy of the final hyperspectral image. By modeling the complementary edge knowledge and instance-level edge context between the first hyperspectral image and the multispectral image, it enhances the understanding of edge information and the ability to represent edge context, effectively reducing the reconstruction error of detailed regions, thereby obtaining a second hyperspectral image with higher resolution, i.e., a second hyperspectral image with higher accuracy.

[0006] This application provides an image processing method for near-site security scenarios, comprising: dividing an acquired first hyperspectral image of the near-site security scenario along bands to obtain multiple hyperspectral image groups, each hyperspectral image group including multiple bands; determining a first shallow feature H0 corresponding to the hyperspectral image group containing the i-th band, and obtaining a second shallow feature M0 corresponding to the multispectral image corresponding to the i-th hyperspectral image group; and determining a first modal edge enhancement feature H at the K-th level based on the first shallow feature H0. K and the second modal edge enhancement feature M at level K k Based on the first mode edge enhancement features H of each level K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features Based on the first modality edge enhancement features and edge context region labeling features at all levels and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of each band, the second hyperspectral image was determined. Among them, the second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

[0007] Optionally, the first modal edge enhancement feature H of the Kth layer is determined layer by layer based on the first shallow layer feature H0. K and the second modal edge enhancement feature M at level K k This includes: extracting semantic features from the first shallow feature H0 to obtain the first semantic feature. Then, semantic features are extracted from the second shallow feature M0 to obtain the second semantic features. First semantic features Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. Based on the second semantic features complementary detail-dependent features After making corrections, we obtain the corrected complementary detail-dependent features. Based on the first semantic features and the corrected complementary detail-dependent features Determine the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level; based on the first modal edge enhancement feature H1 of the (k-1)th level... k-1 and the second modal edge enhancement feature M of the (k-1)th level k-1 Calculate the first modal edge enhancement feature H at level k. k and the second modal edge enhancement feature M at level k k .

[0008] Optionally, based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features Includes: based on hyperspectral mode segmentation maps The second modal edge enhancement feature M at level K K Pooling is performed on the corresponding superpixel regions to obtain the instance region features m; based on the hyperspectral modality segmentation map... Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features.

[0009] Optionally, the first semantic features Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. Includes: processing the first semantic features through an edge-aware branch. Local semantic feature extraction is performed to obtain the detail texture feature estimation result D1; the detail texture feature estimation result D1 is then averaged along the channel dimension to obtain the first detail texture feature. The detailed texture feature estimation result D1 is then convolved along the channel dimension to obtain the second detailed texture feature. Based on the first detail texture features Second detail texture features Determine detailed texture blending features Based on detailed texture fusion features First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features

[0010] Optionally, based on the first semantic feature and the corrected complementary detail-dependent features Determining the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1 includes: within a preset channel range, based on the first semantic feature... The corresponding first deep semantic feature and the corrected complementary detail-dependent features The corresponding second deep semantic features The splicing result f u u∈{1,2,3,4} represents the four parts of the feature partitioned along the channel dimension; four convolutional kernels of different sizes are used to perform multi-scale feature fusion on the four concatenated results to obtain the fused result. Based on the fusion results and the first deep semantic features Determine the first-level first-modal edge enhancement feature H1; and based on the second-level deep semantic features... Determine the second modality edge enhancement feature M1 of the first level.

[0011] Optionally, based on the hyperspectral mode segmentation map Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features. Includes: based on superpixel center features s TDetermine the i-th hyperspectral image group G i Corresponding first graph node features And based on the instance region feature m, determine the multispectral image. Corresponding node features in the second graph Based on the hyperspectral mode segmentation map Determine the i-th hyperspectral image group G i The corresponding first adjacency matrix ε hsi And based on the instance region feature m, determine the multispectral image. The corresponding second adjacency matrix ε msi Based on the node features in the first image Second image node features First adjacency matrix ε hsi Second adjacency matrix ε msi Determine the i-th hyperspectral image group G from the output of N-layer graph convolution. i Corresponding graph node features Based on hyperspectral mode segmentation map Graph Node Features Determine edge context region labeling features

[0012] Optionally, based on the first detail texture features Second detail texture features Determine detailed texture blending features Includes: features of first detail texture Second detail texture features Perform splicing and absolute value operations to obtain the first detail features. For the first detailed feature The maximum value at each spatial location along the channel dimension is selected to generate the second detail feature. For first detail texture features Second detail texture features Second detailed features Perform stitching and convolution operations to obtain detailed texture fusion features.

[0013] Optionally, features are blended based on detailed textures. First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features Includes: determining the first semantic features Corresponding first mode frequency domain features The corresponding first hash code result and second semantic features Corresponding second mode frequency domain features The corresponding second hash code result Based on the first hash encoding result kth value Second hash encoding result kth value Determine the first hash code result With the second hash encoding result Hamming distance d between H According to the Hamming distance d H Generate frequency domain affinity coefficients Based on frequency domain affinity coefficient Features that blend with detailed textures Determine detailed affinity features Based on the first detail texture features Second detail texture features and details of affinity features Determine complementary detail-dependent features

[0014] This application embodiment also provides an image processing device for near-site security scenarios, including: a band acquisition module, used to divide the acquired first hyperspectral image for near-site security scenarios along bands to obtain multiple hyperspectral image groups, each hyperspectral image group including multiple bands; and an edge perception and enhancement module, used to determine the corresponding first shallow feature H0 according to the hyperspectral image group where the i-th band is located, and to obtain the corresponding second shallow feature M0 according to the multispectral image corresponding to the i-th hyperspectral image group, and to determine the first modal edge enhancement feature H at the K-th level according to the first shallow feature H0. K and the second modal edge enhancement feature M at level K k The topology inference module is used to enhance the edge features H of the first modality at each level. K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features The reconstruction module is used to determine the edge enhancement features and edge context region labels based on the first modality edge enhancement features and all levels. and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of each band, the second hyperspectral image was determined. Among them, the second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

[0015] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image processing method for local security scenarios as described above.

[0016] The image processing method, apparatus, and electronic device for near-site security scenarios provided in this application determine multiple levels of first-mode edge enhancement features and second-mode edge enhancement features based on first shallow features of a first hyperspectral image and second shallow features of a multispectral image for near-site security scenarios. Based on the final level of first-mode edge enhancement features and preset superpixel center features, a segmentation map and superpixel center features are determined. Then, combined with the final level of second-mode edge enhancement features, calibration features are determined. Based on the final level of first-mode edge enhancement features, calibration features, and corresponding spectral context information, a second hyperspectral image is determined. By modeling the edge complementarity knowledge and instance-level edge context between the first hyperspectral image and the multispectral image, the understanding of edge information and the ability to represent edge context are enhanced, effectively reducing reconstruction errors in detailed regions, thereby obtaining a second hyperspectral image with higher resolution, i.e., a second hyperspectral image with higher accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an overview diagram of the image processing method for on-site security scenarios provided in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of an image processing method for on-site security scenarios provided in an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating the image processing method for on-site security scenarios provided in this application embodiment;

[0021] Figure 4 This is a schematic diagram of the edge sensing module provided in an embodiment of this application;

[0022] Figure 5 This is the super-resolution reconstruction result with a magnification factor of 4 on the Harvard and CAVE datasets provided in the embodiments of this application;

[0023] Figure 6 This is the super-resolution reconstruction result on the Chikusei dataset with a magnification factor of 4, provided in the embodiments of this application;

[0024] Figure 7 This is the super-resolution reconstruction result with a magnification factor of 4 on the Houston dataset provided in this application embodiment;

[0025] Figure 8 This is a schematic diagram of the structure of the image processing device for on-site security scenarios provided in the embodiments of this application;

[0026] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To better understand the embodiments of this application, the technical field will first be described in detail:

[0029] This application primarily relates to the field of image data processing technology, specifically to the field of near-site security technology architecture. Starting from computer vision and pattern recognition technologies, it focuses on research into hyperspectral image super-resolution reconstruction methods for near-site security applications. By enhancing the understanding of boundary knowledge and contextual instances within the imaging region, reconstruction errors in boundary areas are mitigated, thereby improving the accuracy and visual quality of hyperspectral image super-resolution reconstruction. This provides high-resolution hyperspectral image information for the effective implementation of typical near-site security applications such as high-precision low-altitude UAV patrols, extreme disaster early warning, and target reconnaissance and identification.

[0030] Secondly, the electronic devices involved in the embodiments of this application will be described in detail:

[0031] In this embodiment, the electronic device includes K cascaded Edge Perception and Enhancement (EPE) units for estimating and fusing effective edge information from auxiliary modalities (such as multispectral images), explicitly enhancing attention to complementary edge details in the auxiliary modalities, and achieving efficient integration of complementary texture information in the auxiliary modalities. That is, it can strengthen attention to complementary edge details and optimize the representation of edge information.

[0032] Considering that the topological relationships between instances are important clues for defining edge details, this electronic device also includes a Topology Inference (TI) unit. This unit constructs instance dependencies based on graph convolutional networks using the distribution attributes of instances, enabling the interaction of instance-level boundary context information. In other words, by constructing instance dependencies based on graph convolutional networks using instance distribution attributes, it enhances the understanding of boundary context information and implicitly strengthens the ability to represent boundary information.

[0033] Optionally, the electronic device may include: a computer, a mobile terminal, and a wearable device, etc.

[0034] Furthermore, the image processing method for near-site security scenarios provided in this application embodiment can also be called the Boundary Topology Perception for Hyperspectral Image Super-resolution (HTPSR) method for near-site security scenarios.

[0035] Experimental results on four benchmark hyperspectral datasets demonstrate that the image processing method for near-site security scenarios provided in this application outperforms other representative image processing methods in both qualitative and quantitative evaluation. Specifically, the image processing method for near-site security scenarios provided in this application can significantly mitigate reconstruction errors in detail regions and reduce the impact of atmospheric scattering, environmental noise, and sample scarcity on the accuracy of hyperspectral super-resolution reconstruction. This provides effective and accurate data support for typical near-site security tasks requiring high-precision hyperspectral imagery, such as low-altitude UAV patrols, disaster early warning and rescue, and target reconnaissance. In other words, the HTPSR method, built based on EPE and TI units, improves the reconstruction accuracy of edges and detail regions in multi-source fusion super-resolution tasks by enhancing the understanding of inter-modal edge knowledge and intra-modal instance-level context.

[0036] It should be noted that the execution entity involved in the embodiments of this application can be an image processing device for local security scenarios or an electronic device. The following uses an electronic device as an example to describe in detail the image processing method for local security scenarios provided in the embodiments of this application:

[0037] For example, Figure 1 This is an overview diagram of the image processing method for on-site security scenarios provided in the embodiments of this application; Figure 2 This is a scene diagram illustrating the image processing method for local security scenarios provided in this application embodiment.

[0038] Combination Figure 1 and Figure 2 , Figure 3 This is a flowchart illustrating an image processing method for on-site security scenarios provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps 101-104.

[0039] Step 101: Divide the acquired first hyperspectral image for the local security scenario along the bands to obtain multiple hyperspectral image groups, each of which includes multiple bands.

[0040] Specifically, the electronic device acquires a first hyperspectral image for a local security scenario. First hyperspectral image Includes B hyperspectral image groups [G1,...,G i ,...,G B ], Let represent the i-th hyperspectral image group, where 3 ≤ i ≤ B - 2, and B is an integer greater than 5. Represents the first hyperspectral image The i-th band.

[0041] Among them, the aforementioned first hyperspectral image It is a low-resolution hyperspectral image, and also an input mode, namely a hyperspectral mode, where B represents the first hyperspectral image. The number of corresponding spectral bands; w represents the first hyperspectral image. Width in the spatial domain; h represents the first hyperspectral image. Height in the spatial domain.

[0042] In step 101, the electronic device can first acquire a first hyperspectral image for the local security scenario. Then, based on the high similarity between neighboring bands, the first hyperspectral image can be... Dividing along the spectral bands yields B neighborhood groups, i.e., B hyperspectral image groups. For boundary bands, such as... and The corresponding neighborhood groups can be generated using a symmetric extension strategy.

[0043] Step 102: Determine the corresponding first shallow feature H0 based on the hyperspectral image group containing the i-th band, and obtain the corresponding second shallow feature M0 based on the multispectral image corresponding to the i-th hyperspectral image group. Determine the first modal edge enhancement feature H0 at the K-th level layer by layer based on the first shallow feature H0. K and the second modal edge enhancement feature M at level K k .

[0044] Specifically, the electronic device can be based on the i-th band. The i-th hyperspectral image group G i The corresponding first shallow feature H0, and the i-th hyperspectral image group G i Corresponding multispectral images The corresponding second shallow layer feature M0 determines the first modal edge enhancement feature H of K levels. k and K-level second-modal edge enhancement features M k k = 1, ..., K, where K is an integer greater than 1; where the first modal edge enhancement feature H1 of the first level and the second modal edge enhancement feature M1 of the first level are calculated based on the first shallow feature H0 and the second shallow feature M0; the first modal edge enhancement feature H1 of the k-th level... k and the second modal edge enhancement feature M at level k k It is based on the first modal edge enhancement feature H of the (k-1)th level. k-1 and the second modal edge enhancement feature M of the (k-1)th level k-1 Calculated.

[0045] Among them, the i-th band mentioned above For the current super-resolution reconstruction band, take the i-th band. The i-th hyperspectral image group G i The corresponding multispectral image Perform super-resolution reconstruction.

[0046] The above multispectral images It is a high-resolution multispectral image, and also an input modality, i.e., a multispectral modality, where b represents the multispectral image. The number of corresponding spectral bands; W represents the multispectral image. Width in the spatial domain; H represents the multispectral image. In the spatial domain, the heights satisfy h << H, w << W, and b << B.

[0047] In step 102, for the i-th band The i-th hyperspectral image group G i The electronic device can process the i-th hyperspectral image group Gi Upsampling is performed using bicubic interpolation, so that the i-th hyperspectral image group G after upsampling... i Spatial dimension and corresponding multispectral image Matching. Furthermore, to fully utilize contextual information, the reconstruction results of the previous band, i.e., the (i-1)th band, are used. Compared with the i-th hyperspectral image group G after the above upsampling i The bands are then concatenated. Based on this, the electronic device can determine the i-th band. The i-th hyperspectral image group G i The corresponding first shallow feature H0. Then, the electronic device can process the i-th hyperspectral image group G mentioned above. i Corresponding multispectral images Perform convolution operations to extract shallow features and obtain the second shallow feature M0.

[0048] Next, the electronic device can determine the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1 based on the first shallow feature H0 and the second shallow feature M0; then, based on the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1, it can determine the second-level first-modal edge enhancement feature H2 and the second-level second-modal edge enhancement feature M2; furthermore, based on the second-level first-modal edge enhancement feature H2 and the second-level second-modal edge enhancement feature M2, it can determine the third-level first-modal edge enhancement feature H3 and the third-level second-modal edge enhancement feature M3. And so on, the electronic device can ultimately determine the first-modal edge enhancement features H1 at K levels. k and K-level second-modal edge enhancement features M k .

[0049] Optionally, the first shallow feature H0 can be expressed using the formula Calculated.

[0050] Among them, G i ↑ indicates the i-th hyperspectral image group G after upsampling. i Cat(·) represents the concatenation operation; Conv(·) represents the convolution operation.

[0051] Optionally, the second shallow layer feature M0 can be expressed using the formula Calculated.

[0052] Optionally, to effectively integrate the rich texture knowledge from the auxiliary modality, i.e., the second shallow feature M0, the electronic device is equipped with K cascaded EPE units, which enhance the attention to complementary edge details in the second shallow feature M0, achieving efficient integration of complementary texture knowledge. Specifically, the first EPE unit is used to determine the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1 based on the first shallow feature H0 and the second shallow feature M0; the second EPE unit is used to determine the second-level first-modal edge enhancement feature H2 and the second-level second-modal edge enhancement feature M2 based on the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1. And so on, the Kth EPE unit is used to determine the first-modal edge enhancement features H1 and M2 at K levels. k and K-level second-modal edge enhancement features M k .

[0053] It should be noted that the timing of the electronic device determining the first shallow feature H0 and the second shallow feature M0 is not limited.

[0054] The following section elaborates on how electronic devices determine the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1 based on the first shallow feature H0 and the second shallow feature M0:

[0055] In some embodiments, the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level are calculated according to the following steps: the electronic device extracts semantic features from the first shallow feature H0 to obtain the first semantic feature. Then, semantic features are extracted from the second shallow feature M0 to obtain the second semantic features. The electronic device will have the first semantic feature Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. The electronic device is based on the second semantic feature complementary detail-dependent features After making corrections, we obtain the corrected complementary detail-dependent features. The electronic device is based on the first semantic feature and the corrected complementary detail-dependent features Determine the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level.

[0056] In this embodiment, the electronic device first inputs the first shallow feature H0 into the residual module to extract semantic features and obtain the first semantic feature. Similarly, the electronic device can input the second shallow feature M0 into the residual module to extract semantic features and obtain the second semantic feature. Then, the electronic device uses an edge-aware branch to process the first semantic features. Second semantic features As input, a detail dependency graph is learned by modeling the relationship between these two semantic features to help explore multispectral imagery. Complementary detail-dependent features Next, inspired by attention mechanisms, the electronic device can utilize complementary detail-dependent features. Treat it as a global correlation descriptor for multispectral images Revised to highlight complementary details, specifically based on second semantic features. complementary detail-dependent features After making corrections, we obtain the corrected complementary detail-dependent features. Finally, the electronic device addresses the first semantic feature. and the corrected complementary detail-dependent features Calculations are performed to obtain the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level.

[0057] Optionally, the first semantic feature Formulas can be used Calculated.

[0058] Where Res(·) represents the residual module; Relu(·) represents the modified linear function.

[0059] Optionally, second semantic features Formulas can be used Calculated.

[0060] Optionally, complementary detail-dependent features Formulas can be used Calculated.

[0061] Among them, B per (·) indicates an edge-aware branch.

[0062] It should be noted that the aforementioned edge-aware branch includes two modules, namely as follows: Figure 4 The diagram shows the Edge Perception Module (EPM) and the Perceptual Hash Module (PHM). The former is used to identify detailed regions, while the latter is used to further adaptively adjust the edge perception map.

[0063] Optionally, the modified complementary detail-dependent features Formula (6) can be used. Calculated.

[0064] It should be noted that the electronic device uses the first modal edge enhancement feature H at the (k-1)th level. k-1 and the second modal edge enhancement feature M of the (k-1)th level k-1 The calculation process is similar to that of the above-mentioned electronic device, which determines the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level based on the first shallow feature H0 and the second shallow feature M0, and will not be described in detail here.

[0065] The following is in conjunction with the above. Figure 4 The EPM shown represents the first semantic feature of the electronic device. Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. To elaborate in detail:

[0066] In some embodiments, the electronic device will use the first semantic feature Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. This can include: electronic devices using edge-aware branches to detect first semantic features. Local semantic feature extraction is performed to obtain the detail texture feature estimation result D1; the electronic device performs a mean operation on the detail texture feature estimation result D1 along the channel dimension to obtain the first detail texture feature. The detailed texture feature estimation result D1 is then convolved along the channel dimension to obtain the second detailed texture feature. The electronic device is based on the first detail texture feature Second detail texture features Determine detailed texture blending features The electronic device is based on detailed texture fusion features. First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features

[0067] In this embodiment, considering that details are relatively sparse in a local area, the electronic device first uses the EPM in the edge-aware branch to process the first semantic features. Local semantic feature extraction is performed to obtain the detailed texture feature estimation result D1, specifically through the extraction of the first semantic feature. Perform average pooling to obtain local statistical results, then subtract the first semantic feature. To highlight areas with dramatic changes in detail, a preliminary estimation of the texture region is achieved, yielding a detail texture feature estimation result D1. Then, the electronic device performs a mean operation on the detail texture feature estimation result D1 along the channel dimension to aggregate the information representations of each channel, obtaining the first channel aggregated result, i.e., the first detail texture feature. The detailed texture feature estimation result D1 is then convolved along the channel dimension to aggregate the information representations of each channel, resulting in the second channel aggregated result, i.e., the second detailed texture feature. Among them, the first detail texture features Second detail texture features The image dimensions are all 1×W×H. Next, the electronic device processes the first detail texture features. Second detail texture features Calculations are performed to obtain detailed texture blending features. Finally, the electronic device determines the first semantic feature through PHM in the edge-aware branch. The corresponding first hash code result and second semantic features The corresponding second hash code result Combined with the above-mentioned detailed texture blending features Determine complementary detail-dependent features

[0068] Optionally, the detailed texture feature estimation result D1 can be obtained using the formula Calculated.

[0069] in, This indicates an average pooling operation; This indicates a partial statistical result.

[0070] Optionally, first detail texture features Formulas can be used Calculated.

[0071] Mean(·) represents the mean operation.

[0072] Optionally, second detail texture features Formulas can be used Calculated.

[0073] The following describes the electronic device based on the first detail texture feature. Second detail texture features Determine detailed texture blending features To elaborate in detail:

[0074] In some embodiments, the electronic device, based on the first detail texture feature Second detail texture features Determine detailed texture blending features This may include: the electronic device's first detail texture features Second detail texture features Perform splicing and absolute value operations to obtain the first detail features. The electronic device has a first detailed feature. The maximum value at each spatial location along the channel dimension is selected to generate the second detail feature. The electronic device has a first detail texture feature Second detail texture features Second detailed features Perform stitching and convolution operations to obtain detailed texture fusion features.

[0075] In this embodiment of the application, the first detail texture feature is taken into consideration. Second detail texture features This includes both positive and negative values; in this case, the absolute degree of detail is more important. Based on this, electronic devices can target the first detail texture features. Second detail texture features An adaptive stitching operation is performed, and then an absolute value operation is used to generate a detail map that reflects the degree of detail, i.e., the first detail feature is generated. in, This represents a set of detail dependency graphs. Then, the electronic device processes the first detail feature... By selecting the maximum value at each spatial location along the channel dimension, a more detailed definition is generated, i.e., a second detail feature is generated. Then, through splicing and convolution operations, this second detail feature is... Compared with the previous first detail texture features Second detail texture features The fusion process yields the detailed representation output of the EPM, i.e., the detailed texture fusion feature.

[0076] Finally, the electronic device addresses the first detail texture features. Second detail texture features Second detailed features Perform stitching and convolution operations to obtain detailed texture fusion features.

[0077] Optionally, first detailed features Formulas can be used Calculated.

[0078] in, This represents the first learnable weight; represents the second learnable weight; |·| represents the absolute value operation.

[0079] Optionally, detail texture blending features Formulas can be used Calculated.

[0080] in, This represents the third learnable weight; This represents the fourth learnable weight; This represents the fifth learnable weight.

[0081] It should be noted that although hyperspectral images are imaging results of the same scene, the degree of similarity between different bands and multispectral images is not consistent.

[0082] The following describes the feature fusion based on detailed textures in electronic devices. First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features To elaborate in detail:

[0083] In some embodiments, the electronic device uses detail texture fusion features. First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features This may include: electronic devices determining the first semantic feature Corresponding first mode frequency domain features The corresponding first hash code result and second semantic features Corresponding second mode frequency domain features The corresponding second hash code result The electronic device is based on the first hash encoding result kth value Second hash encoding result kth value Determine the first hash code result With the second hash encoding result Hamming distance d between H The electronic device is based on the Hamming distance d. H Generate frequency domain affinity coefficients This electronic device is based on frequency domain affinity coefficient Features that blend with detailed textures Determine detailed affinity features The electronic device is based on the first detail texture feature Second detail texture features and details of affinity features Determine complementary detail-dependent features

[0084] In this embodiment of the application, firstly, the electronic device detects the first semantic feature. The mean is applied along the channel and downsampled to a preset size (e.g., 8×8) to reduce computational complexity. Then, a Discrete Cosine Transform (DCT) operation is applied to obtain the frequency domain features of the first mode. And based on the same process, the second semantic feature is determined. Corresponding second mode frequency domain features Then, the electronic device performs frequency domain characteristics on the first mode. Second mode frequency domain features Perform hash operations on each DCT coefficient, specifically comparing each DCT coefficient with the mean DCT value (DCT coefficients greater than the mean DCT value are assigned a value of 1, otherwise assigned a value of 0), to obtain the corresponding binary hash value, which is the first hash encoding result. Second hash encoding result Subsequently, in order to determine the first hash encoding result With the second hash encoding result The difference can be determined by judging the first hash encoding result. kth value Second hash encoding result kth value To determine the first hash code result, we need to check if they are the same. With the second hash encoding result Hamming distance d between H Next, the electronic device measures the Hamming distance d. H Calculations were performed to obtain the frequency domain affinity coefficient. This frequency domain affinity coefficient The value ranges from 0 to 1. Furthermore, the electronic device incorporates detail texture fusion features. Normalized to the range [0,1], and the generated frequency domain affinity coefficients are used. Modulation is performed using a threshold to obtain preliminary detail affinity features.

[0085] Based on the following formula, we can see the detail texture blending feature. Medium to large The value of will be enhanced, while the rest will be suppressed. Therefore, the higher the affinity between modes, i.e., the higher the frequency domain affinity coefficient, the better. The larger the size, the better the detail texture blending features. The more regions are enhanced, the better. To improve the integrity of the representation of detailed regions, electronic devices can use detail affinity features. With first detail texture features Second detail texture features By combining these, we obtain the final complementary detail-dependent features.

[0086] Optionally, the first mode frequency domain features Formulas can be used Calculated.

[0087] Where ↓ represents bicubic downsampling operation; DCT(·) represents DCT operation.

[0088] Optionally, the second mode frequency domain features Formulas can be used Calculated.

[0089] Optionally, the first hash encoding result Formulas can be used Calculated.

[0090] Here, hash(·) represents a hash operation.

[0091] Optionally, the second hash encoding result Formulas can be used Calculated.

[0092] Optionally, the Hamming distance d H Formulas can be used Calculated.

[0093] Where n represents the length of the hash code.

[0094] Optionally, frequency domain affinity coefficient Formulas can be used Calculated.

[0095] Optionally, detail affinity features Formulas can be used Calculated.

[0096] Optionally, the final complementary detail-dependent features Formulas can be used Calculated.

[0097] The following describes electronic devices based on their first semantic features. and the corrected complementary detail-dependent features The first-level modal edge enhancement feature H1 and the first-level second-level edge enhancement feature M1 are defined and described in detail:

[0098] In some embodiments, the electronic device, based on the first semantic feature and the corrected complementary detail-dependent features Determining the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level can include: the electronic device determining the first semantic feature within a preset channel range. The corresponding first deep semantic feature and the corrected complementary detail-dependent features The corresponding second deep semantic features The splicing result f u u∈{1,2,3,4} represents the four parts of the feature partitioned along the channel dimension; the electronic device uses four convolutional kernels of different sizes to perform multi-scale feature fusion on the splicing result, obtaining the fused result. The electronic device is based on the fusion results and the first deep semantic features Determine the first-level first-modal edge enhancement feature H1; and based on the second-level deep semantic features... Determine the second modality edge enhancement feature M1 of the first level.

[0099] In this embodiment of the application, firstly, in order to obtain a deeper level of feature representation, the electronic device can determine the first semantic features through the residual module. The corresponding first deep semantic feature and the corrected complementary detail-dependent features The corresponding second deep semantic features Then, the first deep semantic features Second deep semantic features Fusion is performed in the edge enhancement branch to enhance the representation of edge information in the hyperspectral modes, thus obtaining the fusion result. At this point, the fusion result can be expressed using the formula. The calculation yields B. en (·) indicates an edge enhancement branch.

[0100] Specifically, the purpose of the edge enhancement branch is to enhance the second-deep semantic features. Integrating first-level deep semantic features The aim is to improve the detail representation capability of hyperspectral reconstruction information. Based on this, the above-mentioned entire determination fusion result The process can specifically determine the first deep semantic feature within the preset channel range. and the second deep semantic feature The splicing result f u These are the concatenated results f1, f2, f3, and f4. Next, the electronic device uses an Edge Enhancement Module (EEM) with four convolutional kernels of different sizes to extract features from each of the four concatenated results. The extracted results are then fused to obtain the final fused result. To improve the efficiency of multi-scale feature fusion, the electronic device can then adaptively combine the first deep-level semantic features using an additive operation to process detailed information while preserving structural content. and fusion results The first modal edge enhancement feature H1 of the first level of the current EPE unit output is determined. Finally, the electronic device determines the second modal edge enhancement feature M1 of the first level of the current EPE unit output through residual operations.

[0101] Optionally, the first deep semantic features Formulas can be used Calculated.

[0102] Optionally, second-level deep semantic features Formulas can be used Calculated.

[0103] Optionally, the splicing result f u Formulas can be used

[0104] Where (u-1)C / 4: uC / 4 represents the preset channel range; C represents the first deep semantic feature. Or second-level deeper semantic features The number of feature channels.

[0105] Optionally, the fusion result Formulas can be used Calculated.

[0106] in, This represents a convolution operation with a kernel size of 1; This indicates a convolution operation with a kernel size of 3; This indicates a convolution operation with a kernel size of 5; This indicates a convolution operation with a kernel size of 7.

[0107] Optionally, the first modal edge enhancement feature H1 of the first level can be expressed using the formula... Calculated.

[0108] in, This represents the sixth learnable weight; This represents the seventh learnable weight.

[0109] Optionally, the second modal edge enhancement feature M1 of the first level can be expressed using the formula... Calculated.

[0110] It should be noted that the implementation process of each EPE unit is the same, and will not be elaborated here. Based on this, the electronic device can ultimately determine the first-mode edge enhancement features H at K levels. k and K-level second-modal edge enhancement features M k .

[0111] Step 103: Based on the first modal edge enhancement feature H of the Kth level. K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features

[0112] Among them, edge context region labeling features It is a result of topological reasoning.

[0113] In step 103, the electronic device enhances its ability to represent boundary information by constructing a TI unit modeling instance-level context interaction mode. Specifically, it can first enhance the edge feature H of the first modality at the Kth level. K and the preset superpixel center feature s t-1 Calculations were performed to obtain a hyperspectral modal segmentation map of labeled spatial pixel categories. and superpixel center features s T Furthermore, it is combined again with the second modal edge enhancement feature M of the Kth level. K To identify edge context region labeling features with high accuracy

[0114] in,

[0115] It should be noted that boundary information can be defined by the topological distribution of objects in a near-site security scenario. Based on this, instances in the near-site security scenario are considered as contextual information of the boundaries, and a TI unit is designed to effectively understand the boundary context, thereby implicitly enhancing the ability to reconstruct edge details. Specifically, firstly, an instance shape dependent on a compact representation of local regions is constructed based on the soft K-means superpixel algorithm, and then an instance-level topological interaction is achieved using a Graph Convolutional Network (GCN).

[0116] Optionally, the electronic device enhances the edge feature H according to the first modality of the Kth level. K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T In the process, firstly, the first modality edge enhancement feature H at the Kth level is processed by the Superpixel Segmentation (SPA) module. K Processing is performed to calculate the pixel and the preset superpixel center feature s. t-1 The relationship between them.

[0117] It should be noted that the SPA module is an expectation-maximization process involving T iterations, where T is an integer greater than 1. Specifically, in the t-th iteration, the superpixel center correlation map A corresponding to the pixel is first calculated. t , Specifically, formula (27) can be used. Calculations show that

[0118] Among them, A t (ij) can also represent the i-th pixel H K (i) and M superpixel center features s t-1 The correlation values ​​between them, where M represents the preset hyperparameter, 0≤i≤HW-1 and 0≤j≤M-1.

[0119] Next, electronic devices can use the formula Update superpixel center features s t -1 The updated superpixel center features s are obtained. t (j).

[0120] Among them, z t (j) represents the first modal edge enhancement feature H at the Kth level. K The normalized term along the column.

[0121] Thus, after T iterations, the electronic device is finally able to determine the superpixel center feature s. T And the relationship diagram A T At this point, the electronic device then retrieves the associated diagram A. T The maximum index in the second dimension (HW) is used as the category of each pixel, and a reshaping operation is applied to obtain a hyperspectral modality segmentation map of pixel categories in the labeled space.

[0122] The following describes the electronic equipment based on the hyperspectral modal segmentation diagram. The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features To elaborate in detail:

[0123] In some embodiments, the electronic device is based on a hyperspectral mode segmentation map. The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features This may include: electronic devices based on hyperspectral mode segmentation maps The second modal edge enhancement feature M at level K K Pooling is performed on the corresponding superpixel regions to obtain the instance region features m; the electronic device then uses the hyperspectral modality segmentation map... Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features.

[0124] In this embodiment, the electronic device uses an adaptive pooling module to perform hyperspectral mode segmentation based on the previously calculated hyperspectral mode segmentation map. The second modal edge enhancement feature M at level K K Pooling is performed on the corresponding superpixel regions to obtain the instance region features m. The region feature m in this instance is highly representative. Based on this, and combined with the hyperspectral modal segmentation map... and superpixel center features s T It can determine edge context region labeling features with high accuracy.

[0125] The above electronic devices are analyzed based on hyperspectral mode segmentation diagrams. Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features. To elaborate further:

[0126] In some embodiments, the electronic device is based on a hyperspectral mode segmentation map. Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features. This may include: electronic devices based on superpixel center features s T Determine the i-th hyperspectral image group G i Corresponding first graph node features And based on the instance region feature m, determine the multispectral image. Corresponding node features in the second graph The electronic device is based on hyperspectral mode segmentation. Determine the i-th hyperspectral image group G i The corresponding first adjacency matrix ε hsi And based on the instance region feature m, determine the multispectral image. The corresponding second adjacency matrix ε msi The electronic device is based on the node features of the first image. Second image node features First adjacency matrix ε hsi Second adjacency matrix ε msi Determine the i-th hyperspectral image group G from the output of N-layer graph convolution. i Corresponding graph node features The electronic device is based on hyperspectral mode segmentation. Graph Node Features Determine edge context region labeling features

[0127] In this embodiment of the application, firstly, the electronic device can determine the superpixel center feature s T Construct the feature embedding of the first graph node, that is, determine the node features of the first graph. Specifically, the electronic device can determine the j-th pixel s T (j), determine the j-th node of the first graph. Then, the features of the first graph nodes are determined based on all the first graph nodes. Similarly, the electronic device constructs a second graph node feature embedding based on the instance region feature m, that is, determines the second graph node features. Specifically, the electronic device can determine the j-th second graph node based on the j-th pixel m(j). Then, the node features of the second graph are determined based on all the nodes in the second graph. Then, the electronic device is based on the hyperspectral mode segmentation map. The determined superpixel spatial adjacency relationship is for the i-th hyperspectral image group G. iDetermine the weight of the first undirected edge, and then construct the first adjacency matrix ε. hsi , and for multispectral images Determine the weights of the second undirected edges, and then construct the second adjacency matrix ε. msi ,

[0128] Next, in order to achieve cross-modal topology interaction, the electronic device can, based on the node features of the first graph... Second graph node features Constructing a cross-modal node matrix in, And according to the first adjacency matrix ε hsi Second adjacency matrix ε msi Construct the cross-modal adjacency matrix ε. in, I represents the identity matrix, indicating the interaction between corresponding node positions between modalities. The entire process enables instance-level interactions within and between modalities. Subsequently, an undirected weighted graph is constructed. It can realize instance-level cross-modal information modeling and propagation. Specifically, it can use N-layer graph convolution on undirected weighted graphs. Feature extraction is performed to obtain the output of each convolutional layer, thereby obtaining the i-th hyperspectral image group G. i Corresponding graph node features N is an integer greater than 1. Optionally, h is the output of the nth layer graph convolution. n+1 Formula (29) can be used. The calculation yields , where σ(·) represents the activation function; Represents a diagonal matrix. W n This represents the learnable parameters of the nth layer graph convolution.

[0129] It should be noted that in the initial layer, the input is set to... The final result after N layers of graph convolution is Based on this, after graph convolution inference, the electronic device can obtain the i-th hyperspectral image group G. i Corresponding graph node features and multispectral images Corresponding graph node features

[0130] Finally, the understanding of edge regions is enhanced by annotating boundary instance context information. Specifically, the electronic device can utilize hyperspectral modal segmentation maps. A topology diffusion mechanism is used to incorporate graph node features. Each superpixel region is assigned to its respective superpixel region to achieve edge context region calibration, resulting in initial edge context region calibration features F. TI (p,q,:), and then the residual module is used to obtain the edge context region labeling features of the final output of the TI unit. The entire process enhances the expressive power of instance features in boundary regions through the interaction of instance-level contextual information. This enables more accurate modeling of cross-instance feature transition relationships, thereby improving the ability to characterize edge regions and obtaining highly accurate edge contextual region labeling features.

[0131] Optionally, the j-th node of the first graph Formulas can be used Calculated.

[0132] Optionally, the j-th node of the second graph Formulas can be used Calculated.

[0133] Optionally, the weight of the first undirected edge can be determined using the formula... Calculated.

[0134] Among them, the weight of the first undirected edge Let ε be the first adjacency matrix hsi The elements in.

[0135] Optionally, the weight of the second undirected edge can be expressed by the formula Calculated.

[0136] Among them, the weight of the second undirected edge The second adjacency matrix ε msi The elements in.

[0137] Optionally, the initial edge context region labeling feature F TI (p,q,:) can be represented by the formula Calculated.

[0138] Among them, F TI (p,q,:) can also represent the feature vector at position (p,q); This represents the superpixel index at position (p, q); Hyperspectral mode segmentation diagram The pixel position in the image.

[0139] Step 104: Based on the first modal edge enhancement features H of K levels k Edge context region labeling features and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of band B, the second hyperspectral image was determined. Second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

[0140] In step 104, the electronic device can first determine the first modal edge enhancement features H based on K levels. k Edge context region labeling features and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction features By learning the global residual, the corresponding reconstruction results are obtained. After processing each band individually, the electronic device can sequentially obtain the corresponding band reconstruction results, thus ultimately determining the reconstruction results of B bands. These B band reconstruction results are then merged to obtain the second hyperspectral image. In other words, the HTPSR method for near-site security scenarios provided in this application can obtain a second hyperspectral image with higher resolution.

[0141] Optionally, reconstruct features Formulas can be used Calculated.

[0142] Among them, w1, w2, ..., w K+2 This represents the learnable weights, which can be dynamically adjusted during training.

[0143] Optionally, the reconstruction results Formulas can be used Calculated.

[0144] in, Indicates the i-th band The results of double triple upsampling.

[0145] In this embodiment, by modeling the edge complementarity knowledge and instance-level edge context between the first hyperspectral image and the multispectral image, the understanding of edge information and the ability to represent edge context are enhanced, effectively reducing the reconstruction error of detailed regions to obtain a second hyperspectral image with higher resolution, i.e., a second hyperspectral image with higher accuracy. The entire process can effectively overcome the impact of atmospheric scattering, environmental noise, and sample scarcity on the accuracy of hyperspectral super-resolution reconstruction. This provides effective and accurate data support for typical on-site security tasks that require high-precision hyperspectral image information, such as low-altitude UAV patrols, disaster early warning and rescue, and target reconnaissance.

[0146] To better understand the embodiments of this application, the following simulation experiment illustrates the image processing method for local security scenarios provided by the embodiments of this application:

[0147] The hyperspectral super-resolution method based on multi-level metrics, specifically the image processing method for near-site security scenarios provided in this application, was simulated on two natural scene datasets (i.e., the Harvard dataset and the Columbia Advanced Vision and Imaging (CAVE) dataset) and two remote sensing datasets (i.e., the Chikusei dataset and the Houston dataset). Both the two natural scene datasets and the two remote sensing datasets are hyperspectral image datasets. The specific steps of the simulation experiment are as follows:

[0148] Step 1: Constructing the training and test sets

[0149] The CAVE dataset mentioned above includes 32 hyperspectral images captured by a charge-coupled device (CCD) camera. Each image has a spatial resolution of 512×512 pixels, covering wavelengths from 400nm to 700nm in 31 bands spaced 10nm apart. The Harvard dataset mentioned above contains 77 real-world hyperspectral images. Each hyperspectral image consists of 31 spectral bands and is 1392×1040 pixels in size. For training and testing, 80% of the hyperspectral images in each dataset were randomly selected as training data, and the remaining hyperspectral images were used for testing. To construct the training set, 64 image patches were randomly cropped from each hyperspectral image in both the CAVE and Harvard datasets, and each patch was downsampled according to a scaling factor of 4 to obtain a low-resolution image with a spatial size of 12×12. During the testing phase, performance tests were conducted on the top-left 480×480×31 region of the test image from the CAVE dataset and the top-left 576×576×31 region of the test image from the Harvard dataset.

[0150] The Chikusei dataset described above consists of remote sensing images acquired by the Headwall hyperspec-VNIR-C imaging sensor in agricultural and urban areas, covering the spectral range from 363 nm to 1018 nm. Unlike natural scene datasets, this Chikusei dataset records a scene with a spatial size of 2517 × 2335 pixels, containing 128 spectral bands. To remove invalid boundaries, the surrounding 50 pixels of the scene were cropped, resulting in effective hyperspectral scene data of size 2417 × 2235 × 128 pixels. Subsequently, the upper 1905 × 2235 × 128 pixel region was used as training data, and the remaining portion was used for testing. To construct the training set, 396 image patches were randomly selected from the training region and spatially downsampled according to a scale factor to obtain low-resolution samples of size 12 × 12 pixels.

[0151] The Houston dataset contains 48 spectral bands with a spatial size of 2384×601 pixels. The left 601×1808 region of the Houston dataset is used as training data, and the remaining portion is used for testing. 200 image patches are randomly selected from the training region to construct the training set, and downsampled according to a scale factor to obtain low-resolution training samples of size 12×12.

[0152] In addition, data augmentation techniques were applied to the training set, including random scrolling, rotation (90°, 180°, 270°), and flipping operations. Bicubic downsampling was used to obtain a low-resolution image of size L×32×32, where L is an integer greater than or equal to 1. Furthermore, various downsampling kernels of different sizes (including 7, 9, 13, and 15) were generated using isotropic or anisotropic Gaussian functions. These kernels were combined with convolutions, and low-resolution samples from the training set were obtained by performing convolution operations of different lengths. During the testing phase, bicubic downsampling was used to generate the corresponding low-resolution images.

[0153] Step 2: Experiment Setup

[0154] In step two, the kernel size is set to 3 and the number of filters is set to 64. The Adam optimizer is used to optimize the algorithm, with parameters β1 = 0.9 and β2 = 0.999.

[0155] The initial learning rate was set to 0.0001 and decayed by half every 40 epochs.

[0156] The number of training cycles is set to 200.

[0157] Step 3: Setting Evaluation Indicators

[0158] To quantitatively evaluate reconstruction performance, three widely used metrics were employed: Peak Signal-to-Noise Ratio (PSNR), Spectral Angle Mapper (SAM), and Structural Similarity Index Measure (SSIM).

[0159] Among them, PSNR and SSIM are used to evaluate the reconstruction performance in the spatial domain, with higher values ​​indicating better performance; SAM is used to measure spectral reconstruction performance, with lower values ​​indicating stronger spectral recovery ability.

[0160] Step 4: Comparative Experiment

[0161] Comparative experiments were conducted on the four publicly available hyperspectral image datasets mentioned above, using representative methods. For example, Figure 5 This is the super-resolution reconstruction result with a magnification factor of 4 on the Harvard and CAVE datasets provided in the embodiments of this application; Figure 6 This is the super-resolution reconstruction result on the Chikusei dataset with a magnification factor of 4, provided in the embodiments of this application; Figure 7 This is the super-resolution reconstruction result provided in this application embodiment when the magnification factor is 4 on the Houston dataset.

[0162] Among them, the above Figure 5 , Figure 6 and Figure 7 The super-resolution reconstruction result involved is the second hyperspectral image mentioned above.

[0163] The experimental results on the Harvard and CAVE datasets are shown in Table 1, and the experimental results on the Chikusei and Houston datasets are shown in Table 2.

[0164] Among them, the representative methods mentioned above include nine super-resolution methods, namely, Hyperspectral Super-resolution (HySure), Spatial Spectral Seconstruction network for hyperspectral and multispectral image fusion Network (SSRNet), Model Guided deep hyperspectral image super-resolution model-guided Deep Convolutional Network (MoG-DCN), Iduced Feature Modulation Super-Resolution (IFMSR), Pyramid Shuffle-and-Reshuffle Transformer for multispectral and hyperspectral image fusion (PSRT), and Multiscale Spatial-Spectral Transformer Network for hyperspectral and multispectral image fusion. The methods include fusion (MSST), Symmetrical Feature Propagation Network for hyperspectral image super-resolution (SFPN), Multiscale factor joint learning for hyperspectral image super-resolution (MulSR), and Kernel-space Non-Nocal convolution for hyperspectral image super-resolution Network (KNLNet).

[0165] Table 1:

[0166]

[0167] Table 2:

[0168]

[0169] Based on the experimental results in Tables 1 and 2, the experimental results on the four publicly available hyperspectral image datasets show that the image processing method for local security scenarios provided in this application enhances the representation of edge details by improving the understanding of edge knowledge and its contextual instance information, resulting in higher reconstruction accuracy and better visual effects throughout the process. This effectively alleviates edge region errors and reduces reconstruction errors in detail regions.

[0170] The image processing apparatus for local security scenarios provided in the embodiments of this application is described below. The image processing apparatus for local security scenarios described below can be referred to in correspondence with the image processing method for local security scenarios described above.

[0171] Figure 8 This is a schematic diagram of the image processing device for on-site security scenarios provided in an embodiment of this application. Figure 8 As shown, the device includes: an acquisition module 801, an edge perception and enhancement module 802, and a topology reasoning module 803.

[0172] The band acquisition module 801 is used to divide the acquired first hyperspectral image for the local security scenario along the bands to obtain multiple hyperspectral image groups, each of which includes multiple bands.

[0173] The edge perception and enhancement module 802 is used to determine the corresponding first shallow feature H0 based on the hyperspectral image group where the i-th band is located, and to obtain the corresponding second shallow feature M0 based on the multispectral image corresponding to the i-th hyperspectral image group. Based on the first shallow feature H0, it then determines the first modal edge enhancement feature H at the K-th level layer by layer. K and the second modal edge enhancement feature M at level K k ;

[0174] Topology inference module 803 is used to enhance the edge features H of the first modality at each level. K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K Kand superpixel center features s T Determine the edge context region labeling features

[0175] Reconstruction module 804 is used to enhance the edge features H of the first modality of the K levels. k The edge context region identification features and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of band B, the second hyperspectral image was determined. The second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

[0176] Optionally, the band acquisition module 801 is used to extract semantic features from the first shallow feature H0 to obtain the first semantic feature. Then, semantic features are extracted from the second shallow feature M0 to obtain the second semantic feature. The first semantic feature and the second semantic feature The input is fed into the edge-aware branch to obtain complementary detail-dependent features. According to the second semantic feature This complementary detail-dependent feature After making corrections, we obtain the corrected complementary detail-dependent features. Based on the first semantic feature and the corrected complementary detail-dependent features Determine the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level.

[0177] Optionally, the topology inference module 803 is specifically used to infer the hyperspectral mode segmentation map. The second modal edge enhancement feature M of the Kth level K Pooling is performed on the corresponding superpixel regions to obtain the instance region features m; based on this hyperspectral modality segmentation map... The superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features.

[0178] Optionally, the edge perception and enhancement module 802 is specifically used to enhance the first semantic feature through the edge perception branch. Local semantic feature extraction is performed to obtain the detail texture feature estimation result D1; the mean operation is performed on the detail texture feature estimation result D1 along the channel dimension to obtain the first detail texture feature. The second detail texture feature is obtained by convolving the estimated detail texture feature D1 along the channel dimension. Based on the first detail texture feature and the second detail texture feature Determine detailed texture blending features Based on detailed texture fusion features The first semantic feature The corresponding first hash code result and the second semantic feature The corresponding second hash code result Determine complementary detail-dependent features

[0179] Optionally, the edge perception and enhancement module 802 is specifically used to determine the first semantic feature within a preset channel range. The corresponding first deep semantic feature and the corrected complementary detail-dependent features The corresponding second deep semantic features The splicing result f u u∈{1,2,3,4} represents the four parts of the feature partitioned along the channel dimension; using four convolutional kernels of different sizes, multi-scale feature fusion is performed on the concatenated result to obtain the fused result. According to the fusion results and the first deep semantic feature Determine the first modality edge enhancement feature H1 of the first level; and based on the second deep semantic feature... Determine the second modal edge enhancement feature M1 of the first level.

[0180] Optionally, the topology inference module 803 is specifically used to infer the superpixel center feature s T Determine the i-th hyperspectral image group G i Corresponding first graph node features And based on the feature m of the instance region, the multispectral image is determined. Corresponding node features in the second graph Based on the hyperspectral mode segmentation diagram Determine the i-th hyperspectral image group G i The corresponding first adjacency matrix ε hsi And based on the feature m of the instance region, the multispectral image is determined. The corresponding second adjacency matrix ε msiBased on the node features of the first graph The second graph node features The first adjacency matrix ε hsi and the second adjacency matrix ε msi Determine the i-th hyperspectral image group G output by the N-layer graph convolution. i Corresponding graph node features Based on the hyperspectral mode segmentation diagram and the node features of the graph Determine the labeling features of the edge context region.

[0181] Optionally, the edge awareness and enhancement module 802 is specifically used for the first detail texture feature. and the second detail texture feature Perform splicing and absolute value operations to obtain the first detail features. Regarding the first detailed feature The maximum value at each spatial location along the channel dimension is selected to generate the second detail feature. This first detail texture feature The second detail texture feature and the second detailed feature By performing stitching and convolution operations, the detailed texture fusion feature is obtained.

[0182] Optionally, the edge perception and enhancement module 802 is specifically used to determine the first semantic feature. Corresponding first mode frequency domain features The corresponding first hash code result and the second semantic feature Corresponding second mode frequency domain features The corresponding second hash code result According to the first hash encoding result kth value and the second hash encoding result kth value Determine the first hash code result With the second hash encoding result Hamming distance d between H According to the Hamming distance d H Generate frequency domain affinity coefficients Based on this frequency domain affinity coefficient and the feature of blending with the detailed texture Determine detailed affinity features Based on the first detail texture feature The second detail texture feature And this detail affinity feature Determine the complementary detail-dependent feature

[0183] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940. The processor 910, communications interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions from the memory 930 to execute image processing methods for local security scenarios.

[0184] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image processing methods for local security scenarios provided by the above methods.

[0186] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the image processing methods for local security scenarios provided by the methods described above.

[0187] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method for near-site security scenarios, characterized in that, include: The first hyperspectral image acquired for the local security scenario is divided along the bands to obtain multiple hyperspectral image groups, each of which includes multiple bands; Based on the hyperspectral image group containing the i-th band, determine the corresponding first shallow feature H0, and based on the multispectral image corresponding to the i-th hyperspectral image group, obtain the corresponding second shallow feature M0. Based on the first shallow feature H0, determine the first modal edge enhancement feature H of the K-th level layer by layer. K and the second modal edge enhancement feature M at level K k ; Based on the first modal edge enhancement feature H of each level K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features Based on the first modality edge enhancement features and edge context region labeling features at all levels and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of each band, the second hyperspectral image was determined. Among them, the second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

2. The image processing method for near-site security scenarios according to claim 1, characterized in that, The first modal edge enhancement feature H of the Kth layer is determined layer by layer based on the first shallow layer feature H0. K and the second modal edge enhancement feature M at level K k ,include: Semantic features are extracted from the first shallow feature H0 to obtain the first semantic feature. Then, semantic features are extracted from the second shallow feature M0 to obtain the second semantic feature. First semantic features Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. Based on the second semantic features complementary detail-dependent features After making corrections, we obtain the corrected complementary detail-dependent features. Based on the first semantic features and the corrected complementary detail-dependent features Determine the first modal edge enhancement feature H1 and the second modal edge enhancement feature M1 of the first level; Based on the first modal edge enhancement feature H of the (k-1)th level k-1 and the second modal edge enhancement feature M of the (k-1)th level k-1 Calculate the first modal edge enhancement feature H at level k. k and the second modal edge enhancement feature M at level k k .

3. The image processing method for near-site security scenarios according to claim 1, characterized in that, The method based on hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and the superpixel center feature s T Determine the edge context region labeling features include: Based on the hyperspectral mode segmentation map The second modal edge enhancement feature M at level K K Pooling is performed on the corresponding superpixel regions to obtain the instance region features m; Based on the hyperspectral mode segmentation map Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features.

4. The image processing method for near-site security scenarios according to claim 2, characterized in that, The first semantic feature Second semantic features The input is fed into the edge-aware branch to obtain complementary detail-dependent features. include: By using the edge-aware branch, the first semantic features Local semantic features are extracted to obtain the detailed texture feature estimation result D1; The first detail texture feature is obtained by averaging the estimation result D1 along the channel dimension. The detailed texture feature estimation result D1 is then convolved along the channel dimension to obtain the second detailed texture feature. Based on the first detail texture features Second detail texture features Determine detailed texture blending features Based on detailed texture fusion features First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features 5. The image processing method for near-site security scenarios according to claim 2, characterized in that, The first semantic feature and the corrected complementary detail-dependent features Determine the first-level first-modal edge enhancement feature H1 and the first-level second-modal edge enhancement feature M1, including: Within the preset channel range, based on the first semantic feature The corresponding first deep semantic feature and the corrected complementary detail-dependent features The corresponding second deep semantic features The splicing result f u u∈{1,2,3,4} represents the four parts of the feature divided along the channel dimension; Four convolutional kernels of different sizes are used to perform multi-scale feature fusion on the four concatenated results to obtain the fused result. Based on the fusion results and the first deep semantic features Determine the first-level first-modal edge enhancement feature H1; and based on the second-level deep semantic features... Determine the second modality edge enhancement feature M1 of the first level.

6. The image processing method for near-site security scenarios according to claim 3, characterized in that, The method based on hyperspectral mode segmentation map Superpixel center feature s T Based on the instance region feature m, determine the edge context region labeling features. include: Based on the superpixel center feature s T Determine the i-th hyperspectral image group G i Corresponding first graph node features And based on the instance region feature m, determine the multispectral image. Corresponding node features in the second graph Based on the hyperspectral mode segmentation map Determine the i-th hyperspectral image group G i The corresponding first adjacency matrix E hsi And based on the instance region feature m, determine the multispectral image. The corresponding second adjacency matrix E msi ; Based on the node characteristics in the first image Second image node features First adjacency matrix E hsi Second adjacency matrix E msi Determine the i-th hyperspectral image group G from the output of N-layer graph convolution. i Corresponding graph node features Based on the hyperspectral mode segmentation map Graph Node Features Determine edge context region labeling features 7. The image processing method for near-site security scenarios according to claim 4, characterized in that, The first detail texture feature Second detail texture features Determine detailed texture blending features include: For first detail texture features Second detail texture features Perform splicing and absolute value operations to obtain the first detail features. For the first detailed feature The maximum value at each spatial location along the channel dimension is selected to generate the second detail feature. For first detail texture features Second detail texture features Second detailed features Perform stitching and convolution operations to obtain detailed texture fusion features.

8. The image processing method for near-site security scenarios according to claim 4, characterized in that, The feature based on detailed texture fusion First semantic features The corresponding first hash code result and second semantic features The corresponding second hash code result Determine complementary detail-dependent features include: Determine the first semantic features Corresponding first mode frequency domain features The corresponding first hash code result and second semantic features Corresponding second mode frequency domain features The corresponding second hash code result Based on the first hash encoding result kth value Second hash encoding result kth value Determine the first hash code result With the second hash encoding result Hamming distance d between H ; According to the Hamming distance d H Generate frequency domain affinity coefficients Based on frequency domain affinity coefficient Features that blend with detailed textures Determine detailed affinity features Based on the first detail texture features Second detail texture features and details of affinity features Determine complementary detail-dependent features 9. An image processing device for on-site security scenarios, characterized in that, include: The band acquisition module is used to divide the acquired first hyperspectral image for the local security scenario along the bands to obtain multiple hyperspectral image groups, each of which includes multiple bands. The edge perception and enhancement module is used to determine the corresponding first shallow feature H0 based on the hyperspectral image group where the i-th band is located, and to obtain the corresponding second shallow feature M0 based on the multispectral image corresponding to the i-th hyperspectral image group. Based on the first shallow feature H0, the module then determines the first modal edge enhancement feature H at the K-th level layer by layer. K and the second modal edge enhancement feature M at level K k ; The topology inference module is used to enhance the edge features H of the first modality at each level. K and the preset superpixel center feature s t-1 Hyperspectral mode segmentation map to determine the spatial pixel categories and superpixel center features s T Based on the hyperspectral mode segmentation map The second modality edge enhancement feature M at level K K and superpixel center features s T Determine the edge context region labeling features The reconstruction module is used to determine the edge enhancement features and edge context region labels based on the first modality edge enhancement features and all levels. and the (i-1)th band Corresponding spectral context information Determine the i-th band Reconstruction results Based on the reconstruction results of each band, the second hyperspectral image was determined. Among them, the second hyperspectral image The resolution is greater than that of the first hyperspectral image. The resolution.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image processing method for local security scenarios as described in any one of claims 1 to 8.

Images