Image fusion method based on ISP image signal processing visual sensor

Abstract

The invention relates to an image fusion method based on an ISP image signal processing visual sensor. The method comprises the following steps: performing spatial feature extraction and semantic feature extraction on an original multi-source image acquired by an ISP visual sensor to obtain a two-dimensional attention feature map; performing multi-scale pyramid pooling processing to obtain a pyramid feature map set; inputting into a gradient bilateral filter for filtering operation to obtain a three-layer fine-coarse structure feature map set; performing Laplacian operator enhancement to obtain an enhanced feature map, and performing weighted fusion through region energy density and local contrast saliency calculation to obtain a multi-level fusion feature map; and calculating a definition gradient value, a local contrast value and an information entropy value of each layer of feature map, and carrying out weighted stacking operation to obtain a target fusion image. The visual quality and the information integrity of the fused image are effectively improved, the balance of the definition, the contrast ratio and the information amount of the fused image is realized, and the overall performance of the fusion result is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and particularly to an image fusion method based on an ISP image signal processing vision sensor. Background Art

[0002] ISP image signal processing vision sensors are widely used in fields such as industrial automation, medical diagnosis, and intelligent security. Their image fusion technology is the key to effectively integrating multi-source image information. Traditional image fusion methods mainly rely on pixel-level fusion and feature-level fusion, and often have problems such as detail loss and edge blurring when dealing with complex scenes, making it difficult to meet the requirements of high-precision visual perception.

[0003] Currently, in low-light or complex texture scenes, traditional image fusion algorithms are prone to generating artifacts during feature extraction and matching, resulting in a decline in the quality of the fused image. Especially when dealing with images with rich detail information, existing methods are difficult to simultaneously maintain the integrity of spatial detail information and semantic information, affecting the subsequent image analysis and recognition effects. In addition, when existing ISP vision sensors perform image fusion, the computational efficiency is low, the real-time performance is insufficient, and the adaptive ability under different lighting conditions is limited. At the same time, due to the lack of an effective feature constraint mechanism, problems such as poor spatial consistency and low edge accuracy are prone to occur during the fusion process. Summary of the Invention

[0004] The main purpose of the present invention is to provide an image fusion method based on an ISP image signal processing vision sensor. The present invention effectively improves the visual quality and information integrity of the fused image, realizes the balance of the clarity, contrast, and information volume of the fused image, and ensures the overall performance of the fusion result.

[0005] To achieve the above object, the present invention provides an image fusion method based on an ISP image signal processing vision sensor, including the following steps: Perform spatial feature extraction and semantic feature extraction on the original multi-source images collected by the ISP vision sensor to obtain a spatial feature map and a semantic feature map, and perform two-dimensional serial attention processing to obtain a two-dimensional attention feature map; Perform edge detection on the two-dimensional attention feature map to obtain edge constraint information, and perform multi-scale pyramid pooling processing to obtain a set of pyramid feature maps; Input the set of pyramid feature maps into a gradient bilateral filter for feature decomposition, and set the spatial kernel standard deviation and the range kernel standard deviation for filtering operations to obtain a set of three-layer fine-coarse structure feature maps and a base layer image; Perform Laplacian operator enhancement on the three-layer fine-coarse structure feature map sets respectively to obtain enhanced feature maps, and perform weighted fusion through regional energy density and local contrast significance calculation to obtain multi-level fusion feature maps; Perform pixel value normalization processing on the multi-level fusion feature maps, calculate the sharpness gradient value, local contrast value and information entropy value of each layer of feature maps, and perform weighted superposition operation on the multi-level fusion feature maps to obtain a target fusion image.

[0006] The present invention also provides an image fusion device based on an ISP image signal processing vision sensor, including: An acquisition module, configured to perform spatial feature extraction and semantic feature extraction on the original multi-source images acquired by the ISP vision sensor to obtain spatial feature maps and semantic feature maps, and perform two-dimensional serial attention processing to obtain two-dimensional attention feature maps; A detection module, configured to perform edge detection on the two-dimensional attention feature maps to obtain edge constraint information, and perform multi-scale pyramid pooling processing to obtain a set of pyramid feature maps; A decomposition module, configured to input the set of pyramid feature maps into a gradient bilateral filter for feature decomposition, and set the spatial kernel standard deviation and range kernel standard deviation for filtering operations to obtain a three-layer fine-coarse structure feature map set and a base layer image; An enhancement module, configured to perform Laplacian operator enhancement on the three-layer fine-coarse structure feature map sets respectively to obtain enhanced feature maps, and perform weighted fusion through regional energy density and local contrast significance calculation to obtain multi-level fusion feature maps; A superposition module, configured to perform pixel value normalization processing on the multi-level fusion feature maps, calculate the sharpness gradient value, local contrast value and information entropy value of each layer of feature maps, and perform weighted superposition operation on the multi-level fusion feature maps to obtain a target fusion image.

[0007] The present invention also provides a computer device, including a memory and a processor, where a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method described in any one of the above are implemented.

[0008] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0009] In summary, the technical solution provided by the present invention BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a schematic diagram of the steps of an image fusion method based on an ISP image signal processing vision sensor in an embodiment of the present invention; Figure 2 It is a structural block diagram of an image fusion device based on an ISP image signal processing vision sensor in an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a computer device in an embodiment of the present invention.

[0010] The realization, functional characteristics and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0011] In order to make the object, technical solution and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0012] Referring to Figure 1 , this embodiment provides an image fusion method based on an ISP image signal processing vision sensor, including the following steps: S1, perform spatial feature extraction and semantic feature extraction on the original multi-source images collected by the ISP vision sensor to obtain a spatial feature map and a semantic feature map, and perform two-dimensional serial attention processing to obtain a two-dimensional attention feature map; Among them, a dual-branch feature extraction network is designed to simultaneously capture the low-level spatial features and high-level semantic features of images. In the spatial feature branch of the dual-branch feature extraction network, the input image is gradually convolved through a series of residual convolutional units. The original multi-source image is input into the first residual convolutional unit, whose convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 64. By balancing the computational complexity and the ability to retain details, the basic spatial features are captured while maintaining the resolution of the input image. At the same time, the residual structure effectively alleviates the problem of gradient disappearance in deep networks by introducing skip connections and enhances the ability of feature learning. The result of the convolution operation generates the first-layer spatial feature map, which mainly focuses on basic information such as edges and textures. The first-layer spatial feature map is input into the second residual convolutional unit. The second residual convolutional unit uses a convolutional kernel size of 3×3 and a stride of 1, but the number of output channels is increased to 128, thereby expanding the expressive ability of the features. This operation refines the spatial features of the image while retaining important structural information. The second-layer spatial feature map is input into the third residual convolutional unit, which still operates with a 3×3 convolutional kernel, the stride remains 1, and the number of output channels is increased to 256 to extract the final spatial feature map. The design of the spatial feature branch emphasizes the process of gradual refinement of features, and realizes the multi-level spatial feature description of the input image through the stacking of residual convolutional units in the structure. At the same time, the semantic feature branch extracts the high-level semantic information of the multi-source image through depthwise separable convolution operations with relatively low computational complexity. The input original multi-source image is processed by the first depthwise separable convolutional layer, whose convolutional kernel size is 3×3, the stride is set to 2, and the number of output channels is 32. Compared with ordinary convolution, depthwise separable convolution decomposes the standard convolution into depthwise convolution and pointwise convolution, thus significantly reducing the computational cost while retaining the sensitivity of convolution to the feature space. The first-layer semantic feature map generated after the first-layer convolution contains preliminary semantic information. The first-layer semantic feature map is input into the second depthwise separable convolutional layer for processing, whose convolutional kernel size and stride are also 3×3 and 2, and the number of output channels is increased to 64 to capture more complex semantic features. After the second-layer semantic feature map passes through the third depthwise separable convolutional layer, the number of output channels is further increased to 128. This layer of operation generates the third-layer semantic feature map to capture more intermediate semantic information. The third-layer semantic feature map is processed by the fourth depthwise separable convolutional layer, and the number of output channels is increased to 256 to generate the fourth-layer semantic feature map. This feature map is fed into the fifth depthwise separable convolutional layer, whose convolutional kernel size is 3×3, the stride is 2, and the number of output channels is further increased to 512, finally generating the semantic feature map. The semantic feature branch provides rich high-level semantic representations for downstream tasks by gradually expanding the number of channels and the feature expressive ability. Two-dimensional serial attention processing is performed on the spatial feature map and the semantic feature map. In the spatial dimension, a channel attention mechanism is adopted to dynamically adjust the channel importance of the spatial feature map by learning the weight distribution of specific channels.The adjustment is based on global average pooling and max pooling operations, which capture the global information and salient information of the feature map respectively, and calculate the weight coefficients through a shared fully connected network. Finally, the expression ability of important channels is enhanced by weighting. In the semantic dimension, a spatial attention mechanism is applied to generate a spatial weight map by modeling the context relationship of each pixel point in the semantic feature map. The spatial attention mechanism uses two-dimensional convolution operation combined with an activation function to highlight the regions rich in semantic information and suppress background noise. In the serial processing process, a channel attention mechanism is applied to the spatial feature map to generate an enhanced spatial feature map; subsequently, the enhanced spatial feature map is combined with the semantic feature map, and the spatial attention mechanism is applied to it to generate the final two-dimensional attention feature map.

[0013] The spatial feature map and the semantic feature map are concatenated in the channel dimension to obtain a fused feature map. The concatenation operation in the channel dimension can preserve the original spatial structure of each feature map, and at the same time integrate multi-source information in feature expression, so that the fused feature map contains more semantic and detailed features in the channels. Global average pooling and global max pooling operations are respectively performed on the fused feature map to generate a first one-dimensional feature vector and a second one-dimensional feature vector. The statistical information of the image is extracted from a global perspective, where global average pooling represents the overall distribution trend of the fused feature map, while global max pooling focuses on the intensity performance of significant features. These two one-dimensional feature vectors complement each other at the information level, making the subsequent feature weight learning more comprehensive and accurate. The above two one-dimensional feature vectors are input into a shared multi-layer perceptron. The multi-layer perceptron contains two fully connected layers. The number of neurons in the first fully connected layer is set to 1 / 16 of the number of input channels. The dimensionality reduction operation reduces the computational complexity by reducing the feature dimension, and at the same time can focus on the core information of the channel features. After passing through the first fully connected layer, the feature vector is input into the second fully connected layer, and the number of neurons in this layer is restored to the number of channels of the original input, so as to ensure the integrity of the feature space. This process performs consistency processing on the feature vectors generated by the two pooling methods through a parameter sharing mechanism, and outputs a feature vector containing channel weight information, that is, a channel attention vector. In order to enhance the expression ability of the channel attention, the sigmoid activation function is applied to the channel attention vector to limit its value between 0 and 1, generating channel weight coefficients. These weight coefficients reflect the importance of each channel in the fused feature map. By performing element-wise multiplication of the channel weight coefficients and the fused feature map, a channel-weighted feature map is generated. Max pooling and average pooling operations along the channel dimension are performed on the channel-weighted feature map to generate a first two-dimensional feature map and a second two-dimensional feature map respectively. These two pooling methods extract different statistical information in the space, where max pooling focuses on the significant feature regions, while average pooling reflects the overall feature distribution. After these two two-dimensional feature maps are concatenated along the channel dimension, they are input into a 7×7 convolutional layer for convolution operation. The use of a 7×7 convolutional kernel can effectively cover a large spatial area, thereby capturing the context relationship of the image. The output feature map after the convolution operation is processed by the sigmoid activation function to generate a spatial attention weight map, representing the importance of each spatial position. The generated spatial attention weight map is multiplied element-wise with the channel-weighted feature map to generate a two-dimensional attention feature map. Combining the advantages of channel attention and spatial attention, both enhancing important features in the channels and highlighting significant regions in the space, a two-dimensional attention feature map is obtained.

[0014] S2. Edge detection is performed on the two-dimensional attention feature map to obtain edge constraint information, and multi-scale pyramid pooling processing is performed to obtain a set of pyramid feature maps; Specifically, perform edge detection operations on the two-dimensional attention feature map in the horizontal and vertical directions through the Sobel operator. The Sobel operator calculates the gradient by applying a 3×3 convolution kernel in the horizontal and vertical directions of the image respectively, generating a horizontal edge gradient feature map and a vertical edge gradient feature map. The horizontal edge gradient feature map reflects the degree of brightness change of the image in the horizontal direction, while the vertical edge gradient feature map represents the degree of brightness change of the image in the vertical direction. Perform a pixel-level sum of squares operation on these two gradient feature maps, and use the formula to calculate the gradient intensity value of each pixel point, where and represent the horizontal and vertical gradient values respectively. The edge intensity feature map generated by taking the square root after the sum of squares can comprehensively represent the edge saliency of the image in space. Perform adaptive threshold segmentation processing on the edge intensity feature map to obtain edge constraint information. The adaptive threshold segmentation method dynamically adjusts the threshold according to the local statistical characteristics (such as local mean or median) of the edge intensity feature map, so as to adapt to the changes in edge characteristics in different regions of the image. Convert the edge intensity feature map into a binary edge constraint map, where each pixel point is marked as an edge or non-edge, forming a clear boundary representation. Perform a 1×1 pooling kernel operation on the two-dimensional attention feature map according to the edge constraint information, maintaining the original resolution of the input image, generating the first-level pyramid feature map, fully retaining the spatial information of the two-dimensional attention feature map, and highlighting the edge significant regions under the influence of edge constraints. Perform downsampling processing on the two-dimensional attention feature map by using a 2×2 pooling kernel, reducing the resolution to 1 / 2 of the original size, and generating the second-level pyramid feature map. Perform a pooling operation on the two-dimensional attention feature map by using a 4×4 pooling kernel, further reducing the resolution to 1 / 4 of the original size, and obtaining the third-level pyramid feature map. Perform a pooling operation on the two-dimensional attention feature map by using an 8×8 pooling kernel, reducing the resolution to 1 / 8 of the original size, and generating the fourth-level pyramid feature map. The downsampling process of each level of the pyramid feature map can compress the information of the feature map, and at the same time ensure that the edge characteristics are retained through edge constraint guidance. To construct a set of pyramid feature maps, perform bilinear interpolation operations on the second-level pyramid feature map, the third-level pyramid feature map, and the fourth-level pyramid feature map respectively, and restore them to the size consistent with the original resolution. The bilinear interpolation method can achieve resolution restoration with less distortion by performing weighted averaging on adjacent pixel values. The restored pyramid feature maps are concatenated with the first-level pyramid feature map in the channel dimension to form a set of pyramid feature maps containing multi-scale information.

[0015] S3. Input the set of pyramid feature maps into the gradient bilateral filter for feature decomposition, and set the spatial kernel standard deviation and the range kernel standard deviation for filtering operations to obtain a set of three-layer fine-coarse structure feature maps and the base layer image; It should be noted that for the set of pyramid feature maps, gradient operator operations are performed. By calculating the gradients of the image in the horizontal and vertical directions, a horizontal gradient map and a vertical gradient map are obtained respectively. By performing a sum-of-squares operation on the horizontal gradient map and the vertical gradient map pixel by pixel and then taking the square root of the result, a gradient magnitude map is obtained, which effectively represents the gradient intensity of each pixel in the image. Apply a gradient bilateral filter to the gradient magnitude map. The bilateral filter combines the characteristics of the spatial domain and the range domain. By setting the spatial kernel standard deviation to 2.0 and the range kernel standard deviation to 0.1, while weighting adjacent pixels in the spatial domain and performing selective smoothing according to the pixel intensity difference in the range domain, noise is removed while edges are preserved. The filtered feature map has clearer boundaries and more uniform regional characteristics. In order to extract high-frequency components and low-frequency components, a 5×5 Gaussian filter kernel is used to perform gradient filtering operations on the filtered feature map. Gaussian filtering can smooth the details in the image and generate a low-frequency component feature map; at the same time, by calculating the difference between the original filtered feature map and the low-frequency component feature map, the corresponding high-frequency component feature map is extracted, and the latter contains the edge and detail information of the image. Apply wavelet transform to the high-frequency component feature map for three-layer decomposition to generate the first-layer fine-coarse structure feature map. Wavelet transform decomposes the high-frequency information of the image into finer-grained fine-structure high-frequency features and relatively rough coarse-structure high-frequency features, which respectively describe the detail changes and overall high-frequency features in the image. Apply wavelet transform to the low-frequency component feature map for three-layer decomposition as well to generate the second-layer fine-coarse structure feature map. The decomposition results of the low-frequency components include fine-structure low-frequency features and coarse-structure low-frequency features, which are used to describe the feature expressions at different levels in the low-frequency region of the image. Weightedly combine the fine-structure high-frequency features and the fine-structure low-frequency features to obtain the third-layer fine-coarse structure feature map. Generate a set of three-layer fine-coarse structure feature maps according to the first-layer fine-coarse structure feature map, the second-layer fine-coarse structure feature map, and the third-layer fine-coarse structure feature map. By performing a subtraction operation on the set of pyramid feature maps, the part belonging to the fine-coarse structure features is removed, and the global low-frequency information and background features are retained, and finally the base layer image is generated. The base layer image and the set of three-layer fine-coarse structure feature maps together constitute the multi-level feature representation of the image, where the base layer image provides global information, and the set of fine-coarse structure feature maps highlights local details and edges.

[0016] S4. Perform Laplacian operator enhancement on the set of three-layer fine-coarse structure feature maps respectively to obtain enhanced feature maps, and perform weighted fusion through the calculation of regional energy density and local contrast saliency to obtain a multi-level fusion feature map; Specifically, the three-layer fine-coarse structure feature map sets are processed separately. A 5×5 Laplacian operator is used to perform convolution operations on each layer of feature maps to obtain enhanced feature maps. The Laplacian operator is a second-order differential operator that can enhance edge and detail features by strengthening the high-frequency components in the image, making the enhanced feature maps better able to highlight the significant change regions in the image. Through the convolution operation, each layer of fine-coarse structure feature maps is transformed into enhanced feature maps with more significant features. An 8×8 window sliding operation is applied to each layer of enhanced feature maps. By calculating the sum of the squared pixel values of each local region pixel by pixel, a regional energy density map is generated. Regional energy density is an important indicator for describing the intensity changes in local regions of an image and can effectively reflect the aggregation degree of high-frequency information and the distribution of feature intensities. At the same time, the significance calculation of local contrast is performed on the enhanced feature maps. By comparing the brightness values of local regions with those of their surrounding regions, a local contrast significance map is generated. The contrast significance map can highlight the relative changes in brightness between regions. The regional energy density map and the local contrast significance map complement each other at the feature level. The former emphasizes local intensity, and the latter focuses on brightness differences. The combination of the two can more comprehensively express the feature significance of the image. The regional energy density map and the local contrast significance map are used as modulation factors and input into the input module of the Pulse Coupled Neural Network (PCNN). The input module of the PCNN interacts these modulation factors with the input signals of each neuron to generate modulated input features. The modulation factors affect the firing threshold and sensitivity of neurons, enabling the input features to pay more attention to significant regions, so that the most meaningful features in the image are preferentially retained in the subsequent processing. The modulated input features are input into the link module of the PCNN, where the link strength is calculated according to the feature relationships between adjacent neurons. The PCNN uses an adaptive weight coefficient to dynamically adjust the calculation of the link strength to ensure that the connections between features can truly reflect their spatial and semantic relationships. Through the processing of the link module, a neuron link strength map is generated. This map represents the association strength between different regions in the feature map and is an important representation of the interaction between PCNN neurons. After the link strength map is generated, the pulse generation module of the PCNN makes a pulse generation judgment on the modulated input features and the link strength map. Based on preset firing conditions, it decides whether each neuron outputs a pulse signal to generate a neuron pulse output map. This pulse output map not only contains the enhanced feature information but also optimizes the feature expression through the coupling and association between neurons and is the key weight coefficient in the multi-level fusion process. Using the neuron pulse output map as a feature fusion coefficient, a weighted fusion operation is performed on the enhanced feature maps to generate fused feature maps for each layer. Different fine-coarse structure feature layers achieve information cooperation and optimization at different scales and frequencies through pulse coupling. Layer-by-layer feature combination is performed on the generated fused feature maps for each layer, and the final multi-level fused feature map is formed through layer-by-layer stacking and fusion.

[0017] S5 normalizes the pixel values of the multi-level fusion feature map, calculates the sharpness gradient value, local contrast value, and information entropy value of each layer of the feature map, and performs a weighted superposition operation on the multi-level fusion feature map to obtain the target fusion image.

[0018] Among them, normalizing the pixel values of the multi-level fusion feature map maps the pixel values in the feature map to the range of [0, 1], unifies the dynamic range of the data, eliminates the calculation deviation caused by the difference in the value ranges of different feature maps, and generates a normalized feature map. Image gradient operation is performed on the normalized feature map to calculate the sharpness gradient value. The gradient operation evaluates the edge strength and detail sharpness of the image by calculating the gray difference between adjacent pixels. The gradient calculation methods include using the Sobel operator or the Prewitt operator, and a gradient map is generated through gradient calculation, where each pixel value represents the intensity of the gray difference of its surrounding pixels. The average value of all pixel values in the gradient map is calculated to obtain the sharpness gradient value. The larger the sharpness gradient value, the richer the image details and the clearer the edges. Based on the gradient operation, the local contrast value of the normalized feature map is further calculated. The local contrast reflects the strength of the brightness change in the image. By calculating the standard deviation of the pixel values of the normalized feature map, the larger the standard deviation, the more significant the local contrast of the image. Information entropy calculation is performed on the gray histogram of the normalized feature map. Information entropy quantifies the complexity and information content of the image by statistically calculating the distribution probability of each gray value. After calculating the sharpness gradient value, local contrast value, and information entropy value, weight coefficients are assigned to these metrics to determine their contribution degrees in the fusion process. The setting of the weight coefficients is adjusted according to the specific application scenario. The weight assignment of the sharpness gradient value can preferentially highlight the image edge and detail areas, the weight of the local contrast value is used to enhance the brightness contrast between regions, and the weight of the information entropy value is used to optimize the overall information content. Through these weight coefficients, the first weight value, the second weight value, and the third weight value are generated respectively. These weight values are respectively multiplied by the normalized feature map through weighted multiplication operation to generate a weighted feature map. The purpose of the weighted operation is to emphasize the importance of sharpness, contrast, and information entropy in the fusion image to ensure that the final target fusion image achieves balanced optimization in these metrics. The weighted feature map realizes the adaptive adjustment of the multi-level feature map by integrating the three weight information. Contrast stretching and detail enhancement operations are performed on the weighted feature map to generate the target fusion image. Contrast stretching expands the gray range of the image through linear transformation or non-linear mapping, making the brightness difference more significant. Detail enhancement highlights the detail texture of the image by increasing the high-frequency components or adjusting the local brightness, and finally obtains the target fusion image.

[0019] In one example, spatial feature extraction and semantic feature extraction are performed on the original multi-source images collected by the ISP vision sensor to obtain a spatial feature map and a semantic feature map, and two-dimensional serial attention processing is performed to obtain a two-dimensional attention feature map, including: Input the original multi-source images collected by the ISP vision sensor into a two-branch feature extraction network, where the two-branch feature extraction network includes a spatial feature branch and a semantic feature branch; Perform convolution operations on the original multi-source images through the first residual convolution unit in the spatial feature branch. The convolution kernel size of the first residual convolution unit is 3×3, the stride is 1, and the number of output channels is 64 to obtain the first-layer spatial feature map; perform convolution operations on the first-layer spatial feature map through the second residual convolution unit in the spatial feature branch. The convolution kernel size of the second residual convolution unit is 3×3, the stride is 1, and the number of output channels is 128 to obtain the second-layer spatial feature map; perform convolution operations on the second-layer spatial feature map through the third residual convolution unit in the spatial feature branch. The convolution kernel size of the third residual convolution unit is 3×3, the stride is 1, and the number of output channels is 256 to obtain the spatial feature map; Perform convolution operations on the original multi-source images through the first depthwise separable convolution layer in the semantic feature branch. The convolution kernel size of the first depthwise separable convolution layer is 3×3, the stride is 2, and the number of output channels is 32 to obtain the first-layer semantic feature map; perform convolution operations on the first-layer semantic feature map through the second depthwise separable convolution layer in the semantic feature branch. The convolution kernel size of the second depthwise separable convolution layer is 3×3, the stride is 2, and the number of output channels is 64 to obtain the second-layer semantic feature map; perform convolution operations on the second-layer semantic feature map through the third depthwise separable convolution layer in the semantic feature branch. The convolution kernel size of the third depthwise separable convolution layer is 3×3, the stride is 2, and the number of output channels is 128 to obtain the third-layer semantic feature map; perform convolution operations on the third-layer semantic feature map through the fourth depthwise separable convolution layer in the semantic feature branch. The convolution kernel size of the fourth depthwise separable convolution layer is 3×3, the stride is 2, and the number of output channels is 256 to obtain the fourth-layer semantic feature map; perform convolution operations on the fourth-layer semantic feature map through the fifth depthwise separable convolution layer in the semantic feature branch. The convolution kernel size of the fifth depthwise separable convolution layer is 3×3, the stride is 2, and the number of output channels is 512 to obtain the semantic feature map; Perform two-dimensional serial attention processing on the spatial feature map and the semantic feature map to obtain a two-dimensional attention feature map.

[0020] In this example, input the original multi-source images collected by the ISP vision sensor , whose size is , where represents the height of the image, represents the width, Indicates the number of channels. The original image is input into a dual-branch feature extraction network, which is divided into a spatial feature branch and a semantic feature branch for extracting low-level spatial information and high-level semantic information of the image respectively. In the spatial feature branch, the image is subjected to convolutional operations through the first residual convolutional unit. The residual convolutional unit consists of a standard convolutional layer and a skip connection. The convolutional operation is expressed as: ; where, is the first-layer spatial feature map, is the size of the convolutional kernel (set to ), is the convolutional stride (set to 1), ReLU is the activation function, and Conv is the standard convolutional operation. Through the convolutional operation, the number of output channels is 64, that is . The residual structure maintains the input information through the skip connection, improving the efficiency of gradient propagation. The first-layer spatial feature map is input into the second residual convolutional unit, and its convolutional formula is similar to that of the first layer: ; where , , and the number of output channels increases to 128, obtaining the second-layer spatial feature map . The second-layer feature map continues through the third residual convolutional unit: ; , , and finally the number of output channels is 256, obtaining the spatial feature map . In the semantic feature branch, semantic information is extracted through depthwise separable convolutions. The first-layer depthwise separable convolution operates on the original image , and the formula is: ; where is the first-layer semantic feature map, , , and the number of output channels is 32, that is . The depthwise separable convolution reduces the computational complexity by decomposing the convolution into a depth convolution and a pointwise convolution. The semantic feature branch sequentially passes through the second, third, fourth, and fifth depthwise separable convolutional layers to extract deeper features. The formula for each layer is as follows: ; For each layer of convolution, the convolutional kernel size , and the stride , the number of output channels is 64, 128, 256, and 512 in sequence. Finally, a semantic feature map is obtained . Through a two-dimensional serial attention mechanism, the spatial feature map and the semantic feature map are fused. The two feature maps are concatenated in the channel dimension to obtain a fused feature map : ; where Concat represents the channel concatenation operation, . Channel attention processing is performed on the fused feature map. One-dimensional feature vectors are generated through global average pooling GAP and global max pooling GMP operations: ; The two types of feature vectors are input into a shared multi-layer perceptron (MLP), and channel weights are generated through a two-layer fully connected network: ; where and are the weight matrices of the fully connected layers, is the Sigmoid activation function, represents the channel weights. Spatial attention processing is performed on the channel-weighted feature map. The spatial relationship is extracted through the convolution of to generate a spatial weight map : ; The channel weights and the spatial weights are combined into the feature map to generate a two-dimensional attention feature map: ; Through the above operations, the generated two-dimensional attention feature map can simultaneously emphasize spatial information and semantic information for downstream image fusion tasks.

[0021] This embodiment further includes a multi-stage pipelined parallel processing step of ISP hardware acceleration: dividing the original multi-source image into multiple image blocks, each image block having a size of 256×256 pixels, and allocating an independent ISP processing unit to each image block to obtain a group of parallel processing units; establishing a three-level cache structure in each ISP processing unit, including an L1 cache with a capacity of 32KB, an L2 cache with a capacity of 256KB, and an L3 cache with a capacity of 2MB, for storing intermediate feature data of the image blocks to obtain a multi-level cache structure; performing data prefetch scheduling on the multi-level cache structure, predicting the data required for the next processing stage based on the spatial correlation of the image blocks, and preloading the data to the corresponding cache level to obtain a cache prefetch policy; performing vectorized parallel operations on the image blocks through the SIMD instruction set, processing 16 pixel data simultaneously in each clock cycle to obtain a vectorized processing result; constructing a systolic array processing architecture, including 16×16 processing units, each processing unit responsible for a specific convolution kernel operation, to implement pipelined processing of the convolution operation and obtain a parallel convolution result; designing a cross-block boundary feature fusion mechanism, using the feature information of the overlapping area to perform adaptive weight calculation to eliminate the stitching trace between blocks and obtain a seamless fusion result; adopting a bidirectional DMA transmission channel to achieve high-speed data exchange between the ISP processing unit and the external memory, with a data transmission bandwidth reaching 10GB / s to obtain a data transmission result; based on the data transmission result, reorganizing the seamless fusion result, combining the processing results of multiple image blocks into a complete image, and obtaining the original multi-source image after hardware acceleration processing.

[0022] In one example, two-dimensional serial attention processing is performed on the spatial feature map and the semantic feature map to obtain a two-dimensional attention feature map, including: Performing channel dimension splicing on the spatial feature map and the semantic feature map to obtain a fused feature map; Performing global average pooling operation on the fused feature map to obtain a first one-dimensional feature vector, and performing global max pooling operation on the fused feature map to obtain a second one-dimensional feature vector; Inputting the first one-dimensional feature vector and the second one-dimensional feature vector into a shared multi-layer perceptron, the shared multi-layer perceptron includes a first fully connected layer and a second fully connected layer, the number of neurons in the first fully connected layer is 1 / 16 of the number of input channels, and the number of neurons in the second fully connected layer is equal to the number of input channels, to obtain a channel attention vector; Performing activation operation on the channel attention vector through the sigmoid function to obtain a channel weight coefficient, and performing per-channel multiplication operation on the channel weight coefficient and the fused feature map to obtain a channel-weighted feature map; Performing max pooling operation on the channel-weighted feature map along the channel dimension to obtain a first two-dimensional feature map, and performing average pooling operation on the channel-weighted feature map along the channel dimension to obtain a second two-dimensional feature map; The first two-dimensional feature map and the second two-dimensional feature map are concatenated in the channel dimension and then subjected to a convolution operation through a 7×7 convolutional layer, and the spatial attention weight map is obtained by activation through the sigmoid function. Then, the spatial attention weight map and the channel-weighted feature map are subjected to an element-wise multiplication operation to obtain the two-dimensional attention feature map.

[0023] In this example, the input spatial feature map and the semantic feature map are concatenated in the channel dimension to form a fused feature map . Among them, represents the height of the feature map, represents the width, and are the number of channels of the spatial feature map and the semantic feature map respectively. The fused feature map is subjected to global average pooling (GAP) operation and global max pooling (GMP) operation. Global average pooling calculates the mean value of the pixel values in each channel of the fused feature map, and the formula is as follows: ; where, is the global average value of the th channel of the fused feature map, and represent the pixel positions. Global max pooling calculates the maximum value of the pixel values in each channel, and the formula is: ; where, is the global maximum value of the th channel. Global average pooling obtains the first one-dimensional feature vector , and global max pooling obtains the second one-dimensional feature vector . The two one-dimensional feature vectors and are integrated through a concatenation operation to generate a joint feature vector: ; The joint feature vector is input into a shared multi-layer perceptron. The multi-layer perceptron contains two fully connected networks. The number of neurons in the first layer is times the number of input channels, aiming to reduce the computational complexity and extract key features. The operation formula is: ; where, is the hidden layer feature, and are the weight matrix and bias term of the first fully connected layer respectively, is the activation function. The second fully connected network restores the channel dimension, and its formula is: ; Among them, is the channel attention vector, and are the weight matrix and bias term of the second fully connected layer respectively, is the Sigmoid activation function, which restricts the channel weight range between [0, 1]. Multiply the channel weight vector by the fused feature map channel by channel to generate the channel-weighted feature map : ; Among them, is the feature map after channel weighting. This operation improves the representation ability of the feature map by enhancing significant channels and suppressing non-significant channels. Perform max pooling and average pooling operations on the channel-weighted feature map along the channel dimension to generate the first two-dimensional feature map and the second two-dimensional feature map . The formula is as follows: ; Concatenate these two two-dimensional feature maps along the channel dimension: ; Then perform a convolution operation on the concatenated feature map through the convolution kernel of . The formula is: ; Among them, is the spatial attention weight map, Conv represents the convolution operation, is the convolution kernel size, is the Sigmoid activation function, which is used to normalize the spatial weight values. Multiply the spatial attention weight map by the channel-weighted feature map pixel by pixel to generate the final two-dimensional attention feature map: ; Through the above steps, the two-dimensional attention feature map integrates the channel and spatial saliency information at the same time, so that the important features in the image are fully enhanced.

[0024] In an example, perform edge detection on the two-dimensional attention feature map to obtain edge constraint information, and perform multi-scale pyramid pooling processing to obtain a set of pyramid feature maps, including: Perform edge detection operations on the two-dimensional attention feature map in the horizontal and vertical directions through the Sobel operator to obtain a horizontal edge gradient feature map and a vertical edge gradient feature map; Perform a square root operation after pixel-level sum of squares on the horizontal edge gradient feature map and the vertical edge gradient feature map to obtain an edge intensity feature map, and perform an adaptive threshold segmentation operation on the edge intensity feature map to obtain edge constraint information; Perform 1×1 pooling kernel operations on the two-dimensional attention feature map according to the edge constraint information, maintaining the original resolution, to obtain the first-level pyramid feature map; perform 2×2 pooling kernel operations on the two-dimensional attention feature map according to the edge constraint information, downsampling to 1 / 2 of the original size, to obtain the second-level pyramid feature map; perform 4×4 pooling kernel operations on the two-dimensional attention feature map according to the edge constraint information, downsampling to 1 / 4 of the original size, to obtain the third-level pyramid feature map; perform 8×8 pooling kernel operations on the two-dimensional attention feature map according to the edge constraint information, downsampling to 1 / 8 of the original size, to obtain the fourth-level pyramid feature map; Perform bilinear interpolation operations on the second-level pyramid feature map, the third-level pyramid feature map, and the fourth-level pyramid feature map to restore them to the original size, and perform channel dimension concatenation with the first-level pyramid feature map to obtain a set of pyramid feature maps.

[0025] In this example, perform edge detection operations on the two-dimensional attention feature map in the horizontal and vertical directions through the Sobel operator. The Sobel operator calculates the gradient of each pixel in two directions through convolution, and the formula is as follows: ; where, and are the horizontal edge gradient feature map and the vertical edge gradient feature map respectively, and are the Sobel horizontal and vertical convolution kernels, defined as: ; Through convolution operations, and represent the gradient components of the feature map in the horizontal and vertical directions respectively. After performing pixel-level sum of squares on the horizontal and vertical gradients and then taking the square root, calculate the edge intensity feature map : ; where, represents the gradient amplitude of each pixel, reflecting the edge intensity. In order to extract significant edge information, perform an adaptive threshold segmentation operation on to generate edge constraint information . Adaptive threshold segmentation dynamically adjusts the threshold in a local area , and converts into a binary image: ; Among them, is calculated through the mean or median of a local window. According to the edge constraint information Perform multi-scale pooling on the two-dimensional attention feature map to construct a set of pyramid feature maps. Through The pooling kernel operates on without changing the resolution to obtain the first-level pyramid feature map : ; Through The pooling kernel downsamples , and the resolution is reduced to 1 / 2 of the original size. The formula is: ; Among them, is the second-level pyramid feature map. Similarly, use and pooling kernels for downsampling to generate the third-level and the fourth-level pyramid feature maps. For the second-level, third-level, and fourth-level pyramid feature maps , restore them to the original resolution through bilinear interpolation. The interpolation formula is as follows: ; Among them, and represent the upper and lower neighborhood pixel positions of the interpolation point, is the interpolated feature map. Concatenate the interpolated feature map with the first-level pyramid feature map in the channel dimension to obtain the set of pyramid feature maps : ; Among them, is a set of multi-scale feature maps that fuse different scale information.

[0026] In an example, input the set of pyramid feature maps into a gradient bilateral filter for feature decomposition, and set the spatial kernel standard deviation and range kernel standard deviation for filtering operations to obtain a three-layer fine-coarse structure feature map set and a base layer image, including: Perform a gradient operator operation on the set of pyramid feature maps to obtain a horizontal gradient map and a vertical gradient map, and perform a square root operation on the sum of squares of the horizontal gradient map and the vertical gradient map to obtain a gradient magnitude map; Set the spatial kernel standard deviation of the gradient magnitude map to 2.0 and the value range kernel standard deviation to 0.1, perform a gradient bilateral filtering operation to obtain a filtered feature map, and use a 5×5 Gaussian filter kernel to perform a gradient filtering operation on the filtered feature map to obtain a high-frequency component feature map and a low-frequency component feature map; Perform three-layer decomposition on the high-frequency component feature map through wavelet transform to obtain a first-layer fine-coarse structure feature map, and the first-layer fine-coarse structure feature map includes fine-structure high-frequency features and coarse-structure high-frequency features; Perform three-layer decomposition on the low-frequency component feature map through wavelet transform to obtain a second-layer fine-coarse structure feature map, and the second-layer fine-coarse structure feature map includes fine-structure low-frequency features and coarse-structure low-frequency features; Perform weighted combination of the fine-structure high-frequency features and the fine-structure low-frequency features to obtain a third-layer fine-coarse structure feature map, and generate a set of three-layer fine-coarse structure feature maps based on the first-layer fine-coarse structure feature map, the second-layer fine-coarse structure feature map, and the three-layer fine-coarse structure feature map; Remove the feature information in the set of three-layer fine-coarse structure feature maps from the set of pyramid feature maps to obtain a base layer image.

[0027] In this example, perform a gradient operator operation on the set of pyramid feature maps Assume that each feature map , where and represent the height and width of the th feature map respectively, and represents the number of channels. Perform a gradient operator operation on each to extract the gradients in the horizontal and vertical directions. Use the Sobel operator to calculate the gradients, and its formula is: ; where and are the horizontal gradient map and the vertical gradient map respectively, and and are the Sobel convolution kernels, which are used to extract the edge information in the horizontal and vertical directions respectively. Calculate the gradient magnitude map through a pixel-level square root operation on the sum of squares: ; where is the magnitude map reflecting the gradient intensity, which can capture the intensity of edges and significant features in the image. To optimize the gradient magnitude map For its smoothness and edge retention effect, gradient bilateral filtering is applied to it. Bilateral filtering is a non-linear filter that takes into account both the spatial domain and range domain characteristics, and the filtering formula is: ; where is the Gaussian weight function in the spatial domain, is the Gaussian weight function in the range domain, and are the spatial kernel standard deviation (set to 2.0) and the range kernel standard deviation (set to 0.1) respectively. The feature map obtained after filtering can retain edge information and remove noise. Perform gradient filtering operation on the filtered feature map using a 5×5 Gaussian filter kernel to separate the high-frequency component and the low-frequency component . The formula is as follows: ; where is the Gaussian filter kernel, used to extract the smooth low-frequency component , while the high-frequency component retains the rapidly changing edge information. Perform three-layer decomposition on the high-frequency and low-frequency components through wavelet transform. For the high-frequency component , the wavelet transform formula is: ; where is the wavelet basis function, is the wavelet coefficient. The first-layer fine and coarse structure feature map obtained by decomposition includes fine-structure high-frequency features and coarse-structure high-frequency features. Similarly, perform wavelet decomposition on the low-frequency component to generate the second-layer fine and coarse structure feature map, which contains fine-structure low-frequency features and coarse-structure low-frequency features. Combine the fine-structure high-frequency features with the fine-structure low-frequency features through weighted combination to generate the third-layer fine and coarse structure feature map, and the formula is: ; where and are the weight coefficients, used to control the contribution ratio of high-frequency and low-frequency features. Combine the first-layer, second-layer, and third-layer fine and coarse structure feature maps to generate a set of three-layer fine and coarse structure feature maps . Remove the feature information in the set of three-layer fine and coarse structure feature maps from the pyramid feature map set to extract the base layer image . The formula for the base layer image is: ; where the base layer image It contains global smooth features for representing the background information of the image.

[0028] In one example, the Laplacian operator is respectively applied to enhance the three-layer fine-coarse structure feature map set to obtain the enhanced feature map, and weighted fusion is performed through the calculation of regional energy density and local contrast saliency to obtain the multi-level fusion feature map, including: A 5×5 Laplacian operator is used to perform convolution operation on the three-layer fine-coarse structure feature map set to obtain the enhanced feature map, and an 8×8 window is slid on the enhanced feature map to calculate the sum of the squares of the pixel values within the window to obtain the regional energy density map; The brightness difference between the local region and the surrounding region of the enhanced feature map is calculated to obtain the local contrast saliency map; The regional energy density map and the local contrast saliency map are used as modulation factors to input into the input module of the pulse-coupled neural network to modulate the input of each neuron to obtain the modulated input features; The modulated input features are input into the link module of the pulse-coupled neural network, and the adaptive weight coefficient is used to calculate the link strength between adjacent neurons to obtain the neuron link strength map; Based on the neuron link strength map and the modulated input features, pulse generation judgment is performed in the pulse generation module to obtain the neuron pulse output map; The neuron pulse output map is used as the feature fusion coefficient to perform weighted fusion operation on the enhanced feature map to obtain the fusion feature maps of each layer, and inter-layer feature combination is performed on the fusion feature maps of each layer to obtain the multi-level fusion feature map.

[0029] In this example, a 5×5 Laplacian operator is used for the three-layer fine-coarse structure feature map set to perform convolution operation. Assume that each layer of the feature map , where and represent the height and width of the feature map respectively. For , the Laplacian operator is used for convolution operation to enhance the edge and detail information. The convolution operation formula of the Laplacian operator is: ; where, is the enhanced feature map, is the Laplacian operator convolution kernel, defined as: ; Through the convolution operation, the enhanced feature map can highlight the detail change regions, such as edges and textures. For the enhanced feature map to perform Window sliding operation is performed to calculate the sum of the squares of pixel values within each window to generate a regional energy density map. . The formula is: ; Where, represents the energy density of the region with pixel as the top left corner. The regional energy density reflects the intensity distribution of local regions in the image and can highlight high-energy regions such as edges and significant targets. At the same time, the brightness difference between the local region and the surrounding regions of the enhanced feature map is calculated to generate a local contrast saliency map . The local contrast is calculated by the following formula: ; Where, represents the surrounding neighborhood, is the number of pixels in the neighborhood. The local contrast saliency map is used to capture the brightness change between a pixel and its neighborhood and can highlight the salient regions. The regional energy density map and the local contrast saliency map are used as modulation factors and input into the input module of the PCNN to modulate the input of each neuron. The input modulation formula of the PCNN is: ; Where, is the modulated input feature, and are the weights of the modulation factors used to balance the effects of regional energy and contrast. The modulated input feature is input into the link module of the PCNN, and the link strength is calculated based on the feature values of adjacent neurons. The formula for the link strength is: ; Where, is the Gaussian function, is the parameter controlling the sensitivity of adjacent neurons. The link strength map represents the connection strength between adjacent neurons. In the pulse generation module, based on the link strength map and the modulated input feature , it is judged whether to generate a pulse output. The pulse generation judgment formula is: ; Where, is the threshold for pulse excitation, It is a neuron pulse output graph, used to mark the feature excitation of significant regions. The neuron pulse output graph is used as a feature fusion coefficient to perform weighted fusion on the enhanced feature map to generate the fused feature maps of each layer : ; The fused feature maps of each layer reflect the significant feature distribution after PCNN optimization. Inter-layer feature combination is performed on the fused feature maps of each layer to generate multi-level fused feature maps : ; Among them, is the weight of each layer of feature map, controlling the contribution of features of different layers in the final fused map.

[0030] In an example, pixel value normalization is performed on the multi-level fused feature map, the clarity gradient value, local contrast value, and information entropy value of each layer of feature map are calculated, and weighted superposition operation is performed on the multi-level fused feature map to obtain the target fused image, including: Perform normalization operation on the pixel values in the multi-level fused feature map, map the pixel values to the interval [0,1] to obtain the normalized feature map; Perform image gradient operation on the normalized feature map, calculate the gray difference between adjacent pixels to obtain the gradient map, and calculate the average value of the gradient map to obtain the clarity gradient value; Calculate the standard deviation of the pixel values of the normalized feature map to obtain the local contrast value, and obtain the information entropy value by calculating the gray histogram of the normalized feature map; Perform weight coefficient assignment operation on the clarity gradient value to obtain the first weight value, perform weight coefficient assignment operation on the local contrast value to obtain the second weight value, and perform weight coefficient assignment operation on the information entropy value to obtain the third weight value; Perform weighted multiplication operation on the first weight value, the second weight value, and the third weight value with the normalized feature map respectively to obtain the weighted feature map, and perform contrast stretching operation and detail enhancement operation on the weighted feature map to obtain the target fused image.

[0031] In this example, perform normalization operation on the pixel values in the multi-level fused feature map , where and represent the height and width of the feature map respectively, is the gray value of the th pixel. To ensure the numerical stability of subsequent operations, the pixel values are normalized to the interval [0,1]. The normalization formula is as follows: ; Among them, is the normalized pixel value, and are respectively the minimum and maximum values in the feature map. After normalization, the generated normalized feature map eliminates the influence of the numerical range on the calculation result. For the normalized feature map perform image gradient operation to calculate the gray difference between pixels and generate a gradient map . The gradient calculation method uses the Sobel operator, and its gradients in the horizontal and vertical directions are respectively: ; Among them, and are Sobel convolution kernels used to calculate the gradient components in the horizontal and vertical directions. The gradient magnitude of the gradient map is calculated through the square sum and square root operations: ; Take the average value of all pixels of the gradient map to obtain the sharpness gradient value : ; The sharpness gradient value describes the edge strength and detail sharpness of the feature map. The larger the value, the richer the edge information of the image. To evaluate the local contrast, calculate the standard deviation of the pixel values for the normalized feature map , and its formula is: ; Among them, is the mean value of the normalized feature map, defined as: ; The standard deviation reflects the brightness distribution range of the image. The larger the value, the more significant the local contrast. By calculating the gray histogram of the normalized feature map, estimate its information entropy . The probability of each gray value in the gray histogram is , then the formula for the information entropy is: ; Among them, represents the occurrence probability of the gray value in the feature map. The information entropy is a measure of the amount of information in the image. The larger the value, the richer the complexity and detail information of the image. According to the sharpness gradient value , local contrast value and information entropy Perform weight coefficient assignment to generate the first weight value respectively , the second weight value and the third weight value . The formula for weight assignment adopts the normalization method: ; Multiply these weight values with the normalized feature map respectively to generate the weighted feature map : ; Perform contrast stretching and detail enhancement on the weighted feature map. The formula for contrast stretching is: ; where and are the new ranges after contrast stretching. Apply the detail enhancement operation to amplify the high-frequency components of the feature map, such as through lifting filtering processing or non-linear transformation. The finally generated target fusion image has both high definition and significant local contrast, and also retains rich global information content.

[0032] This embodiment further includes ISP adaptive quality assessment and feedback correction steps: dividing the target fused image into N×N evaluation blocks, where N ranges from 8 to 16, calculating the structural similarity index SSIM and peak signal-to-noise ratio PSNR for each evaluation block to obtain block-level quality assessment indicators; constructing a quality assessment neural network, including 3 encoder units and 1 decoder unit, the encoder units adopt depthwise separable convolution structures, the decoder unit adopts a transposed convolution structure, and inputting the block-level quality assessment indicators to obtain a quality feature map; detecting quality abnormal regions in the quality feature map, generating a quality defect mask based on regions with SSIM lower than 0.85 or PSNR lower than 30 dB to obtain an area identifier to be corrected; performing adaptive grouping on the area to be corrected, clustering similar regions based on the quality defect type, and using the K-means algorithm for feature clustering with K set to 4 to obtain defect type groups; designing compensation strategies for each type of defect region, adopting adaptive sharpening for regions with missing details, non-local means filtering for regions with noise pollution, and local histogram equalization for regions with color deviation to obtain regional compensation parameters; constructing an ISP parameter mapping table, mapping the regional compensation parameters to parameters such as gain control, gamma correction, and color mapping of the ISP image signal processor to obtain ISP correction parameters; performing adaptive enhancement and correction processing on the target fused image according to the ISP correction parameters to obtain a target enhanced image with optimized quality; inputting the target enhanced image with optimized quality into the quality assessment neural network for iterative evaluation, and stopping the iteration when the improvement amplitude of the continuous three evaluation results is less than 1% to obtain the final fused enhanced image.

[0033] Among them, the steps of constructing an ISP parameter mapping table to map regional compensation parameters to parameters such as gain control, gamma correction, and color mapping of the ISP image signal processor to obtain ISP correction parameters include: constructing a hierarchical ISP parameter mapping table, including a basic control layer, a scene adaptation layer, and a quality optimization layer, each layer setting an independent parameter update period, the update period of the basic control layer is 1 ms, the update period of the scene adaptation layer is 10 ms, and the update period of the quality optimization layer is 100 ms, to obtain a hierarchical parameter structure; designing a parameter conflict detection mechanism, performing compatibility analysis on parameter combinations at different levels through a Decision Tree, sorting and selecting parameter combinations with conflicts by priority, to obtain a parameter compatibility list; establishing a parameter constraint model based on hardware characteristics, calculating the effective value range of each parameter according to the working voltage, timing requirements, and temperature range of the ISP hardware, and generating parameter constraint conditions, to obtain hardware constraint rules; constructing an adaptive parameter learning network, adopting an LSTM recurrent neural network structure, inputting historical parameter adjustment sequences and corresponding image quality evaluation results, predicting the optimal parameter combination, to obtain a parameter prediction model; designing a parameter smooth transition strategy, calculating the parameter adjustment step size and transition time based on the parameter change rate and the degree of influence on image quality, to avoid image jitter caused by parameter mutations, to obtain a smooth transition plan; establishing a multi-scene parameter template library, presetting parameter combination templates for different lighting conditions, shooting scenes, and image features, and matching the optimal template according to the real-time scene, to obtain a scene-based parameter template; implementing a parameter self-calibration mechanism, dynamically adjusting the weight coefficient of the mapping relationship by online evaluating the parameter mapping effect, and performing real-time optimization and update on the parameter mapping table, to obtain an optimized parameter mapping table; establishing a fault tolerance mechanism for parameter mapping, when a parameter mapping at a certain level is abnormal, automatically switching to a backup parameter combination, and recording the abnormal parameters for subsequent analysis and optimization, to obtain ISP correction parameters.

[0034] Referring to Figure 2 , this embodiment provides an image fusion device based on an ISP image signal processing vision sensor, including: An acquisition module 1, configured to perform spatial feature extraction and semantic feature extraction on the original multi-source images collected by the ISP vision sensor, obtain a spatial feature map and a semantic feature map, and perform two-dimensional serial attention processing to obtain a two-dimensional attention feature map; A detection module 2, configured to perform edge detection on the two-dimensional attention feature map to obtain edge constraint information, and perform multi-scale pyramid pooling processing to obtain a set of pyramid feature maps; A decomposition module 3, configured to input the set of pyramid feature maps into a gradient bilateral filter for feature decomposition, and set the spatial kernel standard deviation and the range kernel standard deviation for filtering operations to obtain a set of three-layer fine-coarse structure feature maps and a base layer image; An enhancement module 4 is configured to perform Laplacian operator enhancement on the three-layer fine and coarse structure feature map sets respectively to obtain enhanced feature maps, and perform weighted fusion through regional energy density and local contrast saliency calculation to obtain a multi-level fusion feature map; A superimposition module 5 is configured to perform pixel value normalization processing on the multi-level fusion feature map, calculate the clarity gradient value, local contrast value, and information entropy value of each layer of the feature map, and perform a weighted superimposition operation on the multi-level fusion feature map to obtain a target fusion image.

[0035] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the description in the above method embodiment, and details are not described herein again.

[0036] Refer to Figure 3 , in an embodiment of the present invention, a computer device is further provided. The computer device may be a server, and its internal structure may be as Figure 3 shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface, and a database connected through a system bus. Among them, the processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0037] Those skilled in the art can understand that Figure 3 the structure shown in

[0038] is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0039] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium provided by the present invention and used in the embodiments can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0040] It should be noted that in this document, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, apparatus, article or method comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article or method. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, apparatus, article or method comprising that element.

[0041] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An image fusion method based on ISP image signal processing visual sensor, characterized in that: The following steps are involved: Perform spatial feature extraction and semantic feature extraction on the original multi-source images collected by the ISP visual sensor to obtain spatial feature maps and semantic feature maps, and perform two-dimensional serial attention processing to obtain two-dimensional attention feature maps; Performing edge detection on the two-dimensional attention feature map to obtain edge constraint information, and performing multi-scale pyramid pooling processing to obtain a pyramid feature map set; Inputting the pyramid feature map set into a gradient bilateral filter for feature decomposition, and setting a spatial kernel standard deviation and a range kernel standard deviation for filtering operation to obtain a three-layer fine-coarse structure feature map set and a base layer image; Performing Laplacian operator enhancement on the three-layer fine-coarse structure feature map set to obtain enhanced feature maps, and performing weighted fusion by regional energy density and local contrast saliency calculation to obtain a multi-level fused feature map; The pixel values ​​of the multi-level fusion feature map are normalized, the clarity gradient value, local contrast value and information entropy value of each layer of the feature map are calculated, and a weighted superposition operation is performed on the multi-level fusion feature map to obtain a target fusion image.

2. The image fusion method based on ISP image signal processing visual sensor according to claim 1 is characterized in that: The method performs spatial feature extraction and semantic feature extraction on the original multi-source image collected by the ISP visual sensor to obtain a spatial feature map and a semantic feature map, and performs two-dimensional serial attention processing to obtain a two-dimensional attention feature map, including: Inputting the original multi-source image collected by the ISP visual sensor into a dual-branch feature extraction network, wherein the dual-branch feature extraction network includes a spatial feature branch and a semantic feature branch; The original multi-source image is convolved by the first residual convolution unit in the spatial feature branch, the convolution kernel size of the first residual convolution unit is 3×3, the step size is 1, and the number of output channels is 64, so as to obtain a first-layer spatial feature map; the first-layer spatial feature map is convolved by the second residual convolution unit in the spatial feature branch, the convolution kernel size of the second residual convolution unit is 3×3, the step size is 1, and the number of output channels is 128, so as to obtain a second-layer spatial feature map; the second-layer spatial feature map is convolved by the third residual convolution unit in the spatial feature branch, the convolution kernel size of the third residual convolution unit is 3×3, the step size is 1, and the number of output channels is 256, so as to obtain a spatial feature map; A convolution operation is performed on the original multi-source image through a first depth-separable convolution layer in the semantic feature branch, wherein the convolution kernel size of the first depth-separable convolution layer is 3×3, the step size is 2, and the number of output channels is 32, so as to obtain a first-layer semantic feature map; a convolution operation is performed on the first-layer semantic feature map through a second depth-separable convolution layer in the semantic feature branch, wherein the convolution kernel size of the second depth-separable convolution layer is 3×3, the step size is 2, and the number of output channels is 64, so as to obtain a second-layer semantic feature map; a convolution operation is performed on the second-layer semantic feature map through a third depth-separable convolution layer in the semantic feature branch, wherein the third depth-separable convolution layer has a convolution kernel size of 3×3, a step size of 2, and the number of output channels is 64, so as to obtain a second-layer semantic feature map. The convolution kernel size of the fourth depth-separable convolution layer is 3×3, the step size is 2, and the number of output channels is 128, so as to obtain a third-layer semantic feature map; the third-layer semantic feature map is convolved by the fourth depth-separable convolution layer in the semantic feature branch, the convolution kernel size of the fourth depth-separable convolution layer is 3×3, the step size is 2, and the number of output channels is 256, so as to obtain a fourth-layer semantic feature map; the fourth-layer semantic feature map is convolved by the fifth depth-separable convolution layer in the semantic feature branch, the convolution kernel size of the fifth depth-separable convolution layer is 3×3, the step size is 2, and the number of output channels is 512, so as to obtain a semantic feature map; Perform two-dimensional serial attention processing on the spatial feature map and the semantic feature map to obtain a two-dimensional attention feature map.

3. The image fusion method based on ISP image signal processing visual sensor according to claim 2 is characterized in that: The performing two-dimensional serial attention processing on the spatial feature map and the semantic feature map to obtain a two-dimensional attention feature map includes: The spatial feature map and the semantic feature map are spliced ​​in channel dimension to obtain a fused feature map; Performing a global average pooling operation on the fused feature map to obtain a first one-dimensional feature vector, and performing a global maximum pooling operation on the fused feature map to obtain a second one-dimensional feature vector; Inputting the first one-dimensional feature vector and the second one-dimensional feature vector into a shared multilayer perceptron, wherein the shared multilayer perceptron includes a first fully connected layer and a second fully connected layer, wherein the number of neurons in the first fully connected layer is 1 / 16 of the number of input channels, and the number of neurons in the second fully connected layer is equal to the number of input channels, to obtain a channel attention vector; Performing an activation operation on the channel attention vector through a sigmoid function to obtain a channel weight coefficient, and performing a channel-by-channel multiplication operation on the channel weight coefficient and the fusion feature map to obtain a channel weighted feature map; Performing a maximum pooling operation on the channel weighted feature map along the channel dimension to obtain a first two-dimensional feature map, and performing an average pooling operation on the channel weighted feature map along the channel dimension to obtain a second two-dimensional feature map; The first two-dimensional feature map and the second two-dimensional feature map are spliced ​​in the channel dimension and then convolved through a 7×7 convolution layer, and a spatial attention weight map is obtained by activation through a sigmoid function. The spatial attention weight map is element-wise multiplied with the channel weighted feature map to obtain a two-dimensional attention feature map.

4. The image fusion method based on ISP image signal processing visual sensor according to claim 3 is characterized in that: The performing edge detection on the two-dimensional attention feature map to obtain edge constraint information, and performing multi-scale pyramid pooling processing to obtain a pyramid feature map set includes: Performing edge detection operations in the horizontal and vertical directions on the two-dimensional attention feature map by using a Sobel operator to obtain a horizontal edge gradient feature map and a vertical edge gradient feature map; Performing a pixel-level square sum operation on the horizontal edge gradient feature map and the vertical edge gradient feature map and then taking a square root to obtain an edge intensity feature map, and performing an adaptive threshold segmentation operation on the edge intensity feature map to obtain edge constraint information; According to the edge constraint information, a 1×1 pooling kernel operation is performed on the two-dimensional attention feature map to maintain the original resolution, so as to obtain a first-level pyramid feature map; according to the edge constraint information, a 2×2 pooling kernel operation is performed on the two-dimensional attention feature map to downsample it to 1 / 2 of the original size to obtain a second-level pyramid feature map; according to the edge constraint information, a 4×4 pooling kernel operation is performed on the two-dimensional attention feature map to downsample it to 1 / 4 of the original size to obtain a third-level pyramid feature map; according to the edge constraint information, an 8×8 pooling kernel operation is performed on the two-dimensional attention feature map to downsample it to 1 / 8 of the original size to obtain a fourth-level pyramid feature map; The second-level pyramid feature map, the third-level pyramid feature map, and the fourth-level pyramid feature map are restored to their original sizes by performing bilinear interpolation operation, and are spliced ​​with the first-level pyramid feature map in channel dimension to obtain a pyramid feature map set.

5. The image fusion method based on ISP image signal processing visual sensor according to claim 4 is characterized in that: The step of inputting the pyramid feature map set into a gradient bilateral filter for feature decomposition, and setting a spatial kernel standard deviation and a range kernel standard deviation for filtering operation to obtain a three-layer fine-coarse structure feature map set and a base layer image includes: Performing a gradient operator operation on the pyramid feature map set to obtain a horizontal gradient map and a vertical gradient map, and performing square and square root operations on the horizontal gradient map and the vertical gradient map to obtain a gradient amplitude map; The spatial kernel standard deviation of the gradient amplitude map is set to 2.0, the range kernel standard deviation is set to 0.1, and a gradient bilateral filtering operation is performed to obtain a filtered feature map, and a 5×5 Gaussian filter kernel is used to perform a gradient filtering operation on the filtered feature map to obtain a high-frequency component feature map and a low-frequency component feature map; Decomposing the high-frequency component feature map into three layers by wavelet transform to obtain a first-layer fine-coarse structure feature map, wherein the first-layer fine-coarse structure feature map includes fine-structure high-frequency features and coarse-structure high-frequency features; Decomposing the low-frequency component feature map into three layers by wavelet transform to obtain a second-layer fine-coarse structure feature map, wherein the second-layer fine-coarse structure feature map includes a fine structure low-frequency feature and a coarse structure low-frequency feature; Performing a weighted combination of the fine structure high frequency feature and the fine structure low frequency feature to obtain a third-layer fine-coarse structure feature map, and generating a three-layer fine-coarse structure feature map set according to the first-layer fine-coarse structure feature map, the second-layer fine-coarse structure feature map and the third-layer fine-coarse structure feature map; The feature information in the three-layer fine-coarse structure feature map set is removed from the pyramid feature map set to obtain a base layer image.

6. The image fusion method based on ISP image signal processing visual sensor according to claim 5 is characterized in that: The three-layer fine-coarse structure feature map set is enhanced by Laplacian operator respectively to obtain enhanced feature maps, and weighted fusion is performed by regional energy density and local contrast saliency calculation to obtain a multi-level fused feature map, including: A 5×5 Laplacian operator is used to perform a convolution operation on the three-layer fine-coarse structure feature map set to obtain an enhanced feature map, and an 8×8 window sliding is performed on the enhanced feature map to calculate the square sum of the pixel values ​​in the window to obtain a regional energy density map; Calculating the brightness difference between the local area and the surrounding area of ​​the enhanced feature map to obtain a local contrast saliency map; Inputting the regional energy density map and the local contrast saliency map as modulation factors into an input module of a pulse coupled neural network, modulating the input of each neuron, and obtaining a modulated input feature; Inputting the modulated input features into the link module of the pulse coupled neural network, calculating the link strength of adjacent neurons using an adaptive weight coefficient, and obtaining a neuron link strength map; Performing pulse generation judgment in a pulse generation module according to the neuron connection strength map and the modulated input characteristics to obtain a neuron pulse output map; The neuron pulse output map is used as a feature fusion coefficient, and a weighted fusion operation is performed on the enhanced feature map to obtain fused feature maps of each layer, and inter-layer feature combination is performed on the fused feature maps of each layer to obtain a multi-level fused feature map.

7. The image fusion method based on ISP image signal processing visual sensor according to claim 6 is characterized in that: The pixel value normalization processing is performed on the multi-level fusion feature map, the clarity gradient value, the local contrast value and the information entropy value of each layer of the feature map are calculated, and the weighted superposition operation is performed on the multi-level fusion feature map to obtain the target fusion image, including: Normalizing the pixel values ​​in the multi-level fusion feature map, mapping the pixel values ​​to the interval [0, 1], and obtaining a normalized feature map; Performing image gradient operation on the normalized feature map, calculating the grayscale difference between adjacent pixels to obtain a gradient map, and calculating an average value of the gradient map to obtain a clarity gradient value; Calculating the standard deviation of pixel values ​​for the normalized feature map to obtain a local contrast value, and obtaining an information entropy value by calculating a grayscale histogram of the normalized feature map; Performing a weight coefficient assignment operation on the clarity gradient value to obtain a first weight value, performing a weight coefficient assignment operation on the local contrast value to obtain a second weight value, and performing a weight coefficient assignment operation on the information entropy value to obtain a third weight value; The first weight value, the second weight value and the third weight value are respectively subjected to weighted multiplication operations with the normalized feature map to obtain a weighted feature map, and contrast stretching operations and detail enhancement operations are performed on the weighted feature map to obtain a target fused image.

8. An image fusion device based on ISP image signal processing visual sensor, characterized in that: For implementing the steps of the method according to any one of claims 1 to 7, the device comprises: The acquisition module is used to extract spatial features and semantic features from the original multi-source images collected by the ISP visual sensor to obtain spatial feature maps and semantic feature maps, and perform two-dimensional serial attention processing to obtain two-dimensional attention feature maps; A detection module, used to perform edge detection on the two-dimensional attention feature map to obtain edge constraint information, and perform multi-scale pyramid pooling processing to obtain a pyramid feature map set; A decomposition module, used for inputting the pyramid feature map set into a gradient bilateral filter for feature decomposition, and setting a spatial kernel standard deviation and a range kernel standard deviation for filtering operation to obtain a three-layer fine-coarse structure feature map set and a base layer image; An enhancement module is used to perform Laplacian operator enhancement on the three-layer fine-coarse structure feature map set to obtain enhanced feature maps, and perform weighted fusion through regional energy density and local contrast saliency calculation to obtain a multi-level fused feature map; The superposition module is used to perform pixel value normalization processing on the multi-level fusion feature map, calculate the clarity gradient value, local contrast value and information entropy value of each layer of the feature map, and perform weighted superposition operation on the multi-level fusion feature map to obtain a target fusion image.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images