A heterogeneous image multi-scale fusion method and system based on master anchor point initialization
Patent Information
- Application Number
- CN202611464095.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-20
- Publication Date
- 2026-10-23
AI Technical Summary
虽然常规多尺度结构能够融合不同层级特征,但若缺少高分辨率融合特征层,后续图像数据处理模块对局部结构的响应仍然可能不足
本发明将彩色图像特征作为主锚点,在融合模块初始化阶段保持彩色图像模型的原有特征表达,使异构图像数据流融合模型能够从稳定的彩色图像表达能力出发进行训练,降低深度图像特征直接参与融合造成模型退化的风险;
Smart Images

Figure CN122966455A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of digital image processing and heterogeneous image fusion enhancement, and in particular to a method and system for multi-scale fusion of heterogeneous images based on master anchor point initialization. Background Technology
[0002] With the development of deep learning technology, neural network-based image processing models have been widely used in tasks such as image enhancement, image reconstruction, and multi-source image fusion. Existing models typically extract multi-scale image features through a backbone network and perform subsequent image processing based on the extracted features. For local details, edge structures, and regions with complex textures, relying solely on color images is easily affected by texture aliasing, appearance overlap, lighting changes, and background interference, making it difficult to stably represent edge, spatial hierarchy, and local structural information in the image.
[0003] Depth images provide information related to spatial structure, layering, and local geometric relationships, thus offering valuable assistance in subsequent image data processing tasks. However, directly inputting depth images as additional channels into a model, or simply adding and concatenating color and depth features, may compromise the representational capabilities of already well-trained color image models. On one hand, depth images may suffer from estimation errors, local missing features, boundary discontinuities, or inconsistencies with the color image space. On the other hand, existing color image models typically possess strong basic expressive capabilities; allowing depth features to participate in fusion in an unconstrained manner during the initial training phase could lead to model degradation or training instability.
[0004] Furthermore, existing multi-scale feature construction processes typically prioritize semantic levels with lower spatial resolution. For small-scale objects, locally visible regions, and densely occluded areas, lower-resolution features may be insufficient to preserve fine-grained spatial information. Although conventional multi-scale structures can fuse features from different levels, without a high-resolution fusion feature layer, subsequent image data processing modules may still have insufficient response to local structures.
[0005] Therefore, there is an urgent need for a method that can safely introduce deep auxiliary structural information and enhance high-resolution feature representation while preserving the stable representation of color images, so as to achieve multi-scale fusion of heterogeneous images and generation of fused images. Summary of the Invention
[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a heterogeneous image multi-scale fusion method and system based on master anchor point initialization, in order to solve the above-mentioned technical problems.
[0007] According to a first aspect of the present invention, a heterogeneous image multi-scale fusion method based on master anchor point initialization is proposed, comprising: S10: Obtain a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; S20: Input the first image data stream into the color main branch network to extract color image feature maps at multiple spatial resolution scales; and input the second image data stream into the depth auxiliary branch network to extract depth image feature maps corresponding to multiple spatial resolution scales. S30: At at least two scales, color image feature maps and depth image feature maps of the same scale are concatenated along the channel dimension and input into a fusion convolutional layer to map and output a fusion image feature map; wherein, during the parameter initialization phase of the fusion convolutional layer, the first mapping sub-matrix corresponding to the color image feature map is initialized to an identity mapping matrix, the second mapping sub-matrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fusion image feature map output in the initial training phase is equal to the color image feature map; the first mapping sub-matrix is the weight sub-matrix used to map the color image feature map in the weight parameters of the fusion convolutional layer, and the second mapping sub-matrix is the weight sub-matrix used to map the depth image feature map in the weight parameters of the fusion convolutional layer; S40: Based on the fused image feature maps at various scales, perform cross-scale feature reconstruction to generate a multi-scale fused feature map set containing a high-resolution fused feature layer and at least one low-resolution semantic fused feature layer; wherein, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct a high-resolution fused feature layer; and S50: Generate a fused image based on a high-resolution fusion feature layer, including: performing channel mapping on the high-resolution fusion feature layer so that the number of mapped feature channels corresponds to the number of image channels in the first image data stream; upsampling the channel-mapped features so that their spatial resolution is restored to the image spatial resolution corresponding to the first image data stream, and obtaining and outputting the fused image.
[0008] In some specific embodiments, before S20, the method further includes: performing at least one spatial alignment process among scaling, cropping, flipping, padding, and size alignment on the first image data stream and the second image data stream; and performing at least one numerical preprocessing operation among normalization, truncation, missing value filling, and valid region marking on the second image data stream.
[0009] In some specific embodiments, the first image data stream is a three-channel image and the second image data stream is a single-channel image; before inputting the second image data stream into the depth auxiliary branch network, the single-channel image is copied along the channel dimension to a three-channel image to match the input channel structure of the color main branch network.
[0010] In some specific embodiments, the parameter initialization step of the deep auxiliary branch network includes: obtaining the parameters of the deep auxiliary branch network by copying or mapping the parameters of the trained color main branch network.
[0011] In some specific embodiments, the color main branch network and the deep auxiliary branch network output image feature maps at four scale levels, respectively; the stride sizes of the four scale levels relative to the input image are 4, 8, 16 and 32, respectively; wherein, a high-resolution fusion feature layer is constructed based on the scale level with a stride size of 4.
[0012] In some specific embodiments, S40 specifically includes: S41: Perform lateral mapping on the fusion image feature map with the highest spatial resolution in the fusion image feature maps at each scale to obtain high-resolution lateral features as lateral feature maps; S42: Upsample adjacent low-resolution semantic features to obtain upsampled semantic features corresponding to the high-resolution lateral feature space size; S43: The high-resolution lateral features and upsampled semantic features are concatenated and fused to obtain a high-resolution fused feature layer.
[0013] In some specific embodiments, the multi-scale fused feature map set is also provided to the subsequent image data processing module, which includes a detection head, which includes a first detection sub-head corresponding to the high-resolution fused feature layer and a second detection sub-head corresponding to the low-resolution semantic fused feature layer.
[0014] In some specific embodiments, the following operations are performed during the parameter initialization phase of the detection head: extracting the task head parameters from the trained color image processing model; copying or mapping the parameters corresponding to the highest spatial resolution detection layer in the original three-layer detection head to the first detection sub-head.
[0015] According to a second aspect of the invention, a computer-readable storage medium is provided on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.
[0016] According to a third aspect of the present invention, a heterogeneous image multi-scale fusion system based on master anchor point initialization is proposed, comprising: The data input module is used to acquire a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; The dual-branch feature extraction module includes a color main branch and a depth auxiliary branch. The color main branch is used to input the first image data stream into the color main branch network and extract color image feature maps at multiple spatial resolution scales. The depth auxiliary branch is used to input the second image data stream into the depth auxiliary branch network and extract depth image feature maps corresponding to multiple spatial resolution scales. The color master anchor point fusion module is used to concatenate color image feature maps and depth image feature maps at at least two scales along the channel dimension and input them into a fusion convolutional layer to map and output a fused image feature map. Specifically, during the parameter initialization phase of the fusion convolutional layer, the first mapping sub-matrix corresponding to the color image feature map is initialized to an identity mapping matrix, the second mapping sub-matrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fused image feature map output in the initial training phase is equal to the color image feature map. The first mapping sub-matrix is the weight sub-matrix used to map the color image feature map in the weight parameters of the fusion convolutional layer, and the second mapping sub-matrix is the weight sub-matrix used to map the depth image feature map in the weight parameters of the fusion convolutional layer. A high-resolution multi-scale fusion feature construction module is used to perform cross-scale feature reconstruction based on fused image feature maps at various scales, generating a multi-scale fusion feature map set containing a high-resolution fusion feature layer and at least one low-resolution semantic fusion feature layer; wherein, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct a high-resolution fusion feature layer; and The fused image generation module is used to perform channel mapping on the high-resolution fused feature layer so that the number of mapped feature channels corresponds to the number of image channels in the first image data stream, and to upsample the channel-mapped features to restore their spatial resolution to the image spatial resolution corresponding to the first image data stream, thereby obtaining and outputting the fused image.
[0017] Compared with the prior art, the present invention has at least the following technical effects: This invention uses color image features as the main anchor point and maintains the original feature expression of the color image model during the initialization stage of the fusion module. This enables the heterogeneous image data stream fusion model to be trained based on stable color image expression capabilities, reducing the risk of model degradation caused by deep image features directly participating in the fusion. This invention introduces depth image features as auxiliary compensation information into multi-scale fusion features, enabling the model to utilize the spatial structure, front and back layers, and local geometric relationships in the depth image to form supplementary representations of occluded areas, overlapping areas, and boundary confusion areas. This invention constructs a high-resolution fusion feature layer and generates a fused image based on the high-resolution fusion feature layer, so that the obtained fused image can simultaneously maintain the local texture and edge details in the color image and the spatial hierarchy and geometric structure information in the depth image, thereby improving the detail preservation ability and structural expression ability of the fused image. This invention supports parameter transfer and task head mapping from existing color image processing models, which can reduce training difficulty, improve the controllability of model initialization, and facilitate deployment in different multi-scale image processing frameworks. This invention does not limit the specific network structure for subsequent tasks and can be applied to RGB-D image data processing models based on convolutional networks, feature pyramid networks, detector heads, or other multi-scale structures, exhibiting good scalability. Attached Figure Description
[0018] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a heterogeneous image multi-scale fusion method based on master anchor point initialization according to an embodiment of this application; Figure 2 This is a flowchart of a heterogeneous image multi-scale fusion method based on master anchor point initialization, which is a specific embodiment of this application; Figure 3 This is a framework diagram of a heterogeneous image multi-scale fusion system based on master anchor point initialization, according to an embodiment of this application. Figure 4 This is a structural diagram of a color main anchor point fusion unit according to a specific embodiment of this application; Figure 5 This is a schematic diagram of a high-resolution multi-scale fusion feature construction module according to a specific embodiment of this application; Figure 6 This is a schematic diagram of parameter migration initialization based on a specific embodiment of the color main branch model of this application; Figure 7 This is a schematic diagram of the structure of a computer system used to implement the electronic devices of the present application embodiments; Figure 8This is a schematic diagram of the heterogeneous image multi-scale fusion process and activation visualization results according to a specific embodiment of this application; wherein, (a) is an RGB input image, (b) is an estimated depth input image, (c) is the activation visualization result of the trained RGB branch, (d) is the activation visualization result of the trained depth branch, (e) is the activation visualization result of the fusion module at the initial checkpoint, and (f) is the activation visualization result of the fusion module at the trained checkpoint. Detailed Implementation
[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0021] Figure 1 A flowchart illustrating a heterogeneous image multi-scale fusion method based on master anchor point initialization according to an embodiment of this application is shown. Figure 1 As shown, the method includes the following steps: S10: Acquire a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream. Specifically, the second image data stream includes any one of the following: a true depth map acquired by a depth sensor, an estimated depth map obtained by a monocular depth estimation model, a depth map reconstructed by binocular or multi-view cameras, and a depth response map.
[0022] In a specific embodiment, before executing the next step, at least one spatial alignment process, including scaling, cropping, flipping, padding, and size alignment, is performed on the first and second image data streams; at least one numerical preprocessing operation, including normalization, truncation, missing value imputation, and valid region labeling, is performed on the second image data stream. Through the above spatial alignment processes, the pixel-level spatial correspondence between the color image and the depth image before input to the network is improved, reducing geometric misalignment errors between modalities; simultaneously, normalization and missing value imputation of the depth data reduce the negative impact of abnormal depth values on the stability of convolution calculation.
[0023] S20: The first image data stream is input into the color main branch network to extract color image feature maps at multiple spatial resolution scales; and the second image data stream is input into the depth auxiliary branch network to extract depth image feature maps corresponding to multiple spatial resolution scales. Specifically, the color main branch network and the depth auxiliary branch network have corresponding feature output levels, and the spatial dimensions and / or channel dimensions of the color image feature maps and depth image feature maps at corresponding scale levels are matched after adaptation processing.
[0024] In a specific embodiment, the first image data stream is a three-channel image, and the second image data stream is a single-channel image. Before inputting the second image data stream into the depth auxiliary branch network, the single-channel image is copied along the channel dimension to become a three-channel image to match the input channel structure of the color main branch network. Without needing to change the existing first-layer convolutional channel structure of the color main branch network, this simple channel copying allows the depth auxiliary network to seamlessly receive single-channel depth map input, ensuring the consistency of the parameter structure.
[0025] In a specific embodiment, the parameter initialization step of the deep auxiliary branch network includes: obtaining the parameters of the deep auxiliary branch network from the parameters of the trained color main branch network through parameter copying or parameter mapping. Alternatively, the parameters of the deep auxiliary branch network are initialized from the parameters of a pre-trained visual model, so that the deep auxiliary branch network has stable feature extraction capabilities in the initial training stage. This setting enables the deep auxiliary branch network to have strong basic image feature extraction capabilities from the beginning of training, significantly reducing the convergence difficulty of the dual-branch heterogeneous network and saving computational costs and training time.
[0026] In a specific embodiment, the color main branch network and the deep auxiliary branch network output image feature maps at four scale levels, respectively; the strides of the four scale levels relative to the input image are 4, 8, 16, and 32, respectively; among them, a high-resolution fusion feature layer is constructed based on the scale level with a stride of 4. A reasonable hierarchical feature pyramid space is constructed, and the high-resolution layer is built using a shallow network with a stride of 4, which helps to control the corresponding computational resource consumption while retaining high spatial resolution.
[0027] S30: At at least two scales, color image feature maps and depth image feature maps of the same scale are concatenated along the channel dimension and input into a fusion convolutional layer to map and output a fusion image feature map; wherein, during the parameter initialization phase of the fusion convolutional layer, the first mapping sub-matrix corresponding to the color image feature map is initialized to an identity mapping matrix or an approximate identity mapping matrix, the second mapping sub-matrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fusion image feature map output in the initial training phase is equal to or approximately equal to the color image feature map.
[0028] As an alternative implementation, the color master anchor point fusion unit can also generate fused image features in the form of residuals: the fused image features are obtained by superimposing color image feature maps and depth auxiliary compensation features; wherein, the depth auxiliary compensation features are generated by depth image feature maps, or by a combination of color image feature maps and depth image feature maps, and the depth auxiliary compensation features have zero contribution, near-zero contribution, or limited contribution during the model initialization stage.
[0029] S40: Based on the fused image feature maps at various scales, perform cross-scale feature reconstruction to generate a multi-scale fused feature map set containing a high-resolution fused feature layer and at least one low-resolution semantic fused feature layer; wherein, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct a high-resolution fused feature layer.
[0030] In a specific embodiment, S40 specifically includes: S41: Perform lateral mapping on the high-resolution fused image feature map to obtain high-resolution lateral features as the lateral feature map; S42: Upsample adjacent low-resolution semantic features to obtain upsampled semantic features corresponding to the high-resolution lateral feature space size; S43: High-resolution lateral features and upsampled semantic features are concatenated and fused to obtain a high-resolution fused feature layer. The precise "spatial localization details (high-resolution lateral features)" of the shallow layer of the backbone network are combined with the rich "global contextual semantics (upsampled semantic features)" of the deep layer at the pixel dimension to construct a high-resolution feature that has both strong semantics and precise localization.
[0031] S50: Generate a fused image based on the high-resolution fusion feature layer, including: performing channel mapping on the high-resolution fusion feature layer so that the number of mapped feature channels corresponds to the number of image channels in the first image data stream; upsampling the channel-mapped features so that their spatial resolution is restored to the image spatial resolution corresponding to the first image data stream, and obtaining and outputting the fused image.
[0032] In a specific embodiment, a fused image is generated based on a high-resolution fusion feature layer. Specifically, channel mapping is performed on the high-resolution fusion feature layer, mapping the number of feature channels of the high-resolution fusion feature layer to the number of image channels corresponding to the first image data stream; then, the channel-mapped features are upsampled to restore their spatial resolution to the spatial resolution of the input image corresponding to the first image data stream, thereby obtaining the fused image. Since the high-resolution fusion feature layer is obtained by fusing color image features and depth image features at the master anchor point and combining them with low-resolution semantic features for cross-scale reconstruction, the generated fused image can contain spatial structure information introduced by the depth image while maintaining the information of the color image. Channel mapping can be implemented through convolution operations, and upsampling can be implemented using any of the following: interpolation upsampling, transposed convolution, or pixel rearrangement.
[0033] In a specific embodiment, subsequent image data processing is executed through a module containing a detection head. The detection head includes a first detection sub-head corresponding to the high-resolution fusion feature layer and a second detection sub-head corresponding to the low-resolution semantic fusion feature layer. Based on the multi-scale fusion feature map set, the detection head outputs target category information and target bounding box location information. This forces the first detection sub-head to focus on small-scale and edge detail extraction, while the second detection sub-head focuses on determining macroscopic subjects and large targets, avoiding feature dilution between targets of different scales within the same receptive field.
[0034] In a specific embodiment, the following operations are performed during the parameter initialization phase of the detection head: extracting the task head parameters from the trained color image processing model; copying or mapping the parameters corresponding to the highest spatial resolution detection layer in the original three-layer detection head to the first detection sub-head. This avoids the instability caused by randomly training the newly added high-resolution first detection sub-head from scratch, and further shortens the training cycle by inheriting the mature parameters of the highest spatial resolution task head of the original model.
[0035] Figure 2 This is a flowchart illustrating the multi-scale feature fusion and optional subsequent processing of a heterogeneous image multi-scale fusion method based on master anchor point initialization, as a specific embodiment of this application. Figure 2 As shown in the figure, this diagram highlights the generation of multi-scale fusion features and the processing path they provide to subsequent image data processing modules; the generation of the fused image can be further performed according to the aforementioned step S50, and the specific flow of this method is as follows: In step S1, the system first acquires the paired data, namely, the color image data stream and the depth image data stream.
[0036] Proceeding to step S2, spatial alignment and numerical preprocessing operations are performed on the acquired heterogeneous image data stream. This step aims to ensure that the color image and depth image are consistent in size and to perform numerical normalization and other processing, thereby ensuring spatial and numerical consistency of the subsequent network input.
[0037] After data preprocessing is completed, the process proceeds to step S3, where dual-branch feature extraction is performed. Specifically, the system uses the main color branch to extract multi-scale color image features Ri, while simultaneously using the depth auxiliary branch to extract the corresponding depth image features Di.
[0038] In step S4, color master anchor point fusion is performed. This step fuses color image features and depth image features at the same scale level at at least two scale levels to obtain fused image features Fi. During the initialization phase of this fusion module, the fusion mapping parameters are initialized (e.g., color mapping is unit mapping, depth mapping is zero mapping) so that the fusion output in the initial state retains the color feature response.
[0039] Step S5 is executed to construct high-resolution fusion features. Based on the fusion image features obtained in the previous steps, the system constructs and generates multi-scale fusion features such as P2, P3, P4, and P5, which include high-resolution fusion feature layers for preserving local spatial details.
[0040] In step S6, the generated multi-scale fusion features are output for use by subsequent image data processing modules. Furthermore, as shown in the dashed box in the figure, this process also supports docking with optional subsequent modules, such as a detection head module, to output the target's category and bounding box location information based on the multi-scale fusion features. It should be noted that... Figure 2 The subsequent image data processing path shown is an optional application path. The multi-scale fusion features can also generate a fused image by performing channel mapping and upsampling processing on the high-resolution fusion feature layer in the aforementioned step S50.
[0041] Figure 3 A framework diagram of a heterogeneous image multi-scale fusion system based on master anchor point initialization according to an embodiment of this application is shown, as follows: Figure 3As shown, the system includes a data input module 301, a dual-branch feature extraction module 302, a color master anchor point fusion module 303, a high-resolution multi-scale fusion feature construction module 304, and a fusion feature output module 305. In this embodiment, after the multi-scale fusion features are generated by the high-resolution multi-scale fusion feature construction module 304, they can be provided as intermediate features to the subsequent image data processing module. Furthermore, according to the aforementioned fusion image generation method, channel mapping and upsampling processing can be performed on the high-resolution fusion feature layer to generate a fusion image. The data input module 301 is used to acquire spatially aligned first and second image data streams, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; the dual-branch feature extraction module 302 includes a color main branch and a depth auxiliary branch. The color main branch is used to input the first image data stream into the color main branch network to extract color image feature maps at multiple spatial resolution scales, and the depth auxiliary branch is used to input the second image data stream into the depth auxiliary branch network to extract depth image feature maps corresponding to multiple spatial resolution scales; the color main anchor point fusion module 303 is used to concatenate color image feature maps and depth image feature maps at the same scale along the channel dimension at at least two scales, and input them into a fusion convolutional layer to map and output a fused image feature map; wherein, during the parameter initialization phase of the fusion convolutional layer, the first image feature map corresponding to the color image feature map is... The mapping submatrix is initialized to an identity mapping matrix or an approximate identity mapping matrix. The second mapping submatrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fused image feature map output in the initial training stage is equal to or approximately equal to the color image feature map. The high-resolution multi-scale fusion feature construction module 304 is used to perform cross-scale feature reconstruction based on the fused image feature maps at each scale, generating a multi-scale fusion feature map set containing a high-resolution fusion feature layer and at least one low-resolution semantic fusion feature layer. Adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct a high-resolution fusion feature layer. The fusion feature output module 305 is used to receive the multi-scale fusion features and output the multi-scale fusion features and / or the fused image generated based on the high-resolution fusion feature layer according to the specific application. This system can be used to execute the specific method steps in the aforementioned method embodiments.
[0042] In the data input module 301, the acquired first image data stream (color image) and second image data stream (depth image) undergo spatial alignment and numerical preprocessing to output high-quality color and depth streams, respectively. Subsequently, these two data streams enter the dual-branch feature extraction module 302 in parallel. The main color branch is responsible for processing the color stream and outputting multi-scale image features R2, R3, R4, and R5, while the auxiliary depth branch is responsible for processing the depth stream and outputting corresponding multi-scale depth features D2, D3, D4, and D5. Furthermore, the figure clearly shows that the parameters of the main color branch can be passed to the auxiliary depth branch through parameter copying or mapping, thus providing an initial structural basis for depth feature extraction.
[0043] During the color master anchor point fusion stage, the same-scale color features Ri and depth features Di extracted from the dual branches are simultaneously fed into the color master anchor point fusion module 303. This module has a specially configured initialization submodule, which issues an W... R =I (unit mapping), W D The initialization constraints of zero (zero mapping) and bias equal to 0 impose deterministic prior constraints on the fusion network, thereby smoothly outputting the same-scale fused image features Fi. Finally, the fused features Fi at each scale are passed to the high-resolution multi-scale fusion feature construction module 304, where they are constructed into a multi-scale fusion feature set including P2, P3, P4, and P5. These features are then uniformly output by the fusion feature output module 305, directly available for downstream optional follow-up modules (such as the detection head module marked with dashed lines in the figure, used to output target category and location information) to perform specific computer vision tasks.
[0044] Figure 4 This is a structural diagram of the color master anchor point fusion unit of a specific embodiment of this application, as shown below. Figure 4 As shown, this fusion unit synchronously receives color image features Ri and depth image features Di from the same scale level. Logically, color image feature Ri serves as the main anchor point for feature representation, while depth image feature Di is positioned as auxiliary compensation information. Upon entering the fusion process, color image feature Ri and depth image feature Di are first concatenated along the channel dimension to form a joint concatenated feature [Ri, Di]. Further, for each scale level, the color image features and depth image features are concatenated along the channel dimension and then mapped to a preset number of output channels through a fusion convolutional layer to calculate and output the fused image feature Fi = Conv([Ri, Di]). The core of this color main anchor point fusion unit lies in its strict parameter initialization constraints and dynamic parameter update mechanism. In the initial stage of network training, the system initializes the first mapping submatrix W corresponding to the color image features in the fusion convolutional layer through the constraint initialization module.R Explicitly set as a unit mapping (i.e., W) R =I), while simultaneously mapping the second submatrix W corresponding to the depth image features. D Set as zero mapping (i.e., W) D =0), and the bias term is directly set to zero. Under this deterministic initialization constraint, the fusion output of the network in the early stage of training is equivalent to the original color image features (i.e., Fi=Ri), thus ensuring the absolute stability of the main anchor features in the initial stage. Subsequently, during the parameter update process of the network, depth compensation information is gradually learned by the model through training. After sufficient training, the fusion convolutional layer can effectively combine depth information, so that the final output fusion image features evolve into Fi=Ri+Δi, that is, depth compensation is superimposed on the color features. This high-quality fusion image feature Fi is finally passed to the optional subsequent module shown by the dashed line in the figure to perform specific image data processing tasks.
[0045] Figure 5 This is a schematic diagram of a high-resolution multi-scale fusion feature construction module according to a specific embodiment of this application, as shown below. Figure 5As shown, this module synchronously receives multiple fusion features with different strides and spatial resolutions from the front-end network. These input features specifically include a high spatial resolution fusion feature F2 with a stride of 4, and fusion features F3, F4, and F5 with strides of 8, 16, and 32, respectively, constructing multi-scale fusion feature layers P2, P3, P4, and P5. Fusion features F3, F4, and F5 are first input into a conventional multi-scale semantic feature construction unit, and after feature processing, generate conventional low-resolution semantic fusion feature layers P3, P4, and P5. To compensate for the lack of spatial detail in conventional semantic features, the system implements an independent reconstruction path for the high spatial resolution fusion feature F2. Fusion feature F2 first undergoes high-resolution lateral mapping (e.g., channel adaptation through lateral convolution) to preserve local spatial details in the original image, resulting in high-resolution lateral features. Simultaneously, the system obtains more macroscopic contextual information from P3 or adjacent low-resolution semantic features and performs upsampling operations on them, enlarging the spatial size of the adjacent semantic features to a dimension matching the high-resolution lateral features. The system then proceeds to the fusion enhancement stage. It performs feature-level fusion (such as concatenation or convolution) between lateral features that retain local details and upsampled semantic features that provide contextual information, successfully constructing a new high-resolution fusion feature layer P2. The output channel count of P2 is set to 64. After this reconstruction, the newly added high-resolution fusion feature P2 is combined with P3, P4, and P5 generated by the conventional multi-scale semantic feature construction unit to form a complete multi-scale fusion feature set (P2 / P3 / P4 / P5). This multi-scale fusion feature set is ultimately used as the module's output for downstream image data processing modules. Figure 5 As shown in the dashed box below, these multi-scale fusion features can be input into optional subsequent image data processing modules (such as object detection heads), enabling the model to fully utilize the high-resolution fusion features of P2 to perform high-precision location determination and feature perception of small-scale objects and local occlusion areas while using P3 to P5 to process large-scale targets.
[0046] Figure 6 This is a schematic diagram of parameter migration initialization based on a specific embodiment of the color main branch model of this application, as shown below. Figure 6As shown, the initialization process starts from a trained color image processing model (containing a backbone network and a three-layer task head) and completes the overall parameter configuration of the heterogeneous fusion model through three independent paths. First, in the backbone branch parameter transfer of path one, the system extracts the backbone parameters of the trained model and loads them into the color main branch. Then, through parameter copying, the color backbone parameters are directly copied or mapped to the deep auxiliary branch, thus completing the initialization of the two branches. This transfer method allows the deep auxiliary branch to inherit the basic feature extraction capabilities of the mature model in the early stages of training, effectively reducing the difficulty of training the two-branch network from scratch. Second, in the deterministic initialization of the fusion unit in path two, the system performs strict deterministic parameter settings for the fusion mapping process after the color feature and deep feature channels are concatenated. Specifically, the system sets the first mapping submatrix W corresponding to the color image features. R Forced to be set to the identity mapping matrix (I), the second mapping submatrix W corresponding to the depth image features D The primary anchor point initialization is completed by setting the zero mapping matrix (0) and setting the bias term of the fusion module to zero. This path constraint mathematically ensures that the output of the fusion module at the start of training is completely equivalent to a stable and reliable color feature response. Finally, in the high-resolution sub-head parameter migration of path three, the system adaptively reconstructs and allocates the task head parameters. For the three-layer task head parameters extracted from the original model (corresponding to P3, P4, and P5 respectively), the system smoothly migrates them to a new four-layer task head (corresponding to P2, P3, P4, and P5) containing high-resolution feature layers. The specific mapping logic is as follows: In the implementation of the four-layer detection head in the subsequent image data processing module, the P3, P4, and P5 parameters of the existing three-layer detection head are mapped to the P3, P4, and P5 sub-heads of the new detection head, and the original P3 sub-head parameters are copied to the newly added P2 sub-head, thereby providing initialization parameters for the newly added P2 sub-head while retaining the original P3 sub-head. Combining the three independent paths mentioned above, the system ultimately constructs an initialized multi-scale fusion processing model for heterogeneous image data streams. This model is fully equipped with a color main branch and a depth auxiliary branch, satisfying W... R =I and W D The main anchor point fusion condition is 0, and it can robustly output P2 / P3 / P4 / P5 multi-scale fusion features.
[0047] Figure 8 This illustration shows a heterogeneous image multi-scale fusion process and activation visualization results according to a specific embodiment of this application. Figure 8 As shown, Figure 8 In the image (a), the input RGB image is shown. Figure 8 In the above, (b) is the estimated depth image aligned with the RGB image space, and the two together serve as heterogeneous input data for the method of this application. Figure 8(c) shows the activation visualization of the trained RGB branch. Figure 8 (d) in the diagram shows the activation visualization of the trained deep branches. Figure 8 (e) shows the activation visualization of the fusion module at the initial checkpoint. Figure 8 Figure (f) shows the activation visualization of the fusion module at the trained checkpoints. The shared color scale below indicates the root mean square activation values in each figure, facilitating a comparison of the responses of different branches and fusion results on a uniform scale. Figure 8 As can be seen, at the initial checkpoint, the activation response of the fusion module maintains the basic response characteristics consistent with the RGB main branch. After training, the fusion module retains the texture information response of the RGB image while introducing the spatial structure information represented by the depth image, thus forming the fused feature response result. This figure illustrates the working process of the main anchor point initialization and heterogeneous image fusion mechanism described in this application from a visual perspective.
[0048] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0049] like Figure 7 As shown, the computer system includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. RAM 703 also stores various programs and data required for system operation. CPU 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.
[0050] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a liquid crystal display (LCD) and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card and a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.
[0051] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0052] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0054] The modules described in the embodiments of this application can be implemented in software or in hardware.
[0055] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; input the first image data stream into a color main branch network to extract color image feature maps at multiple spatial resolution scales; and input the second image data stream into a depth auxiliary branch network to extract depth image feature maps corresponding to multiple spatial resolution scales; at at least two scales, concatenate the color image feature maps and depth image feature maps of the same scale along the channel dimension and input them into a fusion convolutional layer to map and output a fused image feature map; wherein, in the parameters of the fusion convolutional layer... In the initialization phase, the first mapping submatrix corresponding to the color image feature map is initialized as an identity mapping matrix or an approximate identity mapping matrix, the second mapping submatrix corresponding to the depth image feature map is initialized as a zero mapping matrix, and the bias is initialized to zero, so that the fused image feature map output in the initial training phase is equal to or approximately equal to the color image feature map. Based on the fused image feature maps at each scale, cross-scale feature reconstruction is performed to generate a multi-scale fused feature map set containing a high-resolution fused feature layer and at least one low-resolution semantic fused feature layer. Among them, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct a high-resolution fused feature layer. The multi-scale fused feature map set is output for subsequent image data processing.
[0056] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A method for multi-scale fusion of heterogeneous images based on master anchor point initialization, characterized in that, include: S10: Obtain a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; S20: Input the first image data stream into the color main branch network to extract color image feature maps at multiple spatial resolution scales; and input the second image data stream into the depth auxiliary branch network to extract depth image feature maps corresponding to the multiple spatial resolution scales. S30: At at least two scales, the color image feature map and the depth image feature map at the same scale are concatenated along the channel dimension and input into a fusion convolutional layer to map and output a fusion image feature map; wherein, during the parameter initialization phase of the fusion convolutional layer, the first mapping sub-matrix corresponding to the color image feature map is initialized to an identity mapping matrix, the second mapping sub-matrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fusion image feature map output in the initial training phase is equal to the color image feature map; the first mapping sub-matrix is the weight sub-matrix in the weight parameters of the fusion convolutional layer used to map the color image feature map, and the second mapping sub-matrix is the weight sub-matrix in the weight parameters of the fusion convolutional layer used to map the depth image feature map; S40: Based on the fused image feature maps at each scale, perform cross-scale feature reconstruction to generate a multi-scale fused feature map set containing a high-resolution fused feature layer and at least one low-resolution semantic fused feature layer; wherein, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct the high-resolution fused feature layer; and S50: Generate a fused image based on the high-resolution fusion feature layer, including: performing channel mapping on the high-resolution fusion feature layer so that the number of mapped feature channels corresponds to the number of image channels in the first image data stream; upsampling the channel-mapped features so that their spatial resolution is restored to the image spatial resolution corresponding to the first image data stream, and obtaining and outputting the fused image.
2. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, Before step S20, the method further includes: performing at least one spatial alignment process among scaling, cropping, flipping, padding, and size alignment on the first image data stream and the second image data stream; and performing at least one numerical preprocessing operation among normalization, truncation, missing value filling, and valid region marking on the second image data stream.
3. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, The first image data stream is a three-channel image, and the second image data stream is a single-channel image. Before inputting the second image data stream into the depth auxiliary branch network, the single-channel image is copied into three channels along the channel dimension to match the input channel structure of the color main branch network.
4. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, The parameter initialization steps of the deep auxiliary branch network include: obtaining the parameters of the deep auxiliary branch network by copying or mapping the parameters of the trained color main branch network.
5. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, The color main branch network and the deep auxiliary branch network respectively output image feature maps at four scale levels; the stride sizes of the four scale levels relative to the input image are 4, 8, 16 and 32 respectively; wherein, the high-resolution fusion feature layer is constructed based on the scale level with a stride size of 4.
6. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, S40 specifically includes: S41: Perform lateral mapping on the fusion image feature map with the highest spatial resolution at each scale to obtain a high-resolution lateral feature map. S42: Upsample adjacent low-resolution semantic features to obtain upsampled semantic features corresponding to the high-resolution lateral feature space size; S43: The high-resolution lateral features and the upsampled semantic features are concatenated and fused to obtain the high-resolution fused feature layer.
7. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 1, characterized in that, The multi-scale fusion feature map set is also provided to a subsequent image data processing module, which includes a detection head. The detection head includes a first detection sub-head corresponding to the high-resolution fusion feature layer and a second detection sub-head corresponding to the low-resolution semantic fusion feature layer.
8. The heterogeneous image multi-scale fusion method based on master anchor point initialization according to claim 7, characterized in that, The following operations are performed during the parameter initialization phase of the detection head: extracting the task head parameters from the trained color image processing model; The parameters corresponding to the highest spatial resolution detection layer in the original three-layer detection head are copied or mapped to the first detection sub-head.
9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that, When the one or more computer programs are executed by a computer processor, they perform the method according to any one of claims 1-8.
10. A heterogeneous image multi-scale fusion system based on master anchor point initialization, characterized in that, include: The data input module is used to acquire a spatially aligned first image data stream and a second image data stream, wherein the first image data stream is a color image data stream and the second image data stream is a depth image data stream; The dual-branch feature extraction module includes a color main branch and a depth auxiliary branch. The color main branch is used to input the first image data stream into the color main branch network to extract color image feature maps at multiple spatial resolution scales. The depth auxiliary branch is used to input the second image data stream into the depth auxiliary branch network to extract depth image feature maps corresponding to the multiple spatial resolution scales. A color master anchor point fusion module is used to concatenate the color image feature map and the depth image feature map at the same scale along the channel dimension at at least two scales, and input them into a fusion convolutional layer to map and output a fused image feature map. Specifically, during the parameter initialization phase of the fusion convolutional layer, a first mapping sub-matrix corresponding to the color image feature map is initialized to an identity mapping matrix, a second mapping sub-matrix corresponding to the depth image feature map is initialized to a zero mapping matrix, and the bias is initialized to zero, so that the fused image feature map output during the initial training phase is equal to the color image feature map. The first mapping sub-matrix is a weight sub-matrix in the weight parameters of the fusion convolutional layer used to map the color image feature map, and the second mapping sub-matrix is a weight sub-matrix in the weight parameters of the fusion convolutional layer used to map the depth image feature map. A high-resolution multi-scale fusion feature construction module is used to perform cross-scale feature reconstruction based on the fused image feature maps at various scales, generating a multi-scale fusion feature map set containing a high-resolution fusion feature layer and at least one low-resolution semantic fusion feature layer; wherein, adjacent low-resolution semantic features are upsampled and fused with lateral feature maps with corresponding spatial resolution to construct the high-resolution fusion feature layer; and The fused image generation module is used to perform channel mapping on the high-resolution fused feature layer so that the number of mapped feature channels corresponds to the number of image channels in the first image data stream, and to upsample the channel-mapped features to restore their spatial resolution to the image spatial resolution corresponding to the first image data stream, thereby obtaining and outputting the fused image.
