Image Fusion Method, Device, Electronic Device and Medium

Abstract

The present disclosure relates to an image fusion method, apparatus, electronic device, and medium; wherein, the method includes: obtaining image data, the image data including: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images; inputting the image data into a pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model; wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, the heterogeneous feature fusion sub-model is used for performing data fusion on the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is used for performing data fusion on the homogeneous feature association data between the same type of images. The embodiments of the present disclosure can effectively improve the image fusion efficiency.

Description

Technical Field

[0001] The present disclosure relates to image processing technologies, and in particular, to an image fusion method, apparatus, electronic device, and medium. Background Art

[0002] Image fusion is to synthesize two or more images into a new image by using a specific algorithm to extract and summarize the effective data in each image. Among them, there may be heterogeneous feature correlation data or homogeneous feature correlation data between two images. Heterogeneous feature correlation data is different types of correlation data existing between different types of images, and homogeneous feature correlation data is the same type of correlation data existing between the same type of images.

[0003] In related technologies, mainly different types of data are converted to obtain the same type of data to achieve image fusion. For example, the heterogeneous feature data of an image is converted into its homogeneous data, or the homogeneous feature data of an image is converted into its heterogeneous data.

[0004] However, there is information loss during the data conversion process, resulting in image fusion accuracy. Summary of the Invention

[0005] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image fusion method, apparatus, electronic device, and medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an image fusion method, including:

[0007] Obtain image data, where the image data includes: heterogeneous feature correlation data between different types of images and / or homogeneous feature correlation data between the same type of images;

[0008] Input the image data into a pre-trained target fusion model, and determine fusion data according to the output result of the target fusion model;

[0009] Wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model. The heterogeneous feature fusion sub-model is used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature correlation data between the same type of images.

[0010] Optionally, the training process of the target fusion model includes:

[0011] Determine a heterogeneous feature fusion sub-model and determine a homogeneous feature fusion sub-model;

[0012] Determine a target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

[0013] Optionally, the determining of the heterogeneous feature fusion sub-model includes:

[0014] Performing collaborative matrix decomposition on the sample image on a first preset network to obtain the heterogeneous representation matrix of the sample image;

[0015] Training the first preset network based on the heterogeneous representation matrix of the sample image to obtain a heterogeneous feature fusion sub-model.

[0016] Optionally, the determining of the homogeneous feature fusion sub-model includes:

[0017] Performing collaborative matrix decomposition on the sample image on a second preset network to obtain the homogeneous representation matrix of the sample image;

[0018] Training the second preset network based on the homogeneous representation matrix of the sample image to obtain a homogeneous feature fusion sub-model.

[0019] Optionally, the determining of the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model includes:

[0020] Determining an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model;

[0021] Performing optimization processing on the initial parameters of the initial fusion model to obtain target parameters;

[0022] Determining the target fusion model according to the target parameters.

[0023] Optionally, the initial parameters include: a first parameter, a second parameter, a third parameter, and a fourth parameter;

[0024] The performing optimization processing on the initial parameters of the initial fusion model to obtain target parameters includes:

[0025] Optimizing one of the initial parameters in sequence based on a preset optimization rule to obtain the target parameter corresponding to the one parameter;

[0026] Wherein, the preset optimization rule includes: a preset number of iterations and / or model convergence.

[0027] Optionally, the optimizing one of the initial parameters to obtain the target parameter corresponding to the one parameter includes:

[0028] Selecting one of the initial parameters as the parameter to be optimized, keeping the other parameters in the initial parameters unchanged, and optimizing the parameter to be optimized based on a preset constraint function;

[0029] Until each parameter in the initial parameters is optimized, the target parameter corresponding to each parameter is obtained.

[0030] In a second aspect, an embodiment of the present disclosure provides an image fusion device, including:

[0031] An acquisition module, configured to acquire image data, where the image data includes: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images;

[0032] A determination module, configured to input the image data into a pre-trained target fusion model, and determine fusion data according to an output result of the target fusion model;

[0033] Wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, the heterogeneous feature fusion sub-model is configured to perform data fusion on the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is configured to perform data fusion on the homogeneous feature association data between the same type of images.

[0034] Optionally, it further includes: a training module; the training module includes: a first determination unit, a second determination unit, and a third determination unit;

[0035] The first determination unit is configured to determine a heterogeneous feature fusion sub-model;

[0036] The second determination unit is configured to determine a homogeneous feature fusion sub-model;

[0037] The third determination unit is configured to determine a target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

[0038] Optionally, the first determination unit is specifically configured to:

[0039] Perform collaborative matrix decomposition on a sample image on a first preset network to obtain a heterogeneous representation matrix of the sample image;

[0040] Train the first preset network based on the heterogeneous representation matrix of the sample image to obtain a heterogeneous feature fusion sub-model.

[0041] Optionally, the second determination unit is specifically configured to:

[0042] Perform collaborative matrix decomposition on a sample image on a second preset network to obtain a homogeneous representation matrix of the sample image;

[0043] Train the second preset network based on the homogeneous representation matrix of the sample image to obtain a homogeneous feature fusion sub-model.

[0044] Optionally, the third determination unit includes: a determination subunit and an optimization subunit;

[0045] The determination subunit is configured to determine an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model;

[0046] The optimization subunit is configured to perform an optimization process on the initial parameters of the initial fusion model to obtain target parameters;

[0047] The determination subunit is further configured to determine a target fusion model according to the target parameters.

[0048] Optionally, the initial parameters include: a first parameter, a second parameter, a third parameter, and a fourth parameter;

[0049] The determination subunit is specifically configured to:

[0050] Based on a preset optimization rule, optimize one parameter in the initial parameters in sequence to obtain the target parameter corresponding to the one parameter;

[0051] Wherein, the preset optimization rule includes: a preset number of iterations and / or model convergence.

[0052] Optionally, the determination subunit is specifically configured to:

[0053] Select one parameter in the initial parameters as the parameter to be optimized, and keep the other parameters in the initial parameters unchanged, and optimize the parameter to be optimized based on a preset constraint function;

[0054] Until each parameter in the initial parameters is optimized to obtain the target parameter corresponding to each parameter.

[0055] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the image fusion method provided by any embodiment of the present disclosure are implemented.

[0056] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image fusion method provided by any embodiment of the present disclosure are implemented.

[0057] The technical solution provided by the embodiments of the present disclosure has the following advantages compared with the prior art: By inputting image data into a pre-trained target fusion model and determining fusion data according to the output result of the target fusion model, where the heterogeneous feature fusion sub-model included in the target fusion model can be used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model can be used to perform data fusion on the homogeneous feature correlation data between the same type of images, the problem of information loss in the fusion process is solved, thereby effectively improving the image fusion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0060] Figure 1 is a schematic flowchart of an image fusion method provided by an embodiment of the present disclosure;

[0061] Figure 2 is a schematic flowchart of another image fusion method provided by an embodiment of the present disclosure;

[0062] Figure 3 is a schematic flowchart of yet another image fusion method provided by an embodiment of the present disclosure;

[0063] Figure 4 is a schematic structural diagram of an image fusion device provided by an embodiment of the present disclosure;

[0064] Figure 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] In order to be able to more clearly understand the above objects, features, and advantages of the present disclosure, the following will further describe the solutions of the present disclosure. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0066] Many specific details are set forth in the following description in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0067] In one embodiment, as Figure 1 shown, an image fusion method is provided. In this embodiment, taking the application of this method to a mobile terminal as an example, it can be understood that this method can also be applied to the server of the mobile terminal, and can also be applied to a system including a mobile terminal and a server, and is implemented through the interaction between the mobile terminal and the server. In this embodiment, the method includes the following steps:

[0068] S110. Obtain image data.

[0069] Among them, the image data may include: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images.

[0070] Exemplarily, the image data may include: heterogeneous feature association data between different types of images, or homogeneous feature association data between the same type of images, or heterogeneous feature association data between different types of images and homogeneous feature association data between the same type of images.

[0071] Among them, assuming there are m types of images, the heterogeneous feature association data between different types of images can be determined by the following formula (1).

[0072] (1)

[0073] In formula (1), represents the i th sample heterogeneous feature association of the th type of image, represents the Hadamard product.

[0074] The homogeneous feature association data between the same type of images can be determined by the following formula (2).

[0075] (2)

[0076] In formula (2), represents the multi-source homogeneous feature association between the i th sample of the th type of image.

[0077] S120. Input the image data into a pre-trained target fusion model, and determine the fusion data according to the output result of the target fusion model.

[0078] Among them, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model. The heterogeneous feature fusion sub-model is used to perform data fusion on the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature association data between the same type of images.

[0079] Among them, the target fusion model can be determined by an algorithm based on Selective Non-Matrix Factorization (SNMF). Specifically, SNMF can collaboratively decompose the co-occurrence matrices of multiple heterogeneous feature correlation networks into low-rank matrices of each type of node, and automatically optimize the weights of different networks. SNMF also draws on the idea of network alignment and performs regularization constraints by aligning these low-rank matrices to multi-source homogeneous feature networks during the optimization process. Furthermore, the optimized low-rank matrices are used to reconstruct the target image representation matrix.

[0080] Furthermore, SNMF can also be applied to image classification, and specifically, the support vector machine implementation method can be selected as the classification method.

[0081] An image fusion method provided in this embodiment inputs image data into a pre-trained target fusion model, and determines fusion data according to the output result of the target fusion model. Among them, the heterogeneous feature fusion sub-model included in the target fusion model can be used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model can be used to perform data fusion on the homogeneous feature correlation data between the same type of images, solving the problem of information loss during the fusion process, and thus effectively improving the image fusion efficiency.

[0082] Figure 2 It is a schematic flowchart of another image fusion method provided by an embodiment of the present disclosure. This embodiment is further extended and optimized on the basis of the above embodiment. Further, before S120, this embodiment may further include:

[0083] S111. Determine the heterogeneous feature fusion sub-model and determine the homogeneous feature fusion sub-model.

[0084] Among them, the heterogeneous feature fusion sub-model can effectively perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model can effectively perform data fusion on the homogeneous feature correlation data between the same type of images.

[0085] In this embodiment, optionally, determining the heterogeneous feature fusion sub-model includes:

[0086] Performing co-occurrence matrix decomposition on the sample images on the first preset network to obtain the heterogeneous representation matrix of the sample images;

[0087] Training the first preset network based on the heterogeneous representation matrix of the sample images to obtain the heterogeneous feature fusion sub-model.

[0088] Among them, the first preset network can be a multi-type heterogeneous feature association network. Performing collaborative matrix decomposition on the multi-type heterogeneous feature association network can mine the potential representation matrix of the image.

[0089] Exemplarily, the heterogeneous feature fusion sub-model can be determined by the following formula (3).

[0090] (3)

[0091] Wherein, represents the low-rank representation of the i th or j th type of image, represents the reduced dimension.

[0092] Since the number of homogeneous feature association networks of different types of image samples collected is different, assuming , is smaller in scale than and can be regarded as a low-rank compressed heterogeneous feature association network. Describe the attribute information of the th object of the i th type of image sample and the attribute information of the th object of the j th type of image sample in the compressed -dimensional space in a real-numbered form.

[0093] Wherein, , is the weight assigned to the heterogeneous feature association networks.

[0094] Formula (3) mines the internal association of the th type or the i th type hidden in j through low-rank matrix decomposition, and then specifically discovers the potential features of the image.

[0095] For the weight assigned to the heterogeneous feature association networks, for , formula (3) ignores the contribution of the homogeneous feature data source to the target association prediction. In this way, the prediction performance may be affected by the noisy homogeneous feature data source and thus lost. Therefore, this embodiment can further introduce a scheme of collaborative matrix decomposition based on network alignment to selectively fuse different homogeneous feature association networks.

[0096] In this embodiment, optionally, determining the homogeneous feature fusion sub-model includes:

[0097] Perform collaborative matrix decomposition on the sample image on the second preset network to obtain a homogeneous representation matrix of the sample image;

[0098] Based on the homogeneous representation matrix of the sample image, the second preset network is trained to obtain a homogeneous feature fusion sub-model.

[0099] Among them, the second preset network can be a homogeneous feature network based on network alignment, and collaborative matrix decomposition on the homogeneous feature network based on network alignment can mine the potential representation matrix of the image.

[0100] Exemplarily, the homogeneous feature fusion sub-model can be determined by the following formula (4).

[0101] (4)

[0102] Among them, for ,Formula (4) can simultaneously consider the impact of different heterogeneous feature association data and different homogeneous feature association data on the target potential association by calculating the weights.

[0103] Indicates i A total of 10000 types of image samples were collected Homogeneous feature-linked data from various sources, among which: Indicates i A total of 10000 types of image samples were collected Homogeneous feature-linked data from various sources. ,if, , then along The main diagonal i The matrix blocks are 0.

[0104] The entity values in the homogeneous feature association matrix take positive values for dissimilar objects and negative values for similar objects. The positive entities are called no-connect constraints because they force different objects of pairs of the same type of image samples to stay away from each other in the low-rank representation space. The negative entities are called must-connect constraints because they force a pair of objects of the same type of image samples to be close in the low-rank space. These homogeneous feature associations can collaboratively guide the low-rank matrix The solution.

[0105] Assume that yes The low-rank representation of image samples, yes The low-rank representation of image features can be reconstructed by To complete the representation reconstruction of the target image.

[0106] S112. Determine the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

[0107] Among them, by combining the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model, the target fusion model can be effectively trained, thereby avoiding the problem of information loss in the process of image data fusion.

[0108] Figure 3 It is a schematic flowchart of another image fusion method provided by an embodiment of the present disclosure. This embodiment is further extended and optimized on the basis of the above embodiment. Among them, a possible implementation manner of S112 is as follows:

[0109] S1121. Determine the initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

[0110] Among them, the initial fusion model can be determined by the following formula (5).

[0111] (5)

[0112] Among them, is the vector obtained by row stacking and splicing , is used to control the and complexity. At the same time, can also help to selectively integrate different heterogeneous data sources and homogeneous data sources.

[0113] S1122. Optimize the initial parameters of the initial fusion model to obtain the target parameters.

[0114] Among them, the objective function of SNMF is non-convex on , so the alternating direction method of multipliers (ADMM) can be used to optimize and solve it to obtain the target parameters.

[0115] Among them, the initial parameters may include: the first parameter, the second parameter, the third parameter, and the fourth parameter.

[0116] In this embodiment, optionally, optimizing the initial parameters of the initial fusion model to obtain the target parameters includes:

[0117] Based on the preset optimization rule, optimize one parameter in the initial parameters in sequence to obtain the target parameter corresponding to one parameter;

[0118] Among them, the preset optimization rule includes: the preset number of iterations and / or model convergence.

[0119] Among them, one parameter in the initial parameters can be optimized in sequence to obtain the target parameters corresponding to all parameters.

[0120] Exemplarily, the second parameter, the third parameter, and the fourth parameter can be fixed first and remain unchanged, and the first parameter is optimized. After obtaining the target parameter corresponding to the first parameter, the second parameter, the third parameter, and the fourth parameter are optimized in the same way in sequence.

[0121] Thus, based on the preset optimization rules, one parameter in the initial parameters is optimized in sequence, and the target parameters corresponding to each parameter are effectively obtained.

[0122] In this embodiment, optionally, optimizing one parameter in the initial parameters to obtain the target parameter corresponding to one parameter includes:

[0123] Select one parameter in the initial parameters and determine it as the parameter to be optimized, and keep the other parameters in the initial parameters unchanged, and optimize the parameter to be optimized based on the preset constraint function;

[0124] Until each parameter in the initial parameters is optimized, the target parameters corresponding to each parameter are obtained.

[0125] Among them, an iterative process can be used to alternately optimize and solve each initial parameter, that is, each time 3 of them are fixed and unchanged, and the other 1 parameter is optimized at the same time until the specified number of iterations or convergence is reached.

[0126] Introduce the constraint Lagrange multiplier , then formula (5) can be equivalent to the following formula (6).

[0127] (6)

[0128] Among them, assuming is known, can be optimized (if ), so take the partial derivative of formula (6) with respect to , and for , make , formula (7) can be obtained.

[0129] (7)

[0130] Assume is known, can be optimized So take the partial derivative of formula (6) with respect to , and for , make , formula (8) can be obtained.

[0131] (8)

[0132] Similarly, assuming is known, the partial derivative of formula (6) with respect to can be obtained, and formula (9) can be obtained.

[0133] (9)

[0134] The polynomial factor can be obtained by setting and from the Karush-Kuhn-Tucker (KKT) conditions, we have: .

[0135] is a fixed-point equation and the solution must satisfy the convergence condition, so we can have:

[0136] (10)

[0137] For , we have:

[0138] (11)

[0139] For , we have:

[0140] (12)

[0141] The positive and negative signs in formula (10), formula (11), and formula (12) can be defined as and . Therefore, the low-rank representation can be updated to:

[0142] (13)

[0143] After the update of , it can be regarded as a known constant and the partial derivatives of formula (5) with respect to and can be continued. When taking the partial derivative, we get:

[0144] (14)

[0145] Let represent the reconstruction loss of the heterogeneous association , and formula (14) can be simplified to:

[0146] (15)

[0147] Same as solving Similarly, when taking the partial derivative with respect to we can further obtain:

[0148] (16)

[0149] Let represent the homogeneous internal constraint of the low-rank representation Equation (16) can be simplified to:

[0150] (17)

[0151] It should be noted that the solution of Equation (15) and Equation (17) can be regarded as a quadratic programming problem with respect to and and the Lagrange multiplier can be introduced for solution similar to the algorithm SNMF.

[0152] S1123. Determine the target fusion model according to the target parameters.

[0153] Among them, after optimizing each initial parameter, the target parameters corresponding to each initial parameter included in the initial fusion model can be obtained. Thus, based on the target parameters corresponding to the initial parameters, the target fusion model can be effectively constructed.

[0154] Figure 4 FIG. is a schematic structural diagram of an image fusion device provided by an embodiment of the present disclosure; the device is configured in an electronic device and can implement the image fusion method described in any embodiment of the present application. The device specifically includes the following:

[0155] An acquisition module 410, configured to acquire image data, where the image data includes: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images;

[0156] A determination module 420, configured to input the image data into a pre-trained target fusion model and determine the fusion data according to the output result of the target fusion model;

[0157] Among them, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model. The heterogeneous feature fusion sub-model is configured to perform data fusion on the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is configured to perform data fusion on the homogeneous feature association data between the same type of images.

[0158] In this embodiment, optionally, the device in this embodiment further includes: a training module; the training module includes: a first determination unit, a second determination unit, and a third determination unit;

[0159] The first determination unit is configured to determine a heterogeneous feature fusion sub-model;

[0160] The second determination unit is configured to determine a homogeneous feature fusion sub-model;

[0161] The third determination unit is configured to determine a target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

[0162] In this embodiment, optionally, the first determination unit is specifically configured to:

[0163] Perform collaborative matrix decomposition on a sample image on a first preset network to obtain a heterogeneous representation matrix of the sample image;

[0164] Train the first preset network based on the heterogeneous representation matrix of the sample image to obtain a heterogeneous feature fusion sub-model.

[0165] In this embodiment, optionally, the second determination unit is specifically configured to:

[0166] Perform collaborative matrix decomposition on a sample image on a second preset network to obtain a homogeneous representation matrix of the sample image;

[0167] Train the second preset network based on the homogeneous representation matrix of the sample image to obtain a homogeneous feature fusion sub-model.

[0168] In this embodiment, optionally, the third determination unit includes: a determination subunit and an optimization subunit;

[0169] The determination subunit is configured to determine an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model;

[0170] The optimization subunit is configured to perform optimization processing on the initial parameters of the initial fusion model to obtain target parameters;

[0171] The determination subunit is further configured to determine a target fusion model according to the target parameters.

[0172] In this embodiment, optionally, the initial parameters include: a first parameter, a second parameter, a third parameter, and a fourth parameter;

[0173] The determination subunit is specifically configured to:

[0174] Based on a preset optimization rule, optimize one of the initial parameters in sequence to obtain a target parameter corresponding to the one parameter;

[0175] Wherein, the preset optimization rule includes: a preset number of iterations and / or model convergence.

[0176] In this embodiment, optionally, the determining subunit is specifically configured to:

[0177] Select one parameter from the initial parameters as the parameter to be optimized, keep the other parameters in the initial parameters unchanged, and optimize the parameter to be optimized based on a preset constraint function;

[0178] Until each parameter in the initial parameters is optimized, the target parameter corresponding to each parameter is obtained.

[0179] Through the image fusion device according to the embodiment of the present invention, by inputting image data into a pre-trained target fusion model and determining fusion data according to the output result of the target fusion model, wherein the heterogeneous feature fusion sub-model included in the target fusion model can be used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature correlation data between the same type of images, the problem of information loss in the fusion process is solved, thereby effectively improving the image fusion efficiency.

[0180] For the specific limitations on the image fusion device, reference can be made to the limitations on the image fusion method in the above text, which will not be elaborated here. Each module in the above image fusion device can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0181] In one embodiment, an electronic device is provided. The electronic device can be a terminal, and its internal structure diagram can be as Figure 5 shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, it realizes a method for adjusting abnormal screen brightness. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the electronic device, or an external keyboard, a touchpad or a mouse, etc.

[0182] Those skilled in the art can understand,Figure 5 The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0183] In one embodiment, the image fusion device provided by this application can be implemented in the form of a computer program, and the computer program can run on an electronic device as shown in Figure 5 the figure. Each program module that makes up the electronic device can be stored in the memory of the electronic device, and the computer program composed of each program module enables the processor to execute the steps in the image fusion method of each embodiment of this application described in this specification.

[0184] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: obtaining image data; wherein, the image data includes: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images; inputting the image data into a pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model; wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, and the heterogeneous feature fusion sub-model is used for data fusion of the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is used for data fusion of the homogeneous feature association data between the same type of images.

[0185] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining image data; determining the heterogeneous feature fusion sub-model and determining the homogeneous feature fusion sub-model; determining the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model; inputting the image data into the pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model.

[0186] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining image data; determining the heterogeneous feature fusion sub-model and determining the homogeneous feature fusion sub-model; determining an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model; optimizing the initial parameters of the initial fusion model to obtain target parameters; determining the target fusion model according to the target parameters; inputting the image data into the pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model.

[0187] In an embodiment of the present disclosure, image data is input into a pre-trained target fusion model, and fusion data is determined according to the output result of the target fusion model. Among them, the heterogeneous feature fusion sub-model included in the target fusion model can be used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature correlation data between the same type of images, solving the problem of information loss during the fusion process. Thus, the image fusion efficiency is effectively improved.

[0188] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining image data; wherein the image data includes: heterogeneous feature correlation data between different types of images and / or homogeneous feature correlation data between the same type of images; inputting the image data into a pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model; wherein the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, and the heterogeneous feature fusion sub-model is used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature correlation data between the same type of images.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining image data; determining the heterogeneous feature fusion sub-model and determining the homogeneous feature fusion sub-model; determining the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model; inputting the image data into a pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining image data; determining the heterogeneous feature fusion sub-model and determining the homogeneous feature fusion sub-model; determining an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model; performing optimization processing on the initial parameters of the initial fusion model to obtain target parameters; determining the target fusion model according to the target parameters; inputting the image data into a pre-trained target fusion model, and determining fusion data according to the output result of the target fusion model.

[0191] In an embodiment of the present disclosure, image data is input into a pre-trained target fusion model, and fusion data is determined according to the output result of the target fusion model. Among them, the heterogeneous feature fusion sub-model included in the target fusion model can be used to perform data fusion on the heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature correlation data between the same type of images, solving the problem of information loss during the fusion process. Thus, the image fusion efficiency is effectively improved.

[0192] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM), etc.

[0193] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0194] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image fusion method, characterized in that, Including: Obtain image data, where the image data includes: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images; Input the image data into a pre-trained target fusion model, and determine fusion data according to the output result of the target fusion model; Wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, the heterogeneous feature fusion sub-model is used to perform data fusion on the heterogeneous feature association data between different types of images, and the homogeneous feature fusion sub-model is used to perform data fusion on the homogeneous feature association data between the same type of images; The heterogeneous feature fusion sub-model is determined by the following formula: Among them, represents the low-rank representation of the i-th or j-th type of image, and k i , k j represents the reduced dimension; is smaller in scale than R ij and is the heterogeneous feature correlation network after low-rank compression; respectively describe the attribute information of n i , k j objects of the i-th type of image sample and the attribute information of n i objects of the j-th type of image sample in a real-numberized form in the compressed k j -dimensional space; W r ∈R m×n , and W r is the weight assigned to the heterogeneous feature correlation networks; The homogeneous feature fusion sub-model is determined by the following formula: Among them, for It is expressed that a total of n i types of homogeneous feature correlation data from the same source are collected for the i-th type of image sample, where t i represents that a total of t i types of the homogeneous feature correlation data from the same source are collected for the i-th type of image sample; If t > t i , then the i-th matrix block along the main diagonal is 0.

2. The method according to claim 1, wherein The training process of the target fusion model includes: Determine the heterogeneous feature fusion sub-model and determine the homogeneous feature fusion sub-model; Determine the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model.

3. The method according to claim 2, wherein The determination of the heterogeneous feature fusion sub-model includes: Perform collaborative matrix decomposition on the sample images on the first preset network to obtain the heterogeneous representation matrix of the sample images; Based on the heterogeneous representation matrix of the sample images, train the first preset network to obtain the heterogeneous feature fusion sub-model.

4. The method according to claim 2, wherein The determination of the homogeneous feature fusion sub-model includes: Perform collaborative matrix decomposition on the sample images on the second preset network to obtain the homogeneous representation matrix of the sample images; Based on the homogeneous representation matrix of the sample images, train the second preset network to obtain the homogeneous feature fusion sub-model.

5. The method according to any one of claims 2-4, characterized in that, The determination of the target fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model includes: Determine an initial fusion model according to the heterogeneous feature fusion sub-model and the homogeneous feature fusion sub-model; Optimize the initial parameters of the initial fusion model to obtain target parameters; Determine the target fusion model according to the target parameters.

6. The method according to claim 5, wherein The initial parameters include: the first parameter, the second parameter, the third parameter, and the fourth parameter; The optimization of the initial parameters of the initial fusion model to obtain target parameters includes: Based on a preset optimization rule, optimize one parameter in the initial parameters in sequence to obtain the target parameter corresponding to the one parameter; Wherein, the preset optimization rule includes: a preset number of iterations and / or model convergence.

7. The method according to claim 6, wherein The optimization of one parameter in the initial parameters to obtain the target parameter corresponding to the one parameter includes: Select one parameter in the initial parameters as the parameter to be optimized, and keep the other parameters in the initial parameters unchanged, and optimize the parameter to be optimized based on a preset constraint function; Until each parameter in the initial parameters is optimized to obtain the target parameter corresponding to each parameter.

8. An image fusion device, characterized in that, Including: An acquisition module, configured to acquire image data, where the image data includes: heterogeneous feature association data between different types of images and / or homogeneous feature association data between the same type of images; A determination module, configured to input the image data into a pre-trained target fusion model, and determine fusion data according to an output result of the target fusion model; Wherein, the target fusion model includes: a heterogeneous feature fusion sub-model and a homogeneous feature fusion sub-model, the heterogeneous feature fusion sub-model is configured to perform data fusion on heterogeneous feature correlation data between different types of images, and the homogeneous feature fusion sub-model is configured to perform data fusion on homogeneous feature correlation data between the same type of images; The heterogeneous feature fusion sub-model is determined by the following formula: Among them, represents the low-rank representation of the i-th or j-th type of image, and k i , k j represents the reduced dimension; is smaller in scale than R ij and is the heterogeneous feature correlation network after low-rank compression; respectively describe the attribute information of n i , k j objects of the i-th type of image sample and the attribute information of n i objects of the j-th type of image sample in the form of real numbers in the compressed k j , k r ∈R m×n , and W r is the weight assigned to the heterogeneous feature correlation networks; The homogeneous feature fusion sub-model is determined by the following formula: Among them, for It means that a total of n i types of homogeneous feature correlation data from the i-th type of image sample are collected. Among them, t i means that a total of t i types of the homogeneous feature correlation data from the i-th type of image sample are collected; If t > t i , then the i-th matrix block along the main diagonal is 0.

9. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the image fusion method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the image fusion method according to any one of claims 1 to 7 are implemented.

Images