Training Method of Image Registration Model, Image Fusion Method and Application

Abstract

The present invention discloses a training method for an image registration model, an image fusion method and applications thereof. Among them, the training method for the image registration model includes: establishing a training sample set, where the training sample set includes image samples of a target object taken from different perspectives; performing global registration on the image samples and generating a sparse graph; training the image registration model based on an objective cost function to determine the model parameters of the image registration model, where the objective cost function includes a reference optical flow field, and the image registration model registers the image samples based on multi-layer iterative solution of the optical flow field, and after solving the optical flow field of the current layer, uses the sparse graph to optimize the optical flow field of the current layer as the reference optical flow field of the objective cost function during the next layer of iterative solution. The training method for the image registration model of the present invention can reduce the occurrence of abnormal problems such as artifacts, distortions, and detail loss that may occur during the registration of multiple images due to parallax.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and specifically relates to a training method for an image registration model, an image fusion method, and an application thereof. Background Art

[0002] In recent years, photographic imaging technology and computational photography have developed rapidly, and multi-camera systems have been widely used in multiple fields. In the consumer electronics field, its core functions include widening the field of view, optimizing the depth of field, and realizing computational photography effects such as portrait mode and zoom by integrating cameras with different focal lengths or apertures. In the fields of autonomous driving and robotics, multi-camera systems can accurately extract depth information by fusing multi-view data, significantly improving the machine's perception ability of complex environments. In the fields of virtual reality (VR) and augmented reality (AR), the system can efficiently complete the reconstruction task of 3D objects or scenes by combining multi-view images.

[0003] Image fusion technology can synthesize multiple images obtained by a multi-camera system into one image to present a more realistic visual effect of the photographed object. When fusing multiple images with different viewpoints, it is necessary to register and align the multiple images to ensure the fusion effect of the multiple images. The registration methods in the prior art cannot achieve pixel-level precise alignment. Especially when there are large viewpoint differences or inconsistent illuminations between images, problems such as artifacts, distortions, and detail loss often occur.

[0004] Therefore, in view of the above technical problems, it is necessary to provide a training method for an image registration model, an image fusion method, and an application thereof. Summary of the Invention

[0005] The purpose of the present invention is to provide a training method for an image registration model, an image fusion method, and an application thereof, which can solve the problems such as artifacts, distortions, and detail loss that often occur when there are large viewpoint differences or inconsistent illuminations between images.

[0006] In order to achieve the above purpose, a specific embodiment of the present invention provides a training method for an image registration model, and the technical solution is as follows:

[0007] A training method for an image registration model, where the image registration model is used to register images with different viewpoints, and the method includes:

[0008] Establish a training sample set, where the training sample set includes image samples of a target object taken from different viewpoints;

[0009] Perform global registration on the image samples and generate a sparse graph, where the sparse graph includes key feature information in the image samples;

[0010] Training an image registration model based on a target cost function to determine the model parameters of the image registration model, wherein the target cost function includes a reference optical flow field, and the image registration model is based on multi-layer iterative solution of the optical flow field to register image samples, and after solving the optical flow field of the current layer, using the sparse graph to optimize the optical flow field of the current layer as the reference optical flow field of the target cost function during the next layer iterative solution.

[0011] In one or more embodiments of the present invention, using the sparse graph to optimize the optical flow field of the current layer specifically includes:

[0012] Generating a guiding field based on the sparse graph, wherein the guiding field includes displacement change information of key features of the sparse graph;

[0013] Constructing a regularization term based on the guiding field and the optical flow field of the current layer to perform a smoothing constraint on the optical flow field of the current layer.

[0014] In one or more embodiments of the present invention, the target cost function includes at least two of a luminance consistency sub-term, a gradient consistency sub-term, and a regularization sub-term; the luminance consistency sub-term represents the luminance error between image samples; the gradient consistency sub-term represents the gradient error between image samples; the regularization sub-term is used to apply regularization to the gradient of the optical flow field.

[0015] In one or more embodiments of the present invention, the image registration model is used to perform local registration on the image samples after global registration.

[0016] In one or more embodiments of the present invention, obtaining the optical flow field of the image samples based on a multi-resolution scale pyramid strategy of optical flow; and / or,

[0017] Extracting features of the image samples based on ORB, and screening the key feature information based on the RANSAC robust matching algorithm to obtain the sparse graph.

[0018] A specific embodiment of the present invention further provides an image fusion method, and the technical solution is as follows:

[0019] An image fusion method includes:

[0020] Obtaining a set of images to be fused, wherein the set of images to be fused includes images of a target object taken at different perspectives with different exposure degrees;

[0021] Fusing the images of the target object taken at the same perspective with different exposure degrees in the set of images to be fused to obtain intermediate fused images for each perspective;

[0022] After registering the intermediate fusion images of each perspective using the image registration model obtained by the above training method, the images are fused to obtain the fused image of the target object.

[0023] In one or more embodiments of the present invention, images of the target object taken at different exposure levels from the same perspective in the set of images to be fused are fused using a weighted average fusion strategy; and the intermediate fusion images of each perspective are fused using a maximum value fusion strategy.

[0024] A specific embodiment of the present invention also provides a training apparatus for an image registration model, and the technical solution is as follows:

[0025] A training apparatus for an image registration model, where the image registration model is used to register images from different perspectives, and the apparatus includes:

[0026] An acquisition module, configured to establish a training sample set, where the training sample set includes image samples of the target object taken from different perspectives;

[0027] A first registration module, configured to perform global registration on the image samples and generate a sparse graph, where the sparse graph includes key feature information in the image samples;

[0028] A second registration module, configured to train the image registration model based on an objective cost function to determine the model parameters of the image registration model, where the objective cost function includes a reference optical flow field, the image registration model registers the image samples based on a multi-layer iterative solution of the optical flow field, and after solving the optical flow field of the current layer, the sparse graph is used to optimize the optical flow field of the current layer as the reference optical flow field for the objective cost function in the next layer of iterative solution.

[0029] A specific embodiment of the present invention also provides an image fusion apparatus, and the technical solution is as follows:

[0030] An image fusion apparatus, including:

[0031] An acquisition module, configured to acquire a set of images to be fused, where the set of images to be fused includes images of the target object taken from different perspectives at different exposure levels;

[0032] A fusion module, configured to fuse images of the target object taken from the same perspective at different exposure levels in the set of images to be fused to obtain intermediate fusion images of each perspective;

[0033] A processing module, based on the image registration model obtained by the above training method, registers the intermediate fusion images of each perspective and then fuses them to obtain the fused image of the target object.

[0034] A specific embodiment of the present invention also provides an electronic device, and the technical solution is as follows:

[0035] An electronic device, comprising:

[0036] At least one processor; and

[0037] A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to execute the training method of the image registration model as described above or the image fusion method as described above.

[0038] A specific embodiment of the present invention further provides a machine-readable storage medium, and the technical solution is as follows:

[0039] A machine-readable storage medium storing executable instructions that, when executed, cause the machine to execute the training method of the image registration model as described above or the image fusion method as described above.

[0040] Compared with the prior art, the present invention includes at least one of the following beneficial technical effects:

[0041] 1. The training method of the image registration model of the present invention trains the image registration model by constructing a target cost function, and at the same time optimizes the current layer optical flow field based on a sparse graph, which can ensure the matching degree of the optical flow field solved by multiple rounds of iteration. Therefore, it can reduce the occurrence of abnormal problems such as artifacts, distortions, and detail loss that may occur during the registration of multiple images due to parallax.

[0042] 2. The image fusion method of the present invention can perform fusion processing and registration processing on multiple pictures containing a target object, and can obtain a high-quality 2D image of the target object with rich details. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 It is a diagram of the implementation environment of the training method of the image registration model and the image fusion method in an embodiment of the present invention;

[0045] Figure 2 It is a flowchart of the training method of the image registration model in an embodiment of the present invention;

[0046] Figure 3 It is a flowchart of the image fusion method in an embodiment of the present invention;

[0047] Figure 4 Schematic layout diagram of a multi-camera system in an embodiment of the present invention;

[0048] Figure 5 Schematic diagram of the steps of an image fusion method in an embodiment of the present invention;

[0049] Figure 6 Module diagram of a training device for an image registration model in an embodiment of the present invention;

[0050] Figure 7 Module diagram of an image fusion device in an embodiment of the present invention;

[0051] Figure 8 It is a hardware structure diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0052] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0053] Against the background of the rapid development of photographic imaging and computational photography technologies, multi-camera imaging systems are becoming increasingly popular in various application fields. A multi-camera system consists of multiple imaging devices, which can collect image information from multiple perspectives and meet diverse application requirements. The embodiments of the present invention focus on registering the images obtained by multiple cameras to better apply them to subsequent fusion steps, aiming to improve the overall quality of the images, enhance the clarity, and achieve a more delicate detailed presentation effect.

[0054] With reference to Figure 4 and Figure 5 , in a specific scenario example, a multi-camera system can photograph a target object from different angles, and then perform step-by-step registration operations on the images from different perspectives: ① globally register the images from different perspectives, ② then perform pixel registration using the image registration model obtained by the image registration model training method of the present application. Finally, fuse the registered multi-perspective images to obtain a single high-definition and detail-rich 2D image.

[0055] Among them, in the embodiments of the present application, the multi-view images for registration can be directly acquired by a multi-camera system from multiple perspectives; alternatively, the images under each perspective can be obtained by HDR fusion of images with the same perspective but different exposure levels, that is, the images under the same perspective are fused into one, and after the fusion of the images with different exposure levels under the same perspective is completed, the registration process is performed on the images.

[0056] Referring to Figure 1 , a schematic diagram of an implementation environment provided by an exemplary embodiment of the present invention is shown. The implementation environment includes a terminal and a server. Among them, data communication is carried out between the terminal and the server through a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network. The image fusion method disclosed in the present invention can be executed by the server. Correspondingly, the image registration model can also be deployed in the server.

[0057] Alternatively, the terminal and the server can cooperate to run the image fusion method provided in the embodiments of the present application to complete the fusion of images. For example, the terminal can be used to perform global registration on images from different perspectives, or the image registration model can be deployed on the terminal to execute the complete image registration process.

[0058] In a system architecture where the terminal can provide matching computing power, the image fusion method disclosed in the present application can also be directly executed by the terminal. Correspondingly, the image registration model can also be deployed on the terminal.

[0059] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0060] Referring to Figure 2 , a training method of the image registration model in an embodiment of the present invention is introduced. Among them, the image registration model is used to register images from different perspectives. The training method of the image registration model includes:

[0061] S101. Establish a training sample set, and the training sample set includes image samples of a target object taken from different perspectives. Specifically, image samples of the target object taken from different perspectives can be obtained by synchronously shooting with different cameras at the same time.

[0062] S102. Perform global registration on the image samples and generate a sparse graph, where the sparse graph includes the key feature information in the image samples. In this embodiment, the features of the image samples can be extracted based on ORB (Oriented FAST and Rotated BRIEF), and the key feature information can be screened based on the RANSAC (Random Sample Consensus) robust matching algorithm to obtain the sparse graph. Among them, ORB is an algorithm for fast feature point extraction and description, and the RANSAC robust matching algorithm is a robust method for estimating the parameters of a mathematical model through random sampling and iteration, which can effectively eliminate the influence of outliers (noise). Based on the ORB algorithm and the RANSAC robust matching algorithm, the sparse graph can be effectively generated.

[0063] Exemplarily, in ORB feature extraction, the FAST algorithm can be used to detect the key points in the image. By comparing the gray values of the pixel points with those of their surrounding neighborhood pixels, the corner points can be quickly found; then the Intensity Centroid method is used to calculate the direction of the key points to make the features rotation-invariant; subsequently, a set of pixel pairs is selected around the key points, and their gray values are compared to generate a binary descriptor; finally, the ORB features extracted from the two images are matched, and usually the Hamming Distance is used as the similarity metric. In the RANSAC algorithm matching, a minimum sample set can be randomly selected from the matching point pairs (for example, for homography matrix estimation, at least 4 pairs of matching points are required), the model parameters are estimated using the selected sample set, and the errors of all matching point pairs under the current model are calculated. The point pairs with errors less than the threshold are marked as inliers, and the rest are outliers; the above process is repeated multiple times, and the model with the largest number of inliers is selected as the final result, and the model parameters can be re-estimated using all inliers to further improve the accuracy. In the generation of the sparse graph, the inlier pairs after RANSAC screening can be retained, and the outlier pairs can be excluded to determine the key points; then the screened key point pairs are used as the nodes of the sparse graph, and the matching relationship is used as the edge.

[0064] S103. Train the image registration model based on the target cost function to determine the model parameters of the image registration model, where the target cost function includes the reference optical flow field. The image registration model is based on multi-layer iterative solution of the optical flow field to register the image samples, and after solving the optical flow field of the current layer, the sparse graph is used to optimize the optical flow field of the current layer as the reference optical flow field of the target cost function in the next layer of iterative solution. The image registration model is used to perform local registration on the globally registered image samples.

[0065] In this embodiment, the optical flow field of the image sample can be obtained based on the multi-resolution scale pyramid strategy of optical flow. Among them, the multi-resolution scale pyramid strategy of optical flow is a method for optimizing optical flow estimation by constructing an image pyramid (from low resolution to high resolution). It first calculates the rough optical flow on the low-resolution image and then gradually refines it to the high-resolution layer, thereby improving the calculation efficiency and the accuracy of motion estimation.

[0066] Specifically, the target cost function includes at least two of the luminance consistency sub-term, the gradient consistency sub-term, and the regularization sub-term; the luminance consistency sub-term represents the luminance error between image samples; the gradient consistency sub-term represents the gradient error between image samples; the regularization sub-term is used to impose regularization on the gradient of the optical flow field. It can be understood that the target cost function can include only the luminance consistency sub-term and the gradient consistency sub-term, or only the luminance consistency sub-term and the regularization sub-term, or only the gradient consistency sub-term and the regularization sub-term, or include all three of the luminance consistency sub-term, the gradient consistency sub-term, and the regularization sub-term.

[0067] The above is the general concept of the target cost function. In this embodiment, the target cost function is further elaborated based on specific parameters, which is not a limitation on the target cost function in this embodiment. Specifically as follows:

[0068] In a specific embodiment, the target cost function includes three terms: the luminance consistency sub-term, the gradient consistency sub-term, and the regularization sub-term. In this embodiment, the image sample may specifically include a target image and a floating image to be registered with the target image.

[0069]

[0070] Among them, ω I 、 ω G and ω R are the weight coefficients of the luminance, gradient, and regularization terms respectively; x represents the position coordinates of each pixel in the image; x = (x, y), where x and y are the horizontal and vertical coordinates of the pixel in the image. F(x) represents the optical flow field, specifically representing the pixel displacement change amount during the registration of the image sample; I r (x) represents the floating image I r at the pixel position x. I t (x + F(x)) represents in the target image I t the pixel value after the pixel at the pixel position x is displaced by the optical flow field F(x).

[0071] Among them, the optical flow field of the current layer is optimized using a sparse graph, which specifically includes: generating a guiding field based on the sparse graph, where the guiding field includes displacement change information of the key features of the sparse graph; constructing a regularization term based on the guiding field and the optical flow field of the current layer to perform a smoothing constraint on the optical flow field of the current layer.

[0072]

[0073] Among them, denotes the optical flow field F corresponding to the solution of the minimum value of E (F); F guide (x) is the guiding field provided by the sparse graph; γ is the smoothness control parameter; SparseMap represents the sparse graph.

[0074] Among them, the above function can optimize the optical flow field obtained by solving the current layer and be used as the initial reference optical flow field for solving the optical flow field of the next layer. Specifically, in this embodiment, the optical flow field of the current layer is solved by finding the solution of the minimum value of the objective cost function, and the value of the optical flow field of the current layer is optimized based on the guiding field provided by the sparse graph. Finally, the optimized value of the optical flow field is used as the initial value for iteratively solving the optical flow field of the next layer to gradually complete the precise optimization of the optical flow field and ensure the output result of the model.

[0075] In this embodiment, the constraint of the guiding field provided by the sparse graph can effectively solve the problem caused by the incorrect estimation of the optical flow field in the sparse feature region and improve the robustness of registration. Among them, the sparse region represents the region with few features and unclear features in the image sample.

[0076] The training method of the image registration model in this embodiment can effectively reduce abnormal problems such as artifacts and distortions in image registration caused by large perspective differences or insufficient local features of the image through the strategies of global registration and local registration, so as to achieve pixel-level precise alignment of the whole image; at the same time, it can also significantly improve the accuracy, versatility, and robustness of registration.

[0077] Refer to Figure 3 to introduce the image fusion method in an embodiment of the present invention. It should be noted in advance that the image registration model involved in the image fusion method of this embodiment can be obtained based on the above training method of the image registration model.

[0078] The following specifically introduces the image fusion method in an embodiment of the present invention, which specifically includes:

[0079] S201. Obtain a set of images to be fused, where the set of images to be fused includes images of the target object taken at different perspectives and with different exposure degrees.

[0080] In this embodiment, images of a target object taken at different exposure levels from different perspectives can be obtained by shooting with a multi-camera system. The layout of the multi-camera can capture the details of the target image from different perspectives. With reference to Figure 4 , the number, position, distance from the target object, and angle of the cameras in this embodiment can be adjusted according to the actual situation to ensure that the detailed information of different perspectives of the target image is presented. Among them, a single camera can synchronously shoot at multiple exposure levels (such as low, medium, and high) at the same time to capture the details of the target object under different lighting conditions.

[0081] S202. Fuse the images of the target object taken at different exposure levels from the same perspective in the image set to be fused to obtain the intermediate fused image for each perspective. In this embodiment, the images taken at different exposure levels from the same perspective can be fused through an image fusion model, such as by performing fusion processing on the images based on a deep learning model. It can be understood that other methods applicable to HDR fusion can also be used for the fusion of the image set to be fused in this embodiment.

[0082] With reference to Figure 5 , a sequence of LDR pictures with different exposure levels for each perspective can be obtained through the multi-camera system. Inputting the LDR picture sequence into the image fusion model can fuse to obtain an HDR image, that is, the intermediate fused image for each perspective can be obtained to ensure the quality of the images obtained from each perspective. Among them, LDR (Low Dynamic Range) refers to a low dynamic range, and the brightness and color range of its images or videos are limited, which is suitable for ordinary display devices. HDR (High Dynamic Range) refers to a high dynamic range, which provides a wider brightness and color range and can present more details.

[0083] S203. Based on the image registration model obtained by the above image registration model training method, register the intermediate fused images for each perspective and then fuse them to obtain the fused image of the target object.

[0084] With reference to Figure 5 , register the obtained intermediate fused images from multiple perspectives through the image registration model, which can align the features in the images from different perspectives and ensure the quality of subsequent image fusion. Among them, in this embodiment, pixel-level registration of the images can be completed through the cooperation of global registration and local registration, which can solve problems such as distortion and artifacts that occur when the perspective differences are large, and perform excellently when processing three-dimensional objects or images with insufficient local features.

[0085] In this embodiment, in the HDR fusion step of image samples under different exposure levels, a weighted average fusion strategy is adopted, and the maximum value fusion strategy is adopted for the fusion of intermediate fusion images of each view. The reason is that for the HDR fusion of image samples under different exposure levels, an image with appropriate exposure needs to be obtained, so the weighted average fusion strategy is adopted. For the fusion of intermediate fusion images of each view, the optimal details, textures, and unique information in all views need to be integrated to finally form a clear and detailed high-quality image, so the maximum value fusion strategy is adopted.

[0086] The image fusion method in an embodiment of the present invention overcomes the defect that the prior art cannot effectively utilize the unique detail information of different views in multi-view image fusion, and proposes a novel multi-view image fusion method. Different from traditional methods that only focus on expanding the field of view or depth information extraction, this method maximally retains and enhances the detail information under different views through carefully designed image acquisition, image registration, and image fusion strategies, thereby significantly improving the quality and clarity of the final image.

[0087] At the same time, compared with the prior art, the image fusion method in the present invention coordinates image fusion and image registration, and makes targeted improvements and optimizations for multi-view scenes, especially complex situations such as large parallax, further improving the authenticity and consistency of the fused image.

[0088] Refer to Figure 6 , and introduce the training device of the image registration model in an embodiment of the present invention. The image registration model is used to register images of different views, and the device includes an acquisition module 301, a first registration module 302, and a second registration module 303.

[0089] The acquisition module 301 is used to establish a training sample set, and the training sample set includes image samples of a target object taken from different views; the first registration module 302 is used to perform global registration on the image samples and generate a sparse map, where the sparse map includes key feature information in the image samples; the second registration module 303 is used to train the image registration model based on a target cost function to determine the model parameters of the image registration model, where the target cost function includes a reference optical flow field, and the image registration model registers the image samples based on multi-layer iterative solution of the optical flow field, and after solving the optical flow field of the current layer, uses the sparse map to optimize the optical flow field of the current layer as the reference optical flow field of the target cost function in the next layer of iterative solution.

[0090] In an optional embodiment, optimizing the optical flow field of the current layer using the sparse map specifically includes: generating a guiding field based on the sparse map, where the guiding field includes displacement change information of key features of the sparse map; constructing a regularization term based on the guiding field and the optical flow field of the current layer to perform smoothing constraints on the optical flow field of the current layer.

[0091] In an alternative embodiment, the target cost function includes at least two of a luminance consistency sub-term, a gradient consistency sub-term, and a regularization sub-term; the luminance consistency sub-term represents the luminance error between image samples; the gradient consistency sub-term represents the gradient error between image samples; and the regularization sub-term is used to apply regularization to the gradient of the optical flow field.

[0092] In an alternative embodiment, the image registration model is used to perform local registration on the globally registered image samples.

[0093] In an alternative embodiment, an optical flow field of the image samples is obtained based on a multi-resolution scale pyramid strategy of optical flow; and / or, features of the image samples are extracted based on ORB, and key feature information is screened based on the RANSAC robust matching algorithm to obtain a sparse map.

[0094] Refer to Figure 7 , and an image fusion device in an embodiment of the present invention is introduced. In this embodiment, the image fusion device includes an acquisition module 401, a fusion module 402, and a processing module 403.

[0095] The acquisition module 401 is configured to acquire a set of images to be fused, where the set of images to be fused includes images of a target object captured at different perspectives and different exposure levels; the fusion module 402 is configured to fuse the images of the target object captured at the same perspective and different exposure levels in the set of images to be fused to obtain intermediate fusion images for each perspective; and the processing module 403 is configured to register the intermediate fusion images for each perspective and then fuse them to obtain a fusion image of the target object.

[0096] In an alternative embodiment, the processing module 403 also fuses the registered intermediate fusion images for each perspective based on the image registration model obtained by the above training method.

[0097] In an alternative embodiment, the images of the target object captured at the same perspective and different exposure levels in the set of images to be fused are fused with a weighted average fusion strategy; and the intermediate fusion images for each perspective are fused with a maximum value fusion strategy.

[0098] As described above with reference to Figures 1 to 5 , a training method for an image registration model and an image fusion method according to an embodiment of this specification have been described. The details mentioned in the above description of the method embodiments also apply to the image fusion device of the embodiments of this specification. The above image fusion device can be implemented in hardware, or in software, or in a combination of hardware and software.

[0099] Refer to Figure 8 , which shows a hardware structure diagram of an electronic device according to an embodiment of this specification. As Figure 8As shown, the electronic device 50 may include at least one processor 51, a memory 52 (such as a non-volatile memory), a memory 53, and a communication interface 54, and the at least one processor 51, the memory 52, the memory 53, and the communication interface 54 are connected together via an internal bus 55. The at least one processor 51 executes at least one computer-readable instruction stored or encoded in the memory 52.

[0100] It should be understood that the computer-executable instructions stored in the memory 52, when executed, cause the at least one processor 51 to perform the various operations and functions described above in the respective embodiments of this specification in conjunction with Figures 1 to 5 the descriptions.

[0101] In the embodiments of this specification, the electronic device 50 may include, but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, and so on.

[0102] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which when executed by the machine, cause the machine to perform the various operations and functions described above in the respective embodiments of this specification in conjunction with Figures 1 - 5 the descriptions. Specifically, a system or device equipped with a readable storage medium may be provided, on which software program code for implementing the functions of any one of the above embodiments is stored, and the computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.

[0103] In this case, the program code read from the readable medium itself can implement the functions of any one of the above embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0104] Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0105] Those skilled in the art should understand that the various embodiments disclosed above can be variously deformed and modified without departing from the essence of the invention. Therefore, the protection scope of this specification should be defined by the appended claims.

[0106] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined as required. The device structures described in the above-mentioned embodiments can be physical structures or logical structures. That is, some units may be implemented by the same physical entity, or some units may be implemented separately by multiple physical entities, or some components in multiple independent devices may be jointly implemented.

[0107] In the above-mentioned embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor can include permanent dedicated circuits or logic (such as a dedicated processor, FPGA, or ASIC) to perform corresponding operations. The hardware unit or processor can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to perform corresponding operations. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0108] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration", and does not mean "preferred" or "advantageous" compared to other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0109] The above description of the present disclosure is provided to enable any ordinary person skilled in the art to implement or use the present disclosure. Various modifications to the present disclosure are obvious to those of ordinary skill in the art, and the general principles corresponding herein can also be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope that conforms to the principles and novel features disclosed herein.

[0110] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A training method for an image registration model, characterized in that, The image registration model is used to register images from different perspectives. The method includes: Establishing a training sample set, where the training sample set includes image samples of a target object taken from different perspectives; Performing global registration on the image samples and generating a sparse graph, where the sparse graph includes key feature information in the image samples; Training the image registration model based on an objective cost function to determine the model parameters of the image registration model. The objective cost function includes a reference optical flow field. The image registration model registers the image samples based on multi-layer iterative solution of the optical flow field. After solving the optical flow field of the current layer, the sparse graph is used to optimize the optical flow field of the current layer as the reference optical flow field for the next-layer iterative solution in the objective cost function; Using the sparse graph to optimize the optical flow field of the current layer, specifically including: Generating a guiding field based on the sparse graph, where the guiding field includes displacement change information of the key features of the sparse graph; Constructing a regularization term based on the guiding field and the optical flow field of the current layer to perform smoothing constraint on the optical flow field of the current layer.

2. The training method of the image registration model according to claim 1, wherein The objective cost function includes at least two of a brightness consistency sub-term, a gradient consistency sub-term, and a regularization sub-term; the brightness consistency sub-term represents the brightness error between image samples; the gradient consistency sub-term represents the gradient error between image samples; the regularization sub-term is used to impose regularization on the gradient of the optical flow field.

3. The training method of the image registration model according to claim 1, wherein The image registration model is used to perform local registration on the image samples after global registration.

4. The training method of the image registration model according to claim 1, wherein Obtaining the optical flow field of the image samples based on a multi-resolution scale pyramid strategy of optical flow; and / or Extracting features of the image samples based on ORB, and screening the key feature information based on the RANSAC robust matching algorithm to obtain the sparse graph.

5. An image fusion method, characterized in that, Including: Obtaining a set of images to be fused, where the set of images to be fused includes images of a target object taken from different perspectives with different exposure degrees; Fusing the images of the target object taken from the same perspective with different exposure degrees in the set of images to be fused to obtain intermediate fused images for each perspective; Registering the intermediate fused images for each perspective based on the image registration model obtained by the training method according to any one of claims 1 to 4, and then fusing them to obtain the fused image of the target object.

6. The image fusion method according to claim 5, wherein The method specifically includes: Fusing the images of the target object taken from the same perspective with different exposure degrees in the set of images to be fused by a weighted average fusion strategy; and Fusing the intermediate fused images for each perspective by a maximum value fusion strategy.

7. An apparatus for training an image registration model, characterized in that, The image registration model is used to register images from different perspectives. The device includes: An acquisition module, configured to establish a training sample set, where the training sample set includes image samples of a target object taken from different perspectives; A first registration module, configured to perform global registration on the image samples and generate a sparse graph, where the sparse graph includes key feature information in the image samples; A second registration module, configured to train an image registration model based on a target cost function to determine model parameters of the image registration model, wherein the target cost function includes a reference optical flow field, and the image registration model is based on multi-layer iterative solution of an optical flow field to register an image sample, and after solving the optical flow field of the current layer, the sparse graph is used to optimize the optical flow field of the current layer as the reference optical flow field of the target cost function in the next-layer iterative solution.

8. An image fusion device, characterized in that, Comprising: An acquisition module, configured to acquire a set of images to be fused, wherein the set of images to be fused includes images of a target object taken at different perspectives and with different exposure levels; A fusion module, configured to fuse the images of the target object taken at the same perspective and with different exposure levels in the set of images to be fused to obtain intermediate fused images for each perspective; A processing module, configured to register the intermediate fused images for each perspective based on the image registration model obtained by the training method according to any one of claims 1 to 4, and then fuse them to obtain a fused image of the target object.

9. An electronic device, characterized in that, Comprising: At least one processor; And A memory, storing instructions which, when executed by the at least one processor, cause the at least one processor to execute the training method of the image registration model according to any one of claims 1 to 4 or the image fusion method according to any one of claims 5 to 6.

10. A machine-readable storage medium, characterized in that, It stores executable instructions which, when executed, cause the machine to execute the training method of the image registration model according to any one of claims 1 to 4 or the image fusion method according to any one of claims 5 to 6.

Images