Evaluation method, terminal and storage medium of image registration algorithm

Abstract

The embodiment of the present application discloses an evaluation method, a terminal and a storage medium for an image registration algorithm. The evaluation method for the image registration algorithm includes: acquiring a first original image and a second original image through a same shooting device; performing image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; performing image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; obtaining an evaluation result based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an evaluation method, terminal and storage medium for an image registration algorithm. Background Art

[0002] Image registration is the process of matching and superimposing two or more images acquired at different times, with different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.). The accuracy of the registration results obtained by different registration methods will vary, so the pros and cons of different registration algorithms can be measured by evaluating the registration results. Therefore, it is extremely important to accurately evaluate the image registration algorithm.

[0003] However, the noise, moving objects and different illuminations in the shooting process often affect the evaluation of the registration effect, resulting in the inability to accurately quantitatively evaluate the image registration algorithm, reducing the accuracy of the evaluation, and thus being unable to accurately measure the pros and cons of the global image registration algorithm, resulting in poor intelligence. Summary of the invention

[0004] The embodiments of the present application provide an evaluation method, terminal and storage medium for an image registration algorithm, which can improve the accuracy of the evaluation, and further accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating an image registration algorithm, the method comprising:

[0007] Acquiring a first original image and a second original image by using the same photographing device;

[0008] Based on a preset interference pattern, image processing is performed on the first original image and the second original image to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information;

[0009] Performing image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information;

[0010] An evaluation result is obtained according to the displacement information, the first original image, and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0011] In a second aspect, an embodiment of the present application provides a terminal, the terminal comprising: an acquisition unit and a generation unit,

[0012] The acquisition unit is used to acquire the first original image and the second original image through the same shooting device;

[0013] The generating unit is used to perform image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information;

[0014] The acquisition unit is further used to perform image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; and obtain an evaluation result based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0015] In a third aspect, an embodiment of the present application provides a terminal, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the evaluation method of the image registration algorithm as described above is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which is applied to a terminal, and when the program is executed by a processor, the evaluation method of the image registration algorithm as described above is implemented.

[0017] The embodiment of the present application provides an evaluation method, terminal and storage medium for an image registration algorithm, wherein the terminal obtains a first original image and a second original image through the same shooting device; based on a preset interference pattern, the first original image and the second original image are subjected to image processing to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; the reference image and the image to be registered are subjected to image registration processing using the algorithm to be evaluated to obtain displacement information; an evaluation result is obtained based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated. It can be seen that in the embodiment of the present application, when evaluating the global image registration algorithm, controllable test data, i.e., the reference image and the image to be registered, can be obtained by simulating the interference information during shooting based on the first original image and the second original image without interference factors through the preset interference pattern, and then after the reference image and the image to be registered are subjected to registration processing according to the algorithm to be evaluated, the registration result is evaluated using the image data without interference factors, thereby obtaining an accurate evaluation result. That is to say, in this application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during taking pictures, thereby improving the accuracy of the evaluation, and then being able to accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the implementation process of the evaluation method for the image registration algorithm Figure 1 ;

[0019] Figure 2 A schematic diagram for obtaining a first original image and a second original image;

[0020] Figure 3 Schematic diagram of the implementation process of the evaluation method for the image registration algorithm Figure 2 ;

[0021] Figure 4 Image processing flow based on preset interference pattern Figure 1 ;

[0022] Figure 5 Image processing flow based on preset interference pattern Figure 2 ;

[0023] Figure 6 Illustration of simulated motion processing Figure 1 ;

[0024] Figure 7 Illustration of simulated motion processing Figure 2 ;

[0025] Figure 8 Illustration of simulated motion processing Figure 3 ;

[0026] Fig. 9 Generate boxes for evaluation images Figure 1 ;

[0027] Fig.10 Generate boxes for evaluation images Figure 2 ;

[0028] Fig.11 is a schematic diagram of the original image information;

[0029] Fig.12 A schematic diagram for evaluating image information;

[0030] Fig.13 The terminal structure is shown in Figure 1. Figure 1 ;

[0031] Fig.14 The terminal structure is shown in Figure 1. Figure 2 . DETAILED DESCRIPTION

[0032] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for the convenience of description, only the parts related to the related applications are shown in the accompanying drawings.

[0033] Image registration is a common method in image processing. In the shooting of multiple frames, due to the movement of the shooting device, the captured images will have slight differences in content. In the multi-frame image processing, it is often necessary to first perform global registration on the images before performing various other processing. In other words, image registration is the process of matching and superimposing two or more images acquired at different times, different sensors (imaging equipment) or different conditions (weather, illumination, camera position and angle, etc.). It has been widely used in remote sensing data analysis, computer vision, image processing and other fields.

[0034] Specifically, the technical solution for image registration is mainly as follows: first, feature extraction is performed on the two images to obtain feature points; matching feature point pairs are found by performing similarity measurement; then image space coordinate transformation parameters are obtained through the matching feature point pairs; and finally, image registration is performed based on the coordinate transformation parameters.

[0035] Image registration methods are mainly divided into three categories: pixel-based registration methods, feature-based registration methods, and model-based registration methods. Among them, the first two methods are global image registration techniques, which require the assumption that changes in objects in the image are often caused by motion. The third method is suitable for local nonlinear deformation correction of object supports in the image. This distortion is usually caused by the nonlinearity of spatial encoding of the imaging system.

[0036] The accuracy of the registration results obtained by different registration methods will be different, so the pros and cons of different registration algorithms can be measured by evaluating the registration results. It can be seen that it is extremely important to accurately evaluate the image registration algorithm.

[0037] At present, when evaluating image registration algorithms, the root mean square error (RMSE) and mean square error (RMSE) between the motion vector calculated by the registration algorithm and its ground truth, or the structural similarity index (SSIM) and the sum of the absolute pixel difference between the registered image and the reference image are usually used as evaluation criteria.

[0038] Although the above indicators can reflect the registration effect of the image registration algorithm to a certain extent. However, the noise, moving objects and different illuminations in the shooting process often affect the evaluation of the registration effect, and it is difficult to accurately and quantitatively evaluate the image registration algorithm. Specifically, for the global registration of continuous shooting or video images, it is difficult to accurately fit through a certain transformation (such as affine transformation or homography transformation, etc.) due to the existence of perspective phenomenon, which makes it difficult to obtain the true value (ground truth) of the pixel movement in the global registration. If the structural similarity or the sum of the absolute value difference of the pixels between the registered image and the reference image is used to evaluate the effect, it will be interfered by different brightness or local moving objects and noise.

[0039] In summary, due to the influence of noise, moving objects and different illumination during the shooting process, it is difficult to make an accurate quantitative evaluation of the registration algorithm of continuous shooting or video images. In order to overcome the above-mentioned defects, the evaluation method of an image registration algorithm proposed in the present application can be based on the first original image and the second original image without interference factors. The interference information during shooting can be simulated by a preset interference mode to obtain controllable test data, that is, a reference image and an image to be registered, and then after the reference image and the image to be registered are registered according to the algorithm to be estimated, the image data without interference factors is used to evaluate the registration result, so as to obtain an accurate evaluation result. That is to say, in the present application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during shooting, so as to improve the accuracy of the evaluation, and then accurately measure the advantages and disadvantages of the global image registration algorithm, and enhance the intelligence of the terminal.

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] An embodiment of the present application provides an evaluation method for an image registration algorithm, and the evaluation method for an image registration algorithm is applied in a terminal. Figure 1 Schematic diagram of the implementation process of the evaluation method for the image registration algorithm Figure 1 ,like Figure 1 As shown, in an embodiment of the present application, the method for a terminal to evaluate an image registration algorithm may include the following steps:

[0042] Step 101: Acquire a first original image and a second original image through the same photographing device.

[0043] In an embodiment of the present application, the terminal may first obtain the first original image and the second original image through the same shooting device. The first original image and the second original image may be acquired by the same shooting device, in the same shooting environment, and using the same shooting parameters, that is, the first original image and the second original image may simulate two frames of images taken in a continuous shooting or two consecutive frames of images in a video.

[0044] It is understood that, in the embodiment of the present application, the first original image and the second original image can be used to generate evaluation data for evaluating the image registration algorithm, and the first original image and the second original image can also be used to evaluate the image registration algorithm to be evaluated. The first original image and the second original image can be original image files, that is, RAW images.

[0045] RAW image files contain the processed data from the image sensor of a digital camera, scanner, or film scanner. They are so named because they have not yet been processed, printed, or used for editing. Typically, RAW images have a wide gamut of internal colors, can be accurately adjusted, and can be modified with some simple changes before being converted to a file format such as Tagged Image File Format (TIFF) or Joint Photographic Experts Group (JPEG) for storage, printing, or further processing. RAW image files, sometimes called digital negatives, play the same role as film negatives, which are not used directly as images, but create an image containing all the information.

[0046] It should be noted that in the embodiments of the present application, the terminal can be any device with communication and storage functions, such as: tablet computers, mobile phones, e-readers, remote controls, personal computers (PCs), laptops, vehicle-mounted devices, Internet TVs, wearable devices, and other devices.

[0047] It can be understood that, in the embodiments of the present application, the terminal may be configured with a photographing device, so that the first original image and the second original image can be acquired by using the photographing device.

[0048] Further, in an embodiment of the present application, when the terminal acquires the first original image and the second original image through the same shooting device, the terminal may first acquire the first image group at the first position through the same shooting device, and then acquire the second image group at the second position. Next, the terminal may continue to determine the first original image according to the first image group, and determine the second original image according to the second image group.

[0049] It can be understood that, in the embodiment of the present application, the distance between the first position for shooting the first image group and the second position for shooting the second image group belongs to the preset offset range. Specifically, since the first original image and the second original image are used to generate evaluation data for evaluating the image registration algorithm, that is, the first original image and the second original image can be used to simulate two consecutive frames of images, the moving distance between the first original image and the second original image cannot be too large, and therefore, the distance between the first position and the second position can be limited to not exceed the preset offset range.

[0050] It should be noted that, in the present application, the first image group includes at least one frame of original image captured at the first position, and correspondingly, the first image group includes at least one frame of original image captured at the second position.

[0051] Exemplarily, in an embodiment of the present application, after acquiring the first image group and the second image group, the terminal may respectively perform averaging processing on the first image group and the second image group to obtain the first original image and the second original image, thereby eliminating the influence of image noise and ensuring that the difference between the first original image and the second original image is only caused by the movement of the shooting device.

[0052] For example, in this application, Figure 2 To obtain a schematic diagram of the first original image and the second original image, the terminal can be fixed to a tripod and a gimbal. The settings of the tripod and the gimbal are as follows: Figure 2, the terminal performs constant exposure continuous shooting of the target scene at the first position to obtain the first image group, then moves the tripod and randomly rotates the gimbal and fixes the terminal again, and then uses the same parameters (auto focus AF, auto exposure AE, automatic white balance AWB, sensitivity ISO, etc.) to perform constant exposure continuous shooting of the target scene at the second position to complete the shooting of the second image group. It should be noted that the moving distance of the tripod and the gimbal should be limited, that is, the distance between the first position and the second position does not exceed the preset offset range, so that the slight movement of the terminal can be simulated. The spatial displacement and rotation amount of the shooting device can be recorded for the evaluation of the subsequent algorithm. Among them, the target scene should be in a static state, and try to ensure that the difference between the first image group and the second image group is only caused by the movement of the terminal between the first position and the second position. During the shooting process, the remote control shooting method can be used to ensure the stability of the terminal. Specifically, the shooting data in the first image group and the second image group need to be recorded in RAW format, that is, linear domain image data, so that the data can be additionally processed (adding noise, adjusting the image grayscale, etc.). Furthermore, the terminal performs average processing on the first image group and the second image group respectively to obtain two new original images, namely, the first original image and the second original image, which can be used to evaluate the image registration algorithm to be evaluated.

[0053] Step 102: Based on a preset interference pattern, perform image processing on the first original image and the second original image to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information.

[0054] In an embodiment of the present application, after acquiring the first original image and the second original image through the same shooting device, the terminal can perform image processing on the first original image and the second original image respectively based on a preset interference pattern, thereby generating a reference image and an image to be registered.

[0055] It should be noted that in the embodiments of the present application, in the actual processing of image registration, there may be many interference situations that affect the accuracy of the registration result, such as noise during shooting, different illuminations, moving objects, etc. Therefore, before using the algorithm to be evaluated for image registration, it is necessary to select and add interference information to the first original image and the second original image of the two frames of simulated continuous shooting or the two consecutive frames of images in the video, so as to generate more realistic simulation data, namely the reference image and the image to be registered.

[0056] It should be noted that, in the embodiments of the present application, the interference information may include simulated noise information, simulated light source information, simulated exposure information and simulated motion information.

[0057] It is understandable that, in an embodiment of the present application, before the terminal performs image processing on the first original image and the second original image based on the preset interference pattern, it is necessary to first select the interference to be added from the interference information and determine the setting parameters corresponding to the interference to be added, and then the preset interference pattern can be determined according to the interference to be added and the setting parameters. That is to say, in the present application, when the terminal adds the interference information, it can selectively simulate various situations that may be encountered during continuous shooting, and therefore, the generated preset interference pattern can correspondingly include at least one of simulated noise processing, simulated light source processing, simulated exposure processing, and simulated motion processing.

[0058] Furthermore, in an embodiment of the present application, when the terminal selectively adds interference information to the first original image and the second original image based on a preset interference pattern, it also performs image signal processing on the first original image and the second original image based on an image signal processing (ISP) process, and finally generates a reference image and an image to be registered.

[0059] It can be understood that in the present application, the reference image and the image to be registered generated by the terminal are the first original image and the second original image after bit compression. Specifically, the terminal can determine the processed, bit-compressed first original image as the image to be registered and the processed, bit-compressed second original image as the reference image, or determine the processed, bit-compressed second original image as the image to be registered and the processed, bit-compressed first original image as the reference image. No specific limitation is made here.

[0060] Exemplarily, in the present application, the reference image and the image to be registered can be 8-bit original images. Accordingly, the reference image and the image to be registered can also simulate two frames of images taken continuously or two consecutive frames of images in a video. Specifically, the reference image and the image to be registered can be used as input data for the algorithm to be evaluated.

[0061] Exemplarily, in an embodiment of the present application, when simulating noise processing, the terminal may use a preset noise model to increase the noise signal to simulate a picture noise effect of a certain captured image.

[0062] Illustratively, in an embodiment of the present application, when simulating light source processing, the terminal may perform local brightness adjustment to simulate the image effect of an artificial light source (such as a flash).

[0063] Exemplarily, in an embodiment of the present application, when simulating exposure processing, the terminal can simulate different exposure shooting by adjusting the gain.

[0064] Exemplarily, in an embodiment of the present application, when simulating motion processing, the terminal may use a mask image to add a moving object to simulate motion blur of the moving object in photographing.

[0065] It should be noted that in an embodiment of the present application, when the terminal determines the preset interference mode, the interference information and setting parameters corresponding to the first original image should be consistent with the interference information and setting parameters corresponding to the second original image (except for the simulated motion information), that is, the interference simulation is performed synchronously on the first original image and the second original image.

[0066] It can be understood that in the embodiment of the present application, the terminal performs image signal processing on the first original image and the second original image, which may specifically include automatic white balance, gama correction, and noise reduction processing.

[0067] Step 103: perform image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information.

[0068] In an embodiment of the present application, after the terminal performs image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered, the terminal can further perform image registration processing on the reference image and the image to be registered using an algorithm to be evaluated to obtain an image registration result, i.e., displacement information. The algorithm to be evaluated can be any image registration algorithm.

[0069] It should be noted that in an embodiment of the present application, after the terminal obtains the reference image and the image to be registered through image processing, the reference image and the image to be registered can be simultaneously input into the algorithm to be evaluated, and image registration processing can be performed on the reference image and the image to be registered, so that the displacement information between the two can be output.

[0070] It can be understood that, in the embodiment of the present application, the displacement information may be a displacement map of all pixels corresponding to the reference image and the image to be registered, or may be a displacement amount generated using the displacement map.

[0071] Further, in the embodiment of the present application, the displacement information may represent the movement amount of all pixels of the image to be registered in the x-axis and y-axis directions based on all pixels of the reference image.

[0072] Step 104: Obtain an evaluation result according to the displacement information, the first original image, and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0073] In an embodiment of the present application, after the terminal uses the algorithm to be evaluated to perform image registration processing on the reference image and the image to be registered and obtains the displacement information, it can further evaluate the algorithm to be evaluated based on the displacement information, the first original image and the second original image, thereby obtaining an evaluation result.

[0074] It should be noted that, in the embodiments of the present application, the evaluation results can be used to evaluate the registration effect of the algorithm to be evaluated. Exemplarily, in the embodiments of the present application, the evaluation results can be determined using a residual index, wherein the smaller the residual index, the better the registration effect can be considered, and correspondingly, the larger the residual index, the worse the registration effect can be considered.

[0075] Furthermore, in an embodiment of the present application, when evaluating the algorithm to be evaluated, the terminal can first use the displacement information to map the first original image or the second original image, so as to generate a corresponding evaluation image, including the first evaluation image or the second evaluation image, and then continue to determine the registration residual based on the evaluation image and the preset evaluation model, so as to generate the final evaluation result based on the registration residual.

[0076] It can be understood that in an embodiment of the present application, when the terminal uses the displacement information to map the first original image, all pixels in the first original image can be mapped into a first evaluation image according to the displacement information, wherein the first evaluation image is a RAW image, which can be used to perform quantitative evaluation of the algorithm to be evaluated.

[0077] Accordingly, in the present application, when the terminal uses the displacement information to map the second original image, all pixels in the second original image can be mapped into a second evaluation image according to the displacement information, wherein the second evaluation image is a RAW image, which can be used to quantitatively evaluate the algorithm to be evaluated.

[0078] Furthermore, in an embodiment of the present application, when the terminal performs mapping processing according to the displacement information, if the displacement amount corresponding to the displacement information is not an integer, interpolation processing is required after mapping. Specifically, the present application can use the more common bilinear interpolation or bicubic interpolation for difference processing, which is not specifically limited here.

[0079] It should be noted that, in the embodiment of the present application, the preset evaluation model is used to evaluate the registration effect of the algorithm to be evaluated. Exemplarily, in the present application, the terminal can establish a preset evaluation model by using a Laplacian pyramid multi-scale residual method to obtain a registration residual, and generate an evaluation result based on the registration residual.

[0080] It can be understood that in the embodiment of the present application, the first original image and the second original image are captured by the same shooting device, in the same shooting environment, and using the same shooting parameters, that is, the first original image and the second original image have the same conditions (grayscale, noise level, etc.), therefore, the first evaluation image and the second original image after mapping processing also have the same conditions, therefore, the terminal can use the second original image and the first evaluation image to quantitatively evaluate the algorithm to be evaluated; accordingly, the second evaluation image and the first original image after mapping processing also have the same conditions. Therefore, the terminal can use the first original image and the second evaluation image to quantitatively evaluate the algorithm to be evaluated.

[0081] In an embodiment of the present application, through the method proposed in the above steps 101 to 104, the terminal can use the simulation of interference factors to obtain two different sets of image data, including a first original image, a second original image, and a reference image and an image to be registered, wherein the first original image and the second original image are image data without eliminating interference such as noise, local moving objects and illumination changes, which are used to evaluate the algorithm to be evaluated, while the reference image and the image to be registered are image data with interference such as simulated image noise, local moving objects and illumination changes added by a preset interference mode, which are used to test the algorithm to be evaluated. It can be seen that the evaluation method proposed in the present application can generate evaluation data and test data by simulating image noise, local moving objects and illumination changes, etc., and it is precisely because the evaluation data and test data are obtained by the terminal through the addition and removal of interference factors, so these interference factors can be excluded during the evaluation to achieve an accurate and quantitative evaluation of the registration effect. In addition, the present application also proposes a new image registration evaluation indicator, namely the registration residual, which can more comprehensively evaluate the registration effect of the image registration algorithm compared with the traditional indicator.

[0082] An evaluation method for an image registration algorithm proposed in an embodiment of the present application, the terminal obtains a first original image and a second original image through the same shooting device; based on a preset interference pattern, the first original image and the second original image are image processed to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; the reference image and the image to be registered are image registered using the algorithm to be evaluated to obtain displacement information; based on the displacement information, the first original image and the second original image, an evaluation result is obtained; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated. It can be seen that in the embodiment of the present application, when evaluating the global image registration algorithm, the interference information during shooting can be simulated by the preset interference pattern based on the first original image and the second original image without interference factors to obtain controllable test data, i.e., the reference image and the image to be registered, and then after the reference image and the image to be registered are registered according to the algorithm to be evaluated, the registration result is evaluated using the image data without interference factors, so as to obtain an accurate evaluation result. That is to say, in this application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during taking pictures, thereby improving the accuracy of the evaluation, and then being able to accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal.

[0083] Based on the above embodiment, in yet another embodiment of the present application, Figure 3 Schematic diagram of the implementation process of the evaluation method for the image registration algorithm Figure 2 ,like Figure 3 As shown, after the terminal acquires the first original image and the second original image through the same shooting device, that is, after step 101, and before performing image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered, that is, before step 102, the method for the terminal to evaluate the image registration algorithm may also include the following steps:

[0084] Step 105: Select interference to be added from the interference information, and determine setting parameters corresponding to the interference to be added.

[0085] In an embodiment of the present application, the terminal may first select interference to be added from the interference information, and simultaneously determine setting parameters corresponding to the added interference. The interference information may include simulated noise information, simulated light source information, simulated exposure information, and simulated motion information.

[0086] That is to say, in an embodiment of the present application, the terminal can select the simulated noise information, the simulated light source information, the simulated exposure information, and the simulated motion information, respectively, and at the same time, can determine specific setting parameters for the selected interference to be added. Exemplarily, the terminal selects the simulated noise information and the simulated light source information as the interference to be added from the interference information, and at the same time, determines the setting parameters corresponding to the noise signal and the setting parameters corresponding to the brightness adjustment.

[0087] Step 106: Determine a preset interference mode according to the interference to be added and the setting parameters.

[0088] In an embodiment of the present application, after the terminal selects the interference to be added from the interference information and determines the setting parameters corresponding to the added interference, it can further determine the preset interference mode based on the interference to be added and the setting parameters; wherein the preset interference mode includes: at least one of simulated noise processing, simulated light source processing, simulated exposure processing and simulated motion processing.

[0089] It can be understood that in an embodiment of the present application, corresponding to interference information including simulated noise information, simulated light source information, simulated exposure information, and simulated motion information, the preset interference mode may include at least one of simulated noise processing, simulated light source processing, simulated exposure processing, and simulated motion processing. Exemplarily, if the terminal selects simulated noise information and simulated light source information as interference to be added from the interference information, and after determining the corresponding setting parameters, the preset interference mode further generated may include simulated noise processing and simulated light source processing.

[0090] That is to say, in an embodiment of the present application, the terminal can make specific settings for the preset interference mode, including selecting simulation processing of different interferences and parameters corresponding to different processing, so that images taken under unequal exposure or different noise conditions can be simulated through the preset interference mode.

[0091] It should be noted that in the embodiment of the present application, after the terminal determines the preset interference mode according to the interference to be added and the setting parameters, it can perform image processing on the first original image and the second original image based on the preset interference mode. Specifically, the terminal can add interference information and image signal processing to the first original image based on the preset interference mode to generate a first processed image; at the same time, add interference information and image signal processing to the second original image to generate a second processed image.

[0092] Figure 4 Image processing flow based on preset interference pattern Figure 1, the terminal needs to perform image processing on the first original image and the second original image to obtain common 8-bit image data. Specifically, the terminal can simulate commonly used image signal processing solutions. At the same time, in order to simulate various interferences that may be encountered during continuous shooting, in the ISP simulation process, the terminal can choose to add several optional simulated interference methods such as simulated noise processing, simulated light source processing, simulated exposure processing, and simulated motion processing. For example, Figure 4 As shown, the preset interference mode determined by the terminal includes simulated noise processing, simulated exposure processing and simulated motion processing. Based on the preset interference mode, after acquiring the first original image and the second original image through the same shooting device, the terminal can simultaneously perform simulated noise processing (step 201), simulated exposure processing (step 202) and simulated motion processing (step 203) on the first original image and the second original image in sequence, and generate a first processed image and a second processed image after ISP processing (image signal processing, which may specifically include automatic white balance (step 204), gama correction (step 205) and noise reduction processing (step 206), etc.).

[0093] Figure 5 Image processing flow based on preset interference pattern Figure 2 , the terminal needs to perform image processing on the first original image and the second original image to obtain common 8-bit image data. Specifically, the terminal can simulate commonly used image signal processing solutions. At the same time, in order to simulate various interferences that may be encountered during continuous shooting, in the ISP simulation process, the terminal can choose to add several optional simulated interference methods such as simulated noise processing, simulated light source processing, simulated exposure processing, and simulated motion processing. For example, Figure 5 As shown, the preset interference mode determined by the terminal includes simulated noise processing, simulated light source processing, simulated exposure processing and simulated motion processing. Based on the preset interference mode, after acquiring the first original image and the second original image through the same shooting device, the terminal can simultaneously perform simulated noise processing (step 301), simulated light source processing (step 302), and simulated exposure processing (step 303) on the first original image and the second original image in sequence, and after ISP processing (image signal processing, which may specifically include automatic white balance (step 304), gama correction (step 305) and noise reduction processing (step 306), etc.), the image data corresponding to the second original image generates a second processed image; and the terminal continues to perform simulated motion processing (step 307) on the image data corresponding to the first original image, and finally generates a first processed image.

[0094] It can be seen from the figures that in the present application, since the simulated motion processing is used to simulate the interference of moving objects during shooting, it is possible to choose to perform simulated motion processing only on the first original image or the second original image. However, for other interference information other than the simulated motion information, when the terminal determines the preset interference mode, the interference information and setting parameters corresponding to the first original image should be consistent with the interference information and setting parameters corresponding to the second original image, that is, interference simulation is performed synchronously on the first original image and the second original image.

[0095] It should be noted that in an embodiment of the present application, after the terminal performs image processing on the first original image and the second original image based on a preset interference pattern to generate a first processed image and a second processed image, the first processed image can be determined as the image to be registered and the second processed image as the reference image, or the first processed image can be determined as the reference image and the second processed image as the image to be registered.

[0096] It can be understood that in the embodiments of the present application, the compressed reference image and the image to be registered can be input into the algorithm to be evaluated as actual test data, while the unprocessed first original image and the second original image can be retained for evaluating the registration effect of the algorithm to be evaluated.

[0097] Furthermore, in an embodiment of the present application, when simulating noise processing, the terminal may use a preset noise model to increase the noise signal to simulate the image noise effect of a certain captured image. The preset noise model corresponds to the image sensor for image capture. For example, in the present application, the preset noise model conforms to the normal distribution and can be expressed as follows:

[0098] y~N(μ=x,σ 2 =λ read +λ shot X) (1)

[0099] Where X is the strength of the real signal. For a specific camera, the noise can be mainly divided into two types: read noise and shot noise. Referring to the following formula, the model parameter λ corresponding to the read noise is read The model parameter λ corresponding to the shot noise shot All can be measured in practice:

[0100]

[0101] λ shot =g a σ a (3)

[0102] Among them, gd is the digital gain of the camera signal, and ga is the analog gain of the camera signal.

[0103] Through the above formulas (1) to (3), the terminal can simulate the image noise effect of an image captured by a certain specific shooting device. In other words, the terminal can estimate the noise level of the shooting device through a preset noise model.

[0104] Furthermore, in an embodiment of the present application, when simulating light source processing, the terminal may directly adjust the brightness of a part of the image, thereby simulating the image effect of an artificial light source (such as a flash).

[0105] Furthermore, in an embodiment of the present application, when the terminal simulates exposure processing, since the RAW image is in a linear domain, the gain of the image can be directly adjusted to simulate shooting with unequal exposure.

[0106] Further, in an embodiment of the present application, when simulating motion processing, a mask image can be used to increase moving objects. Specifically, the terminal can first perform semantic segmentation on the image (which can be done manually or automatically by an algorithm), and then use part or all of the foreground object after segmentation as a mask, thereby generating a random vector with random size and direction, and then moving the pixels within the mask range along the vector direction, thereby producing the effect of simulating a moving object. Further, in order to make the simulation effect more realistic, the moving vector of the pixel points in the mask can also have a certain degree of randomness, that is, the size and direction vary within a certain range, and blur the moving pixel points along the direction of movement to simulate the motion blur of the moving object in the photo. Further, for the blank area left after the mask is moved, the terminal can use an image repair method to repair it, and the union of the mask area and the area after the mask is moved can be used as a mask image. Specifically, the grayscale value of the mask-covered area can be set to 255, and the remaining grayscale values ​​can be set to 0.

[0107] For example, Figure 6 Illustration of simulated motion processing Figure 1 , Figure 7 Illustration of simulated motion processing Figure 2 , Figure 8 Illustration of simulated motion processing Figure 3 ,like Figure 6 and Figure 7 As shown, the terminal can first segment the image to obtain the foreground object, and use part or all of the foreground object as a mask. Then the terminal can move the pixels within the mask range along the vector direction. Figure 8 In the embodiment, the terminal may use the union of the mask area and the area after the mask is moved as the mask image, wherein the terminal may repair the blank area left after the mask is moved.

[0108] It is understandable that in this application, deep learning has achieved very good results in image restoration and provided many ways of image restoration. The terminal can adopt better image restoration methods when performing image restoration, such as the image watermark removal tool inpaint, which is not specifically limited in this application.

[0109] It should be noted that, in the embodiment of the present application, the first original image, the second original image and the mask image can be used to evaluate the algorithm to be evaluated, and the reference image and the image to be registered can be input into the algorithm to be evaluated for testing the algorithm to be evaluated. That is to say, the image registration algorithm of the present application proposes a test data set including the first original image, the second original image, the reference image and the image to be registered, wherein the first original image and the second original image are image data without eliminating interference such as noise, local moving objects and illumination changes, which are used to evaluate the algorithm to be evaluated, and the reference image and the image to be registered are image data with interference such as simulated image noise, local moving objects and illumination changes added by a preset interference mode, which are used to test the algorithm to be evaluated. It can be seen that the evaluation method proposed in the present application can generate evaluation data and test data by pairing by simulating image noise, local moving objects and illumination changes, etc., and because the evaluation data and the test data are obtained by the terminal through the addition and removal of interference factors, these interference factors can be excluded during the evaluation to achieve an accurate and quantitative evaluation of the registration effect. In addition, this application also proposes a new image registration evaluation indicator, namely the registration residual, which can more comprehensively evaluate the registration effect of the image registration algorithm compared to traditional indicators.

[0110] An evaluation method for an image registration algorithm proposed in an embodiment of the present application, the terminal obtains a first original image and a second original image through the same shooting device; based on a preset interference pattern, the first original image and the second original image are image processed to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; the reference image and the image to be registered are image registered using the algorithm to be evaluated to obtain displacement information; based on the displacement information, the first original image and the second original image, an evaluation result is obtained; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated. It can be seen that in the embodiment of the present application, when evaluating the global image registration algorithm, the interference information during shooting can be simulated by the preset interference pattern based on the first original image and the second original image without interference factors to obtain controllable test data, i.e., the reference image and the image to be registered, and then after the reference image and the image to be registered are registered according to the algorithm to be evaluated, the registration result is evaluated using the image data without interference factors, so as to obtain an accurate evaluation result. That is to say, in this application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during taking pictures, thereby improving the accuracy of the evaluation, and then being able to accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal.

[0111] Based on the above embodiments, an evaluation method for an image registration algorithm proposed in an embodiment of the present application, if the terminal determines the first processed image as the image to be registered and the second processed image as the reference image, then after the terminal uses the algorithm to be evaluated to perform image registration processing on the reference image and the image to be registered to obtain displacement information, the terminal can continue to use the displacement information to perform mapping processing on the first original image, thereby obtaining a first evaluation image corresponding to the first original image, and then based on the first evaluation image and the second original image, determine the corresponding registration residual of the algorithm to be evaluated according to a preset evaluation model, so that the evaluation result of the registration effect of the algorithm to be evaluated can be generated through the registration residual.

[0112] Fig. 9 Generate boxes for evaluation images Figure 1 ,like Fig. 9As shown, when the terminal performs registration processing (step 401) based on the first processed image and the second processed image using the algorithm to be evaluated, the first processed image is input into the algorithm to be evaluated as the image to be registered, and at the same time, the second processed image is input into the algorithm to be evaluated as the reference image, so as to realize the registration processing of the two and output displacement information. After the terminal obtains the displacement information of the two, it can use the displacement information to map all the pixels in the first original image (step 402), so as to obtain the corresponding first evaluation image, wherein the first evaluation image can be used to quantitatively evaluate the algorithm to be evaluated.

[0113] In an embodiment of the present application, further, if the terminal determines the second processed image as the image to be registered and the first processed image as the reference image, then after the terminal uses the algorithm to be evaluated to perform image registration processing on the reference image and the image to be registered and obtains the displacement information, it can continue to map the second original image using the displacement information, thereby obtaining a second evaluation image corresponding to the second original image, and then based on the second evaluation image and the first original image, determine the corresponding registration residual of the algorithm to be evaluated according to a preset evaluation model, so that the evaluation result of the registration effect of the algorithm to be evaluated can be generated through the registration residual.

[0114] Fig.10 Generate boxes for evaluation images Figure 2 ,like Fig.10 As shown, when the terminal performs registration processing (step 501) based on the first processed image and the second processed image using the algorithm to be evaluated, the second processed image is input into the algorithm to be evaluated as the image to be registered, and at the same time, the first processed image is input into the algorithm to be evaluated as the reference image to achieve registration processing of the two and output displacement information. After the terminal obtains the displacement information of the two, it can use the displacement information to map all the pixels in the second original image (step 502), so as to obtain the corresponding second evaluation image, wherein the second evaluation image can be used to quantitatively evaluate the algorithm to be evaluated.

[0115] It can be understood that in the embodiment of the present application, when the terminal uses the residual indicator to evaluate the registration effect of the algorithm to be evaluated, it can specifically use the Laplace pyramid multi-scale residual method to generate a preset evaluation model.

[0116] Exemplarily, in the present application, the method for determining the registration residual according to a preset evaluation model based on the first evaluation image and the second original image by the terminal may specifically include the following steps:

[0117] Step 201: Perform color channel splitting and Laplacian pyramid division on the second original image and the first evaluation image respectively, to obtain all L layers in four color channels.

[0118] In an embodiment of the present application, the terminal may split the second original image into four color channels, for example, into four color channels of R, G1, G2, and B. At the same time, the terminal may also split the first evaluation image into the same four color channels.

[0119] Further, the terminal may divide each color channel of the second original image into a Laplacian pyramid, and obtain L layers of Laplacian pyramids in each color channel; wherein L is a positive integer. At the same time, the terminal may divide each color channel of the first evaluation image into a Laplacian pyramid, and obtain L layers of Laplacian pyramids in each color channel. Exemplarily, the R channel is used as an example, and the other channels are all in the same way: the terminal divides the R channel of the second original image into 3 layers of Laplacian pyramids, and at the same time divides the R channel of the first evaluation image into 3 layers of Laplacian pyramids.

[0120] Step 202: Generate a residual value corresponding to the first color channel in the first layer according to the first pixel parameter and the first weight parameter corresponding to the first color channel of the second original image in the first layer and the second pixel parameter corresponding to the first color channel of the first evaluation image in the first layer.

[0121] In an embodiment of the present application, after splitting and acquiring each layer of the Laplacian pyramid of each color channel of the first evaluation image and the second original image, the terminal can first determine the first pixel parameter and the first weight parameter corresponding to the first color channel of the second original image in the first layer, and at the same time, can also determine the second pixel parameter corresponding to the first color channel of the first evaluation image in the first layer.

[0122] Illustratively, in steps 202 to 206 , the R channel is used as the first color channel for illustration.

[0123] It should be noted that, in the embodiments of the present application, the Laplacian pyramid may be composed of two parts: image information and edge information of different scales, so that the pixel parameters and weight parameters may be determined according to the image information and edge information of different scales.

[0124] It can be understood that in the embodiment of the present application, the terminal can determine the pixel parameters using the image information in the Laplacian pyramid. Specifically, the first pixel parameter is determined using the image information of the R channel of the second original image in the first layer of the Laplacian pyramid; correspondingly, the terminal can determine the second pixel parameter using the image information of the R channel of the first evaluation image in the first layer of the Laplacian pyramid.

[0125] Further, in an embodiment of the present application, the terminal may determine a weight parameter using edge information in the Laplacian pyramid. Specifically, when determining the first weight parameter corresponding to the R channel of the second original image in the first layer, the terminal may first perform Gaussian blur on the edge information of the R channel of the second original image in the first layer of the Laplacian pyramid to obtain blurred edge information, wherein the blurred edge information filters out part of the noise and diffuses the edge, and then calculates the first weight parameter W for the blurred edge information using the following formula: x , y :

[0126] W x，y =w1,if|B x，y |>thresh; else W x，y =w2 (4)

[0127] Among them, for a pixel with coordinates (x, y), B x , y is the intensity value of the blurred edge information. If the absolute value of the intensity value corresponding to the blurred edge information is greater than the preset intensity threshold thresh, the first weight parameter w1 will be assigned. Otherwise, the first weight parameter w2 will be assigned. Since the human eye is more sensitive to edge information, w1>w2 is set to increase the weight of the registration near the edge.

[0128] It should be noted that, in the embodiment of the present application, the terminal only needs to decompose the R channel of the first evaluation image into image information of different scales in the first layer of the Laplace pyramid, and there is no need to calculate the edge information of the first evaluation image, and thus there is no need to determine the weight of the first evaluation image.

[0129] Furthermore, in an embodiment of the present application, after determining the first pixel parameter and the first weight parameter corresponding to the R channel of the second original image in the first layer, and the second pixel parameter corresponding to the R channel of the first evaluation image in the first layer, the terminal can generate a residual value corresponding to the R channel in the first layer based on the first pixel parameter, the second pixel parameter and the first weight parameter.

[0130] Exemplarily, in the present application, the terminal may use the following formula to generate the residual value of the residual value corresponding to the R channel in the first layer:

[0131]

[0132] Among them, for the R channel, for the pixel points (x, y) outside the mask image mask, Rse L is the residual value corresponding to the Lth layer (the first layer takes L = 1), N is the number of pixels outside the mask image, R x，y，L R is the second pixel parameter, Ax，y，L is the first pixel parameter, W x，y，L is the first weight parameter, α is the norm, and can be taken between 1 and 2 according to the sensitivity to noise.

[0133] Step 203: traverse all L layers corresponding to the first color channel of the second original image and the first evaluation image to obtain all L residual values ​​corresponding to the first color channel in all L layers.

[0134] Step 204: Generate a residual sum value of the first color channel according to all L residual values.

[0135] In an embodiment of the present application, after generating the residual value corresponding to the R channel in the first layer, the terminal can traverse all L layers corresponding to the R channel of the second original image and the first evaluation image according to the method of the above-mentioned step 202, and obtain all L residual values ​​corresponding to the R channel in all L layers, and then generate the residual sum value of the R channel based on all L residual values.

[0136] It should be noted that in the embodiment of the present application, when the terminal generates the residual sum value of the R channel from all L residual values, it is necessary to perform weighted average calculation on the residual value of each layer. Specifically, the residual sum value Rse can be calculated according to the following formula:

[0137]

[0138] Among them, LW L is the weight value of the Lth layer, and l is the layer sequence with a value between 1 and L. Since the human eye is more sensitive to the structural information of the image, the larger l is set, the higher the LW L The larger the value of .

[0139] Fig.11 is a schematic diagram of the original image information. Fig.12 A schematic diagram for evaluating image information, such as Fig.11 and Fig.12 As shown in the figure, a color channel of the original image is divided into three layers of Laplacian pyramid, and at the same time, a color channel of the evaluation image is divided into three layers of Laplacian pyramid. The Laplacian pyramid consists of two parts: image information of different scales and edge information. For the original image, the edge information is Gaussian blurred to obtain the blurred edge information. The blurred edge will filter out some noise and diffuse the edge, and then the weight is calculated using the blurred edge information. For the evaluation image and mask, it only needs to be decomposed into image information of different scales through the Laplacian pyramid, without Gaussian blurring and calculating weights.

[0140] Step 205 , traverse all four color channels of the second original image and the first evaluation image to obtain all residual sum values ​​corresponding to all four color channels.

[0141] Step 206: Generate a registration residual based on all residual sum values.

[0142] In an embodiment of the present application, after calculating the residual sum value of the first color channel, the terminal can traverse all four color channels of the second original image and the first evaluation image according to the method of steps 202 to 204 above to obtain all the residual sum values ​​corresponding to all four color channels, so that the registration residual can be generated using all the residual sum values ​​corresponding to the four color channels.

[0143] It should be noted that in an embodiment of the present application, after the terminal generates each residual sum value of each channel based on all L residual values ​​of each color channel, it can add all 4 residual sum values ​​corresponding to all 4 color channels, and then the sum of the 4 residual sum values ​​is determined as the registration residual.

[0144] Furthermore, in an embodiment of the present application, the registration residual can be used to evaluate the registration effect of the algorithm to be evaluated, wherein the smaller the registration residual is, the better the registration effect of the algorithm to be evaluated is, and the larger the registration residual is, the worse the registration effect of the algorithm to be evaluated is.

[0145] In an embodiment of the present application, further, when the terminal determines the registration residual based on the second evaluation image and the first original image according to the preset evaluation model, it can respectively perform color channel splitting and Laplacian pyramid division on the first original image and the second evaluation image in sequence to obtain all L layers in the four color channels; wherein L is a positive integer; according to the third pixel parameter and the second weight parameter corresponding to the first color channel of the first original image in the first layer, and the fourth pixel parameter corresponding to the first color channel of the second evaluation image in the first layer, generate the residual value corresponding to the first color channel in the first layer; traverse all L layers corresponding to the first color channel of the first original image and the second evaluation image, and obtain all L residual values ​​corresponding to the first color channel in all L layers; generate the residual sum value of the first color channel according to all L residual values; traverse all four color channels of the first original image and the second evaluation image, and obtain all residual sum values ​​corresponding to all four color channels; generate the registration residual according to all residual sum values.

[0146] That is to say, in the embodiment of the present application, based on the method of steps 201 to 206 above, the terminal may also determine the registration residual according to the second evaluation image, the first original image and the preset evaluation model.

[0147] An evaluation method for an image registration algorithm proposed in an embodiment of the present application, the terminal obtains a first original image and a second original image through the same shooting device; based on a preset interference pattern, the first original image and the second original image are image processed to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; the reference image and the image to be registered are image registered using the algorithm to be evaluated to obtain displacement information; based on the displacement information, the first original image and the second original image, an evaluation result is obtained; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated. It can be seen that in the embodiment of the present application, when evaluating the global image registration algorithm, the interference information during shooting can be simulated by the preset interference pattern based on the first original image and the second original image without interference factors to obtain controllable test data, i.e., the reference image and the image to be registered, and then after the reference image and the image to be registered are registered according to the algorithm to be evaluated, the registration result is evaluated using the image data without interference factors, so as to obtain an accurate evaluation result. That is to say, in this application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during taking pictures, thereby improving the accuracy of the evaluation, and then being able to accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal.

[0148] Based on the above embodiment, in yet another embodiment of the present application, Fig.13 The terminal structure is shown in Figure 1. Figure 1 ,like Fig.13 As shown, the terminal 10 proposed in the embodiment of the present application may include an acquisition unit 11, a generation unit 12, and a determination unit 13.

[0149] The acquisition unit 11 is used to acquire the first original image and the second original image through the same shooting device;

[0150] The generating unit 12 is used to perform image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information;

[0151] The acquisition unit 11 is further used to perform image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; and obtain an evaluation result based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0152] Furthermore, in an embodiment of the present application, the interference information includes: simulated noise information, simulated light source information, simulated exposure information and simulated motion information.

[0153] Furthermore, in an embodiment of the present application, the determination unit 13 is used to perform image processing on the first original image and the second original image based on a preset interference pattern, and before generating a reference image and an image to be registered, select interference to be added from the interference information, and determine setting parameters corresponding to the interference to be added; determine the preset interference pattern based on the interference to be added and the setting parameters; wherein the preset interference pattern includes: at least one of simulated noise processing, simulated light source processing, simulated exposure processing, and simulated motion processing.

[0154] Furthermore, in an embodiment of the present application, the generating unit 12 is specifically used to add the interference information and image signal processing to the first original image based on the preset interference pattern to generate a first processed image; and to add the interference information and image signal processing to the second original image to generate a second processed image.

[0155] Furthermore, in an embodiment of the present application, the generating unit 12 is further specifically configured to determine the first processed image as the image to be registered, and determine the second processed image as the reference image.

[0156] Furthermore, in an embodiment of the present application, the generating unit 12 is further specifically configured to determine the first processed image as the reference image, and determine the second processed image as the image to be registered.

[0157] Furthermore, in an embodiment of the present application, the acquisition unit 11 is specifically used to map the first original image using the displacement information to obtain a first evaluation image; based on the first evaluation image and the second original image, determine the registration residual according to a preset evaluation model; and generate the evaluation result through the registration residual.

[0158] Furthermore, in an embodiment of the present application, the acquisition unit 11 is also specifically used to use the displacement information to map the second original image to obtain a second evaluation image; based on the second evaluation image and the first original image, determine the registration residual according to a preset evaluation model; and generate the evaluation result through the registration residual.

[0159] Further, in an embodiment of the present application, the acquisition unit 11 is also specifically used to perform color channel splitting and Laplacian pyramid division on the second original image and the first evaluation image respectively, to obtain all L layers in the four color channels; wherein L is a positive integer; according to the first pixel parameter and the first weight parameter corresponding to the first color channel of the second original image in the first layer, and the second pixel parameter corresponding to the first color channel of the first evaluation image in the first layer, generate the residual value corresponding to the first color channel in the first layer; traverse all L layers corresponding to the first color channel of the second original image and the first evaluation image, to obtain all L residual values ​​corresponding to the first color channel in all L layers; generate the residual sum value of the first color channel according to the all L residual values; traverse all four color channels of the second original image and the first evaluation image, to obtain all residual sum values ​​corresponding to all four color channels; generate the registration residual according to the all residual sum values.

[0160] Further, in an embodiment of the present application, the acquisition unit 11 is also specifically used to perform color channel splitting and Laplacian pyramid division on the first original image and the second evaluation image respectively, to obtain all L layers in the four color channels; wherein L is a positive integer; according to the third pixel parameter and the second weight parameter corresponding to the first color channel of the first original image in the first layer, and the fourth pixel parameter corresponding to the first color channel of the second evaluation image in the first layer, generate the residual value corresponding to the first color channel in the first layer; traverse all L layers corresponding to the first color channel of the first original image and the second evaluation image, to obtain all L residual values ​​corresponding to the first color channel in all L layers; generate the residual sum value of the first color channel according to the all L residual values; traverse all four color channels of the first original image and the second evaluation image, to obtain all residual sum values ​​corresponding to all four color channels; generate the registration residual according to the all residual sum values.

[0161] Furthermore, in an embodiment of the present application, the acquisition unit 11 is also specifically used to acquire a first image group at a first position and a second image group at a second position through the same shooting device; wherein the distance between the first position and the second position belongs to a preset offset range; a first original image is determined based on the first image group, and a second original image is determined based on the second image group.

[0162] Fig.14 The terminal structure is shown in Figure 1. Figure 2 ,like Fig.14As shown, the terminal 10 proposed in the embodiment of the present application may also include a processor 14 and a memory 15 storing executable instructions of the processor 14. Further, the terminal 10 may also include a communication interface 16 and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0163] In the embodiment of the present application, the processor 14 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor may also be other, and the embodiment of the present application is not specifically limited. The terminal 10 may also include a memory 15, which may be connected to the processor 14, wherein the memory 15 is used to store executable program code, the program code includes computer operation instructions, and the memory 15 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0164] In the embodiment of the present application, the bus 17 is used to connect the communication interface 16, the processor 14 and the memory 15, and the mutual communication between these devices.

[0165] In the embodiment of the present application, the memory 15 is used to store instructions and data.

[0166] Furthermore, in an embodiment of the present application, the processor 14 is used to acquire a first original image and a second original image through the same shooting device; based on a preset interference pattern, perform image processing on the first original image and the second original image to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; perform image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; obtain an evaluation result based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0167] In practical applications, the memory 15 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0168] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0169] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0170] A terminal is proposed in an embodiment of the present application, which obtains a first original image and a second original image through the same shooting device; based on a preset interference pattern, the first original image and the second original image are subjected to image processing to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; the reference image and the image to be registered are subjected to image registration processing using an algorithm to be evaluated to obtain displacement information; an evaluation result is obtained based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated. It can be seen that in an embodiment of the present application, when evaluating the global image registration algorithm, controllable test data, i.e., a reference image and an image to be registered, can be obtained by simulating the interference information during shooting through a preset interference pattern based on the first original image and the second original image without interference factors, and then after the reference image and the image to be registered are subjected to registration processing according to the algorithm to be evaluated, the registration result is evaluated using the image data without interference factors, thereby obtaining an accurate evaluation result. That is to say, in this application, the terminal obtains two sets of image data to test and evaluate the global image registration algorithm by simulating various interference conditions such as noise, different illumination and moving objects during taking pictures, thereby improving the accuracy of the evaluation, and then being able to accurately measure the advantages and disadvantages of the global image registration algorithm and enhance the intelligence of the terminal.

[0171] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the evaluation method of the image registration algorithm as described above.

[0172] Specifically, the program instructions corresponding to the evaluation method of an image registration algorithm in this embodiment can be stored in a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to the evaluation method of an image registration algorithm in the storage medium are read or executed by an electronic device, the following steps are included:

[0173] Acquiring a first original image and a second original image by using the same photographing device;

[0174] Based on a preset interference pattern, image processing is performed on the first original image and the second original image to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information;

[0175] Performing image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information;

[0176] An evaluation result is obtained according to the displacement information, the first original image, and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0178] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0181] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for evaluating an image registration algorithm, characterized in that: The method comprises: Acquiring a first original image and a second original image by using the same photographing device; Based on a preset interference pattern, image processing is performed on the first original image and the second original image to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; Performing image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; Obtaining an evaluation result according to the displacement information, the first original image, and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated; The obtaining an evaluation result according to the displacement information, the first original image, and the second original image includes: Performing mapping processing on the first original image using the displacement information to obtain a first evaluation image; Sequentially performing color channel splitting and Laplacian pyramid division on the second original image and the first evaluation image to obtain all L layers in four color channels; wherein L is a positive integer; Generate a residual value corresponding to the first color channel in the first layer according to a first pixel parameter and a first weight parameter corresponding to the first color channel of the second original image in the first layer, and a second pixel parameter corresponding to the first color channel of the first evaluation image in the first layer; Traversing all L layers corresponding to the first color channel of the second original image and the first evaluation image to obtain all L residual values ​​corresponding to the first color channel in all L layers; Generate a residual sum value of the first color channel according to all L residual values; Traversing all four color channels of the second original image and the first evaluation image to obtain all residual sum values ​​corresponding to all four color channels; Generate a registration residual according to all the residual sum values; The evaluation result is generated by the registration residual.

2. The method according to claim 1, characterized in that The interference information includes: simulated noise information, simulated light source information, simulated exposure information and simulated motion information.

3. The method according to claim 2, characterized in that Before performing image processing on the first original image and the second original image based on the preset interference pattern to generate a reference image and an image to be registered, the method further includes: Selecting interference to be added from the interference information, and determining setting parameters corresponding to the interference to be added; The preset interference mode is determined according to the interference to be added and the setting parameters; wherein the preset interference mode includes: at least one of simulated noise processing, simulated light source processing, simulated exposure processing and simulated motion processing.

4. The method according to claim 3, characterized in that The performing image processing on the first original image and the second original image based on the preset interference pattern includes: Based on the preset interference pattern, the interference information and image signal processing are added to the first original image to generate a first processed image; and the interference information and image signal processing are added to the second original image to generate a second processed image.

5. The method according to claim 4, characterized in that The generating of the reference image and the image to be registered comprises: The first processed image is determined as the image to be registered, and the second processed image is determined as the reference image.

6. The method according to claim 4, characterized in that The generating of the reference image and the image to be registered comprises: The first processed image is determined as the reference image, and the second processed image is determined as the image to be registered.

7. The method according to claim 6, characterized in that The obtaining an evaluation result according to the displacement information, the first original image, and the second original image includes: Performing mapping processing on the second original image using the displacement information to obtain a second evaluation image; Based on the second evaluation image and the first original image, determining a registration residual according to a preset evaluation model; The evaluation result is generated by the registration residual.

8. The method according to claim 7, characterized in that The determining the registration residual based on the second evaluation image and the first original image according to a preset evaluation model includes: Sequentially performing color channel splitting and Laplacian pyramid division on the first original image and the second evaluation image to obtain all L layers in four color channels; wherein L is a positive integer; Generate a residual value corresponding to the first color channel in the first layer according to a third pixel parameter and a second weight parameter corresponding to the first color channel of the first original image in the first layer, and a fourth pixel parameter corresponding to the first color channel of the second evaluation image in the first layer; Traversing all L layers corresponding to the first color channel of the first original image and the second evaluation image to obtain all L residual values ​​corresponding to the first color channel in all L layers; Generate a residual sum value of the first color channel according to all L residual values; Traversing all four color channels of the first original image and the second evaluation image to obtain all residual sum values ​​corresponding to all four color channels; The registration residual is generated according to the total residual sum value.

9. The method according to claim 1, characterized in that: The method of acquiring the first original image and the second original image by using the same photographing device includes: Acquire a first image group at a first position and acquire a second image group at a second position by the same photographing device; wherein the distance between the first position and the second position belongs to a preset offset range; A first original image is determined according to the first image group, and a second original image is determined according to the second image group.

10. A terminal, characterized in that: The terminal comprises: an acquisition unit and a generation unit, The acquisition unit is used to acquire the first original image and the second original image through the same shooting device; The generating unit is used to perform image processing on the first original image and the second original image based on a preset interference pattern to generate a reference image and an image to be registered; wherein the preset interference pattern is used to simulate interference information; The acquisition unit is further used to perform image registration processing on the reference image and the image to be registered using the algorithm to be evaluated to obtain displacement information; and obtain an evaluation result based on the displacement information, the first original image and the second original image; wherein the evaluation result is used to evaluate the registration effect of the algorithm to be evaluated; The acquisition unit is further specifically used to perform color channel splitting and Laplacian pyramid division on the second original image and the first evaluation image respectively, to obtain all L layers in the four color channels; wherein L is a positive integer; according to the first pixel parameter and the first weight parameter corresponding to the first color channel of the second original image in the first layer, and the second pixel parameter corresponding to the first color channel of the first evaluation image in the first layer, generate the residual value corresponding to the first color channel in the first layer; traverse all L layers corresponding to the first color channel of the second original image and the first evaluation image, to obtain all L residual values ​​corresponding to the first color channel in all L layers; generate the residual sum value of the first color channel according to the all L residual values; traverse all four color channels of the second original image and the first evaluation image, to obtain all residual sum values ​​corresponding to all four color channels; generate a registration residual according to the all residual sum values; and generate the evaluation result through the registration residual.

11. A terminal, characterized in that: The terminal includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a program stored thereon, applied to a terminal, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images