Image fusion method, device and electronic equipment

Abstract

The embodiment of the present application provides an image fusion method, device and electronic device, the method comprising: extracting feature points of the original telephoto image and the amplified short-focus image respectively as the first feature point and the second feature point; taking the first feature point and the temporary matching point of the first feature point as the first temporary matching pair; taking the second feature point and the temporary matching point of the second feature point as the second temporary matching pair; if there is a second temporary matching pair consistent with the first temporary matching pair, taking the temporary matching pair as the feature matching pair; based on each feature matching pair, determining the area matching the original telephoto image in the amplified short-focus image to obtain the first matching area; determining the third matching area that meets the preset matching condition; mapping the original telephoto image to the third matching area in the amplified short-focus image to obtain the fused image. Applying the technical solution provided by the embodiment of the present application can improve the quality of the fused image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device and electronic equipment. Background Art

[0002] A telephoto lens can capture distant dynamics, while a short-focus lens can record nearby details in detail. Therefore, in scenarios such as road safety management, vehicle autonomous driving, and vehicle-road collaboration that require attention to both distant and near targets, a telephoto lens and a short-focus lens are usually used simultaneously for shooting. The long-focus image taken by the telephoto lens and the short-focus image taken by the short-focus lens are then fused to obtain a fused image. In this way, more comprehensive scene information can be obtained by analyzing the fused image.

[0003] Although there are many methods for fusing long-focus images with short-focus images in the related technology, the quality of the fused images obtained is low. The low quality of the fused images will affect the accuracy of the analysis results. Therefore, how to improve the quality of the fused images has become a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image fusion method, device and electronic device to improve the quality of fused images. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an image fusion method, the method comprising:

[0006] Acquire an original long-focus image and an original short-focus image; and amplify the original short-focus image to obtain an amplified short-focus image; wherein the real space size corresponding to each pixel in the amplified short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image;

[0007] Extracting feature points in the original long-focus image as first feature points; and extracting feature points in the amplified short-focus image as second feature points;

[0008] Determine the temporary matching point of each of the first feature points respectively, and use the first feature point and the temporary matching point of the first feature point as a first temporary matching pair; wherein the temporary matching point of the first feature point is the second feature point that is closest to the first feature point;

[0009] respectively determining temporary matching points of the second feature points, and taking the second feature points and the temporary matching points of the second feature points as second temporary matching pairs;

[0010] If there is a second temporary matching pair that is consistent with the first temporary matching pair, taking the first temporary matching pair or the second temporary matching pair as a feature matching pair;

[0011] Based on each of the feature matching pairs, determining an area in the amplified short-focus image that matches the original long-focus image to obtain a first matching area;

[0012] Determine at least one second matching area in the neighborhood of the first matching area; wherein the size of the second matching area is the same as the size of the first matching area;

[0013] Calculating the similarity distances between the first matching area and each of the second matching areas and the original telephoto image respectively as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each of the second matching areas; wherein the similarity distance is used to indicate the degree of brightness difference;

[0014] According to the brightness difference between the first matching area and the second matching area, determining a third matching area that meets a preset matching condition in the first matching area and the second matching area; wherein the preset matching condition is that the brightness difference between the matching areas is less than a preset threshold;

[0015] The original long-focus image is mapped to the third matching area in the amplified short-focus image to obtain a fused image.

[0016] In some embodiments, based on each of the feature matching pairs, determining an area in the amplified short-focus image that matches the original long-focus image to obtain a first matching area includes:

[0017] According to the feature matching pair, determining an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area;

[0018] Controlling the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and using the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area;

[0019] respectively calculating similarity distances between the fourth matching region and each of the fifth matching regions and the original telephoto image as third brightness difference degrees corresponding to the fourth matching region and fourth brightness difference degrees corresponding to each of the fifth matching regions;

[0020] According to the brightness difference between the fourth matching area and the fifth matching area, determining a matching area satisfying the preset matching condition within the fourth matching area and the fifth matching area as the first matching area;

[0021] The determining of at least one second matching area in the neighborhood of the first matching area includes:

[0022] The first matching area is controlled to translate a second preset number of times along a second preset direction according to a preset translation unit, and an area occupied by the first matching area in the enlarged short-focus image after each translation is used as the second matching area.

[0023] In some embodiments, the coordinates at which each pixel in the original telephoto image should be located when mapped to the third matching area are determined by:

[0024] The mapping relationship between the third matching area and the original telephoto image is expressed as the following 3×3 coordinate mapping matrix:

[0025] ;

[0026] Wherein, t00, t01, t02 are parameters in the coordinate mapping matrix;

[0027] According to the coordinate mapping matrix, the coordinates at which each pixel point in the original telephoto image should be located when mapped to the third matching area are determined by the following formula:

[0028] ;

[0029] Among them, j_long represents the x-axis coordinate of the pixel point in the original telephoto image; i_long represents the y-axis coordinate of the pixel point in the original telephoto image; tx represents the coordinate on the x-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; ty represents the coordinate on the y-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; tz represents the coordinate on the z-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area.

[0030] In some embodiments, mapping the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image includes:

[0031] Performing low-pass filtering within a small radius on the third matching area and the original telephoto image respectively to obtain a denoised third matching area and a denoised telephoto image;

[0032] Counting the average brightness value of the third matching area after noise reduction as a first average brightness value; and counting the average brightness value of the telephoto image after noise reduction as a second average brightness value;

[0033] Calculating a ratio of the first brightness average value to the second brightness average value as a target brightness adjustment value;

[0034] Calculating the product of the brightness value of each pixel in the denoised telephoto image and the target brightness adjustment value as the target brightness value corresponding to each pixel in the denoised telephoto image;

[0035] Adjusting the brightness value of each pixel in the denoised telephoto image to a target brightness value corresponding to each pixel to obtain a target telephoto image;

[0036] The target telephoto image is mapped to the third matching area after noise reduction to obtain a fused image.

[0037] In some embodiments, the method further comprises,

[0038] In the fused image, for a boundary of the third matching area, a neighborhood of the boundary facing the inside of the boundary is divided into a plurality of first candidate blocks, and a neighborhood of the boundary facing the outside of the boundary is divided into a plurality of second candidate blocks;

[0039] For each of the first candidate blocks, a plurality of continuous first candidate blocks centered on the first candidate block are determined as first target blocks corresponding to the first candidate blocks, and a second candidate block located on a normal direction of a boundary of each of the first target blocks is determined as a second target block corresponding to the first target block, wherein the boundary direction is parallel to the normal direction of the boundary at the first target block and faces outside the boundary;

[0040] For each of the first candidate blocks, respectively, calculate a third brightness average value of each of the first target blocks and the second target block corresponding to the first candidate block;

[0041] For each of the first candidate blocks, the brightness of each of the first target blocks and the second target block corresponding to the first candidate block is adjusted in a direction close to the third brightness average.

[0042] In some embodiments, adjusting the brightness of each of the first target blocks and the second target block corresponding to each of the first candidate blocks toward a direction close to the third brightness average value includes:

[0043] The brightness mean of each of the first candidate blocks is calculated as the fourth brightness mean; and the brightness mean of the second candidate blocks is calculated as the fifth brightness mean, and the difference of the brightness mean is calculated by the following formula:

[0044] ;

[0045] Wherein, a_i represents the fourth brightness mean value, or the fifth brightness mean value; m_ave represents the third brightness mean value, and dif_i represents the difference between the brightness mean values;

[0046] If the difference in the brightness mean is greater than 0, the brightness value to be adjusted to the target block is calculated using the following formula:

[0047] ;

[0048] Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted to the target block;

[0049] If the difference in the brightness mean is less than 0, the brightness value to be adjusted to the target block is calculated using the following formula:

[0050] ;

[0051] Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted for the target block; abs() means taking the absolute value;

[0052] The brightness value of the target block is adjusted to the brightness value to be adjusted.

[0053] In some embodiments, the brightness difference between each matching area and the original telephoto image is calculated by the following formula:

[0054] ;

[0055] Wherein, distance represents the brightness difference between the matching area and the original telephoto image; a_i represents the brightness value of the i-th pixel in the matching area; b_i represents the brightness value of the pixel covering the i-th pixel in the matching area when the original telephoto image is mapped to the matching area; the value range of i is 1, 2, …, n, where n is the number of pixels in the matching area.

[0056] In a second aspect, an embodiment of the present application further provides an image fusion device, the device comprising:

[0057] An image acquisition module, used to acquire an original long-focus image and an original short-focus image; and to amplify the original short-focus image to obtain an amplified short-focus image; wherein the real space size corresponding to each pixel in the amplified short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image;

[0058] A feature extraction module, used to extract feature points in the original long-focus image as first feature points; and extract feature points in the amplified short-focus image as second feature points;

[0059] a matching pair acquisition module, used to respectively determine temporary matching points of each of the first feature points, and use the first feature point and the temporary matching point of the first feature point as a first temporary matching pair; wherein the temporary matching point of the first feature point is the second feature point that is closest to the first feature point;

[0060] The matching pair acquisition module is further used to respectively determine temporary matching points of each of the second feature points, and use the second feature point and the temporary matching point of the second feature point as a second temporary matching pair;

[0061] The matching pair acquisition module is further configured to use the first temporary matching pair or the second temporary matching pair as a feature matching pair if there is a second temporary matching pair that is consistent with the first temporary matching pair;

[0062] a matching region determining module, configured to determine, based on each of the feature matching pairs, a region in the amplified short-focus image that matches the original long-focus image, to obtain a first matching region;

[0063] The matching region determination module is used to determine at least one second matching region in the neighborhood of the first matching region; wherein the size of the second matching region is the same as the size of the first matching region;

[0064] a similarity distance calculation module, used to respectively calculate the similarity distances between the first matching area and each of the second matching areas and the original telephoto image as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each of the second matching areas; wherein the similarity distance is used to represent the degree of brightness difference;

[0065] The matching region determination module is further configured to determine, according to the brightness difference between the first matching region and the second matching region, a third matching region that meets a preset matching condition within the first matching region and the second matching region; wherein the preset matching condition is that the brightness difference between the matching regions is less than a preset threshold;

[0066] An image fusion module is used to map the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image.

[0067] In some embodiments, the matching area determination module is specifically used to:

[0068] According to the feature matching pair, determining an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area;

[0069] Controlling the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and using the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area;

[0070] respectively calculating similarity distances between the fourth matching region and each of the fifth matching regions and the original telephoto image as third brightness difference degrees corresponding to the fourth matching region and fourth brightness difference degrees corresponding to each of the fifth matching regions;

[0071] According to the brightness difference between the fourth matching area and the fifth matching area, determining a matching area satisfying the preset matching condition within the fourth matching area and the fifth matching area as the first matching area;

[0072] The determining of at least one second matching area in the neighborhood of the first matching area includes:

[0073] The first matching area is controlled to translate a second preset number of times along a second preset direction according to a preset translation unit, and an area occupied by the first matching area in the enlarged short-focus image after each translation is used as the second matching area.

[0074] In some embodiments, the coordinates at which each pixel in the original telephoto image should be located when mapped to the third matching area are determined by:

[0075] The mapping relationship between the third matching area and the original telephoto image is expressed as the following 3×3 coordinate mapping matrix:

[0076] ;

[0077] Wherein, t00, t01, t02 are parameters in the coordinate mapping matrix;

[0078] According to the coordinate mapping matrix, the coordinates at which each pixel point in the original telephoto image should be located when mapped to the third matching area are determined by the following formula:

[0079] ;

[0080] Among them, j_long represents the x-axis coordinate of the pixel point in the original telephoto image; i_long represents the y-axis coordinate of the pixel point in the original telephoto image; tx represents the coordinate on the x-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; ty represents the coordinate on the y-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; tz represents the coordinate on the z-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area.

[0081] In some embodiments, the image fusion module is specifically used to:

[0082] Performing low-pass filtering within a small radius on the third matching area and the original telephoto image respectively to obtain a denoised third matching area and a denoised telephoto image;

[0083] Counting the average brightness value of the third matching area after noise reduction as a first average brightness value; and counting the average brightness value of the telephoto image after noise reduction as a second average brightness value;

[0084] Calculating a ratio of the first brightness average value to the second brightness average value as a target brightness adjustment value;

[0085] Calculating the product of the brightness value of each pixel in the denoised telephoto image and the target brightness adjustment value as the target brightness value corresponding to each pixel in the denoised telephoto image;

[0086] Adjusting the brightness value of each pixel in the denoised telephoto image to a target brightness value corresponding to each pixel to obtain a target telephoto image;

[0087] The target telephoto image is mapped to the third matching area after noise reduction to obtain a fused image.

[0088] In some embodiments, the image fusion device further includes:

[0089] In the fused image, for a boundary of the third matching area, a neighborhood of the boundary facing the inside of the boundary is divided into a plurality of first candidate blocks, and a neighborhood of the boundary facing the outside of the boundary is divided into a plurality of second candidate blocks;

[0090] For each of the first candidate blocks, a plurality of continuous first candidate blocks centered on the first candidate block are determined as first target blocks corresponding to the first candidate blocks, and a second candidate block located on a normal direction of a boundary of each of the first target blocks is determined as a second target block corresponding to the first target block, wherein the boundary direction is parallel to the normal direction of the boundary at the first target block and faces outside the boundary;

[0091] For each of the first candidate blocks, respectively, calculate a third brightness average value of each of the first target blocks and the second target block corresponding to the first candidate block;

[0092] For each of the first candidate blocks, the brightness of each of the first target blocks and the second target block corresponding to the first candidate block is adjusted in a direction close to the third brightness average.

[0093] In some embodiments, the brightness adjustment module is specifically used to:

[0094] The brightness mean of each of the first candidate blocks is calculated as the fourth brightness mean; and the brightness mean of the second candidate blocks is calculated as the fifth brightness mean, and the difference of the brightness mean is calculated by the following formula:

[0095] ;

[0096] Wherein, a_i represents the fourth brightness mean value, or the fifth brightness mean value; m_ave represents the third brightness mean value, and dif_i represents the difference between the brightness mean values;

[0097] If the difference in the brightness mean is greater than 0, the brightness value to be adjusted to the target block is calculated using the following formula:

[0098] ;

[0099] Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted to the target block;

[0100] If the difference in the brightness mean is less than 0, the brightness value to be adjusted to the target block is calculated using the following formula:

[0101] ;

[0102] Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted for the target block; abs() means taking the absolute value;

[0103] The brightness value of the target block is adjusted to the brightness value to be adjusted.

[0104] In some embodiments, the brightness difference between each matching area and the original telephoto image is calculated by the following formula:

[0105] ;

[0106] Wherein, distance represents the brightness difference between the matching area and the original telephoto image; a_i represents the brightness value of the i-th pixel in the matching area; b_i represents the brightness value of the pixel covering the i-th pixel in the matching area when the original telephoto image is mapped to the matching area; the value range of i is 1, 2, …, n, where n is the number of pixels in the matching area.

[0107] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0108] Memory, used to store computer programs;

[0109] The processor is used to implement any of the above-mentioned image fusion methods when executing the program stored in the memory.

[0110] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned image fusion methods is implemented.

[0111] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described image fusion methods.

[0112] Beneficial effects of the embodiments of the present application:

[0113] In the technical solution provided by the embodiment of the present application, an original long-focus image and an original short-focus image are obtained, and the original short-focus image is enlarged to obtain an enlarged short-focus image, so that the real space size corresponding to each pixel in the enlarged short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image, and the fusion effect of the original long-focus image and the enlarged short-focus image is better; feature points in the original long-focus image are extracted as first feature points, and feature points in the enlarged short-focus image are extracted as second feature points; temporary matching points of each first feature point are determined respectively, that is, the second feature point closest to the first feature point, and the temporary matching points of the first feature point and the first feature point are used as a first temporary matching pair; then, temporary matching points of each second feature point are determined respectively, and the temporary matching points of the second feature point and the second feature point are used as a first temporary matching pair. The matching points are used as a second temporary matching pair. If there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Based on each feature matching pair, the area that matches the original telephoto image can be determined in the enlarged short-focus image to obtain a first matching area. When there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Then, based on each feature matching pair, the area that matches the original telephoto image in the enlarged short-focus image is determined. In this way, the registration accuracy between the original telephoto image and the enlarged short-focus image can be greatly improved, and the registration error can be reduced. That is, the first matching area that matches the original telephoto image can be determined more accurately, and the position where each pixel point in the original telephoto image should be when mapped to the enlarged short-focus image can be accurately determined.

[0114] After determining the first matching area, at least one second matching area with the same size as the first matching area is determined in the neighborhood of the first matching area, and the similarity distances between the first matching area and each second matching area and the original telephoto image are calculated respectively as the first brightness difference corresponding to the first matching area and the second difference corresponding to each second matching area; then, based on the brightness difference corresponding to the first matching area and the second matching area, a third matching area that meets the preset matching condition is determined within the first matching area and the second matching area, that is, a matching area whose brightness difference is less than a preset threshold; then, the original telephoto image is mapped to the third matching area in the enlarged short-focus image to obtain a fused image. It can be understood that the aforementioned feature matching pair-based method more accurately determines the area in the enlarged short-focus image that matches the original telephoto image, that is, the first matching area. Therefore, at least one second matching area is subsequently determined in the neighborhood of the first matching area, and the brightness difference between the first matching area, the second matching area and the original telephoto image is determined. The area with a brightness difference less than a preset threshold is used as the third matching area, and the original telephoto image is mapped to the third matching area to obtain a fused image. That is, after determining the first matching area that matches the original telephoto image in the enlarged short-focus image based on the feature matching pair from the perspective of feature points, the position of the first area in the enlarged short-focus image that matches the original telephoto image is adjusted from the perspective of brightness value to obtain a third matching area that better matches the original telephoto image. In this way, after mapping the original telephoto image to the third matching area, the quality of the fused image can be improved.

[0115] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0117] Figure 1 A schematic diagram of a traffic binocular radar vision product provided in an embodiment of the present application;

[0118] Figure 2 A schematic diagram of a first process flow of the image fusion method provided in an embodiment of the present application;

[0119] Figure 3 A schematic diagram of the positional relationship between the long-focus image and the short-focus image during fusion provided for the implementation of the present application;

[0120] Figure 4A schematic diagram of feature point extraction of an original telephoto image provided in an embodiment of the present application;

[0121] Figure 5 A schematic diagram of extracting feature points of an original short-focus image provided in an embodiment of the present application;

[0122] Figure 6 A schematic diagram of feature point matching provided in an embodiment of the present application;

[0123] Figure 7 A schematic diagram of a fused image provided in an embodiment of the present application;

[0124] Figure 8 A detailed schematic diagram of the above steps S26 and S27 provided in an embodiment of the present application;

[0125] Fig. 9 A schematic diagram of determining a fifth matching area provided in an embodiment of the present application;

[0126] Fig.10 A schematic diagram of determining a second matching area provided in an embodiment of the present application;

[0127] Fig.11 A schematic diagram of a process for processing a fused image provided in an embodiment of the present application;

[0128] Fig.12 A schematic diagram of a dynamic border stitching provided in an embodiment of the present application;

[0129] Fig.13 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application;

[0130] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0131] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0132] Traffic binocular radar vision products are intelligent traffic equipment that combines radar technology and video technology. They can accurately identify the movement trajectory and direction of vehicles, and perceive traffic flow and other information in real time, so as to provide users with rich and accurate traffic data. Figure 1 As shown, Figure 1A schematic diagram of a traffic binocular radar product provided in an embodiment of the present application, wherein the traffic binocular radar product is equipped with a telephoto lens 11 and a short focal lens 12, and has 8 lights composed of light emitting diodes (LEDs) at the lower end, for example Figure 1 The LED light 13 is shown in FIG. The telephoto lens 11 is built with a gimbal, which can control the telephoto lens 11 to move up, down, left, and right within a certain range, and adjust the shooting scene at any time according to the area of ​​interest to the user, wherein the resolution size of the telephoto lens 11 and the short-focus lens 12 are both 4096×2160.

[0133] It is understandable that the traffic binocular radar product is equipped with both a telephoto lens and a short-focus lens. The fused image obtained by fusing the telephoto image taken by the telephoto lens and the short-focus image taken by the short-focus lens can provide richer information. In the related technology, the following two methods are usually used for the fusion of long-focus and short-focus images:

[0134] Method 1: Determine the reduction coefficient of the telephoto image according to the focal lengths of the telephoto lens and the short-focus lens, and reduce the telephoto image to obtain a reduced telephoto image. Then, based on the relative angle between the telephoto lens and the short-focus lens, calculate the position where the reduced telephoto image should be located in the short-focus image during fusion, and fuse the reduced telephoto image to cover this position to achieve fusion of the long-focus and short-focus images.

[0135] In this method, the position of the reduced telephoto image in the short-focus image during fusion is determined based on the relative angle between the telephoto lens and the short-focus lens. This is relatively rough and cannot accurately determine the area in the short-focus image where the reduced telephoto image should actually be fused. Therefore, after mapping the reduced telephoto image to the short-focus image, serious pixel misalignment of the same object may occur, resulting in poor quality of the resulting fused image.

[0136] Method 2: Determine the magnification factor of the short-focus image according to the focal lengths of the telephoto lens and the short-focus lens, and amplify the short-focus image to obtain an amplified short-focus image; extract feature points in the telephoto image and the short-focus image respectively, and take each feature point in the telephoto image as a reference, calculate the feature points in the amplified short-focus image that are most similar to each feature point in the telephoto image in turn, and use them as feature matching points for each feature point in the telephoto image. Afterwards, each feature point in the telephoto image and the corresponding feature matching point constitute a feature matching pair. According to the feature matching pair, determine the position where the telephoto image should be located in the amplified short-focus image during fusion, and fuse the telephoto image to cover this position to achieve long- and short-focus image fusion.

[0137] In this method, the position where the long-focus image should be located in the amplified short-focus image during fusion is determined from the perspective of feature points. Compared with the above method 1, the position where the long-focus image should be located in the amplified short-focus image during fusion determined by method 2 is more accurate. However, the feature matching pairs determined based only on the feature points in the long-focus image are not accurate enough. For example, there are feature points 1, 2, and 3 in the long-focus image, and there are feature points 4, 5, and 6 in the amplified short-focus image. Taking the long-focus image as a reference, if the feature matching point of feature point 1 is determined to be feature point 4, the feature matching point of feature point 2 is also feature point 4, and the feature matching point of feature point 3 is feature point 6; in this case, (feature point 1, feature point 4), (feature point 2, feature point 6) are used as the basis. 4), (feature point 3, feature point 6) as feature matching pairs for image fusion, but in actual circumstances, the similarity between feature point 2 and feature point 4 is greater than the similarity between feature point 1 and feature point 4. Therefore, the quality of the image obtained by image fusion with (feature point 2, feature point 4), (feature point 3, feature point 6) as feature matching pairs should be higher than the quality of the image obtained by image fusion with (feature point 1, feature point 4), (feature point 2, feature point 4), (feature point 3, feature point 6) as feature matching pairs. It can be seen that the feature matching pairs determined only based on the feature points in the telephoto image are not accurate enough. Therefore, in the fused image obtained by method 2, there may be serious pixel misalignment of the same object, and the quality of the fused image obtained is still poor.

[0138] In order to improve the quality of fused images, an embodiment of the present application provides an image fusion method, which is applied to an electronic device, and the electronic device may be a desktop computer, a portable computer, a smart mobile terminal, etc.

[0139] Specifically, in some embodiments, the image fusion method provided in the embodiments of the present application can be applied to the above-mentioned binocular radar vision products. For example, the image fusion method provided in the present application is applied to a traffic binocular radar vision product in a traffic scene. After obtaining the fused image, all license plate information within a range of 100 meters can be identified based on the fused image, thereby achieving strict traffic control of 100 meters; or, in some embodiments, the image fusion method provided in the embodiments of the present application can also be applied to a robot vision system. The obtained fused image can help the robot better obtain long-distance and short-distance information, thereby improving its environmental perception and task execution capabilities; or, in some embodiments, the image fusion method provided in the embodiments of the present application can also be applied to an intelligent driving system. The obtained fused image can help an autonomous driving vehicle comprehensively and accurately detect the surrounding environment.

[0140] In addition, it is understandable that there are significant differences between telephoto lenses and short-focus lenses in terms of viewing angle, magnification effect, depth of field control, etc. Therefore, in order to better handle the difference between images at different focal lengths during image fusion, the focal length variation range of the telephoto lens and the short-focus lens used in the image fusion method provided in the embodiment of the present application should be large enough. Among them, the focal length variation range of the telephoto lens and the short-focus lens is large enough, which means that when the short-focus lens can capture the whole picture, the telephoto lens can still capture a farther distance or closer details.

[0141] The image fusion method provided by the present application is described in detail below through specific examples.

[0142] See also Figure 2 , Figure 2 The first flow chart of the image fusion method provided in the embodiment of the present application may include the following steps:

[0143] Step S21: obtaining an original long-focus image and an original short-focus image; and amplifying the original short-focus image to obtain an amplified short-focus image;

[0144] The real space size corresponding to each pixel in the enlarged short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image;

[0145] Step S22: extracting feature points in the original long-focus image as first feature points; and extracting feature points in the enlarged short-focus image as second feature points;

[0146] Step S23: respectively determining temporary matching points of each first feature point, and taking the first feature point and the temporary matching point of the first feature point as a first temporary matching pair;

[0147] The temporary matching point of the first feature point is the second feature point that is closest to the first feature point;

[0148] Step S24: respectively determining temporary matching points of each second feature point, and taking the second feature point and the temporary matching point of the second feature point as a second temporary matching pair;

[0149] Step S25: If there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair;

[0150] Step S26: Based on each feature matching pair, determine an area in the amplified short-focus image that matches the original long-focus image to obtain a first matching area;

[0151] Step S27: determining at least one second matching area in the neighborhood of the first matching area;

[0152] Wherein, the size of the second matching area is the same as the size of the first matching area;

[0153] Step S28: calculating the similarity distances between the first matching area and each second matching area and the original telephoto image respectively as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each second matching area;

[0154] Step S29: determining a third matching area that meets a preset matching condition within the first matching area and the second matching area according to the brightness difference between the first matching area and the second matching area;

[0155] The preset matching condition is that the brightness difference of the matching area is less than a preset threshold;

[0156] Step S210: Mapping the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image.

[0157] In the technical solution provided by the embodiment of the present application, an original long-focus image and an original short-focus image are obtained, and the original short-focus image is enlarged to obtain an enlarged short-focus image, so that the real space size corresponding to each pixel in the enlarged short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image, and the fusion effect of the original long-focus image and the enlarged short-focus image is better; feature points in the original long-focus image are extracted as first feature points, and feature points in the enlarged short-focus image are extracted as second feature points; temporary matching points of each first feature point are determined respectively, that is, the second feature point closest to the first feature point, and the temporary matching points of the first feature point and the first feature point are used as a first temporary matching pair; then, temporary matching points of each second feature point are determined respectively, and the temporary matching points of the second feature point and the second feature point are used as a first temporary matching pair. The matching points are used as a second temporary matching pair. If there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Based on each feature matching pair, the area that matches the original telephoto image can be determined in the enlarged short-focus image to obtain a first matching area. When there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Then, based on each feature matching pair, the area that matches the original telephoto image in the enlarged short-focus image is determined. In this way, the registration accuracy between the original telephoto image and the enlarged short-focus image can be greatly improved, and the registration error can be reduced. That is, the first matching area that matches the original telephoto image can be determined more accurately, and the position where each pixel point in the original telephoto image should be when mapped to the enlarged short-focus image can be accurately determined.

[0158] After determining the first matching area, at least one second matching area with the same size as the first matching area is determined in the neighborhood of the first matching area, and the similarity distances between the first matching area and each second matching area and the original telephoto image are calculated respectively as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each second matching area; then, according to the brightness difference corresponding to the first matching area and the second matching area, a third matching area that meets the preset matching condition is determined within the first matching area and the second matching area, that is, a matching area whose brightness difference is less than a preset threshold; then, the original telephoto image is mapped to the third matching area in the enlarged short-focus image to obtain a fused image. It can be understood that the aforementioned feature matching pair-based method more accurately determines the area in the enlarged short-focus image that matches the original telephoto image, that is, the first matching area. Therefore, at least one second matching area is subsequently determined in the neighborhood of the first matching area, and the brightness difference between the first matching area, the second matching area and the original telephoto image is determined. The area with a brightness difference less than a preset threshold is used as the third matching area, and the original telephoto image is mapped to the third matching area to obtain a fused image. That is, after determining the first matching area that matches the original telephoto image in the enlarged short-focus image based on the feature matching pair from the perspective of feature points, the position of the first area in the enlarged short-focus image that matches the original telephoto image is adjusted from the perspective of brightness value to obtain a third matching area that better matches the original telephoto image. In this way, after mapping the original telephoto image to the third matching area, the quality of the fused image can be improved.

[0159] In the above step S21, the original telephoto image is an image captured by a telephoto lens, and the short-focus image is an image captured by a short-focus lens. The scene captured by the lens capturing the original telephoto image and the scene captured by the lens capturing the original short-focus image are the same scene, that is, the original telephoto image and the original short-focus image are images obtained by capturing the same scene by different lenses. In some embodiments, the original telephoto image and the original short-focus image can be understood as images captured by the telephoto lens and the short-focus lens in the traffic binocular radar vision product.

[0160] The short-focus image has a wide frame range and small objects, while the long-focus image has a small frame range and large objects. Therefore, the real-space size corresponding to each pixel in the short-focus image is larger than the real-space size corresponding to each pixel in the long-focus image. Therefore, after the electronic device obtains the original long-focus image and the original short-focus image, it first enlarges the original short-focus image to make the real-space size corresponding to each pixel in the enlarged short-focus image consistent with the real-space size corresponding to each pixel in the original long-focus image. In this way, the real-space size corresponding to the pixels in the original long-focus image is consistent with the real-space size corresponding to the pixels in the enlarged short-focus image. Furthermore, under the condition of consistent size, the feature matching pair can be determined by comparing the distance between each feature point of the original long-focus image and each feature point of the enlarged short-focus image. Thereafter, the area in the enlarged short-focus image that matches the original long-focus image is accurately determined based on the feature matching pair (for details, see the description in steps S22 to S26 below), so as to avoid large errors in the matching area caused by inconsistent real-space sizes corresponding to pixels.

[0161] Specifically, taking the license plate as an example, since the focal lengths of the telephoto lens and the short-focus lens are different, according to the principle of optical imaging, the real space reflected by one pixel in the original telephoto image captured by the telephoto lens is smaller than the real space reflected by one pixel in the original short-focus image captured by the short-focus lens, and the size of the license plate in the real space is certain, so the license plate occupies more pixels in the original telephoto image and fewer pixels in the original short-focus image. If the original telephoto image is directly superimposed on the original short-focus image, the area where the license plate is located in the original telephoto image cannot completely overlap with the area where the license plate is located in the original short-focus image, so the original telephoto image cannot be directly superimposed on the original short-focus image.

[0162] In an embodiment of the present application, a magnification factor can be set in advance. After acquiring the original telephoto image and the original short-focus image, the electronic device can amplify the short-focus image according to the preset magnification factor to obtain an enlarged short-focus image, wherein the preset magnification factor can be set by a technician based on actual experience, such as usually being set to 2 times, 3 times, etc.; it can also be the magnification factor required for the short-focus image calculated in advance based on the focal length of the telephoto lens and the focal length of the short-focus lens.

[0163] Alternatively, in some embodiments, it is not necessary to pre-set the magnification factor. After the electronic device acquires the original long-focus image and the original short-focus image, it can directly analyze and calculate the original long-focus image and the original short-focus image, determine the magnification factor suitable for the original short-focus image, and amplify the original short-focus image according to the magnification factor to obtain the amplified short-focus image. For example, after the electronic device acquires the original long-focus image and the original short-focus image, it analyzes the real space size corresponding to each pixel in the original long-focus image and the original short-focus image, respectively, and then, according to the real space size corresponding to each pixel in the original long-focus image and the real space size corresponding to each pixel in the original short-focus image, calculates the magnification factor suitable for the original short-focus image, and amplifies the original short-focus image according to the magnification factor to obtain the amplified short-focus image.

[0164] To facilitate understanding of the subsequent content, an example is used to illustrate the positional relationship between the long-focus image and the short-focus image during image fusion. For example, see Figure 3 , Figure 3 A schematic diagram of the positional relationship between the long-focus image and the short-focus image during fusion provided for the implementation of the present application, wherein the width of the original long-focus image 31 acquired by the electronic device is w, and the length is h; the width of the original short-focus image 32 acquired is w, and the length is h; the original short-focus image is enlarged by 2 times to obtain an enlarged short-focus image 33, and the width of the enlarged short-focus image 33 is 2w, and the length is 2h. When performing image fusion, the original long-focus image 31 should be mapped to a sub-area in the enlarged short-focus image 33.

[0165] In the above step S22, the first feature point is a pixel point in the original long-focus image extracted by the electronic device, and the second feature point is a feature point in the amplified short-focus image extracted by the electronic device. Therefore, in the subsequent steps, it is necessary to determine the feature matching pair according to the distance between each feature point. Therefore, in order to improve the reliability of the determined feature matching pair, the same feature extraction algorithm is used to extract the first feature point and the second feature point in this embodiment.

[0166] See also Figure 4 and Figure 5 , Figure 4 A schematic diagram of feature point extraction of an original telephoto image provided in an embodiment of the present application; Figure 5 A schematic diagram of extracting feature points of an original short-focus image provided in an embodiment of the present application, Figure 4 and Figure 5 In the figure, black dots (e.g. Figure 4 Midpoint 41 and Figure 5 The black point shown in the midpoint 51 is the extracted feature point. It can be understood that Figure 4 and Figure 5There are many feature points shown in FIG. 4 , so only point 41 and point 51 are marked and explained; in addition, the extracted feature points may include feature points of any area in the image, for example, Figure 4 and Figure 5 Taking the scene shown as an example, the areas where the extracted feature points can be located include but are not limited to the vehicles, green belts, roads and other areas shown in the image. Figure 4 and Figure 5 This is only a schematic diagram of a feature point. To keep the image clear, it is not shown. Figure 4 and Figure 5 All possible feature points in , and Figure 4 and Figure 5 The form of the feature points shown in is only for the convenience of distinguishing from other areas in the image. In actual application, the representation form of the feature points is not limited.

[0167] In the above step S23, the temporary matching point of the first feature point is the second feature point that is closest to the first feature point; the first temporary matching pair is composed of the first feature point and the temporary matching point of the first feature point; wherein the first temporary matching pair composed of the first feature point and the temporary matching point of the first feature point can be stored in the form of a point pair, for example, each first feature point and the temporary matching point of the first feature point are stored as a pair of data. Specifically, if the first feature point includes feature point 1 and feature point 2, the second feature point includes feature point 3 and feature point 4, and the temporary matching point of feature point 1 is feature point 3, and the temporary matching point of feature point 2 is feature point 4, then (feature point 1, feature point 3) can be stored as a point pair, and (feature point 2, feature point 4) can be stored as a point pair.

[0168] Alternatively, the first temporary match consisting of the first feature point and the temporary matching point of the first feature point can also be stored in other forms other than point pairs, for example, in the form of two sequences, with each first feature point being recorded in the first sequence, and each sequence in the second sequence recording a temporary matching point of the first feature point with the same sequence as that in the first sequence; or, in the form of a key-value table, in each key-value pair, the key is used to record the first feature point, and the value is used to record the temporary matching point with the first feature point referred to by the key. It can be understood that no matter in what form the first temporary matching pair is stored, it is sufficient to be able to query the temporary matching point of the first feature point in subsequent queries.

[0169] In the embodiment of the present application, after the electronic device determines the first feature point and the second feature point, it can use each first feature point as a reference, calculate the distance between each second feature point and the first feature point in sequence, and determine the second feature point closest to each first feature point as the temporary matching point of each first feature point. (For the description of the distance between feature points, please refer to the description in step S24 below)

[0170] In the above step S24, the temporary matching point of the second feature point is the first feature point that is closest to the second feature point; the second temporary matching pair is composed of the second feature point and the temporary matching point of the second feature point; wherein, the specific storage form of the second temporary matching pair and the first temporary matching pair may be the same or different, and the specific storage form may refer to the description in the above step S23, such as the first temporary matching pair and the second temporary matching pair may both be stored in the form of a key-value table, or, alternatively, the first temporary matching pair may be stored in the form of a key-value table, and the second temporary matching may be stored in the form of a sequence; it is sufficient to be able to queried the temporary matching point of the second feature point in subsequent queries.

[0171] In the embodiment of the present application, after the electronic device determines the first feature point and the second feature point, it can use each second feature point as a reference to calculate the distance between each first feature point and the second feature point in sequence, and determine the first feature point that is closest to each second feature point as a temporary matching point for each second feature point.

[0172] For the specific process of determining the temporary matching point of the first feature point in step S23 and the specific process of determining the temporary matching point of the second feature point in step S24, please refer to the following Figure 6 In addition, the distance between the first feature point and the second feature point described in the above steps S23 and S24 actually refers to the similarity between the feature points. For example, determining the first feature point closest to a second feature point actually refers to determining the first feature point with the highest similarity to the second feature point; wherein the distance between feature points can be determined by Manhattan distance, Euclidean distance, etc.

[0173] Understandably, Figure 2 The figure only shows a possible flow chart of the image fusion method provided by the present application. In other possible embodiments, step S23 may not be executed first and then step S24. For example, step S24 may be executed first and then step S23, or step S23 and step S24 may be executed simultaneously. There is no limitation on this.

[0174] In the above step S25, the second temporary matching pair consistent with the first temporary matching pair can also be understood as: the first feature point constituting the second temporary matching pair is the same as the first feature point constituting the first temporary matching pair, and the second feature point constituting the second temporary matching pair is the same as the second feature point constituting the first temporary matching pair, that is, if the first feature point and the second feature point in a second temporary matching pair are consistent with the first feature point and the second feature point in the first temporary matching pair, then the second temporary matching pair is a second temporary matching pair consistent with the first temporary matching pair.

[0175] The feature matching pair can be understood as a matching pair consisting of the first feature point and the second feature point in the consistent first temporary matching and second temporary matching pairs.

[0176] The following is a specific example to illustrate the contents of the above steps S22 to S25:

[0177] For example, see Figure 6 , Figure 6 A schematic diagram of feature point matching provided in an embodiment of the present application, Figure 6 (a) is a schematic diagram of feature points in the original telephoto image. It can be seen that the width of the original telephoto image is w, the length is h, and the first feature points extracted include feature point 1, feature point 2, feature point 3, and feature point 4; Figure 6 (b) is a schematic diagram of feature points in the magnified short-focus image. It can be seen that the width of the magnified short-focus image is 2w and the length is 2h. The extracted second feature points include feature point 5, feature point 6, feature point 7, feature point 8, and feature point 9.

[0178] The electronic device calculates the similarity distances between feature points 5-9 and feature point 1 respectively, determines the feature point with the smallest similarity distance to feature point 1 as the temporary matching point of feature point 1; calculates the similarity distances between feature points 5-9 and feature point 2 respectively, determines the feature point with the smallest similarity distance to feature point 2 as the temporary matching point of feature point 2; calculates the similarity distances between feature points 5-9 and feature point 3 respectively, determines the feature point with the smallest similarity distance to feature point 3 as the temporary matching point of feature point 3; calculates the similarity distances between feature points 5-9 and feature point 4 respectively, determines the feature point with the smallest similarity distance to feature point 4 as the temporary matching point of feature point 4. This corresponds to the above step S23.

[0179] Furthermore, the electronic device calculates the similarity distances between feature points 1-4 and feature point 5, determines the feature point with the smallest similarity distance to feature point 5 as the temporary matching point of feature point 5; calculates the similarity distances between feature points 1-4 and feature point 6, determines the feature point with the smallest similarity distance to feature point 6 as the temporary matching point of feature point 6; ...; calculates the similarity distances between feature points 1-4 and feature point 9, determines the feature point with the smallest similarity distance to feature point 9 as the temporary matching point of feature point 9. This corresponds to the above step S24.

[0180] Therefore, if the temporary matching point of the feature point 1 calculated above is feature point 5, then feature point 1 and feature point 5 are a first temporary matching pair; if the temporary matching point of the feature point 5 calculated above is feature point 1, then feature point 1 and feature point 5 are a second temporary matching pair. At this time, the electronic device can detect the existence of a consistent first temporary matching pair and a second temporary matching pair, in which the feature points are feature point 1 and feature point 5, so feature point 1 and feature point 5 can constitute a feature matching pair (corresponding to step S25 above).

[0181] In addition, if the temporary matching point of the feature point 1 calculated above is the feature point 5, and the temporary matching point of the feature point 5 calculated above is the feature point 2, then the feature point 1 and the feature point 5 cannot form a feature matching pair.

[0182] It can be seen from the above content that the feature points in the feature matching pairs determined based on the above steps S22-S25 are the feature points with the closest distance. Compared with the related art (see the above method 2) which only uses the feature points of the telephoto image as a reference, the feature matching pairs determined are more accurate. Therefore, the position where the telephoto image determined by the feature matching pairs determined based on the embodiment of the present application should be mapped in the enlarged short-focus image is more accurate, and the image fusion effect will be better.

[0183] In addition, since the above-mentioned method 2 takes the feature points of the telephoto image as a reference to determine the feature points that are most similar to the feature points in the telephoto image, and then obtains the feature matching degree, the number of feature matching pairs obtained in the above-mentioned method 2 should be the same as the data of the feature points in the telephoto image, and the feature matching determined based on the embodiment of the present application is only the best feature matching pair. For example, the feature matching pairs determined according to the above-mentioned method 2 are (feature point 1, feature point 4), (feature point 2, feature point 4), (feature point 3, feature point 6), and the feature matching pairs determined based on the scheme provided in the present application only have (feature point 2, feature point 4), (feature point 3, feature point 6). Therefore, the number of feature matching pairs determined based on the embodiment of the present application is less than the number of feature points determined in the above-mentioned method 2. It can be understood that in the embodiments of the present application, for the convenience of description, only a small number of feature points are used as an example for illustration, while in actual applications, the number of feature points in the image is relatively large. In actual applications, the number of feature matching pairs determined by the above-mentioned method 2 is approximately 2 to 3 times the number of feature matching pairs determined in the embodiments of the present application. It can be seen that the feature matching pair calculation method provided in the embodiments of the present application can not only improve the accuracy of image fusion, but also greatly reduce the amount of calculation when determining the first matching area in the subsequent step S26, thereby improving the computing efficiency.

[0184] In the above step S26, based on each feature matching pair, the electronic device can determine the area in the enlarged short-focus image that matches the original telephoto image to obtain a first matching area. It can be understood that the first matching area is the area in the enlarged short-focus image that matches the original telephoto image, determined based on each feature matching pair. Thus, the first matching area can be the area where the original telephoto image should be located in the enlarged short-focus image, obtained by directly calculating by the electronic device based on each feature matching pair. Alternatively, the first matching area can also be the area obtained by slightly adjusting the area where the original telephoto image should be located in order to make the original telephoto image more accurately integrated into the position of the enlarged short-focus image after the electronic device determines the area where the original telephoto image should be located based on each feature matching pair. For details, please refer to the following Figure 8 Description in .

[0185] In the above step S27, the neighborhood of the first matching area may refer to an area formed with the first matching area as the center. In a possible embodiment, in order to determine as many second matching areas as possible, the neighborhood of the first matching area should be large enough. For example, during the process of rotating the first matching area 0-360° in a clockwise direction, the area passed by the first matching area in the enlarged short-focus image should all be located within the neighborhood of the first matching area.

[0186] The second matching area is at least one area with the same size as the first matching area determined by the electronic device in the neighborhood of the first matching area, wherein the second matching area can be obtained by performing operations such as translation and rotation on the first matching area. For example, the first matching area can be translated a certain distance in the enlarged short-focus image, and the area occupied by the first matching area after translation in the enlarged short-focus image is used as the second matching area; or, the first matching area can be rotated a certain angle in the enlarged short-focus image, and the area occupied by the first matching area after rotation in the enlarged short-focus image is used as the second matching area; or, the first matching area can be rotated a certain angle in the enlarged short-focus image, and then the rotated first matching area can be translated a certain angle, and the area occupied by the first matching area after rotation and translation in the enlarged short-focus image is used as the second matching area. For details, please refer to the following Figure 8 Description in .

[0187] In the above step S28, the first brightness difference is the brightness difference between the first matching area calculated by the electronic device and the original telephoto image; the second brightness difference is the brightness difference between each second matching area calculated by the electronic device and the original telephoto image, that is, each second matching area corresponds to a second brightness difference, and each second brightness difference represents the brightness difference between the corresponding second matching area and the original telephoto image.

[0188] In the application embodiment, the similarity distance is used to indicate the degree of brightness difference between the matching area and the original telephoto image, wherein the similarity distance can be calculated by Euclidean distance, cosine distance, Hamming distance, brightness variance, etc. Below, the brightness variance and Euclidean distance are used as examples to illustrate the calculation method of the similarity distance:

[0189] When calculating the similarity distance by brightness variance, the electronic device may use the brightness value of each pixel in the first matching area and the brightness value of each pixel in the original telephoto image as a data set, calculate the variance of the data set as the first brightness difference. It is understandable that the data set includes the brightness value of each pixel in the first matching area and the brightness value of each pixel in the original telephoto image. And the electronic device may use the brightness value of each pixel in the second matching area and the brightness value of each pixel in the original telephoto image as a data set, calculate the variance of the data set as the second brightness difference corresponding to the second matching area. It is understandable that the data set includes the brightness value of each pixel in the second matching area and the brightness value of each pixel in the original telephoto image. Thus, the second brightness difference of each second matching area can be calculated respectively.

[0190] When calculating the similarity distance using Euclidean distance, it can be calculated using the following formula (1):

[0191] (1)

[0192] Wherein, distance represents the brightness difference between the matching area and the original telephoto image; a_i represents the brightness value of the i-th pixel in the matching area; b_i represents the brightness value of the pixel covering the i-th pixel in the matching area when the original telephoto image is mapped to the matching area; the value range of i is 1, 2, …, n, where n is the number of pixels in the matching area.

[0193] In the above step S29, the third matching area is the first matching area or the second matching area that meets a preset condition; the preset matching condition is that the brightness difference of the matching area is less than a preset threshold.

[0194] It can be understood that the brightness difference determined in the above step S28 can represent the brightness difference between the matching area and the original telephoto image. The smaller the brightness difference, the smaller the brightness difference between the matching area and the original telephoto image. For two images of the same scene taken at the same time, theoretically, the brightness of the same area in the two images should be almost the same. Therefore, the smaller the brightness difference between the matching area and the original telephoto image, the more likely it is that the matching area is the area in the telephoto image that should be mapped to the short-focus image.

[0195] Based on this, a threshold can be set, and each time a matching area is determined, the brightness difference between the matching area and the original telephoto image is calculated. When the brightness difference is less than a preset threshold, the matching area corresponding to the brightness difference is directly used as the third matching area, and the search for other matching areas is stopped; alternatively, the electronic device can also first determine all possible second matching areas, and then respectively calculate the brightness difference between the first matching area and each second matching area and the original telephoto image, and then, select any matching area from the matching areas whose corresponding brightness difference is less than the preset threshold as the third matching area, or select the brightness difference with the smallest value from the matching areas whose corresponding brightness difference is less than the preset threshold, and use the matching area corresponding to the brightness difference as the third matching area.

[0196] In the embodiment of the present application, since the second matching area is a matching area determined within the neighborhood of the first matching area, no matter whether the first matching area or any second matching area is used as the third matching area, in fact, the position of the original telephoto image in the enlarged short-focus image has only changed slightly, that is, the main position of the original telephoto image in the enlarged short-focus image is still determined based on the feature matching pair. Therefore, it can be understood that after the main position of the original telephoto image in the enlarged short-focus image is determined based on the feature matching pair, the position of the original telephoto image in the enlarged short-focus image is slightly adjusted based on the brightness difference. In this way, the position of the original telephoto image in the enlarged short-focus image is determined from the two perspectives of feature points and brightness values, thereby improving the accuracy of mapping the original telephoto image to the position in the enlarged short-focus image, thereby enabling the quality of the image to be fused.

[0197] In the above step S210, after determining the third matching area, the electronic device may map the original long-focus image to the third matching area in the amplified short-focus image, thereby obtaining a fused image.

[0198] In some embodiments, the mapping relationship between the third matching area and the original telephoto image can be expressed as a 3×3 mapping matrix, as shown in the following formula (2):

[0199] (2)

[0200] Among them, t00, t01, t02, ..., t22 are parameters in the coordinate mapping matrix;

[0201] According to the coordinate mapping matrix shown in the above formula (2), the coordinates where each pixel point in the original telephoto image should be located when mapping to the third matching area can be determined by the following formulas (3)-(4):

[0202] (3)

[0203] (4)

[0204] Among them, j_long represents the x-axis coordinate of the pixel point in the original telephoto image; i_long represents the y-axis coordinate of the pixel point in the original telephoto image; tx represents the coordinate on the x-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; ty represents the coordinate on the y-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; tz represents the coordinate on the z-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area.

[0205] In some embodiments, in the process of mapping the original telephoto image to the third matching area to obtain the fused image (see the above step S210), the electronic device may also perform low-pass filtering processing on the third matching area and the original telephoto image in the small half-warp range, respectively, to obtain the third matching area and the telephoto image after low-pass filtering processing. Since low-pass filtering can filter the noise in the image, the third matching area after low-pass filtering processing is called the third matching area after noise reduction, and the telephoto image after low-pass filtering processing is called the telephoto image after noise reduction. After that, the telephoto image after noise reduction is mapped to the third matching area after noise reduction to reduce the impact of noise on the quality of the fused image. It can be understood that in addition to low-pass filtering in the small half-warp range, other low-pass filtering can also be used to reduce the noise of the image, such as median filtering, Gaussian filtering, etc., but low-pass filtering in the small half-warp range has a fast calculation speed and can retain as many edges of objects in the image as possible, thereby reducing noise while avoiding the impact on image quality.

[0206] In addition, after the electronic device obtains the denoised telephoto image and the third matching area after denoising, it can also count the average brightness value of the third matching area after denoising as the first brightness average value; and count the average brightness value of the telephoto image after denoising as the second brightness average value; then, calculate the ratio of the first brightness average value to the second brightness average value as the target brightness adjustment value; calculate the product of the brightness value of each pixel in the denoised telephoto image and the target brightness adjustment value to obtain the target brightness value corresponding to each pixel in the denoised telephoto image; and adjust the brightness value of each pixel in the denoised telephoto image to the target brightness value corresponding to each pixel to obtain the target telephoto image, and map the target telephoto image to the third matching area after denoising, so that while achieving noise reduction, the brightness value of the original telephoto image is adjusted (that is, the brightness value of the fusion area is adjusted), so that in the fusion image, the brightness of the fusion area (that is, the area obtained after the denoised telephoto image is mapped to the third matching area) and the brightness outside the fusion area (such as the periphery of the boundary of the fusion area) are relatively consistent, thereby improving the quality of the fusion image.

[0207] The method of adjusting the brightness value of the telephoto image after noise reduction is further described by taking an example. For example, the average brightness value of the third matching area after noise reduction (i.e., the first brightness average value) is luma_ave_short; the average brightness value of the telephoto image after noise reduction (i.e., the second brightness average value) is luma_ave_long; then the target brightness adjustment value can be calculated by the following formula (5):

[0208] (5)

[0209] Among them, luma_ave_long is the average brightness value of the telephoto image after noise reduction; luma_ave_short is the average brightness value of the third matching area after noise reduction, and y_ratio is the target brightness adjustment value.

[0210] Therefore, based on the target brightness adjustment value calculated by the above formula (5), the target brightness value corresponding to each pixel in the telephoto image after noise reduction can be determined by the following formula (6):

[0211] (6)

[0212] Among them, long_img_y_new i is the target brightness value corresponding to the i-th pixel in the telephoto image after denoising, long_img_y i is the brightness value of the ith pixel in the denoised telephoto image. Therefore, after determining the target brightness value corresponding to the ith pixel in the denoised telephoto image, the brightness value of the ith pixel in the denoised telephoto image is adjusted to the target brightness value corresponding to the ith pixel. Based on this, after adjusting the brightness values ​​of each pixel in the denoised telephoto image, the telephoto image obtained is the target telephoto image.

[0213] Understandably, the above Figure 2 Although the method shown can greatly improve the registration accuracy of the telephoto image during fusion, in actual use, especially in the on-site calibration configuration, the scene is complex and changeable, so there may still be some errors, for example, Figure 7 As shown, Figure 7 A schematic diagram of a fused image provided in an embodiment of the present application, Figure 7 There is a small misalignment of the boundary of the black line in the fusion area shown in the middle frame 71 (see the area with diagonal lines in the area shown in the frame 71). Based on this, the embodiment of the present application also provides a detailed implementation step for the above steps S26 and S27 (as shown below): Figure 8 As shown in the figure, the registration accuracy of the telephoto image in the fused image can be further improved, thereby improving the quality of the fused image.

[0214] See also Figure 8 , Figure 8 A detailed schematic diagram of the above steps S26 and S27 provided in the embodiment of the present application may include the following steps:

[0215] Step S81: determining, according to the feature matching pair, an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area;

[0216] Step S82: controlling the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and using the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area;

[0217] Step S83: calculating the similarity distances between the fourth matching area and each of the fifth matching areas and the original telephoto image respectively as the third brightness difference corresponding to the fourth matching area and the fourth brightness difference corresponding to each of the fifth matching areas;

[0218] Step S84: determining a matching area satisfying a preset matching condition in the fourth matching area and the fifth matching area according to the brightness difference between the fourth matching area and the fifth matching area as the first matching area;

[0219] Step S85: controlling the first matching area to translate a second preset number of times along a second preset direction according to a preset translation unit, and taking the area occupied by the first matching area in the enlarged short-focus image after each translation as the second matching area.

[0220] In the technical solution provided in the embodiment of the present application, on the basis of determining the fourth matching area according to the feature matching pair, the position of the fourth matching area is adjusted based on the brightness difference between the original telephoto image and the fourth matching area and the fifth matching area through operations such as rotation, and the first matching area is obtained. Then, based on the first matching area, multiple second matching areas are determined through operations such as translation, that is, the second matching areas are obtained based on the brightness difference adjustment after translation and rotation operations on the basis of the fourth matching area. Therefore, the third matching area calculated based on the brightness difference of the second matching area has a higher matching degree with the original telephoto image, and the quality of the fused image obtained after mapping the original telephoto image to the third matching area is higher.

[0221] In the above step S81, the fourth matching area is an area that matches the original long-focus image and is determined by the electronic device in the enlarged short-focus image according to the feature matching pair.

[0222] In the above step S82, the preset rotation unit is the preset angle to be rotated each time; the first preset direction is the preset rotation direction for each rotation; the fifth matching area is the area occupied by the fourth matching area in the enlarged short-focus image after the preset rotation unit is rotated toward the first preset direction each time. Among them, the preset rotation unit can be 1 degree (°), 2°, 3°, etc.; the preset rotation direction can be clockwise or counterclockwise; the first preset number of times can be directly set to 60 times, 100 times, 200 times, etc.; or, the number of times to be rotated can be determined by the range to be rotated and the preset rotation unit. For example, the rotation range can be set to 0°~360°, and the preset rotation unit can be set to 1°. In this way, the preset number of times can be set to 360 times.

[0223] In the embodiment of the present application, the electronic device can control the fourth matching area to rotate a preset number of times along the first preset direction according to a preset rotation unit, and use the area occupied by the fourth matching area after each rotation in the enlarged short-focus image as the fifth matching area.

[0224] The following is a specific example of the method for determining the fifth matching area provided in the embodiment of the present application. For example, see Fig. 9 , Fig. 9 A schematic diagram of determining a fifth matching area provided in an embodiment of the present application, Fig. 9 , the preset rotation unit 91 is set to 1°, and the first preset direction is set to the clockwise direction 92. Then, a fifth matching area can be obtained by controlling the fourth matching area 93 to rotate 1° along the clockwise direction 92. That is, after the fourth matching area 93 is rotated 1° along the clockwise direction, the area in the enlarged short-focus image occupied by the current fourth matching area 93 is a fifth matching area, which can be called matching area 1; thereafter, on the basis of having rotated 1°, the fourth matching area 93 is controlled to rotate 1° along the clockwise direction 92 again, that is, the current fourth matching area 93 has rotated 2°. At this time, the area in the enlarged short-focus image occupied by the current fourth matching area 93 is a fifth matching area again, which can be called matching area 2; thereafter, the fourth matching area 93 is controlled to rotate 1° along the clockwise direction 92 again on the basis of having rotated 2°, and a fifth matching area can be obtained, which can be called matching area 3. By analogy, after the first preset number of rotations, the first preset number of fifth matching areas can be obtained.

[0225] In the above step S83, the third brightness difference is the calculated brightness difference between the fourth matching area and the original telephoto image; the fourth brightness difference is the calculated brightness difference between the fifth matching area and the original telephoto image. It can be understood that there are multiple fifth matching areas, and each fifth matching area corresponds to a fourth brightness difference.

[0226] In the embodiment of the present application, the principle of calculating the brightness similarity distance is the same as the principle of calculating the similarity distance in the above step S28. The only difference is that the matching area in the above step S28 is required to refer to the first matching area or the second matching area; while the matching area in the embodiment of the present application refers to the fourth matching area or the fifth matching area.

[0227] In the above step S84, according to the brightness difference between the fourth matching area and the fifth matching area, a matching area that meets the preset matching condition can be determined in the fourth matching area and the fifth matching area as the first matching area. The principle of determining the first matching area is the same as the principle of determining the third matching area, and the details can be referred to the description in the above step S29.

[0228] In the above step S85, the preset translation unit can be understood as the preset number of pixel points to be translated each time, such as 10, 15, 20, etc.; the second preset direction is the preset direction of each translation, and the second preset direction can be upward, downward, left, or right; the second matching area is the area occupied by the first matching area in the enlarged short-focus image after the first matching area is translated by the preset translation unit in the first preset direction each time; the second preset number of times is the preset number of translations in the second preset direction, such as 5 times, 10 times, 15 times, etc.

[0229] In the embodiment of the present application, the electronic device can control the first matching area to translate a second preset number of times along a second preset direction according to a preset translation unit, and use the area occupied by the first matching area after each translation in the enlarged short-focus image as the second matching area.

[0230] The following is a specific example of the method for determining the second matching area provided in the embodiment of the present application. For example, see Fig.10 , Fig.10 A schematic diagram of determining a second matching area provided in an embodiment of the present application, Fig.10 In the embodiment, the preset translation unit is set to 10 pixels, the second preset directions include upward 101, downward 102, left 103, and right 104, and the second preset number of times is set to 10 times. The electronic device needs to control the first matching area 105 to translate upward 101, downward 102, left 103, and right 104 10 times respectively, and a total of 40 second matching areas can be obtained.

[0231] The specific translation process is described in one direction. Taking the translation in the direction as an example, the first matching area 105 is controlled to translate 10 pixels upward 101, and the area in the enlarged short-focus image currently occupied by the first matching area 105 is a second matching area; then, on the basis of the translation of 10 pixels upward 101, the first matching area 105 is controlled to translate another 10 pixels upward 101, that is, the first matching area 105 has been translated 20 pixels upward 101. At this time, the enlarged short-focus image currently occupied by the first matching area 105 is The area in is another second matching area; after that, the first matching area 105 is controlled to be translated upward 101 by 10 pixels on the basis of being translated upward 101 by 20 pixels, and then a second matching area can be obtained, and so on, until the translation is performed 10 times, 10 second matching areas are obtained, and then starting from the initial first matching area 101 (that is, the first matching area that has not been translated), the first matching area is controlled to move in the next direction until the second preset number of times is reached, thereby traversing each direction indicated by the second preset direction in turn.

[0232] Understandably, Figure 8 What is shown is only a possible detailed schematic diagram of step S26-step S27 provided by the present application. In other possible embodiments, as described in the above step S26, the first matching area can be the area in the enlarged short-focus image determined according to the feature matching pair that matches the original telephoto image. In this case, the subsequent second matching area can be obtained in accordance with the method shown in the above step S82, and it is only necessary to regard the fourth matching area in step S82 as the first matching area, and the fifth matching area as the second matching area; alternatively, the second matching area can also be determined by referring to the above step S85, the only difference being that the first matching area at this time is the area in the enlarged short-focus image determined according to the feature matching pair that corresponds to the original telephoto image.

[0233] Alternatively, in some embodiments, after determining the fourth matching area, the second matching area can be determined by referring to step S82 and step S85 at the same time. For example, the fourth area is first rotated by a preset unit, and then the fourth matching area is translated by a preset translation unit. In this way, a second matching area is obtained, and so on, multiple second matching areas can be obtained.

[0234] Alternatively, in other embodiments, the first matching area in the above-mentioned step S26 may be the first matching area determined in step S85 after the fourth matching area is determined according to step S81. In this case, it is only necessary to regard the first matching area in step S85 as the fourth matching area, and the second matching area as the fifth matching area. Then, the first matching area is determined in the fourth matching area and the fifth matching area. After that, the existing step S85 is replaced by step S82, that is, the fourth matching area in step S82 is regarded as the first matching area, and the fifth matching area is regarded as the second matching area.

[0235] In addition, in the above Figure 8 In the illustrated embodiment, after the electronic device determines the fourth matching area, it can also directly map the telephoto image to the fourth matching area based on the mapping relationship between the fourth matching area and the original telephoto image. In the subsequent steps, the fused original telephoto image is directly regarded as the first matching area, and the position of the fused telephoto image is adjusted. When it is adjusted to the third matching area, the fused image can be obtained.

[0236] In some embodiments, after the fused image is obtained through the above step S210, the fused image may be further processed. Fig.11 , Fig.11 A schematic diagram of a process for processing a fused image provided in an embodiment of the present application may include the following steps:

[0237] Step S111: In the fused image, for a boundary of the third matching area, a neighborhood of the boundary facing the inside of the boundary is divided into a plurality of first candidate blocks, and a neighborhood of the boundary facing the outside of the boundary is divided into a plurality of second candidate blocks;

[0238] Step S112: for each first candidate block, determining a plurality of continuous first candidate blocks centered on the first candidate block as first target blocks corresponding to the first candidate blocks, and determining a second candidate block located on a normal direction of a boundary of each first target block as a second target block corresponding to the first target block;

[0239] Wherein, the boundary direction is parallel to the normal direction of the boundary at the first target block and faces the outside of the boundary;

[0240] Step S113: for each first candidate block, calculating the third brightness average of each first target block and the second target block corresponding to the first candidate block;

[0241] Step S114: for each first candidate block, adjusting the brightness of each first target block and second target block corresponding to the first candidate block towards a direction close to the third brightness average.

[0242] In the technical solution provided in the embodiment of the present application, in order to reduce the brightness difference of pixel values ​​inside and outside each boundary of the third matching area in the fused image, for the boundary of the third matching area, a plurality of first candidate blocks are divided on the neighborhood inside the boundary, and the neighborhood outside the boundary is divided into a plurality of second candidate blocks; a plurality of first target blocks are determined from the first candidate blocks, and a second target block corresponding to each first target block is determined from the second candidate blocks, the third brightness mean of the first target block and the second target block is calculated, and for each first candidate block, the brightness of each first target block and the second target block corresponding to the first candidate block is adjusted in a direction close to the third brightness mean.

[0243] In the above step S111, the neighborhood inside the boundary can be understood as an area extending from the boundary as the starting point to the inside of the boundary; the neighborhood outside the boundary can be understood as an area extending from the boundary as the starting point to the outside of the boundary, wherein the area inside the boundary should be as consistent as possible with the neighborhood size outside the boundary, the inside of the boundary can be understood as the inside of the third matching area, and the outside of the boundary can be understood as the outside of the third matching area in the fused image. The first candidate block is a candidate block obtained by dividing the neighborhood inside the boundary of the third matching area; the second candidate block is a candidate block obtained by dividing the neighborhood outside the boundary.

[0244] It can be understood that the third matching area has four boundaries, namely, top, bottom, left, and right. Each boundary has a corresponding first candidate block and a second candidate block. The first candidate block and the second candidate block in the same boundary are corresponding candidate blocks. In this article, the first candidate block and the second candidate block that appear at the same time when processing in the following steps are both defaulted to candidate blocks in the same boundary.

[0245] In the above step S112, for each first candidate block, multiple consecutive first candidate blocks are taken with each candidate block as the center as the first target block corresponding to the first candidate block; wherein the number of candidate blocks in the first target block can be set according to actual needs.

[0246] After determining the first target block, the electronic device can determine a second candidate block on the boundary normal of each first target block as the second target block corresponding to the first target block, wherein the boundary direction is parallel to the normal of the boundary at the first target block and faces outside the boundary.

[0247] It can be understood that a plurality of first target blocks can be obtained from the first candidate block, and each first target block has a corresponding second target block.

[0248] In the above step S113, for each first candidate block, the electronic device can determine the brightness mean of each first target block and the second target block corresponding to the first target block as the third brightness mean. That is, the third brightness mean is the brightness mean of the first target block and the second target block corresponding to the first target block. For example, there are first target block 1 and first target block 2, the second target block corresponding to the first target block 1 is second target block 3, and the second target block corresponding to the first target block 2 is second target block 4. Then the brightness mean of the first target block 1 and the second target block 3 is the third brightness mean corresponding to the first target block 1 and the second target block 3, and the brightness mean of the first target block 2 and the second target block 4 is the third brightness mean corresponding to the first target block 2 and the second target block 4.

[0249] In the above step S114, for each first candidate block, the electronic device can adjust the brightness values ​​of each first target block and second target block corresponding to the first candidate block in the direction of approaching the third brightness mean. Continuing with the example shown in the above step S53, that is, the brightness values ​​of each block in the first target block 1 and the second target block 3 are adjusted toward the third brightness mean corresponding to the first target block 1 and the second target block 3; the brightness values ​​of each block in the first target block 2 and the second target block 4 are adjusted toward the third brightness mean corresponding to the first target block 2 and the second target block 4. In this way, the brightness values ​​of each first candidate block and each second candidate block are adjusted.

[0250] In some embodiments, the electronic device can specifically implement the above step S114 in the following manner. For ease of description, the following only describes the adjustment method of a pair of first target blocks and second target blocks. The adjustment principle of other target blocks is the same. Specifically,

[0251] The electronic device may calculate the average brightness value of the first candidate block as the fourth average brightness value; and calculate the average brightness value of the second candidate block as the fifth average brightness value, and then calculate the difference of the average brightness values ​​by the following formula (7):

[0252] (7)

[0253] Among them, a_i represents the fourth brightness mean, or the fifth brightness mean; m_ave represents the third brightness mean, and dif_i represents the difference in brightness mean.

[0254] If the difference in the mean brightness is greater than 0, the brightness value of the target block to be adjusted is calculated using the following formula (8):

[0255] (8)

[0256] Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; and a' is the brightness value to be adjusted to the target block.

[0257] If the difference in the mean brightness is less than 0, the brightness value to be adjusted to the target block is calculated using the following formula (9):

[0258] (9)

[0259] Among them, a is the brightness value of the target block; dif_i is the difference in brightness mean; a' is the brightness value to be adjusted for the target block; abs() means taking the absolute value.

[0260] In the embodiment of the present application, after the brightness value to be adjusted for each target block is determined, the brightness value of each target block is adjusted to the brightness value to be adjusted.

[0261] Next, a specific example is used to describe the method of adjusting the brightness values ​​inside and outside the boundary of the third matching area (i.e. Fig.11 For ease of description, in the embodiment of the present application, only the upper boundary of the third matching area is used as an example for illustration, and the principles of other boundaries are the same and will not be repeated here.

[0262] See also Fig.12 , Fig.12 A schematic diagram of a dynamic border stitching provided in an embodiment of the present application, such as Fig.12 As shown, the upper and lower areas of the upper boundary of the third fusion area 122 (i.e., the area obtained after the long-focus image is mapped to the third matching area) in the enlarged short-focus image 121 are respectively divided into n small areas, that is, divided into n first candidate blocks and n second candidate blocks, respectively, wherein a1, a2, a3, a4, a5, a6, a7…an can be understood as the second candidate blocks, and b1, b2, b3, b4, b5, b6, b7…bn can be understood as the second candidate blocks ( Fig.12 In order to make the image clear, only a1-a8 and b1-b8 are shown, and a9-an and b9-bn are not shown); the electronic device can calculate the brightness average of a1-an, that is, the fifth brightness average, recorded as b_i; and can calculate the brightness average of b1-bn, that is, the fourth brightness average, recorded as a_i. Among them, the width of the enlarged short-focus image 121 is 2w, and the length is 2h; the width of the third fusion area 122 is w, and the length is h.

[0263] Take m blocks (i.e., the first target block) from the first candidate block, and take m candidate blocks (i.e., the second target block) corresponding to the m blocks from the second candidate block to calculate the average brightness of the 2m blocks (i.e., the third average brightness), which is recorded as m_ave. For example, if m is 5, the average brightness of the 10 blocks a1, a2, a3, a4, a5, b1, b2, b3, b4, and b5 is calculated.

[0264] Calculate the difference between the third brightness mean and the fourth brightness mean (b_i-m_ave)=dif_b_i respectively; and the difference between the third brightness mean and the fifth brightness mean (a_i-m_ave)=dif_a_i; if dif_a_i is greater than 0, then a1=a1-(dif_a_i / 2), if dif_a_i is less than 0, then a1=a1+(abs(dif_a_i) / 2), a2-a5 is calculated in the same way as a1, and thus the brightness value to be adjusted to of a1-a5 is calculated; if dif_b_i is greater than 0, then b1=b1-(dif_b_i / 2), if dif_b_i is less than 0, then b1=b1+(abs(dif_b_i) / 2), b2-b5 is calculated in the same way as b1, and thus the brightness value to be adjusted to of b1-b5 is calculated.

[0265] Based on the above adjustment steps for a1-a5 and b1-b5, each time the continuous m block area is calculated, t blocks are slid in sequence. For example, t is set to 3, that is, the average brightness of a1, a2, a3, a4, and a5 is taken for the first time, the average brightness of a4, a5, a6, a7, and a8 is taken for the second time, and the average brightness of a7, a8, a9, a10, and a11 is taken for the third time, and so on, and the brightness values ​​to be adjusted to each first candidate block and each second candidate block are calculated.

[0266] Corresponding to the above-mentioned image fusion method, the present application embodiment also provides an image fusion device, see Fig.13 , Fig.13 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application, the device includes:

[0267] Image acquisition module 131: used to acquire an original long-focus image and an original short-focus image; and to amplify the original short-focus image to obtain an amplified short-focus image; wherein the real space size corresponding to each pixel in the amplified short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image;

[0268] The feature extraction module 132 is used to extract feature points in the original long-focus image as first feature points; and extract feature points in the enlarged short-focus image as second feature points;

[0269] The matching pair acquisition module 133 is used to determine the temporary matching points of each first feature point respectively, and use the first feature point and the temporary matching point of the first feature point as a first temporary matching pair; wherein the temporary matching point of the first feature point is the second feature point that is closest to the first feature point;

[0270] The matching pair acquisition module 133 is further used to respectively determine temporary matching points of each second feature point, and use the second feature point and the temporary matching point of the second feature point as a second temporary matching pair;

[0271] The matching pair acquisition module 133 is further configured to use the first temporary matching pair or the second temporary matching pair as a feature matching pair if there is a second temporary matching pair that is consistent with the first temporary matching pair;

[0272] A matching region determining module 134 is used to determine, based on the feature matching pair, a region in the amplified short-focus image that matches the original long-focus image, to obtain a first matching region;

[0273] The matching region determination module 134 is further configured to determine at least one second matching region in the neighborhood of the first matching region; wherein the size of the second matching region is the same as the size of the first matching region;

[0274] A similarity distance calculation module 135, used to calculate the similarity distances between the first matching area and each second matching area and the original telephoto image, as the first brightness difference corresponding to the first matching area, and the second brightness difference corresponding to each second matching area; wherein the similarity distance is used to indicate the degree of brightness difference;

[0275] The matching region determination module 134 is further configured to determine, according to the brightness difference between the first matching region and the second matching region, a third matching region that meets a preset matching condition within the first matching region and the second matching region; wherein the preset matching condition is that the brightness difference between the matching regions is less than a preset threshold;

[0276] The image fusion module 136 is used to map the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image.

[0277] In the technical solution provided by the embodiment of the present application, an original long-focus image and an original short-focus image are obtained, and the original short-focus image is enlarged to obtain an enlarged short-focus image, so that the real space size corresponding to each pixel in the enlarged short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image, and the fusion effect of the original long-focus image and the enlarged short-focus image is better; feature points in the original long-focus image are extracted as first feature points, and feature points in the enlarged short-focus image are extracted as second feature points; temporary matching points of each first feature point are determined respectively, that is, the second feature point closest to the first feature point, and the temporary matching points of the first feature point and the first feature point are used as a first temporary matching pair; then, temporary matching points of each second feature point are determined respectively, and the temporary matching points of the second feature point and the second feature point are used as a first temporary matching pair. The matching points are used as a second temporary matching pair. If there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Based on each feature matching pair, the area that matches the original telephoto image can be determined in the enlarged short-focus image to obtain a first matching area. When there is a second temporary matching pair that is consistent with the first temporary matching pair, the temporary matching pair is used as a feature matching pair. Then, based on each feature matching pair, the area that matches the original telephoto image in the enlarged short-focus image is determined. In this way, the registration accuracy between the original telephoto image and the enlarged short-focus image can be greatly improved, and the registration error can be reduced. That is, the first matching area that matches the original telephoto image can be determined more accurately, and the position where each pixel point in the original telephoto image should be when mapped to the enlarged short-focus image can be accurately determined.

[0278] After determining the first matching area, at least one second matching area with the same size as the first matching area is determined in the neighborhood of the first matching area, and the similarity distances between the first matching area and each second matching area and the original telephoto image are calculated respectively as the first brightness difference corresponding to the first matching area and the second difference corresponding to each second matching area; then, based on the brightness difference corresponding to the first matching area and the second matching area, a third matching area that meets the preset matching condition is determined within the first matching area and the second matching area, that is, a matching area whose brightness difference is less than a preset threshold; then, the original telephoto image is mapped to the third matching area in the enlarged short-focus image to obtain a fused image. It can be understood that the aforementioned feature matching pair-based method more accurately determines the area in the enlarged short-focus image that matches the original telephoto image, that is, the first matching area. Therefore, at least one second matching area is subsequently determined in the neighborhood of the first matching area, and the brightness difference between the first matching area, the second matching area and the original telephoto image is determined. The area with a brightness difference less than a preset threshold is used as the third matching area, and the original telephoto image is mapped to the third matching area to obtain a fused image. That is, after determining the first matching area that matches the original telephoto image in the enlarged short-focus image based on the feature matching pair from the perspective of feature points, the position of the first area in the enlarged short-focus image that matches the original telephoto image is adjusted from the perspective of brightness value to obtain a third matching area that better matches the original telephoto image. In this way, after mapping the original telephoto image to the third matching area, the quality of the fused image can be improved.

[0279] In some embodiments, the matching area determination module 134 is specifically configured to:

[0280] According to the feature matching pair, determining an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area;

[0281] Control the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and use the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area;

[0282] respectively calculating similarity distances between the fourth matching region and each of the fifth matching regions and the original telephoto image as third brightness difference corresponding to the fourth matching region and fourth brightness difference corresponding to each of the fifth matching regions;

[0283] According to the brightness difference between the fourth matching area and the fifth matching area, determining a matching area that meets a preset matching condition in the fourth matching area and the fifth matching area as a first matching area;

[0284] The first matching area is controlled to translate a second preset number of times along a second preset direction according to a preset translation unit, and the area occupied by the first matching area in the enlarged short-focus image after each translation is used as the second matching area.

[0285] In some embodiments, the coordinates at which each pixel in the original telephoto image should be located when mapped to the third matching area are determined by:

[0286] The mapping relationship between the third matching area and the original telephoto image is expressed as the following 3×3 coordinate mapping matrix:

[0287]

[0288] Among them, t00, t01, and t02 are parameters in the coordinate mapping matrix;

[0289] According to the coordinate mapping matrix, the coordinates where each pixel in the original telephoto image should be located when mapped to the third matching area are determined by the following formula:

[0290]

[0291] Among them, j_long represents the x-axis coordinate of the pixel point in the original telephoto image; i_long represents the y-axis coordinate of the pixel point in the original telephoto image; tx represents the coordinate on the x-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; ty represents the coordinate on the y-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; tz represents the coordinate on the z-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area.

[0292] In some embodiments, the image fusion module 136 is specifically configured to:

[0293] Performing low-pass filtering within a small radius on the third matching area and the original telephoto image respectively to obtain a denoised third matching area and a denoised telephoto image;

[0294] Counting the average brightness of the third matching area after noise reduction as the first average brightness; and counting the average brightness of the telephoto image after noise reduction as the second average brightness;

[0295] Calculate the ratio of the first brightness average value to the second brightness average value as the target brightness adjustment value;

[0296] Calculate the product of the brightness value of each pixel in the denoised telephoto image and the target brightness adjustment value as the target brightness value corresponding to each pixel in the denoised telephoto image;

[0297] The brightness value of each pixel in the denoised telephoto image is adjusted to the target brightness value corresponding to each pixel to obtain a target telephoto image;

[0298] The target telephoto image is mapped to the third matching area after noise reduction to obtain a fused image.

[0299] In some embodiments, the image fusion device further includes:

[0300] A candidate block division module is used to divide the neighborhood of the boundary facing the inside of the boundary into a plurality of first candidate blocks, and divide the neighborhood of the boundary facing the outside of the boundary into a plurality of second candidate blocks in the fused image for the boundary of the third matching area;

[0301] a target block determination module, for determining, for each first candidate block, a plurality of continuous first candidate blocks centered on the first candidate block as the first target block corresponding to the first candidate block, and determining a second candidate block located on the boundary normal direction of each first target block as the second target block corresponding to the first target block, wherein the boundary direction is parallel to the normal direction of the boundary at the first target block and faces the outside of the boundary;

[0302] A brightness mean value calculation module, used for calculating, for each first candidate block, a third brightness mean value of each first target block and a second target block corresponding to the first candidate block;

[0303] The brightness adjustment module is used to adjust the brightness of each first target block and the second target block corresponding to each first candidate block towards a direction close to a third brightness average value.

[0304] In some embodiments, the brightness adjustment module is specifically used to:

[0305] The brightness mean of each first candidate block is calculated as the fourth brightness mean; and the brightness mean of the second candidate block is calculated as the fifth brightness mean, and the difference of the brightness mean is calculated by the following formula:

[0306]

[0307] Wherein, a_i represents the fourth brightness mean value, or the fifth brightness mean value; m_ave represents the third brightness mean value, and dif_i represents the difference between the brightness mean values;

[0308] If the difference in the mean brightness is greater than 0, the brightness value to be adjusted to the target block is calculated using the following formula:

[0309]

[0310] Where a is the brightness value of the target block; dif_i is the difference in the mean brightness; a is the brightness value to be adjusted for the target block;

[0311] If the difference in the mean brightness is less than 0, the following formula is used to calculate the brightness value of the target block to be adjusted:

[0312]

[0313] Where a is the brightness value of the target block; dif_i is the difference in the mean brightness; a' is the brightness value to be adjusted for the target block; abs() means taking the absolute value;

[0314] The brightness value of the target block is adjusted to the brightness value to be adjusted.

[0315] In some embodiments, the brightness difference between each matching area and the original telephoto image is calculated by the following formula:

[0316]

[0317] Wherein, distance represents the brightness difference between the matching area and the original telephoto image; a_i represents the brightness value of the i-th pixel in the matching area; b_i represents the brightness value of the pixel covering the i-th pixel in the matching area when the original telephoto image is mapped to the matching area; the value range of i is 1, 2, …, n, where n is the number of pixels in the matching area.

[0318] The present application also provides an electronic device, such as Fig.14 As shown, including:

[0319] A memory 141, for storing computer programs;

[0320] The processor 142 is used to implement any of the above-mentioned image fusion methods when executing the program stored in the memory 141.

[0321] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 142, the communication interface, and the memory 141 communicate with each other via the communication bus.

[0322] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0323] The communication interface is used for communication between the above electronic device and other devices.

[0324] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0325] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0326] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image fusion methods are implemented.

[0327] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image fusion method described in any one of the above embodiments.

[0328] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0329] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0330] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0331] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire an original long-focus image and an original short-focus image; and amplify the original short-focus image to obtain an amplified short-focus image; wherein the real space size corresponding to each pixel in the amplified short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image; Extracting feature points in the original long-focus image as first feature points; and extracting feature points in the amplified short-focus image as second feature points; Determine the temporary matching point of each of the first feature points respectively, and use the first feature point and the temporary matching point of the first feature point as a first temporary matching pair; wherein the temporary matching point of the first feature point is the second feature point that is closest to the first feature point; respectively determining temporary matching points of the second feature points, and taking the second feature points and the temporary matching points of the second feature points as second temporary matching pairs; If there is a second temporary matching pair that is consistent with the first temporary matching pair, taking the first temporary matching pair or the second temporary matching pair as a feature matching pair; Based on each of the feature matching pairs, determining an area in the amplified short-focus image that matches the original long-focus image to obtain a first matching area; Determine at least one second matching area in the neighborhood of the first matching area; wherein the size of the second matching area is the same as the size of the first matching area; Calculating the similarity distances between the first matching area and each of the second matching areas and the original telephoto image respectively as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each of the second matching areas; wherein the similarity distance is used to indicate the degree of brightness difference; According to the brightness difference between the first matching area and the second matching area, determining a third matching area that meets a preset matching condition in the first matching area and the second matching area; wherein the preset matching condition is that the brightness difference between the matching areas is less than a preset threshold; The original long-focus image is mapped to the third matching area in the amplified short-focus image to obtain a fused image.

2. The method according to claim 1, characterized in that The step of determining, based on each of the feature matching pairs, an area in the amplified short-focus image that matches the original long-focus image to obtain a first matching area includes: According to the feature matching pair, determining an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area; Controlling the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and using the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area; respectively calculating similarity distances between the fourth matching region and each of the fifth matching regions and the original telephoto image as third brightness difference degrees corresponding to the fourth matching region and fourth brightness difference degrees corresponding to each of the fifth matching regions; According to the brightness difference between the fourth matching area and the fifth matching area, determining a matching area satisfying the preset matching condition within the fourth matching area and the fifth matching area as the first matching area; The determining of at least one second matching area in the neighborhood of the first matching area includes: The first matching area is controlled to translate a second preset number of times along a second preset direction according to a preset translation unit, and an area occupied by the first matching area in the magnified short-focus image after each translation is used as the second matching area.

3. The method according to claim 1, characterized in that: The coordinates at which each pixel point in the original telephoto image should be located when mapped to the third matching area are determined in the following manner: The mapping relationship between the third matching area and the original telephoto image is expressed as the following 3×3 coordinate mapping matrix: ; Wherein, t00, t01, t02 are parameters in the coordinate mapping matrix; According to the coordinate mapping matrix, the coordinates at which each pixel point in the original telephoto image should be located when mapped to the third matching area are determined by the following formula: ; Among them, j_long represents the x-axis coordinate of the pixel point in the original telephoto image; i_long represents the y-axis coordinate of the pixel point in the original telephoto image; tx represents the coordinate on the x-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; ty represents the coordinate on the y-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area; tz represents the coordinate on the z-axis where the pixel point in the original telephoto image should be located when mapped to the third matching area.

4. The method according to claim 1, characterized in that: Mapping the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image includes: Performing low-pass filtering within a small radius on the third matching area and the original telephoto image respectively to obtain a denoised third matching area and a denoised telephoto image; Counting the average brightness of the third matching area after noise reduction as the first brightness average; and calculating a brightness average of the denoised telephoto image as a second brightness average; Calculating a ratio of the first brightness average value to the second brightness average value as a target brightness adjustment value; Calculating the product of the brightness value of each pixel in the denoised telephoto image and the target brightness adjustment value as the target brightness value corresponding to each pixel in the denoised telephoto image; Adjusting the brightness value of each pixel in the denoised telephoto image to a target brightness value corresponding to each pixel to obtain a target telephoto image; The target telephoto image is mapped to the third matching area after noise reduction to obtain a fused image.

5. The method according to claim 1, characterized in that The method further comprises, In the fused image, for a boundary of the third matching area, a neighborhood of the boundary facing the inside of the boundary is divided into a plurality of first candidate blocks, and a neighborhood of the boundary facing the outside of the boundary is divided into a plurality of second candidate blocks; For each of the first candidate blocks, a plurality of continuous first candidate blocks centered on the first candidate block are determined as first target blocks corresponding to the first candidate blocks, and a second candidate block located on a normal direction of a boundary of each of the first target blocks is determined as a second target block corresponding to the first target block, wherein the boundary direction is parallel to the normal direction of the boundary at the first target block and faces outside the boundary; For each of the first candidate blocks, respectively, calculate a third brightness average value of each of the first target blocks and the second target block corresponding to the first candidate block; For each of the first candidate blocks, the brightness of each of the first target blocks and the second target block corresponding to the first candidate block is adjusted in a direction close to the third brightness average.

6. The method according to claim 5, characterized in that The adjusting, for each of the first candidate blocks, the brightness of each of the first target blocks and the second target block corresponding to the first candidate blocks in a direction close to the third brightness average value, includes: The brightness mean of each of the first candidate blocks is calculated as the fourth brightness mean; and the brightness mean of the second candidate blocks is calculated as the fifth brightness mean, and the difference of the brightness mean is calculated by the following formula: ; Wherein, a_i represents the fourth brightness mean value, or the fifth brightness mean value; m_ave represents the third brightness mean value, and dif_i represents the difference between the brightness mean values; If the difference in the brightness mean is greater than 0, the brightness value to be adjusted to the target block is calculated using the following formula: ; Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted to the target block; If the difference in the brightness mean is less than 0, the brightness value to be adjusted to the target block is calculated using the following formula: ; Wherein, a is the brightness value of the target block; dif_i is the difference of the brightness mean; a' is the brightness value to be adjusted for the target block; abs() means taking the absolute value; The brightness value of the target block is adjusted to the brightness value to be adjusted.

7. The method according to claim 1, characterized in that The brightness difference between each matching area and the original telephoto image is calculated by the following formula: ; Wherein, distance represents the brightness difference between the matching area and the original telephoto image; a_i represents the brightness value of the i-th pixel in the matching area; b_i represents the brightness value of the pixel covering the i-th pixel in the matching area when the original telephoto image is mapped to the matching area; the value range of i is 1, 2, …, n, where n is the number of pixels in the matching area.

8. An image fusion device, characterized in that: The device comprises: An image acquisition module, used to acquire an original long-focus image and an original short-focus image; and to amplify the original short-focus image to obtain an amplified short-focus image; wherein the real space size corresponding to each pixel in the amplified short-focus image is consistent with the real space size corresponding to each pixel in the original long-focus image; A feature extraction module, used to extract feature points in the original long-focus image as first feature points; and extract feature points in the amplified short-focus image as second feature points; a matching pair acquisition module, used to respectively determine temporary matching points of each of the first feature points, and use the first feature point and the temporary matching point of the first feature point as a first temporary matching pair; wherein the temporary matching point of the first feature point is the second feature point that is closest to the first feature point; The matching pair acquisition module is further used to respectively determine temporary matching points of each of the second feature points, and use the second feature point and the temporary matching point of the second feature point as a second temporary matching pair; The matching pair acquisition module is further configured to use the first temporary matching pair or the second temporary matching pair as a feature matching pair if there is a second temporary matching pair that is consistent with the first temporary matching pair; a matching region determining module, configured to determine, based on each of the feature matching pairs, a region in the amplified short-focus image that matches the original long-focus image, to obtain a first matching region; The matching region determination module is used to determine at least one second matching region in the neighborhood of the first matching region; wherein the size of the second matching region is the same as the size of the first matching region; a similarity distance calculation module, used to respectively calculate the similarity distances between the first matching area and each of the second matching areas and the original telephoto image as the first brightness difference corresponding to the first matching area and the second brightness difference corresponding to each of the second matching areas; wherein the similarity distance is used to represent the degree of brightness difference; The matching region determination module is further configured to determine, according to the brightness difference between the first matching region and the second matching region, a third matching region that meets a preset matching condition within the first matching region and the second matching region; wherein the preset matching condition is that the brightness difference between the matching regions is less than a preset threshold; An image fusion module is used to map the original long-focus image to the third matching area in the amplified short-focus image to obtain a fused image.

9. The device according to claim 8, characterized in that The matching area determination module is specifically used to: According to the feature matching pair, determining an area in the amplified short-focus image that matches the original long-focus image as a fourth matching area; Controlling the fourth matching area to rotate a first preset number of times along a first preset direction according to a preset rotation unit, and using the area occupied by the fourth matching area in the enlarged short-focus image after each rotation as the fifth matching area; respectively calculating similarity distances between the fourth matching region and each of the fifth matching regions and the original telephoto image as third brightness difference degrees corresponding to the fourth matching region and fourth brightness difference degrees corresponding to each of the fifth matching regions; According to the brightness difference between the fourth matching area and the fifth matching area, determining a matching area satisfying the preset matching condition within the fourth matching area and the fifth matching area as the first matching area; The determining of at least one second matching area in the neighborhood of the first matching area includes: The first matching area is controlled to translate a second preset number of times along a second preset direction according to a preset translation unit, and an area occupied by the first matching area in the magnified short-focus image after each translation is used as the second matching area.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

Images