Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium. The method comprises the following steps: carrying out superpixel segmentation on an image to be processed to obtain a plurality of superpixels; classifying different superpixels to obtain at least one depth-of-field region; wherein different depth-of-field areas correspond to different depths of field; based on the reference image, determining a target homography matrix of each field depth area; wherein the reference image is an image aligned with the image to be processed; and performing image warping processing on the corresponding depth-of-field region by using the target homography matrix of each depth-of-field region. According to the scheme provided by the invention, alignment of different depth-of-field planes can be realized at the same time, and the image alignment effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, electronic device, chip and storage medium. Background Art

[0002] In the related art, by performing global feature extraction and feature matching on the two frames of images, the homography matrix between the two frames of images can be obtained, so that the two frames of images can be aligned using the homography matrix. However, the related art cannot align the contents in different depth of field planes at the same time, and the image alignment effect is poor. Summary of the invention

[0003] The present disclosure provides an image processing method, device, electronic device, chip and storage medium to solve the problems in the related art, and at the same time achieve the alignment of different depth of field planes to improve the image alignment effect.

[0004] The first aspect of the present disclosure provides an image processing method, device, electronic device, chip and storage medium. The method includes:

[0005] Performing super-pixel segmentation on the image to be processed to obtain multiple super-pixels;

[0006] Classifying different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field;

[0007] Determine a target homography matrix for each depth of field region based on a reference image; wherein the reference image is an image to be aligned with the image to be processed;

[0008] The target homography matrix of each depth of field area is used to perform image warping processing on the corresponding depth of field area.

[0009] In some embodiments of the present disclosure, the classifying different superpixels to obtain at least one depth of field region includes:

[0010] Determine whether the depth of field of the first superpixel is the same as that of a corresponding second superpixel; wherein the second superpixel is a superpixel corresponding to a neighborhood of the first superpixel;

[0011] When the depth of field of the first superpixel is the same as that of the corresponding second superpixel, the first superpixel and the second superpixel are divided into the same depth of field area.

[0012] In some embodiments of the present disclosure, determining whether the depth of field of the first superpixel is the same as that of the corresponding second superpixel includes:

[0013] Based on the reference image, determining a feature point pair of the first superpixel;

[0014] Calculating a first homography matrix of the first superpixel using the feature point pairs of the first superpixel;

[0015] Acquire a feature point from the second superpixel to obtain a test feature point, and determine a pixel point matching the test feature point in the reference image;

[0016] Mapping the test feature points using the first homography matrix to obtain first mapping points;

[0017] Calculating an error metric between the first mapping point and a pixel point in the reference image that matches the test feature point, and determining whether the error metric is less than a preset threshold;

[0018] When the error metric is less than a preset threshold, it is determined that the depth of field of the first superpixel is the same as that of the second superpixel.

[0019] In some embodiments of the present disclosure, determining a target homography matrix for each depth of field region based on the reference image includes:

[0020] Based on the reference image, determining a feature point pair of each superpixel in the depth of field area to obtain a feature point pair set of the depth of field area;

[0021] The target homography matrix of the depth of field area is calculated using the set of feature point pairs.

[0022] In some embodiments of the present disclosure, the step of calculating the target homography matrix of the depth of field region by using the set of feature point pairs includes:

[0023] Calculating a regional homography matrix of the depth of field region using the feature point pair set;

[0024] Using the feature point pair of each superpixel in the feature point pair set, calculate a first homography matrix of each superpixel;

[0025] The first homography matrix of each superpixel is updated by using the regional homography matrix to obtain the second homography matrix of each superpixel, and the second homography matrix is ​​used as the target homography matrix of the depth of field region.

[0026] In some embodiments of the present disclosure, the updating of the first homography matrix of each superpixel by using the regional homography matrix to obtain the second homography matrix of each superpixel includes:

[0027] The product of the regional homography matrix and the preset first weight and the product of each first homography matrix and the preset second weight are added together to obtain a corresponding second homography matrix.

[0028] In some embodiments of the present disclosure, the step of calculating the target homography matrix of the depth of field region by using the set of feature point pairs includes:

[0029] The first homography matrix of each superpixel is calculated by using the feature point pair of each superpixel in the feature point pair set, and the first homography matrix is ​​used as the target homography matrix of the depth of field area.

[0030] A second aspect of the present disclosure provides an image processing device, the device comprising:

[0031] A segmentation unit, used for performing super-pixel segmentation on the image to be processed to obtain a plurality of super-pixels;

[0032] A classification unit, used to classify different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field;

[0033] A determination unit, used for determining a target homography matrix of each depth of field region based on a reference image;

[0034] The processing unit is used to perform warping processing on the image of the corresponding depth of field area by using the target homography matrix of each depth of field area.

[0035] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0036] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0037] A fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which executes the method described in the first aspect embodiment of the present disclosure when a processor is used to execute the computer program.

[0038] The sixth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0039] In summary, the image processing method, device, electronic device, chip and storage medium proposed in the present invention can realize the alignment processing of different depth of field planes separately by dividing the image into superpixels and classifying different superpixels into different depth of field planes, thereby realizing effective alignment of different depth of field planes and improving the image alignment effect.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0042] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0043] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0044] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0045] Figure 3a A schematic diagram of a set of feature point pairs in different depths of field regions obtained in an embodiment of the present disclosure;

[0046] Figure 4 A flowchart of an image alignment method based on superpixel segmentation and feature point distribution provided for an application embodiment of the present disclosure;

[0047] Figure 4a A schematic diagram of a process for calculating superpixel segmentation of an image in an application embodiment of the present disclosure;

[0048] Figure 4b A schematic diagram of a process of using superpixel information to guide the completion of global alignment of more than one plane in an application embodiment of the present disclosure;

[0049] Figure 4c A schematic diagram of a process for completing global alignment fusion of multiple depths of field based on superpixels and image warping in an application embodiment of the present disclosure;

[0050] Figure 4d A schematic diagram of the effect of alignment using an image alignment method in the related art;

[0051] Figure 4e A schematic diagram of the effect of alignment using the image alignment method in the application embodiment of the present disclosure;

[0052] Figure 5 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0053] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0054] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0056] In the related art, when aligning images, it is often assumed that the scene is on the same plane, and the image is aligned using a global alignment method; specifically, a global feature extraction and feature matching method is used to obtain a single homography matrix, and the scene is warped using a single homography matrix. However, in actual scenes, there are often multiple different depths of field, such as the foreground and background of the scene, or in a large depth scene with a large depth of field span, the scene content cannot fall on the same plane; in this case, using a global alignment method for alignment processing cannot effectively use a single global alignment information to adapt to the different alignment requirements caused by local depth of field differences, and cannot take into account the alignment of different depth of field planes, resulting in some areas in the image not being fully aligned, resulting in poor image effects. For example, the foreground of the scene is aligned but the background is not aligned, or some positions in the large depth scene are aligned, but another area that is the same coherent area in the image content is actually not aligned, resulting in problems such as corridor scenes, and cannot be effectively aligned.

[0057] In order to solve the problems existing in the related technology, the present disclosure proposes an image processing method, which divides the image into superpixels and classifies different superpixels into different depth of field planes, so as to realize separate alignment of different depth of field planes, thereby realizing effective alignment of different depth of field planes and improving the image alignment effect.

[0058] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image processing method in an embodiment. Figure 1As shown, the application scenario includes an electronic device 104, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires the image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image is sent to the target application. Among them, the electronic device 104 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0059] In some optional embodiments, the electronic device may also be a vehicle-mounted device or a vehicle networking device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.

[0060] A camera can be installed on an electronic device, and images can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of an electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on an electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.

[0061] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.

[0062] Several applications can be installed in electronic devices. Applications refer to software written for a certain application purpose in electronic devices. Electronic devices can provide services for users through applications. When an application needs to capture an image, it will initiate an image capture instruction. The electronic device will call the camera module according to the image capture instruction to capture the image. Image capture instructions refer to instructions used to trigger image capture operations.

[0063] The electronic device is also provided with a processor, and the image processing module in the processor can perform correction processing on the image captured by the camera module, for example, execute the image processing method provided in the present disclosure.

[0064] The embodiments of the present disclosure can be applied to scenarios that require image processing, for example, image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, image processing in night scene shooting or video mode, and the like. For another example, the embodiments can also be applied to scenarios where the image signal processing (ISP) module is a key module that affects the processor and high-quality edge sharpening control is required during the image processing process of the image pipeline.

[0065] Figure 2 An image processing method is provided in an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0066] Step S201: performing super-pixel segmentation on the image to be processed to obtain a plurality of super-pixels.

[0067] In some embodiments, a simple linear iterative clustering (SLIC) method may be used to perform superpixel segmentation on the image to be processed.

[0068] In some embodiments, the image to be processed may also be referred to as a reference image or a base image.

[0069] Step 202: Classify different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field.

[0070] In some embodiments, whether to classify each superpixel and its adjacent superpixel into the same depth of field region can be determined by whether the depth of field corresponding to each superpixel and its adjacent superpixel is the same.

[0071] Step 203: Determine a target homography matrix for each depth of field region based on a reference image; wherein the reference image is an image to be aligned with the image to be processed.

[0072] In some embodiments, the feature point pairs of each superpixel in the depth of field area can be first determined according to the reference image to obtain a set of feature point pairs of the depth of field area, and the homography matrix of the depth of field area is calculated according to the feature point pairs in the feature point pair set, that is, the homography matrix obtained based on the fitting of the entire depth of field area, and the homography matrix of the corresponding superpixels in the depth of field area is calculated according to the feature point pairs of each superpixel in the feature point pair set, that is, the homography matrix obtained based on the fitting of each superpixel; finally, the homography matrix of each superpixel can be iteratively fused using the homography matrix of the depth of field area as an initial value to obtain the optimized homography matrix of each superpixel, that is, the target homography matrix.

[0073] In some embodiments, the feature point pairs of each superpixel in the depth of field area are first determined based on the reference image to obtain a set of feature point pairs in the depth of field area. Then, the homography matrix of the corresponding superpixel in the depth of field area can be calculated based on the feature point pairs of each superpixel in the feature point pair set, that is, the homography matrix obtained based on the fitting of each superpixel, and the calculated homography matrix of each superpixel is used as the target homography matrix.

[0074] It should be noted that the reference image is an image aligned with the image to be processed. For example, in a motion estimation application scenario, the reference image may be a previous frame of the image to be processed. The reference image may also be referred to as a Reference Image.

[0075] In practical applications, the homography matrix may also be called a homography matrix or a Homography matrix.

[0076] Step 204: using the target homography matrix of each depth of field area, perform warping processing on the image of the corresponding depth of field area.

[0077] In some embodiments, in the process of performing warping processing on the image of the corresponding depth of field area using the target homography matrix of each depth of field area, the homography matrix of each superpixel in the target homography matrix can be used to perform warping processing on the corresponding superpixel, thereby achieving alignment of the image to be processed.

[0078] In summary, the image processing method proposed in the embodiment of the present disclosure, by dividing the image into superpixels and classifying different superpixels into different depth of field planes, can realize separate alignment processing of different depth of field planes, achieve effective alignment of different depth of field planes, and improve the image alignment effect.

[0079] based on Figure 2 The embodiment shown, Figure 3 A flowchart of an image processing method proposed in the present disclosure is further shown. For ease of understanding, Figure 4 FIG. 1 is a flow chart showing an image processing method provided by an embodiment of the present disclosure. Figure 3 As shown, the method includes:

[0080] Step S301: performing superpixel segmentation on the image to be processed to obtain a plurality of superpixels.

[0081] It should be pointed out that superpixel segmentation refers to the process of subdividing a digital image into multiple image sub-regions (a collection of pixels); a superpixel is a small area composed of a series of pixels with adjacent positions and similar characteristics such as color, brightness, and texture. Most of these small areas retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image; if a pixel and a neighboring pixel belong to the same superpixel, where the neighboring pixel represents the pixel adjacent to the pixel position, it means that the two have a high similarity and are likely to belong to the same object or the same plane, so that the two can be affine transformed using similar or identical models.

[0082] In some embodiments, the SLIC method may be used to perform superpixel segmentation on the image to be processed.

[0083] In some embodiments, a superpixel coordinate map of each superpixel can also be generated, and the size of the superpixel coordinate map is the same as the image size; wherein, a superpixel label (Superpixel_label) can be defined for each superpixel position in the superpixel coordinate map, for example, if the whole image includes 100 superpixels, the value range of Superpixel_label is 1 to 100; by generating the superpixel coordinate map, it can be determined whether the current pixel and the surrounding area pixels belong to the same superpixel, that is, they can be distinguished by whether they have the same Superpixel_label.

[0084] Step 302: Classify different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field.

[0085] In some embodiments, whether to classify each superpixel and its adjacent superpixel into the same depth of field region can be determined by whether the depth of field corresponding to each superpixel and its adjacent superpixel is the same.

[0086] Based on this, in some embodiments, classifying different superpixels to obtain at least one depth of field area may include:

[0087] Determine whether the depth of field of the first superpixel is the same as that of a corresponding second superpixel; wherein the second superpixel is a superpixel corresponding to a neighborhood of the first superpixel;

[0088] When the depth of field of the first superpixel is the same as that of the corresponding second superpixel, the first superpixel and the second superpixel are divided into the same depth of field area.

[0089] In some embodiments, determining whether the depth of field of the first superpixel is the same as that of the corresponding second superpixel may include:

[0090] Based on the reference image, determining a feature point pair of the first superpixel;

[0091] Calculating a first homography matrix of the first superpixel using the feature point pairs of the first superpixel;

[0092] Acquire a feature point from the second superpixel to obtain a test feature point, and determine a pixel point matching the test feature point in the reference image;

[0093] Mapping the test feature points using the first homography matrix to obtain first mapping points;

[0094] Calculating an error metric between the first mapping point and a pixel point in the reference image that matches the feature point, and determining whether the error metric is less than a preset threshold;

[0095] When the error metric is less than a preset threshold, it is determined that the depth of field of the first superpixel is the same as that of the second superpixel.

[0096] It should be pointed out that the first superpixel may be the superpixel currently being classified, and the first superpixel may be any superpixel in the image to be processed; the first homography matrix is ​​the homography matrix of the superpixel currently being classified, that is, for each superpixel in the image to be processed, the corresponding first homography matrix can be calculated and generated based on the feature point pairs between the superpixel and the reference image.

[0097] In some embodiments, feature extraction can be first performed based on the image to be processed and the reference image to obtain a set of feature point pairs Feature_all{Num} of all locations in the image to be processed where feature points can be extracted, where Num is the total number of features extracted, and after superpixel segmentation is performed on the image to be processed, the feature point pairs between each superpixel and the reference image are obtained; or the image to be processed can be first segmented into superpixels, and features can be extracted on each superpixel to obtain the feature point pairs of the superpixel.

[0098] It should be noted that the test feature point and the pixel point in the reference image that matches the test feature point are the feature point pair corresponding to the test feature point.

[0099] In some embodiments, three feature point pairs with the same Superpixel_label can be selected to estimate the homography matrix between the current superpixel and the reference image, that is, the first homography matrix, and then load a feature point pair of the neighborhood superpixel, wherein the loaded feature point pair includes two pixel points, one is the feature point of the image to be processed, and the other is the pixel point in the reference image that matches the feature point of the image to be processed; the feature points of the image to be processed in the loaded feature point pair are mapped using the first homography matrix, that is, the points of the image to be processed in the loaded feature point pair are warped to obtain the first mapping point, and the coordinate error between the first mapping point and the pixel point of the reference image in the acquired feature point pair is calculated. The error metric between the first mapping point and the pixel point of the reference image in the acquired feature point pair can also be called a warping error metric, or a warping error.

[0100] In some embodiments, after calculating the warping error metric between each superpixel and its neighboring superpixels, when the warping error metric is less than a preset threshold, it can be considered that the pixels of the Superpixel_label and the Superpixel_label area of ​​its neighborhood are located in the same depth plane, and these feature point pairs can be merged to achieve classification of different superpixels and corresponding feature point pairs; through the calculation of the warping error metric, the feature points of different depth planes (for example, foreground and background areas) can be first tested based on the superpixel, and then tested across superpixels, so that the superpixels and corresponding feature point pairs of different depth planes can be classified, and feature point pair sets of different depth planes are obtained, that is, feature point pair sets of different depth areas, for example, the feature point pair set matching pair_foreground{num_foreground_feature} of the foreground area and the feature point pair set matching pair_background{num_background_feature} of the background area are obtained.

[0101] In some embodiments, the preset thresholds of different superpixels may be different or the same. The value of the preset threshold may be determined based on the specific application scenario and the degree of difference between the depths of field to be divided, which is not limited in this embodiment of the present disclosure.

[0102] For example, the warping error metric can be calculated by the following formula:

[0103] Warping Error=warping(matching pair4_current frame_pt)-matchingpair4_base frame_pt

[0104] Among them, matching pair4_current frame_pt represents the point of the reference image in the acquired point pair, and warping(matching pair4_current frame_pt) represents the warping operation on the point matching pair4_current frame_pt using the homography matrix.

[0105] It should be pointed out that if the point pairs of the neighborhood are located on the same depth plane as the first three point pairs used to estimate the homography matrix of the current superpixel, the homography matrix of the current superpixel calculated in the previous order will be able to well represent the mapping relationship between the neighborhood point pairs. At this time, the homography matrix of the current superpixel calculated in the previous order is used to map the points of the current frame (i.e., the image to be processed) in the neighborhood point pairs toward the reference frame (i.e., the reference image), and points very close to the points on the reference frame in the neighborhood point pairs will be obtained. Therefore, the warping error metric will be relatively small; accordingly, if the point pairs of the neighborhood are located on different depth planes from the first three point pairs used to estimate the homography matrix of the current superpixel, the homography matrix of the current superpixel calculated in the previous order is actually unable to accurately represent the mapping relationship between the neighborhood point pairs. At this time, the homography matrix of the current superpixel calculated in the previous order is used to map the points of the current frame in the neighborhood point pairs toward the reference frame, and points very close to the points on the reference frame in the neighborhood point pairs will not be obtained. Therefore, the warping error metric will be relatively large. It can be seen that the depth plane corresponding to the superpixel can be accurately distinguished through the warping error metric.

[0106] Step 303: Based on the reference image, determine the feature point pair of each superpixel in the depth of field area to obtain a feature point pair set of the depth of field area.

[0107] It should be noted that, when feature segmentation is first performed on the image to be processed and the reference image to obtain a set of feature point pairs of all locations in the image to be processed where feature points can be extracted, the feature point pairs of each superpixel can be obtained from the set of feature point pairs; or the image to be processed can be first segmented into superpixels, and features can be extracted from each superpixel to obtain the feature point pairs of the superpixel. For example, when the depth of field area includes a foreground area and a background area, the feature point pair sets of different depth of field areas can be obtained as follows: Figure 3aAs shown, the X-shaped mark area is the foreground area, the light-colored feature points corresponding to the X-shaped mark are the feature points of the foreground area, the triangle mark area is the background area, and the dark-colored feature points corresponding to the triangle mark are the feature points of the background area; Figure 3a It can be seen that through the embodiments of the present disclosure, the feature points of different depth of field planes can be classified, thereby achieving effective classification of different depth of field planes.

[0108] Step 304: Calculate the regional homography matrix of the depth of field region using the set of feature point pairs.

[0109] In some embodiments, the feature points of all superpixels in the depth of field area can be integrated to obtain a set of feature point pairs in the depth of field area, and the regional homography matrix of the entire depth of field area can be calculated based on the set of feature point pairs in the depth of field area, that is, the homography matrix obtained by fitting the entire depth of field area. Exemplarily, a global alignment algorithm can be used to perform feature matching, RANSAC and homography matrix calculation on the feature point pairs in the foreground area to obtain the homography matrix Homography_foreground of the foreground area, and feature matching, RANSAC and homography matrix calculation on the feature point pairs in the background area to obtain the homography matrix Homography_background of the background area.

[0110] Step 305: Calculate a first homography matrix of each superpixel using the feature point pair of each superpixel in the feature point pair set.

[0111] In some embodiments, for each superpixel, three non-collinear feature point pairs may be selected from the feature point pairs of the superpixel in the feature point pair set, and the selected feature point pairs may be used to estimate the homography matrix of the corresponding superpixel.

[0112] Step 306: Use the regional homography matrix to update the first homography matrix of each superpixel to obtain the second homography matrix of each superpixel, and use the second homography matrix as the target homography matrix of the depth of field region.

[0113] In some embodiments, the regional homography matrix may be used as an initial value to iteratively fuse the homography matrix of each superpixel, thereby obtaining an optimized homography matrix of each superpixel, that is, a target homography matrix.

[0114] In some embodiments, weights may be defined for the regional homography region and the first homography matrix, and the two may be fused to obtain a second homography matrix corresponding to the superpixel.

[0115] Based on this, in some embodiments, using the regional homography matrix to update the first homography matrix of each superpixel to obtain the second homography matrix of each superpixel may include:

[0116] The product of the regional homography matrix and the preset first weight and the product of each first homography matrix and the preset second weight are added together to obtain a corresponding second homography matrix.

[0117] In some embodiments, the sum of the first weight and the second weight may be 1.

[0118] For example, if the current superpixel is located in the foreground area, the second homography matrix of the current superpixel can be calculated by the following formula:

[0119] Homo2_cur_superpixel=alpha*Homo1_cur_superpixel+(1-alpha)*Homography_foreground;

[0120] If the current pixel is located in the background area, the second homography matrix of the current superpixel can be calculated by the following formula:

[0121] Homo2_cur_superpixel=alpha*Homo1_cur_superpixel+(1-alpha)*Homography_background;

[0122] Among them, Homo2_cur_superpixel represents the second homography matrix of the current superpixel, Homo1_cur_superpixel represents the first homography matrix of the current superpixel, Homography_foreground represents the homography matrix of the foreground area, Homography_background represents the homography matrix of the background area, and alpha represents the weight of the first homography matrix, that is, the first weight.

[0123] In some embodiments, the first weight and the second weight can be determined according to preset configuration information; specifically, the values ​​of the first weight and the second weight can be directly configured; or the correspondence between the depth of field and the weight can be configured, so that the weight corresponding to the depth of field plane where the current superpixel is located can be determined according to the configuration information.

[0124] In some embodiments, after obtaining the set of feature point pairs of the depth of field area, that is, after executing step 303, each feature point pair in the set of feature point pairs can be directly used to calculate the first homography matrix corresponding to the superpixel, and the first homography matrix can be used as the target homography matrix corresponding to the depth of field area.

[0125] Step 307: Using the target homography matrix of each depth of field area, perform warping processing on the image of the corresponding depth of field area.

[0126] In some embodiments, in the process of performing warping processing on the image of the corresponding depth of field area using the target homography matrix of each depth of field area, the homography matrix of each superpixel in the target homography matrix can be used to perform warping processing on the corresponding superpixel, thereby achieving alignment of the image to be processed.

[0127] In summary, the image processing method proposed in the embodiment of the present disclosure, by dividing the image into superpixels and classifying different superpixels into different depth of field planes, can realize separate alignment processing of different depth of field planes, achieve effective alignment of different depth of field planes, and improve the image alignment effect.

[0128] The technical solution of the present disclosure is further described below in conjunction with specific application examples.

[0129] Figure 4 An image alignment method based on superpixel segmentation and feature point distribution is provided for an application embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0130] Step 401: Input the Base Image and Reference Image, and then execute step 402.

[0131] Here, Base Image is the image to be processed, and Reference Image is the reference image, specifically, the previous frame image of Base Image.

[0132] Step 402: Calculate the superpixel segmentation of the image, obtain the superpixel information of the image, and then execute step 403.

[0133] like Figure 4a As shown, the process of using the SLIC method to perform superpixel segmentation may specifically include:

[0134] Step 4021: Initialize superpixel calculation, and then execute step 4022.

[0135] Specifically, the seed points (cluster centers) are initialized.

[0136] Step 4022: Superpixel core calculation, then execute step 4023.

[0137] Specifically, a seed point is reselected within the n*n neighborhood of the seed point (generally n=3); then, a class label is assigned to each pixel point in the neighborhood around each seed point (i.e., which cluster center it belongs to); then, a distance metric (including color distance and spatial distance) is performed, i.e., for each searched pixel point, the distance between it and the seed point is calculated respectively; then, iterative optimization is performed, i.e., the above steps are continuously iterated until the error converges (which can be understood as until the cluster center of each pixel point no longer changes).

[0138] Step 4023: Post-processing of superpixel information.

[0139] Specifically, connectivity is enhanced, that is, a new labeling table is created, all elements in the table are -1, and discontinuous superpixels and superpixels with too small sizes are reallocated to adjacent superpixels in a "Z"-shaped direction (from left to right, from top to bottom), and the traversed pixels are assigned to corresponding labels until all points are traversed.

[0140] Step 403: Use superpixel information to guide the global alignment of more than one plane, obtain multiple global alignment matrices, and then execute step 404.

[0141] like Figure 4b As shown, step 403 may specifically include the following steps:

[0142] Step 4031: Select feature points based on the superpixel area, and then execute step 4032.

[0143] Here, based on the superpixel information generated in step 402 and in combination with the feature point information calculated by global alignment, feature points can be selected for the foreground object and the background scene area in a targeted manner.

[0144] Specifically, according to the input front and back frame image information (i.e., Reference Image and Base Image), the basic steps of global image alignment, i.e., feature extraction, feature matching, and Homography matrix calculation (e.g., feature extraction and matching technology based on ORB, etc.), are used to obtain a set of feature points Feature_all{Num} of all feature point positions that can be extracted in the image scene, where Num is the total number of extracted features.

[0145] Step 4032: perform multi-region global alignment core calculation, and then execute step 4033.

[0146] At this time, since the superpixel coordinate map of each pixel in the image has been obtained in step 402, Superpixel_label, its size is the same as the image size, and the Superpixel_label of the pixel is saved at each position. If there are 100 superpixels in the whole image, the range of Superpixel_label is 1-100. Therefore, it is possible to know whether the current pixel and the surrounding pixels belong to the same superpixel according to the superpixel coordinate map, that is, to distinguish them by whether they have the same Superpixel_label. If the current pixel and the neighboring pixel a have the same Superpixel_label, then in terms of semantic information, the two have a high similarity and are likely to belong to the same object, or belong to the same plane, so that the two can be affine transformed using similar or identical models.

[0147] On this basis, the concept of warping error metric can be introduced to better complete the selection of foreground and background pixels based on superpixels; when three point pairs with the same superpixel label are selected, the Homography matrix is ​​calculated using the Homography matrix estimation method, and then a point pair matching pair4 in the neighborhood of these three points is loaded, and the calculated Homography matrix is ​​used to map the point matching pair4_current frame_pt of the current frame in the point pair, and the error between the mapped point and the point on the reference frame in the point pair is calculated, which is the warping error metric between the current superpixel and the superpixel in the neighborhood.

[0148] The formula for calculating the warping error metric is:

[0149] Warping Error=warping(matching pair4_current frame_pt)-matchingpair4_base frame_pt;

[0150] Among them, the warping() operation is to use the homography matrix to warp the points.

[0151] It can be seen that the warping error metric is closely related to whether the selected fourth point pair is located in the same plane as the previous three point pairs; specifically, if the fourth point pair is actually located on the same depth plane as the previous three point pairs, the homography matrix calculated by the previous point pairs with the same superpixel label can actually well represent the mapping relationship between the fourth point pairs, and using such a homography matrix to map the points of the current frame in the fourth point pair to the reference frame will also obtain points very close to the points on the reference frame, and the corresponding warping error metric here will be relatively small; conversely, if the fourth point pair is actually located on a different depth plane from the previous three point pairs, the homography matrix calculated by the previous point pairs with the same superpixel label cannot actually well represent the mapping relationship between the fourth point pairs, and using such a homography matrix to map the points of the current frame in the fourth point pair to the reference frame will not obtain points very close to the points on the reference frame, and the corresponding warping error metric here is relatively large.

[0152] Through the above-mentioned warping error metric calculation method, the test of feature points within a superpixel can be completed, and the superpixel label m and the pixels with superpixel label n in its neighborhood can be subjected to the above-mentioned test. If its warping error metric is less than the preset threshold, it can be considered that the pixels in the superpixel label m and the superpixel label n area in its neighborhood are also in the same depth plane, and these feature points can be merged to jointly calculate Homography.

[0153] Through the above operations, the feature points of the foreground area and the background area are first tested within the superpixel and then tested across superpixels, so that the feature points of the foreground area and the background area can be classified to obtain the foreground feature point pair set matching pair_foreground{num_foreground_feature} and the background feature point pair set matching pair_background{num_background_feature}.

[0154] Step 4033: Global alignment calculation post-processing.

[0155] Here, the model is fitted for the foreground feature point pair set matching pair_foreground{num_foreground_feature} and the background feature point pair set matching pair_background{num_background_feature}; in the process of model fitting, non-collinear feature points are selected, and then the model is fitted using these selected feature points, that is, the Homography matrix is ​​fitted. Among them, feature matching, RANSAC and Homography matrix calculation of the global alignment algorithm can be used to obtain the matrix Homography_foreground of the foreground area and the matrix Homography_background of the background area.

[0156] It should be pointed out that in the process of fitting the Homography matrix, the Homography matrix can be fitted superpixel by superpixel, that is, the corresponding Homography matrix is ​​fitted for each superpixel to obtain the Homography matrix with the same number of superpixels, and these matrices are used to warp the corresponding superpixel areas; it is also possible to integrate the feature points of the foreground and background, and fit the corresponding Homography matrix for each depth plane (foreground or background).

[0157] Step 404: Complete the global alignment fusion of multiple depths of view based on superpixels and image warping.

[0158] Specifically, Figure 4c As shown, multiple global alignment information can be fused according to superpixel information and global alignment information of multiple depths of field, and the image can be warped using the fused global alignment information.

[0159] In particular, with respect to the method of fitting the Homography matrix superpixel by superpixel in step 403, the Homography matrix of each superpixel may be used to perform warping on the superpixel.

[0160] With respect to the method of fitting the corresponding Homography matrix of each depth plane (foreground or background) in step 403, the Homography matrix of each superpixel in the foreground area can be fitted according to the Homography matrix Homography_foreground of the foreground area, and the Homography matrix of each superpixel in the background area can be fitted according to the Homography matrix Homography_background of the background area, and the corresponding superpixel can be warped using the obtained Homography matrix of each superpixel.

[0161] Specifically, the Homography matrix of each pixel (or current superpixel area) is first fitted one by one, and then the Homography matrix Homography_foreground of the foreground area or the Homography matrix Homography_background of the background area is used as the initial value of the iteration to fuse the Homography matrix of each pixel (or current superpixel area);

[0162] If the current pixel (or superpixel area) is on the superpixel in the foreground area, the fused Homography matrix can be calculated by the following formula:

[0163] Homo_cur_superpixel=alppa*Homo_cur_superpixel+(1-alpha)*Homography_foreground;

[0164] If the current pixel (or superpixel area) is on the superpixel of the background area, the fused Homography matrix can be calculated by the following formula:

[0165] Homo_cur_superpixel=alppa*Homo_cur_superpixel+(1-alpha)*Homography_background.

[0166] Figure 4d This is a schematic diagram of the effect of alignment using the image alignment method in the related art. Figure 4e This is a schematic diagram of the effect of using the image alignment method in the application embodiment of the present disclosure to perform alignment. Figure 4d and Figure 4eIt can be seen from the content in the box that when the image alignment method of the related technology is used for alignment, since the global alignment can only align a single depth of field plane, objects with different depths of field cannot actually be aligned at the same time, and overlapping problems will occur; using the image alignment method of the application embodiment of the present disclosure, through local segmentation of superpixels and local Homography matrix fitting, the problem area in the 4D box can be optimized, avoiding the problems of overlap and ghosting. It can be seen that the image alignment method of the application embodiment of the present disclosure can effectively improve the fusion effect of images and videos.

[0167] Figure 5 FIG. 5 is a schematic diagram of the structure of an image processing device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the image processing device comprises:

[0168] The segmentation unit 501 is used to perform super-pixel segmentation on the image to be processed to obtain a plurality of super-pixels;

[0169] A classification unit 502 is used to classify different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field;

[0170] A determination unit 503 is used to determine a target homography matrix of each depth of field region based on a reference image; wherein the reference image is an image to be aligned with the image to be processed;

[0171] The processing unit 504 is used to perform a warping process on the image of each depth of field area by using the target homography matrix of each depth of field area.

[0172] In some embodiments, the classification unit 502 may be specifically used to:

[0173] Determine whether the depth of field of the first superpixel is the same as that of a corresponding second superpixel; wherein the second superpixel is a superpixel corresponding to a neighborhood of the first superpixel;

[0174] When the depth of field of the first superpixel is the same as that of the corresponding second superpixel, the first superpixel and the second superpixel are divided into the same depth of field area.

[0175] In some embodiments, the classification unit 502 may be specifically used to:

[0176] Based on the reference image, determining a feature point pair of the first superpixel;

[0177] Calculating a first homography matrix of the first superpixel using the feature point pairs of the first superpixel;

[0178] Acquire a feature point from the second superpixel to obtain a test feature point, and determine a pixel point matching the test feature point in the reference image;

[0179] Mapping the test feature points using the first homography matrix to obtain first mapping points;

[0180] Calculating an error metric between the first mapping point and a pixel point in the reference image that matches the test feature point, and determining whether the error metric is less than a preset threshold;

[0181] When the error metric is less than a preset threshold, it is determined that the depth of field of the first superpixel is the same as that of the second superpixel.

[0182] In some embodiments, the determining unit 503 may be specifically configured to:

[0183] Based on the reference image, determining a feature point pair of each superpixel in the depth of field area to obtain a feature point pair set of the depth of field area;

[0184] The target homography matrix of the depth of field area is calculated using the set of feature point pairs.

[0185] In some embodiments, the determining unit 503 may be specifically configured to:

[0186] Calculating a regional homography matrix of the depth of field region using the feature point pair set;

[0187] Using the feature point pair of each superpixel in the feature point pair set, calculate a first homography matrix of each superpixel;

[0188] The first homography matrix of each superpixel is updated by using the regional homography matrix to obtain the second homography matrix of each superpixel, and the second homography matrix is ​​used as the target homography matrix of the depth of field region.

[0189] In some embodiments, the determining unit 503 may be specifically configured to:

[0190] The product of the regional homography matrix and the preset first weight and the product of each first homography matrix and the preset second weight are added together to obtain a corresponding second homography matrix.

[0191] In some embodiments, the determining unit 503 may be specifically configured to:

[0192] The first homography matrix of each superpixel is calculated by using the feature point pair of each superpixel in the feature point pair set, and the first homography matrix is ​​used as the target homography matrix of the depth of field area.

[0193] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0194] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0195] Figure 6 is a block diagram of an electronic device 600 for implementing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0196] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0197] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0198] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0199] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0200] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0201] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0202] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0203] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0204] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0205] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0206] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0207] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.

[0208] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the image processing method described in the above embodiment of the present disclosure when a processor is used.

[0209] The embodiment of the present disclosure also provides a chip, which can be seen in Figure 7 Schematic diagram of the chip structure shown. Figure 7 The chip shown includes a processor 701 and an interface circuit 702. The number of the processor 701 can be one or more, and the number of the interface circuit 702 can be one or more.

[0210] Optionally, the chip also includes a memory 703, which is used to store necessary computer programs and data; the interface circuit 702 is used to receive signals from the memory 703 and send signals to the processor 701, and the signals include computer instructions stored in the memory 703. When the processor 701 executes the computer instructions, the electronic device executes the image processing method described in the above embodiments of the present disclosure.

[0211] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0212] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0213] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0214] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0215] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0216] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0217] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0218] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Performing super-pixel segmentation on the image to be processed to obtain multiple super-pixels; Classifying different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field; Determine a target homography matrix for each depth of field region based on a reference image; wherein the reference image is an image to be aligned with the image to be processed; The target homography matrix of each depth of field area is used to perform image warping processing on the corresponding depth of field area.

2. The method according to claim 1, characterized in that The classifying different superpixels to obtain at least one depth of field region includes: Determine whether the depth of field of the first superpixel is the same as that of a corresponding second superpixel; wherein the second superpixel is a superpixel corresponding to a neighborhood of the first superpixel; When the depth of field of the first superpixel is the same as that of the corresponding second superpixel, the first superpixel and the second superpixel are divided into the same depth of field area.

3. The method according to claim 2, characterized in that The determining whether the depth of field of the first superpixel is the same as that of the corresponding second superpixel comprises: Based on the reference image, determining a feature point pair of the first superpixel; Calculating a first homography matrix of the first superpixel using the feature point pairs of the first superpixel; Acquire a feature point from the second superpixel to obtain a test feature point, and determine a pixel point matching the test feature point in the reference image; Mapping the test feature points using the first homography matrix to obtain first mapping points; Calculating an error metric between the first mapping point and a pixel point in the reference image that matches the test feature point, and determining whether the error metric is less than a preset threshold; When the error metric is less than a preset threshold, it is determined that the depth of field of the first superpixel is the same as that of the second superpixel.

4. The method according to claim 1, characterized in that The step of determining a target homography matrix for each depth of field region based on the reference image comprises: Based on the reference image, determining a feature point pair of each superpixel in the depth of field area to obtain a feature point pair set of the depth of field area; The target homography matrix of the depth of field area is calculated using the set of feature point pairs.

5. The method according to claim 4, characterized in that The step of calculating the target homography matrix of the depth of field region by using the set of feature point pairs includes: Calculating a regional homography matrix of the depth of field region using the feature point pair set; Using the feature point pair of each superpixel in the feature point pair set, calculate a first homography matrix of each superpixel; The first homography matrix of each superpixel is updated using the regional homography matrix to obtain a second homography matrix of each superpixel, and the second homography matrix is ​​used as a target homography matrix of the depth of field region.

6. The method according to claim 5, characterized in that The method of updating the first homography matrix of each superpixel by using the regional homography matrix to obtain the second homography matrix of each superpixel comprises: The product of the regional homography matrix and the preset first weight and the product of each first homography matrix and the preset second weight are added together to obtain a corresponding second homography matrix.

7. The method according to claim 4, characterized in that The step of calculating the target homography matrix of the depth of field region by using the set of feature point pairs includes: The first homography matrix of each superpixel is calculated by using the feature point pair of each superpixel in the feature point pair set, and the first homography matrix is ​​used as the target homography matrix of the depth of field area.

8. An image processing device, characterized in that: The device comprises: A segmentation unit, used for performing super-pixel segmentation on the image to be processed to obtain a plurality of super-pixels; A classification unit, used to classify different superpixels to obtain at least one depth of field region; wherein different depth of field regions correspond to different depths of field; A determination unit, used for determining a target homography matrix of each depth of field region based on a reference image; The processing unit is used to perform warping processing on the image of the corresponding depth of field area by using the target homography matrix of each depth of field area.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.

Images