Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses an image processing method and device, electronic equipment and storage medium, the method comprises: matching the image content of a first image and a second image in a target region to obtain a matching relationship corresponding to the target region; each frame of image in the first image and the second image comprises at least two regions, each region included in the first image corresponds to each region included in the second image respectively, and the at least two regions are divided according to the depth value of the pixel points in each region; the target region is any one of the at least two regions; and image registration is performed on the first image and the second image according to the matching relationship corresponding to the at least two regions respectively. The embodiment of the application can improve the accuracy of image registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, in particular to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Image registration can align one image with another image, and image registration can be used for video analysis, pattern recognition, object tracking and other image processing. However, it is found in practice that the current image registration method still has the problem of low precision. SUMMARY

[0003] Embodiments of the present application disclose an image processing method and device, electronic equipment and storage medium, which can improve the precision of image registration.

[0004] Embodiments of the present application disclose an image processing method, which comprises: matching image contents of a first image and a second image in a target region to obtain a matching relationship corresponding to the target region; each frame of image in the first image and the second image comprises at least two regions, each region included in the first image corresponds to each region included in the second image, and the at least two regions are divided according to depth values of pixel points in each region; the target region is any one of the at least two regions; and image registration processing is performed on the first image and the second image according to the matching relationship corresponding to the at least two regions respectively.

[0005] Embodiments of the present application disclose an image processing device, which comprises: a matching module, configured to match image contents of a first image and a second image in a target region to obtain a matching relationship corresponding to the target region; each frame of image in the first image and the second image comprises at least two regions, each region included in the first image corresponds to each region included in the second image, and the at least two regions are divided according to depth values of pixel points in each region; the target region is any one of the at least two regions; and a registration module, configured to perform image registration processing on the first image and the second image according to the matching relationship corresponding to the at least two regions respectively.

[0006] Embodiments of the present application disclose an electronic equipment, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor realize any one of the image processing methods disclosed by the embodiments of the present application.

[0007] Embodiments of the present application disclose a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize any one of the image processing methods disclosed by the embodiments of the present application.

[0008] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0009] The electronic device can divide the first image and the second image into at least two regions according to the depth values, and match the image contents of the first image and the second image in the regions to obtain a matching relationship corresponding to each region respectively. The matching relationship corresponding to each region respectively can be used for image registration of the first image and the second image. The image content matching in the regions can improve the matching accuracy, and is beneficial to improving the accuracy of image registration. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0011] Figure 1 FIG. 1 is a structural schematic diagram of an image processing circuit disclosed by an embodiment;

[0012] Figure 2 FIG. 2 is a method flow schematic diagram of an image processing method disclosed by an embodiment;

[0013] Figure 3A FIG. 3 is an example diagram of feature point matching of a foreground region disclosed by an embodiment;

[0014] Figure 3B FIG. 4 is an example diagram of feature point matching of a background region disclosed by an embodiment;

[0015] Figure 3C FIG. 5 is an example diagram of full-image feature point matching in the related art;

[0016] Figure 4 FIG. 6 is a method flow schematic diagram of another image processing method disclosed by an embodiment;

[0017] Figure 5A FIG. 7 is a comparison example diagram of registration effects of a foreground region disclosed by an embodiment;

[0018] Figure 5B FIG. 8 is a comparison example diagram of registration effects of a background region disclosed by an embodiment;

[0019] Figure 6 FIG. 9 is an example diagram of performing portrait recognition on a first image disclosed by an embodiment;

[0020] Figure 7Fig. 1 is a schematic diagram of a method flow of another image processing method disclosed in an embodiment;

[0021] Figure 8 Fig. 2 is a schematic diagram of a structure of an image processing apparatus disclosed in an embodiment;

[0022] Figure 9 Fig. 3 is a schematic diagram of a structure of an electronic device disclosed in an embodiment. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present application and the drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed, or can optionally further include other steps or units inherent to the process, method, product or device.

[0025] Image registration refers to a process of determining a translation and rotation relationship between two images of the same object, and aligning the two images in space based on the translation and rotation relationship.

[0026] In the related art, image registration is mainly performed through feature point matching of the whole image. That is, the image registration scheme in the related art identifies mutually matching feature points in two different images when performing feature point matching, estimates a mapping matrix corresponding to the whole image according to the identified feature points, and performs mapping transformation on all pixel points included in the image according to the mapping matrix, so as to achieve registration of the two images.

[0027] Whole image feature point matching refers to matching feature points without distinguishing the image region where the feature points are located, but directly taking the whole image as the search region for feature point matching. The registration method performed through whole image feature point matching can achieve good results in some scenarios, but still has the problem of insufficient accuracy in other scenarios.

[0028] For example, when the shooting subject is a portrait, if the photographed person stands at a position far from the imaging device, the image obtained by shooting can be registered based on full-image feature point matching, and a relatively accurate registration result can be obtained. If the photographed person stands at a position close to the imaging device, the registration accuracy of the image obtained by shooting based on full-image feature point matching decreases.

[0029] Embodiments of the present application disclose an image processing method and device, electronic equipment and a storage medium, which can improve the accuracy of image registration. The following will be described in detail.

[0030] Please refer to Figure 1 , Figure 1 is a structural schematic diagram of an image processing circuit disclosed by an embodiment. The image processing circuit can be applied to electronic equipment such as a smart phone, a smart tablet, a smart watch, but is not limited thereto. As shown in Figure 1 , the image processing circuit can include an imaging device (camera) 110, a posture sensor 120, an image memory 130, an image signal processing (ISP) processor 140, a logic controller 150, and a display 160.

[0031] The image processing circuit includes the ISP processor 140 and the control logic 150. The image data captured by the imaging device 110 is first processed by the ISP processor 140, which analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 can include one or more lenses 112 and an image sensor 114. The image sensor 114 can include a color filter array (such as a Bayer filter), and the image sensor 114 can obtain light intensity and wavelength information captured by each imaging pixel and provide a set of raw image data (RAW image data) that can be processed by the ISP processor 140. The posture sensor 120 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) can provide image processing parameters (such as anti-shake parameters) collected by the posture sensor 120 to the ISP processor 140 based on the posture sensor 120 interface type. The posture sensor 120 interface can use an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0032] In addition, the image sensor 114 can also send the raw image data to the posture sensor 120, which can provide the raw image data to the ISP processor 140 based on the posture sensor 120 interface type, or the posture sensor 120 stores the raw image data in the image memory 130.

[0033] The ISP processor 140 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 140 can perform one or more image processing operations on the raw image data, collect statistics about the image data. Among others, the image processing operations can be performed at the same or different bit depth precisions.

[0034] The ISP processor 140 can also receive image data from the image memory 130. For example, the raw image data is sent to the image memory 130 by the pose sensor 120 interface, and the raw image data in the image memory 130 is provided to the ISP processor 140 for processing. The image memory 130 can be part of a memory device, a storage device, or a separate dedicated memory within the electronic device, and can include a DMA (Direct Memory Access) feature.

[0035] When receiving the raw image data from the image sensor 114 interface or from the pose sensor 120 interface or from the image memory 130, the ISP processor 140 can perform one or more image processing operations, such as temporal filtering. The processed image data can be sent to the image memory 130 for further processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130 and performs image data processing in the raw domain and in one or more color spaces, such as YUV, RGB, YCbCr, etc. The processed image data from the ISP processor 140 can be output to the display 160 for viewing by a user and / or further processing by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 140 can also be sent to the image memory 130, and the display 160 can read the image data from the image memory 130. In one embodiment, the image memory 130 can be configured to implement one or more frame buffers.

[0036] The statistics determined by the ISP processor 140 can be sent to the control logic 150. For example, the statistics can include image sensor 114 statistics such as vibration frequency of the gyroscope, auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 112 shading correction, etc. The control logic 150 can include a processor and / or microcontroller that executes one or more routines (e.g., firmware) that can determine control parameters for the imaging device 110 and control parameters for the ISP processor 140 based on the received statistics. For example, the control parameters for the imaging device 110 can include attitude sensor 120 control parameters (e.g., gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (e.g., focus or zoom focal length), or a combination of these parameters. The ISP control parameters can include gain levels and color correction matrices for auto white balance and color adjustment (e.g., during YUV processing), and lens 112 shading correction parameters.

[0037] In one embodiment, the ISP processor 140 can acquire two different images from the imaging device 110, which are respectively a first image and a second image. The first image and the second image can be two different images including a same photographed object.

[0038] For example, the first image and the second image can be two images acquired by the imaging device 110 in time sequence; or the first image and the second image can be two images acquired by the imaging device 110 from different photographing angles for a same object; or the first image and the second image can be two images acquired by the imaging device 110 when tracking and photographing a moving object, which is not limited in particular.

[0039] After acquiring the first image and the second image, the ISP processor 140 can match image contents in the target region in the first image and the second image to obtain a matching relationship corresponding to the target region. Each of the first image and the second image includes at least two regions, which are divided according to depth values of pixel points in each region; each region included in the first image corresponds to each region included in the second image, and the target region is any one of the at least two regions.

[0040] After acquiring the matching relationship corresponding to each region, the ISP processor 140 can perform image registration processing on the first image and the second image according to the matching relationship corresponding to each region.

[0041] Please refer to Figure 2 , Figure 2Fig. 1 is a schematic diagram of a method flow of an image processing method according to an embodiment, which can be applied to any of the electronic devices described above. As shown in Fig. 1, the method can include the following steps: Figure 2

[0042] 210, match the image content of the first image and the second image in the target region to obtain a matching relationship corresponding to the target region.

[0043] The first image and the second image include the same photographed object, but due to different photographing times or photographing angles, the image coordinates of the same photographed object in the first image and the second image are different. Therefore, in the first image and the second image, the same image content at different image coordinates can be mutually matched image content. Wherein, the image content can include feature points or image blocks, etc., which are not limited in detail.

[0044] Each frame of image in the first image and the second image can be divided into at least two regions based on the depth value of the pixel point. Wherein, the depth value corresponding to the pixel point can be used to represent the physical distance between the pixel point corresponding to the photographed object and the imaging device.

[0045] For example, the first image can include a foreground region and a background region, and the depth value corresponding to the pixel point in the foreground region can be smaller than the depth value corresponding to the pixel point in the background region. When the first image is an image obtained by the imaging device photographing a portrait, the foreground region can be a portrait region where the portrait is located, and the background region can be the remaining image region except the portrait region.

[0046] For example, the first image can also include a foreground region, a close-up region and a long shot region, and the depth value corresponding to each pixel point in the foreground region can be smaller than the depth value corresponding to the close-up region, and the depth value corresponding to the close-up region can be smaller than the depth value corresponding to the long shot region. When the first image is an image obtained by the imaging device photographing a portrait, the foreground region can be a portrait region where the portrait is located, the close-up region can be a background plate set up when photographing the portrait, and the long shot region can be a forest behind the background plate.

[0047] Since the second image can include the same photographed object as the first image, similar to the first image, the second image can also be divided into at least two regions, and each region included in the second image can correspond to each region included in the first image respectively.

[0048] The target region described above can be any one of the at least two regions included in the first image. For example, the target region can be the foreground region, or the background region. The electronic device can obtain the matching relationship corresponding to the target region after matching the image content for each target region.

[0049] ​For example, assuming that the matching of image content includes feature point matching, if the first image and the second image respectively include a foreground region and a background region, refer to Figure 3A and Figure 3B , Figure 3A is an example diagram of feature point matching of a foreground region according to an embodiment, Figure 3B is an example diagram of feature point matching of a background region according to an embodiment.

[0050] If the target region is a foreground region, an example of pixel point matching in the foreground region of the first image and the second image can be as shown in Figure 3A The electronic device can identify the feature point pairs 310 that match each other in the foreground region of the first image and the second image. If the target region is a background region, an example of pixel point matching in the background region of the first image and the second image can be as shown in Figure 3B The electronic device can identify the feature point pairs 320 that match each other in the background region of the first image and the second image.

[0051] It can be seen that in the embodiments of the present application, the electronic device can perform image content matching in different image regions respectively, and obtain the matching relationship corresponding to each different image region respectively. The matching relationship corresponding to each region can include a plurality of feature point pairs, and each feature point pair can include two feature points that match each other. Alternatively, the matching relationship corresponding to each region can include a plurality of image block pairs, and each image block pair can include two image blocks that match each other, which is not limited in detail.

[0052] In the embodiments of the present application, the image can be divided into regions in advance according to the depth values of the pixel points, and the matching of image content can be performed in different image regions respectively, so that the pixel points with larger depth values in the first image are matched with the pixel points with larger depth values in the second image, and the pixel points with smaller depth values in the first image are matched with the pixel points with smaller depth values in the second image, thereby improving the matching accuracy of image content.

[0053] 220, performing image registration on the first image and the second image by using the matching relationship corresponding to each region of the at least two regions.

[0054] The matching relationship corresponding to each region of the at least two regions can be used for image registration, and the matching relationship corresponding to each region can be used to indicate the offset amount of each region between two frames of images, i.e., the translation and rotation relationship of each region between two frames of images.

[0055] The electronic device can utilize the matching relationship corresponding to each region to perform registration on the first image, so that the first image is aligned with the second image based on the second image; or the electronic device can also utilize the matching relationship corresponding to each region to perform registration on the second image, so that the second image is aligned with the first image based on the first image, and the specific implementation is not limited.

[0056] It can be understood that in a scene with a large difference in image depth, the offset of the image region corresponding to different depth values between two frames of images can also be different and large. For example, when the imaging device is shooting at a fixed shooting angle, the portrait close to the imaging device moves, and the offset of the portrait region with a small depth value between two frames of images is large, but the offset of the background region with a small depth value between two frames of images is small. If the offset is calculated only by using a matching relationship corresponding to the whole image, the calculated offset can be greatly different from the actual offset corresponding to different image regions, thereby reducing the accuracy of image registration.

[0057] In the embodiments of the present application, the electronic device can calculate the offset of each region between two frames of images according to the matching relationship corresponding to each region based on the depth value division, and perform image transformation processing on each image region based on the offset corresponding to each region, thereby improving the accuracy of image registration.

[0058] In one embodiment, the first image can be a reference image, and the second image can be a to-be-registered image, and the to-be-registered image is aligned with the reference image based on the reference image. The offset can be represented by a mapping matrix. Therefore, the electronic device can determine the mapping matrix corresponding to each region according to the matching relationship corresponding to each region in the at least two regions, and perform image transformation processing on each image region in the second image by using the mapping matrix corresponding to each region, so that the second image is aligned with the first image based on the first image.

[0059] The mapping matrix corresponding to each region can include a homography matrix. Taking feature point matching as an example, if the electronic device identifies at least four groups of mutually matched feature point pairs in each region, the homography matrix can be calculated according to the identified feature point pairs.

[0060] It can be seen that after the electronic device executes the image processing method disclosed in the embodiments of the present application, a plurality of matching relationships corresponding to different image regions can be obtained, and each matching relationship corresponding to a region can include a plurality of mutually matched feature point pairs or a plurality of mutually matched image blocks. In the related art, if the image regions are not distinguished and feature point matching is performed based on the whole image, only one matching relationship corresponding to the whole image can be obtained. For example, please refer to Figure 3C , Figure 3Cis an example diagram of full-image feature point matching in the related art.

[0061] Therefore, in the embodiments of the present application, the electronic device can calculate two or more mapping matrices based on two or more different matching relationships, while full-image feature point matching can only calculate one mapping matrix corresponding to the full image. The calculation of the mapping matrix in the sub-regions can reduce the calculation error of the offset caused by the calculation of the mapping matrix based on the full-image feature point matching, and is beneficial to improve the registration accuracy of the registered image obtained after the final registration.

[0062] It can be seen that, by implementing the image processing method disclosed in the embodiments of the present application, the electronic device can improve the matching accuracy when the depth difference of the pixel points in the image is large by respectively performing image content matching in different image regions and performing image registration according to the matching relationship corresponding to each region, thereby improving the accuracy of the image registration based on the matching relationship, and making the image registration more accurate. Moreover, the matching error of the image content can be compatible to a certain extent, and the influence of the matching error on the image registration can be reduced.

[0063] In some embodiments, the electronic device can further perform one or more image processing operations such as image synthesis, background blurring, and motion object tracking using the registered image. The registered image with higher accuracy is beneficial to improve the accuracy of a series of image processing operations based on the registered image, for example, the clarity of the image synthesis can be improved.

[0064] In some embodiments, each frame of image in the first image and the second image can be divided into two regions, i.e., a foreground region and a background region, and the following content is based on this to introduce the image processing method disclosed in the embodiments of the present application.

[0065] Please refer to Figure 4 , Figure 4 is a method flow diagram of another image processing method disclosed in an embodiment, which can be applied to any one of the electronic devices described above. As Figure 4 shown, the method can include the following steps:

[0066] 410. Matching the image content of the first image and the second image in the foreground region, and determining the first mapping matrix corresponding to the foreground region according to the matching relationship corresponding to the foreground region.

[0067] 420. Matching the image content of the first image and the second image in the background region, and determining the second mapping matrix corresponding to the background region according to the matching relationship corresponding to the background region.

[0068] The implementation of matching the image content in the foreground region and the background region can refer to the implementation of the step 210 of the foregoing embodiments, and the following content will not be repeated.

[0069] It should be noted that the foregoing steps 410 and 420 do not necessarily exist in logical sequence. In other embodiments, the electronic device can first perform step 420 and then perform step 410, or the electronic device can perform steps 410 and 420 simultaneously.

[0070] 430, multiplying the first image coordinates of the foreground region in the second image by the first mapping matrix to obtain the foreground region of the registered image.

[0071] The multiplication result of the first image coordinates and the first mapping matrix can be the image coordinates of the foreground region in the registered image.

[0072] 440, multiplying the second image coordinates of the background region in the second image by the second mapping matrix to obtain the background region of the registered image.

[0073] The multiplication result of the second image coordinates and the second mapping matrix can be the image coordinates of the background region in the registered image.

[0074] It can be seen that the foregoing registered image can be obtained by performing image transformation processing on the second image, and the registered image is aligned with the first image based on the first image.

[0075] It should be noted that step 430 can be performed after step 410, and step 440 can be performed after step 420. Steps 430 and 440 do not necessarily exist in logical sequence, and step 440 can be performed before step 430 or simultaneously.

[0076] It can be seen that the electronic device can match the image content of the foreground region and the background region respectively, and perform image transformation on the second image to be registered based on the matching relationship corresponding to the foreground region and the background region respectively, so that the transformed registered image can be accurately and highly precisely aligned with the first image.

[0077] For example, please refer to Figure 5A , Figure 5A is a registration effect comparison example of a foreground region disclosed in an embodiment. As Figure 5A shown, taking the first image 510 as a reference image, the second image is registered based on a full-image matching registration method to obtain a registered image 520, and the second image is registered based on the image processing method disclosed in the foregoing embodiment to obtain a registered image 530.

[0078] As Figure 5AAs shown, partial images are cropped at the same position in the foreground regions of the first image 510, the registered image 520, and the registered image 530 to obtain the foreground region 510a in the first image 510, the foreground region 520a in the registered image 520, and the foreground region 530a in the registered image 530.

[0079] like Figure 5A As shown, there is a significant difference between the image position of pixel i in the foreground region 510a and the image position of pixel i in the foreground region 520a; while the image position of pixel i in the foreground region 510a and the image position of pixel i in the foreground region 530a tend to be consistent.

[0080] For example, please refer to Figure 5B , Figure 5B This is a comparison example of the registration effect of a background area disclosed in one embodiment. Figure 5B The first image 510, the registered image 520, and the registered image 530 in the image are... Figure 5A The same applies to Chinese, so the following content will not be repeated.

[0081] like Figure 5B As shown, partial images are cropped at the same positions in the background areas of the first image 510, the registered image 520, and the registered image 530 to obtain the background area 510b in the first image 510, the background area 520b in the registered image 520, and the background area 530b in the registered image 530.

[0082] like Figure 5B As shown, the image position of pixel j in background region 510b is significantly different from that in background region 520b; while the image position of pixel j in background region 510b is more consistent with that in background region 530b.

[0083] The higher the accuracy of image registration, the closer the positions of the matched pixels are on different images. Therefore, implementing the image processing method disclosed in the embodiments of this application can improve the accuracy of image registration.

[0084] In one embodiment, before performing step 310, the electronic device may first divide the image regions in the first and second images. It should be noted that although the image regions are divided according to the depth values ​​of the pixels within each region, the electronic device does not necessarily need to perform depth estimation on the captured image to recover the pixel depth values. The following provides two optional image region division methods, but is not limited to these.

[0085] In addition, since the respective regions included in the first image and the second image correspond respectively, the electronic device can perform the operations shown in the following embodiments on both the first image and the second image respectively, perform region division on the first image and the second image respectively, or perform the operations shown in the following embodiments on only one frame of image, and the other frame of image can be region-divided according to the divided image region.

[0086] The following describes the two image region division manners with the first image as an example.

[0087] As an optional embodiment, the first image and the second image can include a portrait, and the electronic device can perform portrait recognition on the first image to obtain a portrait region where the portrait is located. Generally, in an image with a portrait as a shooting subject, the portrait is usually a foreground region. Therefore, the electronic device can determine the portrait region of the first image as the foreground region of the first image, and determine the remaining image region of the first image except the portrait region as the background region of the first image.

[0088] The electronic device can perform portrait recognition by using a portrait recognition algorithm such as feature extraction-based or machine learning-based portrait matting, portrait segmentation, etc., and the specific implementation is not limited.

[0089] For example, refer to Figure 6 , Figure 6 is an example diagram of performing portrait recognition on the first image according to an embodiment. As shown in Figure 6 , after performing portrait matting on the first image 610, a portrait matting result 620 corresponding to the first image 610 can be obtained. The portrait matting result 620 can be a binary image, in which a pixel point with a pixel value of 0 (black) can be used to indicate a portrait region, and a pixel point with a pixel value of 1 (white) can be used to indicate a background region.

[0090] The embodiment of determining the foreground region and the background region in the second image based on portrait recognition can refer to the embodiment of determining the foreground region and the background region in the first image based on portrait recognition, and the following content will not be repeated.

[0091] It can be seen that in the embodiments of the present application, the foreground regions of the first image and the second image can be portrait regions determined by portrait recognition; wherein the foreground region of the first image can be determined by performing portrait recognition on the first image; and the foreground region of the second image can be determined by performing portrait recognition on the second image.

[0092] Correspondingly, the background region of the first image and the second image can be an image region other than the portrait region. The background region of the first image can be an image region other than the portrait region of the first image, and the background region of the second image can be an image region other than the portrait region of the second image.

[0093] As another optional implementation, the electronic device can perform depth estimation on the image to divide the image region according to the recovered depth value.

[0094] The electronic device can perform monocular depth estimation on the first image by using a machine learning algorithm such as a convolutional neural network (Network) or a U-Net, or can perform binocular depth estimation on the first image and the second image to recover the depth value of each pixel point in the first image by relying on the parallax between the first image and the second image.

[0095] The electronic device can identify, as the pixel point of the foreground region, the pixel point in the first image whose depth value is less than a first depth threshold, to obtain the foreground region of the first image. The first depth threshold can be set according to actual business requirements, and is not limited specifically. Alternatively,

[0096] The electronic device can also identify, as the pixel point of the background region, the pixel point in the first image whose depth value is greater than or equal to a second depth threshold, to obtain the background region of the first image. The second depth threshold can also be set according to actual business requirements. The second depth threshold can be the same as the first depth threshold, or the second depth threshold can be greater than the first depth threshold, and is not limited specifically.

[0097] Optionally, the electronic device can first identify the foreground region of the first image according to the first depth threshold. After identifying the foreground region, the electronic device can determine the remaining image region of the first image other than the foreground region as the background region, or can identify the background region of the first image according to the second depth threshold.

[0098] Optionally, the electronic device can first identify the background region of the first image according to the second depth threshold. After identifying the background region, the electronic device can determine the remaining image region of the first image other than the background region as the foreground region, or can identify the foreground region of the first image according to the first depth threshold.

[0099] The implementation of determining the foreground region and the background region in the second image based on depth estimation can refer to the implementation of determining the foreground region and the background region in the first image based on depth estimation, and the following content will not be repeated.

[0100] It can be seen that the depth values of the pixel points in the first image and the second image can be obtained through depth estimation. The foreground region of the first image and the second image can be an image region in which the depth value of the pixel point is less than the first depth threshold; and / or, the background region of the first image and the second image can be an image region in which the depth value of the pixel point is less than the second depth threshold.

[0101] It can be seen that in the foregoing embodiments, the electronic device can divide the image regions of the first image and the second image through portrait recognition or depth estimation. The image region division method based on portrait recognition can quickly distinguish the foreground region and the background region of the image, reduce the calculation amount, and improve the real-time performance of the algorithm. The image division method based on depth estimation can accurately distinguish the foreground region and the background region of the image when the photographed object of the image is a non-portrait object, which is beneficial to expand the application scenarios of the image processing method disclosed in the embodiments of the present application.

[0102] In the image processing method disclosed in the embodiments of the present application, the electronic device matches the image content in the target region, which can include feature point matching or image block matching. The following content is introduced separately.

[0103] In one embodiment, the electronic device can perform feature point matching in the target region of the first image and the second image, which can include the following steps:

[0104] The electronic device extracts feature points from the first image and the second image respectively to extract first feature points of the first image and second feature points of the second image. The extraction of feature points can not be distinguished by image region, and can be performed in the full image of the first image and the second image.

[0105] For example, the electronic device can extract feature points from the first image and the second image through a feature point detection algorithm such as a Features from Accelerated Segments Test (FAST) algorithm, a Scale-Invariant Feature Transform (SIFT) algorithm, etc.

[0106] After the electronic device divides the first image and the second image into regions based on any one of the foregoing image region division methods, the electronic device can identify third feature points in the first image target region from the first feature points, and identify fourth feature points in the second image target region from the second feature points. Matching the third feature points and the fourth feature points can obtain a feature point matching relationship corresponding to the target region. In addition, the electronic device can simultaneously perform feature point matching and image region division, which can reduce time consumption and improve the efficiency of image registration.

[0107] Exemplarily, the electronic device can describe the third feature point and the fourth feature point through a BRIEF (Binary Robust Independent Elementary Features) feature description algorithm. The matching of the feature points can be converted into calculating the Hamming distance between two feature descriptors, and the Hamming distance between the feature descriptors corresponding to the two feature points matched with each other is the smallest.

[0108] The feature point matching algorithm combining the FAST and the BRIEF is generally called an ORB (Oriented Fast and Rotated Brief) feature matching algorithm. The ORB algorithm has the advantages of being fast and being affected to a certain extent by noise and image transformation. On the basis of performing image content matching in a region, the matching of the image feature points is performed in combination with the ORB feature matching algorithm, which can further improve the accuracy of the image content matching and thus improve the precision of the image registration.

[0109] Exemplarily, please refer to Figure 7 , Figure 7 is a method flow diagram of another image processing method disclosed in an embodiment. As shown in Figure 7 , the first image 710 and the second image 720 can be subjected to feature point extraction based on ORB respectively to obtain the first feature point 730 of the first image 710 and the second feature point 740 of the second image 720.

[0110] Meanwhile, the electronic device can perform portrait matting on the first image 710 to obtain a portrait matting result 750. The portrait matting result 750 can be used to distinguish the foreground region and the background region in the first image and the second image.

[0111] The electronic device can identify the third feature point 730a belonging to the foreground region and the third feature point 730b belonging to the background region in combination with the first feature point 730 and the portrait matting result 750. In addition, the electronic device can identify the fourth feature point 740a belonging to the foreground region and the fourth feature point 750b belonging to the background region in combination with the second feature point 740 and the portrait matting result 750.

[0112] The electronic device matches the third feature point 730a belonging to the foreground region and the fourth feature point 740a belonging to the foreground region to obtain a feature point matching relationship corresponding to the foreground region, and matches the third feature point 730b belonging to the background region and the fourth feature point 740b belonging to the background region to obtain a feature point matching relationship corresponding to the background region.

[0113] The electronic device can further calculate a first mapping matrix 760 corresponding to the foreground region according to the feature point matching relationship corresponding to the foreground region, and calculate a second mapping matrix 770 corresponding to the background region according to the feature point matching relationship corresponding to the background region. The first mapping matrix 760 is multiplied by the first image coordinates of the foreground region of the second image, and the second mapping matrix 770 is multiplied by the second image coordinates of the background region of the second image, to obtain a registered image 780.

[0114] In some embodiments, the feature points in the first image and the second image can be less. For example, when the texture features of the images are less, the images include a large number of color blocks of pure colors, it is difficult to extract and match the feature points. In order to solve the problem caused by too few feature points, the electronic device can perform image block matching in the target region of the first image and the second image, which can include the following steps:

[0115] The electronic device respectively performs image blocking in the target region of the first image to obtain a plurality of first image blocks in the target region of the first image, and performs image blocking in the target region of the second image to obtain a plurality of second image blocks in the target region of the second image; and matches the first image blocks and the second image blocks to obtain an image block matching relationship corresponding to the target region.

[0116] The electronic device can divide the target region of the first image into a plurality of first image blocks of the same size, and divide the target region of the second image into a plurality of second image blocks of the same size. The size of the first image blocks and the size of the second image blocks can also be the same.

[0117] After image blocking, the electronic device can match each first image block and second image block based on a square difference measurement method or the like, and the square difference between the mutually matched first image block and second image block is the smallest.

[0118] Based on image blocking, the matching of image content can still calculate a more accurate matching relationship in the scene where the feature points are less, and can solve the problem of reduced image registration accuracy caused by reduced feature points.

[0119] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an image processing device disclosed in an embodiment, which can be applied to any one of the electronic devices described above. As shown in Figure 8 , the image processing device 800 can include a matching module 810 and a registration module 820.

[0120] The matching module 810 can be configured to match image contents of the first image and the second image in a target region to obtain a matching relationship corresponding to the target region. Each frame of image in the first image and the second image includes at least two regions, and each region included in the first image corresponds to a region included in the second image. The at least two regions are divided according to depth values of pixel points in each region. The target region is any one of the at least two regions.

[0121] The registration module 820 can be configured to perform image registration processing on the first image and the second image according to the matching relationship corresponding to each region of the at least two regions.

[0122] In an embodiment, the registration module 820 can include a matrix calculation unit and a transformation unit.

[0123] The matrix calculation unit can be configured to determine a mapping matrix corresponding to each region of the at least two regions according to the matching relationship corresponding to each region.

[0124] The transformation unit can be configured to perform image transformation processing on each region in the second image by using the mapping matrix corresponding to each region to obtain a registration image. The registration image is aligned with the first image with the first image as a reference.

[0125] In an embodiment, the at least two regions include a foreground region and a background region, and a depth value corresponding to a pixel point in the foreground region is less than a depth value corresponding to a pixel point in the background region. The mapping matrix corresponding to each region of the at least two regions includes a first mapping matrix corresponding to the foreground region and a second mapping matrix corresponding to the background region.

[0126] The transformation unit can be further configured to multiply first image coordinates of the foreground region in the second image by the first mapping matrix to obtain a foreground region of the registration image, and multiply second image coordinates of the background region in the second image by the second mapping matrix to obtain a background region of the registration image.

[0127] In an embodiment, the image processing apparatus 800 can further include a determination module.

[0128] The determination module can be configured to perform portrait recognition on the first image to obtain a portrait region where a portrait is located before the matching module 810 matches image contents of the first image and the second image in a target region to obtain a matching relationship corresponding to the target region. The portrait region is determined as a foreground region of the first image, and the remaining image region of the first image except the portrait region is determined as a background region of the first image.

[0129] In one embodiment, the aforementioned determining module can also be used to perform depth estimation on the first image before the matching module 810 matches the image content of the first image and the second image within the target area to obtain a matching relationship corresponding to the target area, to obtain the depth value of each pixel in the first image; and to identify pixels in the first image whose depth value is less than a first depth threshold as pixels in the foreground area, thereby obtaining the foreground area of ​​the first image; and / or to identify pixels in the first image whose depth value is greater than or equal to a second depth threshold as pixels in the background area, thereby obtaining the background area of ​​the first image.

[0130] In one embodiment, the image content includes: feature points; the matching module 810 may include: an extraction unit and a matching unit.

[0131] The extraction unit can be used to extract feature points from the first image and the second image respectively, so as to extract the first feature points of the first image and the second feature points of the second image; and to identify the third feature points located in the target region of the first image from the first feature points, and to identify the fourth feature points located in the target region of the second image from the second feature points.

[0132] The matching unit can be used to match the third and fourth feature points to obtain the feature point matching relationship with the target region.

[0133] In one embodiment, the image content includes: image blocks; the aforementioned matching module 810 may further include: block segmentation units.

[0134] The segmentation unit can be used to segment the image into blocks within the target area of ​​the first image, thereby obtaining multiple first image blocks within the target area of ​​the first image and multiple second image blocks within the target area of ​​the second image.

[0135] The matching unit can also be used to match the first image block and the second image block to obtain the image block matching relationship corresponding to the target region.

[0136] As can be seen, the image processing apparatus disclosed in the embodiments of this application can improve matching accuracy when pixel depth differences in the image are large by performing image content matching in different image regions, thereby improving the accuracy of image registration based on matching relationships and making image registration more accurate. Furthermore, it can also accommodate image content matching errors to a certain extent, reducing the impact of matching errors on image registration.

[0137] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device disclosed in one embodiment. For example... Figure 9 As shown, the electronic device 900 may include:

[0138] a memory 910 storing executable program code;

[0139] a processor 920 coupled with the memory 910;

[0140] The processor 920 invokes the executable program code stored in the memory 910 to implement any of the image processing methods disclosed in the embodiments of the present application.

[0141] It should be noted that, Figure 9 The mobile terminal shown can also include a power supply, input keys, a camera, a speaker, a screen, RF circuitry, a Wi-Fi module, a Bluetooth module, sensors, and other components not shown, and the embodiments of the present application will not be described here.

[0142] The embodiments of the present application disclose a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to enable the processor to implement any of the image processing methods disclosed in the embodiments of the present application.

[0143] The embodiments of the present application disclose a computer program product including a non-transitory computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement any of the image processing methods disclosed in the embodiments of the present application.

[0144] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0145] In various embodiments of the present application, it should be understood that the size of the serial number of the above processes does not mean the inevitable sequence of execution, and the execution sequence of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0146] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed to multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiments of the present application.

[0147] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0148] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or parts of the technical solutions contributing to the prior art, or all or a part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc., and specifically can be a processor in the computer device) to perform all or part of the steps of the methods according to the embodiments of the present application.

[0149] A person of ordinary skill in the art can understand that all or part of the steps of the methods in the above embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, including a Read-Only Memory (ROM), a Random Access Memory (RAM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM), or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other medium of a computer-readable type.

[0150] The image processing method and device, the electronic device, and the storage medium disclosed by the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that, The method includes: The image content of the first image and the second image within the target area is matched to obtain a matching relationship corresponding to the target area; each frame of the first image and the second image includes at least two regions, and each region included in the first image corresponds to each region included in the second image. The at least two regions are divided according to the depth value of the pixels in each region. The depth value of the pixels can be used to characterize the physical distance between the object being photographed and the imaging device corresponding to the pixel; the target area is any one of the at least two regions. Image registration is performed on the first image and the second image based on the matching relationship corresponding to the at least two regions respectively; At least two regions of both the first image and the second image include a foreground region and a background region, wherein the depth value of pixels in the foreground region is less than the depth value of pixels in the background region; the foreground region of the first image and the second image is an image region where the depth value of pixels is less than a first depth threshold; and / or, the background region of the first image and the second image is an image region where the depth value of pixels is greater than or equal to a second depth threshold. The depth values ​​of pixels in the first and second images are obtained through depth estimation.

2. The method according to claim 1, characterized in that, The step of performing image registration on the first image and the second image based on the matching relationship corresponding to the at least two regions includes: Determine the mapping matrix corresponding to each region based on the matching relationship between each region in the at least two regions; The image transformation process is performed on each region in the second image using the mapping matrix corresponding to each region to obtain a registration image; the registration image is aligned with the first image as a reference.

3. The method according to claim 1, characterized in that, The foreground regions of the first image and the second image are human portrait regions determined by human portrait recognition; the background regions of the first image and the second image are image regions other than human portrait regions.

4. The method according to any one of claims 1 to 3, characterized in that, The image content includes: feature points; the matching of the image content of the first image and the second image within the target region to obtain a matching relationship corresponding to the target region includes: Feature points are extracted from the first image and the second image respectively, so as to extract the first feature point of the first image and the second feature point of the second image; A third feature point located in the first image target region is identified from the first feature point, and a fourth feature point located in the second image target region is identified from the second feature point; The third feature point and the fourth feature point are matched to obtain the feature point matching relationship corresponding to the target region.

5. The method according to any one of claims 1 to 3, characterized in that, The image content includes: image blocks; the matching of the image content of the first image and the second image within the target area to obtain a matching relationship corresponding to the target area includes: The image is divided into blocks within the target area of ​​the first image to obtain multiple first image blocks within the target area of ​​the first image and multiple second image blocks within the target area of ​​the second image. The first image block and the second image block are matched to obtain the image block matching relationship corresponding to the target region.

6. An image processing apparatus, characterized in that, include: The matching module is used to match the image content of the first image and the second image within the target area to obtain a matching relationship corresponding to the target area; each frame of the first image and the second image includes at least two regions, and each region included in the first image corresponds to each region included in the second image. The at least two regions are divided according to the depth value of the pixels in each region, and the depth value of the pixels can be used to characterize the physical distance between the shooting object corresponding to the pixel and the imaging device. The target area is any one of the at least two areas; The registration module is used to perform image registration processing on the first image and the second image according to the matching relationship corresponding to the at least two regions respectively; Both the first image and the second image include at least two regions, one foreground region and one background region, wherein the depth value of the pixels in the foreground region is less than the depth value of the pixels in the background region. The foreground region of the first image and the second image is an image region where the depth value of a pixel is less than a first depth threshold; And / or, the background region of the first image and the second image is an image region where the depth value of a pixel is greater than or equal to a second depth threshold; The depth values ​​of pixels in the first and second images are obtained through depth estimation.

7. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images