Image processing method, device, storage medium and electronic device

Abstract

The present application discloses an image processing method, device, storage medium and electronic device, wherein the method comprises: determining a first feature region in an image, obtaining a depth image of the image and a segmented image of a second feature region in the image, the second feature region including the first feature region; obtaining feature depth information of the first feature region according to the depth image; and correcting the second feature region in the segmented image according to the feature depth information of the first feature region. By adopting the present application, the accuracy of the feature region in the segmented image can be enhanced, and the reliability of image processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] In recent years, photography has risen, become popular, and then become universal. With the development of social networks and self-media, photography has become an indispensable part of the digital age, and image processing technology has also emerged. Image segmentation technology, as an important part of image processing technology, has also received increasing attention. For example, image segmentation technology can be used to segment the portrait area in an image, separate the portrait area from the background, and highlight or blur the portrait area. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, storage medium and electronic device, which can enhance the accuracy of segmenting feature areas in an image and improve the reliability of image processing. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Determine a first feature region in an image, obtain a depth image of the image and a segmented image of a second feature region in the image, wherein the second feature region includes the first feature region;

[0006] Acquire feature depth information corresponding to the first feature area according to the depth image;

[0007] The second feature area in the segmented image is corrected according to feature depth information corresponding to the first feature area.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0009] An image acquisition module, used to determine a first feature area in an image, acquire a depth image of the image and a segmented image of a second feature area in the image, wherein the second feature area includes the first feature area;

[0010] An information acquisition module, configured to acquire feature depth information corresponding to the first feature area according to the depth image;

[0011] An image correction module is used to correct the second feature area in the segmented image according to the feature depth information corresponding to the first feature area.

[0012] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0014] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0015] According to the feature depth information of the first feature area, the second feature area segmented from the image is corrected, that is, the depth information is used to provide a reference for image segmentation, so as to obtain a more accurate second feature area, thereby improving the errors and instability caused by a single algorithm that only uses depth information or a segmentation algorithm in the related technology, avoiding the influence of erroneous segmentation on the imaging performance of the image, and improving the reliability of image processing at a low computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is an image schematic diagram of an image processing provided by an embodiment of the present application;

[0018] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0019] Figure 3A is a schematic diagram of an image including a first characteristic area provided in an embodiment of the present application;

[0020] Figure 3B is a schematic diagram of a depth image corresponding to an image provided in an embodiment of the present application;

[0021] Figure 3C is a schematic diagram of a segmented image including a second feature area provided in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0023] Figure 5is a schematic diagram of alignment of an image and a depth image provided by an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a depth image including depth feature information provided by an embodiment of the present application;

[0025] Figure 7 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of correcting a second feature area according to depth feature information provided by an embodiment of the present application;

[0027] Fig. 9 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0028] Fig.10 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0029] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0032] The present application is described in detail below with reference to specific embodiments.

[0033] In recent years, photography has risen, become popular, and then become universal. With the development of social networks and self-media, photography has become an indispensable part of the digital age, and image processing technology has also emerged. Image segmentation technology, as an important part of image processing technology, has also received increasing attention. For example, image segmentation technology can be used to segment the portrait area in an image, separate the portrait area from the background, and highlight or blur the portrait area.

[0034] like Figure 1 As shown, it is an image schematic diagram of an image processing provided by an embodiment of the present application. The left picture is a photo taken by a user. When the user wants to realize the "background blur" function, that is, when the processor receives the trigger condition on the background blur control, it is first necessary to segment the portrait in the left picture, that is, to obtain the portrait area shown in the right picture, so as to further blur the background area outside the portrait area.

[0035] There are three main segmentation technologies in the related image segmentation technologies. The first is the traditional color space method, the edge feature-based method, the wavelet transform-based method, etc. However, this technology is difficult to analyze for areas with similar colors, changes in light and shadow, or interference from image noise. Some improvements can be obtained by referring to a wider range of information, but this also increases the computational cost. The second is semantic segmentation and instance segmentation based on deep learning (deep leaming). However, deep learning algorithms such as semantic segmentation mainly rely on the design of the network architecture and the training data. In particular, in the training data set, all target scenes must be covered as much as possible. However, the behavior of taking photos is ever-changing, and different viewing angles, irregular portrait occlusion (multi-person scenes) and uncertain texture features, etc., many variables may affect the segmentation results, so there is a problem of insufficient stability. The third is the segmentation method based on the depth map. The segmentation method based on the depth map can distinguish different objects at different distances, but it cannot analyze different objects at the same distance. And for the depth generated by stereo vision, there is a problem of parallax occlusion, which is not completely reliable.

[0036] Therefore, based on the above problems, the present application proposes an image processing method, which can enhance the accuracy of segmenting feature areas in an image and improve the reliability of image processing.

[0037] In one embodiment, Figure 2FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application. The method can be implemented by a computer program and can be run on an image processing device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0038] Specifically, the image processing method includes:

[0039] S101. Acquire a depth image of the image and a segmented image of a second feature region in the image according to a first feature region in the image, where the second feature region includes the first feature region.

[0040] An image is a similar and vivid description or portrait of natural things or objective objects (people, animals, plants, landscapes, etc.), or an image can be understood as a way of representing natural things or objective objects (people, animals, plants, landscapes, etc.), which contains relevant information about the described object. Usually, an image is a picture with visual effects, which can be a photo, painting, clip art, map, satellite cloud map, film and television screen, X-ray, electroencephalogram, electrocardiogram, etc.

[0041] The first characteristic region in the image can be understood as a specific region in the region of interest of the image, the region of interest is also the second characteristic region, the second characteristic region includes the first characteristic region, and the region can be a single pixel or a collection of multiple pixels. For example, if the region of interest is a portrait region, the first characteristic region is a face region, if the region of interest is a car region, the first characteristic region is a front region, if the region of interest is a chest region, the first characteristic region is a heart region.

[0042] It can be understood that the following figure takes the first feature area as a face area and the second feature area as a portrait area as an example, and the present application also includes other types and contents of first feature areas and second feature areas. The image processing method described in the present application is applicable to any image including a first feature area and a second feature area.

[0043] The first feature region in the image is obtained by any one or more feature extraction algorithms in the related art, such as face feature extraction algorithm, deep learning network, local binary pattern (Local Binary Pattern, LBP) algorithm, etc., and this application does not impose any restrictions on this. Figure 3A , which is a schematic diagram of an image including a first feature area provided in an embodiment of the present application, wherein the first feature area 101 is also the image area corresponding to the face in the figure.

[0044] Obtain a depth image of the image. The depth image includes depth information. The depth information can be understood as the distance value or depth value from any point in the image to the camera, that is, the three-dimensional coordinate information of any point. The depth information is obtained by any one or more methods such as stereo image matching, depth camera or deep learning, and this application does not impose any restrictions on this. Figure 3B , which is a schematic diagram of a depth image corresponding to an image provided in an embodiment of the present application, wherein areas of different shades of color in the figure correspond to different depth values.

[0045] A segmented image of the second characteristic region in the image is obtained. The segmented image includes the second characteristic region and at least one other region. The segmented image can be obtained by any one or more methods such as a histogram threshold method, a region growing method, an image-based random field model method, and a relaxed labeled region segmentation method, and the application does not impose any restrictions on this.

[0046] like Figure 3C FIG. 1 is a schematic diagram of a segmented image including a second feature region provided in an embodiment of the present application. The second feature region 102 is a portrait region obtained based on an image segmentation method. Figure 3C The segmented image shown also includes a first region 103, a second region 104, a third region 105, and a fourth region 106. The fourth region 106 is a background region obtained based on the image segmentation method. It can be understood that Figure 3C The divisions of the multiple regions shown are for illustration only.

[0047] S102: Acquire feature depth information corresponding to the first feature area according to the depth image.

[0048] The feature depth information corresponding to the first feature region can be understood as information representing the distance value between the first feature region and the camera. The feature depth information corresponding to the first feature region is obtained according to the depth image and the first feature region extracted from the image.

[0049] For example, Figure 3A The first feature area 101 in the image shown in FIG. Figure 3B The depth map shown in FIG. 1 is used to obtain the feature depth information corresponding to the first feature area 101 .

[0050] S103 . Correct the second feature region in the segmented image according to feature depth information corresponding to the first feature region.

[0051] Since the segmentation accuracy of the second feature area included in the segmented image in the related image segmentation technology is not satisfactory, the second feature area in the segmented image is corrected according to the feature depth information corresponding to the first feature area.

[0052] The correction principle can be understood as: due to the correlation between the first feature area and the second feature area, that is, the second feature area and the first feature area are in the same plane, or the difference between the depth value of the second feature area and the depth value of the first feature area is less than a preset threshold, for example, when a person takes a selfie, the face and the portrait are in the same plane; then according to the feature depth information corresponding to the first feature area, the second feature area is inferred in the depth map, the second feature area corresponding to the depth map is compared with the second feature area in the segmented image, and the second feature area in the segmented image is corrected according to the comparison result.

[0053] In one embodiment, the correction method is: determining a target region in the segmented image belonging to the second feature region according to feature depth information corresponding to the first feature region; and correcting the second feature region according to the target region. The number of the target regions is one or more, and the target region is a single pixel or a collection of multiple pixels.

[0054] For example, according to the feature depth information corresponding to the first feature area 101, Figure 3B The second feature area and the background area are obtained from the depth image shown; Figure 3B The second feature area in the depth image is Figure 3C Compared with the segmented image shown, Figure 3C The segmented image shown includes a second characteristic area 102, a first area 103, a second area 104, a third area 105, and a fourth area 106, wherein the fourth area 106 is a background area, and the target area is judged to be the first area 103, the second area 104, and the third area 105; the original second characteristic area 102 is corrected according to the target area.

[0055] In this embodiment, the second feature area of ​​the segmented image is corrected to solve the problem that different objects at the same distance cannot be distinguished because the second feature area is obtained based on the feature depth information of the first feature area in the depth image. Figure 3B According to the depth feature information corresponding to the face area (i.e., the first feature area), the portrait area (i.e., the second feature area) includes a star-shaped decoration area 107. The star-shaped decoration area 107 can be understood as many falling star lights, and the user is taking a selfie among many star lights. Figure 3B The portrait area corresponding to the depth image shown is Figure 3CWhen the portrait area 102 corresponding to the segmented image is corrected, the star-shaped decoration area in the segmented image belongs to the background area 106, and the portrait area 102 is excluded, and the target area for further correction of the portrait area 102 does not include the star-shaped decoration area. Therefore, the present application solves the problem that when the image is segmented only based on the depth image, it is impossible to distinguish different objects at the same distance, which causes inaccuracy in image segmentation.

[0056] In one embodiment, modifying the second feature region according to the target region includes: fusing the target region into the second feature region. Figure 3C In the segmentation diagram shown, the first region 103 , the second region 104 , and the third region 105 as the target region are merged into the second feature region 102 .

[0057] In one embodiment, the image processing technology proposed in this application can be applied to the field of video encoding. Specifically, each frame of video included in the video is segmented into a background area and an area of ​​interest through image processing technology. The area of ​​interest is the second characteristic area mentioned above, and the background area and the area of ​​interest of each frame of video image are independently encoded, for example, a low-distortion encoding method is used for the area of ​​interest, and a simple and efficient encoding method is used for the background area. Through this embodiment, the encoding efficiency of the video is improved and the memory space is saved.

[0058] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0059] In one embodiment, Figure 4 As shown, a flowchart of an image processing method provided by an embodiment of the present application is provided, and the method can be implemented by a computer program and can be run on an image processing device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0060] S201. Acquire a depth image of the image and a segmented image of a second feature region in the image according to a first feature region in the image, where the second feature region includes the first feature region.

[0061] S201 refers to the above-mentioned S101 and will not be described in detail here.

[0062] S202: Acquire a depth image region corresponding to the first feature region according to the aligned depth image and image.

[0063] The depth image and the image are aligned, and the depth image area of ​​the first feature area in the image in the depth image is obtained according to the aligned depth image and the image. The image and depth image alignment method includes one or more feature extraction algorithms such as ORB (Oriented FAST and Binary Robust Independent Elementary Features) feature extraction method, SURF (Speeded-Up Robust Features) algorithm, feature matching algorithm, calculation of homography matrix and twisting picture algorithm, etc. This application does not impose any restrictions on the method of aligning images and depth images.

[0064] like Figure 5 FIG. 1 is a schematic diagram of aligning an image and a depth image provided by an embodiment of the present application. Figure 5 The upper part shows that the image including the first feature region and the depth image are aligned, and the depth image region corresponding to the first feature region, that is, the depth image region 201 is obtained according to the aligned depth image and the image.

[0065] S203 . Acquire feature depth information corresponding to the first feature region according to the depth image region corresponding to the first feature region.

[0066] In one embodiment, according to the depth image region corresponding to the first feature region, the depth values ​​corresponding to the depth image region are averaged to obtain average depth information corresponding to the first feature region; and according to the average depth information, feature depth information corresponding to the first feature region is obtained.

[0067] For example, Figure 5 The depth image region 201 shown includes N pixels, each pixel corresponds to a depth value X, and the N depth values ​​are averaged to obtain an average depth value corresponding to the depth image region 201.

[0068] In another embodiment, based on the depth image area corresponding to the first feature area, a weighted calculation is performed on the depth value corresponding to the depth image area to obtain the depth information corresponding to the first feature area; based on the depth information, the feature depth information corresponding to the first feature area is obtained. For example, the weight is negatively correlated with the distance value of the target pixel point from the center pixel point in the first feature area. The farther the distance value of the target pixel point from the center pixel point is, the smaller the weight is assigned to the depth value corresponding to the target pixel point during calculation. The present application also includes any other weighted calculation method.

[0069] In one embodiment, obtaining feature depth information corresponding to the first feature region according to the average depth information includes: performing compensation calculation on the depth value corresponding to the average depth information to obtain the feature depth information corresponding to the first feature region. The compensation calculation is used to perform range compensation on the depth value corresponding to the feature depth information corresponding to the first feature region.

[0070] For example, Figure 5 As shown, the average depth value D of the depth image area 201 corresponding to the first feature area is obtained, and the average depth value D is compensated by ±10% (it can be understood that the compensation value is only an example), and the depth feature information corresponding to the first feature area is obtained as a depth value range [0.9D, 1.1D], and the area in the depth image where the depth value does not belong to the depth value range is marked as null, and the following is obtained. Figure 6 The depth image schematic diagram shown is Figure 6 A schematic diagram of a depth image including depth feature information provided in an embodiment of the present application includes a second feature area 202 corresponding to the depth feature information after compensation calculation.

[0071] In the present application, feature depth information corresponding to the first feature area is obtained based on the average depth information, and the depth value corresponding to the average depth information is compensated and calculated to obtain the feature depth information corresponding to the first feature area, so that the depth feature information corresponding to the first feature area is more accurate and reliable.

[0072] S204 : Correct the second feature region in the segmented image according to the feature depth information corresponding to the first feature region.

[0073] S204 refers to the above-mentioned S103 and will not be described in detail here.

[0074] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0075] In one embodiment, Figure 7 As shown, a flowchart of an image processing method provided by an embodiment of the present application is provided, and the method can be implemented by a computer program and can be run on an image processing device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0076] S301: Determine a first characteristic region in an image, and obtain a depth image of the image and a segmented image of a second characteristic region in the image.

[0077] S301 refers to the above-mentioned S101 and will not be described in detail here.

[0078] S302: Acquire feature depth information corresponding to the first feature area according to the depth image.

[0079] S302 refers to the above-mentioned S202 and S203, which will not be described in detail here.

[0080] S303A: Determine a first candidate region in the segmented image whose confidence level is within a first confidence level range.

[0081] The segmented image includes a second feature region and at least one other region, and each region corresponds to a confidence level belonging to the second feature region. The distribution of the confidence level conforms to a normal distribution model, and the confidence level is divided into three ranges, namely, a first confidence level range (0, T1], a second confidence level range (T1, T2), and a third confidence level range [T2, 1], wherein T1 and T2 are any values ​​set according to requirements.

[0082] In the present application, when correcting the second feature area in the segmented image, the area with a confidence level of 0 or approaching 0 is not considered, and the area with a confidence level within the third confidence level range is corrected to the second feature area, or a correction calculation is performed on whether the area belongs to the second feature area based on other relevant image segmentation algorithms and image segmentation models.

[0083] like Figure 8 , which is a schematic diagram of correcting the second feature area according to the depth feature information provided by an embodiment of the present application. Figure 8 The segmented image shown in the upper part includes the second feature region 102 and the first region 103, the second region 104, the third region 105, and the fourth region 106. Among them, the confidence that the first region 103 belongs to the second feature region 102 is 0.5, the confidence that the second region 104 belongs to the second feature region 102 is 0.6, the confidence that the third region 105 is 0.2, and the confidence that the fourth region 106 is 0, and the fourth region 106 is determined to be the background region.

[0084] Determine a first candidate region in the segmented image whose confidence is in a first confidence range, where the first confidence range is (0, T1], where T1 is a certain value set according to requirements. For example, the first confidence range is (0, 0.3], Figure 8 In the segmented image shown in the upper part, the first candidate region is the third region 105 with a confidence level of 0.2. It can be understood that the first candidate region can be a single pixel or a set of pixels, and the segmented regions shown in this application are only examples.

[0085] S304A: Determine a first target region in the segmentation map that belongs to the second feature region according to the feature depth information corresponding to the first feature region and the first candidate region.

[0086] According to the feature depth information corresponding to the first feature area and the first candidate area, a first target area in the first candidate area belonging to the second feature area is determined, and the condition of the first target area is that the corresponding depth image area is included in the area determined by the feature depth information, that is, the condition of the first target area is that the corresponding depth value belongs to the depth value range in the feature depth information corresponding to the first feature area.

[0087] For example, Figure 8 The third region 105 as the first candidate region in the segmented image shown in the upper part is included in the region 301 determined by the feature depth information corresponding to the first feature region in the depth image, so the third region 105 belongs to the first target region.

[0088] S303B: Determine a second candidate region in the segmented image whose confidence level is within a second confidence level range.

[0089] According to S303A and S304A, a first target region in a first candidate region whose confidence is within a first confidence range in the segmented image is obtained, and according to steps S303B, S304B and S305B, a second target region in a second candidate region whose confidence is within a second confidence range is obtained. It can be understood that the present application also includes other fusion algorithms for determining the second target region, and the following is only one possible implementation method.

[0090] Determine a second candidate region in the segmented image whose confidence is within a second confidence range, where the second confidence range is (T1, T2), where T1 and T2 are any values ​​set according to requirements. For example, the second confidence range is (0.3, 0.7]. Figure 8 In the segmented image shown in the upper part, the second candidate region is the first region 103 with a confidence of 0.5 and the second region 104 with a confidence of 0.6. It can be understood that the second candidate region can be a single pixel or a set of pixels, and the segmented regions shown in this application are only examples.

[0091] S304B: Acquire feature depth information corresponding to the second candidate region according to the depth image, and recalculate the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region.

[0092] The confidence of the second candidate area is recalculated according to the feature depth information corresponding to the second candidate area. The confidence value and the corresponding depth value of the second candidate area are considered in the calculation fusion, and the confidence value and the depth value are fused and calculated to recalculate the confidence of the second candidate area.

[0093] For example, calculating Figure 8 The absolute error α between the average depth value of the region 202 and the face region in the upper half of the depth image, that is, the absolute error α between the depth value of the target region in the region 202 and the average depth value corresponding to the face region; α is respectively integrated into the first region 103 and the second region 104 included in the second candidate region for calculation, and the calculation formula includes the following formula;

[0094] S(x)=(1-α)×M(x)+α×P(x);

[0095] Among them, M(x) indicates whether region x in the second candidate region belongs to the second feature region determined by the feature depth information in the depth image. If yes, M(x)=1, if not, M(x)=0; P(x) indicates the confidence of region x in the second candidate region.

[0096] In other words, according to the error between the depth value corresponding to each area in the second candidate area and the average depth value of the face area (that is, the first feature area), different weights are assigned to the confidence of each area in the second candidate area. The larger the error, the smaller the weight assigned to the confidence, and vice versa.

[0097] For example, the second candidate area is the first area 103 with a confidence of 0.5 and the second area 104 with a confidence of 0.6. The confidence corresponding to the first area 103 is recalculated as 0.6 according to the depth value corresponding to the first area 103, and the confidence corresponding to the second area 104 is recalculated as 0.7 according to the depth value corresponding to the second area 104.

[0098] It can be understood that the above is only an example of a fusion calculation method for fusing the confidence value and the depth value to recalculate the confidence of the second candidate area, and the present application also includes any other fusion calculation method.

[0099] S305B: Determine a second target region belonging to the second feature region in the segmented image according to the recalculated confidence level of the second candidate region.

[0100] According to the confidence of the recalculated second candidate area, the area with a confidence higher than the correction threshold is determined as the second target area belonging to the second feature area. For example, the second candidate area is the first area 103 with a confidence of 0.5 and the second area 104 with a confidence of 0.6. The confidence corresponding to the first area 103 is recalculated as 0.6 according to the depth value corresponding to the first area 103, and the confidence corresponding to the second area 104 is recalculated as 0.7 according to the depth value corresponding to the second area 104; the area with a confidence higher than 0.55 is determined as the second target area, and the first area 103 and the second area 104 are determined as the second target area.

[0101] S306: Correct the second characteristic region according to the target region.

[0102] In one embodiment, the second characteristic region is modified according to the first target region and the second target region. In another embodiment, the target region only includes the first target region, or the target region only includes the second target region, and the second characteristic region is modified according to the target region.

[0103] The second feature region is modified according to the target region, including: fusing the target region into the second feature region. Figure 8 As shown, the fourth region 106 as the first target region and the first region 103 and the second region 104 as the second target region are merged into the second characteristic region 105 to obtain Figure 8 The second characteristic region 302 is shown in the lower half.

[0104] In the present application, the candidate areas with confidence levels in different confidence ranges in the segmented image are calculated separately, and the confidence levels of the candidate areas are calculated by integrating the depth information of the depth image, thereby further improving the accuracy of the second feature area in the modified segmented image.

[0105] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0106] In one embodiment, Fig. 9 FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application. The method can be implemented by a computer program and can be run on an image processing device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0107] Specifically, the image processing method includes:

[0108] S401. Acquire a depth image of the image and a segmented image of a second feature region in the image according to a first feature region in the image, where the second feature region includes the first feature region.

[0109] S401 refers to the above-mentioned S101 and will not be described in detail here.

[0110] S402: Acquire a depth image region corresponding to the first feature region according to the aligned depth image and image.

[0111] S402 refers to the above-mentioned S202 and will not be described in detail here.

[0112] S403 . Acquire feature depth information corresponding to the first feature region according to the depth image region corresponding to the first feature region.

[0113] According to the depth image area corresponding to the first feature area, the depth values ​​corresponding to the depth image area are averaged to obtain the average depth information corresponding to the first feature area; the depth values ​​corresponding to the average depth information are compensated to obtain the feature depth information corresponding to the first feature area.

[0114] S403 refers to the above-mentioned S203 and will not be described in detail here.

[0115] S404A: Determine a first candidate region in the segmented image whose confidence level is within a first confidence level range.

[0116] S404A refers to the above-mentioned S303A and will not be described in detail here.

[0117] S405A: Determine a first target region in the segmentation map that belongs to the second feature region according to the feature depth information corresponding to the first feature region and the first candidate region.

[0118] S405A refers to the above-mentioned S304A and will not be described in detail here.

[0119] S404B: Determine a second candidate region in the segmented image whose confidence level is within a second confidence level range.

[0120] S404B refers to the above-mentioned S303B and will not be described in detail here.

[0121] S405B: Acquire feature depth information corresponding to the second candidate region according to the depth image, and recalculate the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region.

[0122] S405B refers to the above-mentioned S304B and will not be described in detail here.

[0123] S406B: Determine a second target region belonging to the second feature region in the segmented image according to the recalculated confidence level of the second candidate region.

[0124] S406B refers to the above-mentioned S305B and will not be described in detail here.

[0125] S407: Correct the second characteristic region according to the target region.

[0126] S407 refers to the above-mentioned S306 and will not be described in detail here.

[0127] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0128] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0129] See also Fig.10 , which shows a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application. The image processing device can be implemented as all or part of the device through software, hardware or a combination of both. The image processing device includes an image acquisition module 1001, an information acquisition module 1002 and an image correction module 1003.

[0130] An image acquisition module 1001 is used to determine a first feature region in an image, acquire a depth image of the image and a segmented image of a second feature region in the image, where the second feature region includes the first feature region;

[0131] An information acquisition module 1002 is used to acquire feature depth information corresponding to the first feature area according to the depth image;

[0132] The image correction module 1003 is used to correct the second feature area in the segmented image according to the feature depth information corresponding to the first feature area.

[0133] In one embodiment, the image correction module 1003 includes:

[0134] a target determination unit, configured to determine a target region in the segmented image belonging to the second feature region according to feature depth information corresponding to the first feature region;

[0135] A target correction unit is used to correct the second characteristic area according to the target area.

[0136] In one embodiment, the target determination unit includes:

[0137] A first candidate subunit, used to determine a first candidate region in the segmented image whose confidence is within a first confidence range;

[0138] The first target subunit is used to determine a first target region belonging to the second feature region in the segmented image according to the feature depth information corresponding to the first feature region and the first candidate region.

[0139] In one embodiment, the target determination unit includes:

[0140] A second candidate subunit, used to determine a second candidate region in the segmented image whose confidence is within a second confidence range;

[0141] a second calculation subunit, configured to obtain feature depth information corresponding to the second candidate region according to the depth image, and recalculate the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region;

[0142] The second target subunit is used to determine the second target region belonging to the second feature region in the segmented image according to the confidence of the second candidate region after recalculation.

[0143] In one embodiment, the second computing subunit is specifically configured to:

[0144] Determining weight information of the feature depth information corresponding to the second candidate region according to the feature depth information corresponding to the first feature region;

[0145] The confidence of the second candidate region is recalculated according to the weight information of the feature depth information corresponding to the second candidate region.

[0146] In one embodiment, the target correction unit is specifically used to:

[0147] The target region is merged into the second feature region.

[0148] In one embodiment, the information acquisition module 1002 includes:

[0149] an alignment unit, configured to obtain a depth image region corresponding to the first feature region according to the aligned depth image and the image;

[0150] An acquiring unit is used to acquire feature depth information corresponding to the first feature area according to the depth image area corresponding to the first feature area.

[0151] In one embodiment, the acquisition unit includes:

[0152] an average calculation subunit, configured to perform an average calculation on the depth values ​​corresponding to the depth image region according to the depth image region corresponding to the first feature region, so as to obtain average depth information corresponding to the first feature region;

[0153] The feature acquisition subunit is used to obtain feature depth information corresponding to the first feature area according to the average depth information.

[0154] In one embodiment, the feature acquisition subunit is specifically used for:

[0155] A compensation calculation is performed on the depth value corresponding to the average depth information to obtain feature depth information corresponding to the first feature area.

[0156] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0157] It should be noted that, when the image processing device provided in the above embodiment executes the image processing method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0158] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 one Fig. 9 The image processing method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 one Fig. 9 The specific description of the illustrated embodiment will not be repeated here.

[0160] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 one Fig. 9 The image processing method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 one Fig. 9 The specific description of the illustrated embodiment will not be repeated here.

[0161] See also Fig.11 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 may include: a processor 1101 , a network interface 1104 , a user interface 1103 , a memory 1105 , and a communication bus 1102 .

[0162] The communication bus 1102 is used to realize the connection and communication between these components.

[0163] The user interface 1103 may include a display screen (Display) and a camera (Camera), and the optional user interface 1103 may also include a standard wired interface and a wireless interface.

[0164] The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0165] Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect various parts within the entire electronic device 1100, and executes various functions and processes data of the electronic device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1101 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1101, and it can be implemented separately through a chip.

[0166] Among them, the memory 1105 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1105 includes a non-transitory computer-readable storage medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for functions (such as touch functions, sound playback functions, image playback functions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may optionally be a storage device located away from the aforementioned processor 1101. As Fig.11 As shown, the memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image correction application.

[0167] exist Fig.11In the electronic device 1100 shown, the user interface 1103 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1101 can be used to call the image processing application stored in the memory 1105 and specifically perform the following operations:

[0168] Determine a first feature region in an image, obtain a depth image of the image and a segmented image of a second feature region in the image, wherein the second feature region includes the first feature region;

[0169] Acquire feature depth information corresponding to the first feature area according to the depth image;

[0170] The second feature area in the segmented image is corrected according to feature depth information corresponding to the first feature area.

[0171] In one embodiment, the processor 1101 performs the correction of the second feature area in the segmented image according to the feature depth information corresponding to the first feature area, specifically performing:

[0172] determining, according to the feature depth information corresponding to the first feature area, a target area in the segmented image belonging to the second feature area;

[0173] The second characteristic region is corrected according to the target region.

[0174] In one embodiment, the processor 1101 determines the target area in the segmented image belonging to the second feature area according to the feature depth information corresponding to the first feature area, and specifically performs:

[0175] Determine a first candidate region in the segmented image whose confidence level is within a first confidence level range;

[0176] A first target region in the segmented image belonging to the second feature region is determined according to the feature depth information corresponding to the first feature region and the first candidate region.

[0177] In one embodiment, the processor 1101 determines the target area in the segmented image belonging to the second feature area according to the feature depth information corresponding to the first feature area, and specifically performs:

[0178] Determine a second candidate region in the segmented image whose confidence level is within a second confidence level range;

[0179] Acquire feature depth information corresponding to the second candidate region according to the depth image, and recalculate the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region;

[0180] A second target region belonging to the second feature region in the segmented image is determined according to the recalculated confidence level of the second candidate region.

[0181] In one embodiment, the processor 1101 recalculates the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region, and specifically performs:

[0182] Determining weight information of the feature depth information corresponding to the second candidate region according to the feature depth information corresponding to the first feature region;

[0183] The confidence of the second candidate region is recalculated according to the weight information of the feature depth information corresponding to the second candidate region.

[0184] In one embodiment, the processor 1101 corrects the second feature area according to the target area by specifically performing:

[0185] The target region is merged into the second feature region.

[0186] In one embodiment, the processor 1101 obtains the feature depth information corresponding to the first feature area according to the depth image by specifically performing:

[0187] Acquire a depth image region corresponding to the first feature region according to the aligned depth image and the image;

[0188] According to the depth image area corresponding to the first feature area, feature depth information corresponding to the first feature area is acquired.

[0189] In one embodiment, the processor 1101 obtains the feature depth information corresponding to the first feature area according to the depth image area corresponding to the first feature area, and specifically performs:

[0190] According to the depth image region corresponding to the first feature region, average calculation is performed on the depth values ​​corresponding to the depth image region to obtain average depth information corresponding to the first feature region;

[0191] According to the average depth information, characteristic depth information corresponding to the first characteristic region is obtained.

[0192] In one embodiment, the processor 1101 obtains the feature depth information corresponding to the first feature area according to the average depth information, and specifically performs:

[0193] A compensation calculation is performed on the depth value corresponding to the average depth information to obtain feature depth information corresponding to the first feature area.

[0194] The present application corrects the second feature area segmented from the image based on the feature depth information of the first feature area, that is, provides a reference basis for image segmentation through the depth information, so as to obtain a more accurate second feature area, improves the errors and instability caused by the single algorithm that only uses depth information or segmentation algorithm in the related technology, avoids the influence of erroneous segmentation on the imaging performance of the image, and has a low computing cost on the basis of improving the reliability of image processing.

[0195] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0196] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a first feature region in an image, obtain a depth image of the image and a segmented image of a second feature region in the image, wherein the second feature region includes the first feature region; Acquire feature depth information corresponding to the first feature area according to the depth image; Correcting the second feature area in the segmented image according to feature depth information corresponding to the first feature area; The correcting the second feature area in the segmented image according to the feature depth information corresponding to the first feature area includes: determining, according to the feature depth information corresponding to the first feature area, a target area in the segmented image belonging to the second feature area; Correcting the second characteristic area according to the target area; The correcting the second characteristic area according to the target area includes: The target region is merged into the second feature region.

2. The method according to claim 1, characterized in that The determining, according to the feature depth information corresponding to the first feature area, a target area in the segmented image belonging to the second feature area includes: Determine a first candidate region in the segmented image whose confidence level is within a first confidence level range; A first target region in the segmented image belonging to the second feature region is determined according to the feature depth information corresponding to the first feature region and the first candidate region.

3. The method according to claim 1 or 2, characterized in that: The determining, according to the feature depth information corresponding to the first feature area, a target area in the segmented image belonging to the second feature area includes: Determine a second candidate region in the segmented image whose confidence level is within a second confidence level range; Acquire feature depth information corresponding to the second candidate region according to the depth image, and recalculate the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region; A second target region belonging to the second feature region in the segmented image is determined according to the recalculated confidence level of the second candidate region.

4. The method according to claim 3, characterized in that The recalculating the confidence of the second candidate region according to the feature depth information corresponding to the second candidate region includes: Determining weight information of the feature depth information corresponding to the second candidate region according to the feature depth information corresponding to the first feature region; The confidence of the second candidate region is recalculated according to the weight information of the feature depth information corresponding to the second candidate region.

5. The method according to claim 1, characterized in that The acquiring, according to the depth image, feature depth information corresponding to the first feature area includes: Acquire a depth image region corresponding to the first feature region according to the depth image and the image after correspondence; According to the depth image area corresponding to the first feature area, feature depth information corresponding to the first feature area is acquired.

6. The method according to claim 5, characterized in that The acquiring, according to the depth image area corresponding to the first feature area, feature depth information corresponding to the first feature area includes: According to the depth image region corresponding to the first feature region, average calculation is performed on the depth values ​​corresponding to the depth image region to obtain average depth information corresponding to the first feature region; According to the average depth information, characteristic depth information corresponding to the first characteristic region is obtained.

7. The method according to claim 6, characterized in that The obtaining, according to the average depth information, feature depth information corresponding to the first feature area comprises: A compensation calculation is performed on the depth value corresponding to the average depth information to obtain feature depth information corresponding to the first feature area.

8. An image processing device, characterized in that: The device comprises: An image acquisition module, used to determine a first feature area in an image, acquire a depth image of the image and a segmented image of a second feature area in the image, wherein the second feature area includes the first feature area; An information acquisition module, configured to acquire feature depth information corresponding to the first feature area according to the depth image; An image correction module, used for correcting the second feature area in the segmented image according to feature depth information corresponding to the first feature area; The image correction module comprises: a target determination unit, configured to determine a target region in the segmented image belonging to the second feature region according to feature depth information corresponding to the first feature region; a target correction unit, configured to correct the second characteristic region according to the target region; The target correction unit is specifically used for: The target region is merged into the second feature region.

9. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 7.

Images