Image processing method and device, storage medium and electronic device

Abstract

This application discloses an image processing method, apparatus, storage medium, and electronic device. The method includes: acquiring an image to be processed and acquiring a historical reference image; performing a difference comparison between the image to be processed and the historical reference image to obtain a comparison result; based on the comparison result, determining difference regions and non-difference regions from the image to be processed, and calculating the depth information of the difference regions using a depth information calculation strategy; and determining the depth information of the regions in the historical reference image corresponding to the non-difference regions as the depth information of the non-difference regions. This application can determine the depth information of an image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an image processing method, apparatus, storage medium and electronic device. Background Technology

[0002] With the rapid development of image processing technology, the demand for 3D image information is increasing. Many fields, such as industrial manufacturing, augmented reality, object reconstruction, and gaming, require depth information from images; therefore, solutions for determining the depth information of images are needed. Summary of the Invention

[0003] This application provides an image processing method, apparatus, storage medium, and electronic device that can determine the depth information of an image.

[0004] In a first aspect, embodiments of this application provide an image processing method, including:

[0005] Acquire the image to be processed and acquire historical reference images;

[0006] The image to be processed and the historical reference image are compared to obtain the comparison result.

[0007] Based on the comparison results, the difference regions and non-difference regions are determined from the image to be processed, and the depth information of the difference regions is calculated using a depth information calculation strategy.

[0008] The depth information of the region in the historical reference image that corresponds to the non-differential region is determined as the depth information of the non-differential region.

[0009] Secondly, embodiments of this application provide an image processing apparatus, comprising:

[0010] The image acquisition module is used to acquire the image to be processed and to acquire historical reference images;

[0011] The difference comparison module is used to perform a difference comparison between the image to be processed and the historical reference image to obtain the comparison result.

[0012] The region determination module is used to determine the difference regions and non-difference regions from the image to be processed based on the comparison results, and to calculate the depth information of the difference regions through a depth information calculation strategy.

[0013] The information determination module is used to determine the depth information of the region in the historical reference image that corresponds to the non-differential region as the depth information of the non-differential region.

[0014] Thirdly, embodiments of this application provide an electronic device, including:

[0015] A front-end image processor is used to acquire an image to be processed and a historical reference image; perform a difference comparison between the image to be processed and the historical reference image to obtain a comparison result; determine the difference region and the non-difference region from the image to be processed based on the comparison result; and calculate the depth information of the difference region.

[0016] An application processor is configured to determine the depth information of the region in the historical reference image corresponding to the non-differential region as the depth information of the non-differential region.

[0017] Fourthly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the processor executes the image processing method provided in embodiments of this application by calling a computer program stored in the memory.

[0018] Fifthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the image processing method provided in embodiments of this application.

[0019] In this embodiment, the process involves acquiring an image to be processed and a historical reference image; performing a difference comparison between the image to be processed and the historical reference image to obtain a comparison result; determining the difference region and the non-difference region from the image to be processed based on the comparison result, and calculating the depth information of the difference region using a depth information calculation strategy; and determining the depth information of the region in the historical reference image corresponding to the non-difference region as the depth information of the non-difference region. Thus, by calculating the depth information of the difference region of the image to be processed using a depth information calculation strategy and determining the depth information of the non-difference region of the image to be processed using the historical reference image, the depth information of the image can be determined. Attached Figure Description

[0020] The technical solution and its beneficial effects will become apparent from the following detailed description of specific embodiments of this application, in conjunction with the accompanying drawings.

[0021] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application.

[0022] Figure 2 This is a schematic diagram of the first scenario of the image processing method provided in the embodiments of this application.

[0023] Figure 3 This is a schematic diagram of a second scenario of the image processing method provided in the embodiments of this application.

[0024] Figure 4 This is a schematic diagram of a third scenario of the image processing method provided in the embodiments of this application.

[0025] Figure 5 This is a schematic diagram of the fourth scenario of the image processing method provided in the embodiments of this application.

[0026] Figure 6 This is a schematic diagram of the fifth scenario of the image processing method provided in the embodiments of this application.

[0027] Figure 7 This is a schematic diagram of the sixth scenario of the image processing method provided in the embodiments of this application.

[0028] Figure 8 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0029] Figure 9 This is a schematic diagram of the first structure of the electronic device provided in the embodiments of this application.

[0030] Figure 10 This is a schematic diagram of a second structure of the electronic device provided in the embodiments of this application.

[0031] Figure 11 This is a schematic diagram of the third structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] It should be noted that the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] This application provides an image processing method, an image processing apparatus, a storage medium, and an electronic device. The entity executing the image processing method can be the image processing apparatus provided in this application, or an electronic device integrating the image processing apparatus. The image processing apparatus can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA, laptop computer, or other device equipped with a processor and possessing image processing capabilities.

[0035] Please see Figure 1 , Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application. The process may include:

[0036] In step 101, the image to be processed is obtained, and the historical reference image is also obtained.

[0037] In this embodiment, the electronic device acquires the image to be processed and acquires historical reference images.

[0038] The image to be processed can be an image whose depth information needs to be determined. The image to be processed can be a RAW image, a YUV image, or an RGB image, etc.

[0039] Historical reference images can be images captured by the electronic device before the image to be processed is acquired. For example, the electronic device can continuously acquire images of the shooting scene, obtaining images M11, M12, M13 and M14 in sequence, where M14 can be the image to be processed, and at least one frame of M11, M12 and M13 can be a historical reference image.

[0040] It should be noted that in the embodiments of this application, each image, such as the image to be processed, the historical reference image, and the historical candidate image, can be a RAW image, a YUV image, or an RGB image, etc.

[0041] In step 102, a difference comparison is performed between the image to be processed and the historical reference image to obtain the comparison results.

[0042] In step 103, based on the comparison results, the difference regions and non-difference regions are determined from the image to be processed, and the depth information of the difference regions is calculated using a depth information calculation strategy.

[0043] In this embodiment, after obtaining the image to be processed and the historical reference image, the electronic device performs a difference comparison between the image to be processed and the historical reference image to obtain a comparison result. After obtaining the comparison result, the electronic device determines the difference region and the non-difference region from the image to be processed based on the comparison result.

[0044] In other words, the electronic device compares the differences between the image to be processed and the historical reference image to determine the regions that are different (different image content) and the regions that are the same (same image content). The regions that are different in the image to be processed are defined as the difference regions, and the regions that are the same in the image to be processed are defined as the non-difference regions.

[0045] In this context, the depth information of an image can be understood as the distance from the image acquisition device to each point in the corresponding shooting scene. When an electronic device launches a shooting application (such as the system application "Camera") based on user input, the scene that its camera is pointing at becomes the shooting scene. For example, if a user launches the "Camera" application by tapping its icon on an electronic device, and then points the camera at a particular scene, that scene becomes the shooting scene. Based on the above description, those skilled in the art should understand that the shooting scene does not refer to a specific scene, but rather to the scene that is pointed at in real-time as the camera is positioned.

[0046] In this embodiment, after identifying the difference regions from the image to be processed, the electronic device calculates the depth information of the difference regions using a depth information calculation strategy. The depth information of the difference regions includes the depth information of each pixel within the difference regions.

[0047] For example, depth information calculation strategies include depth estimation models. After identifying the difference region from the image to be processed, the electronic device can segment the difference region from the image to be processed and input the difference region into the pre-trained depth estimation model to obtain the depth information of the difference region.

[0048] For example, depth information calculation strategies include binocular stereo vision strategies. An electronic device acquires an image of a scene to be processed using one camera, and simultaneously acquires an image of the scene to be segmented using another camera at a certain distance. After identifying the difference region from the image to be processed, the electronic device can segment the difference region from the image to be processed, and then segment the corresponding segmented region from the image to be segmented. Subsequently, the electronic device uses a stereo matching algorithm to find the corresponding pixels in the difference region and the segmented region, and then calculates the disparity information based on the triangulation principle. This disparity information, after transformation, can be used to represent the depth information of objects in the scene, thereby obtaining the depth information of the difference region.

[0049] It should be noted that depth information calculation strategies may also include other strategies that can calculate the depth information of an image. Those skilled in the art can adopt corresponding depth information calculation strategies to calculate the depth information of the difference region according to the actual situation, and no specific restrictions are imposed here.

[0050] In 104, the depth information of the region corresponding to the non-differential region in the historical reference image is determined as the depth information of the non-differential region.

[0051] In this embodiment, after identifying the non-disparate region from the image to be processed, the electronic device determines the depth information of the region corresponding to the non-disparate region in the historical reference image as the depth information of the non-disparate region. The depth information of the non-disparate region includes the depth information of each pixel within the non-disparate region.

[0052] It is understood that in this embodiment, the depth information of the historical reference image is already determined, while the depth information of the image to be processed needs to be determined in real time. In the steps described above, the depth information of the difference regions has already been calculated using a depth information calculation strategy. Therefore, it is also necessary to determine the depth information of the non-difference regions. Since the non-difference regions are identical to their corresponding regions in the historical reference image (i.e., the image content of the two regions is the same), when determining the depth information of the non-difference regions, the depth information of the corresponding regions in the historical reference image can be directly used as the depth information of the non-difference regions, thus eliminating the need to calculate the depth information of the non-difference regions using a depth information calculation strategy.

[0053] It is understandable that directly determining the depth information of the corresponding region in the historical reference image as the depth information of the non-differential region can reduce the data processing pressure compared to the scheme of calculating the depth information of the non-differential region through a depth information calculation strategy.

[0054] Furthermore, in this embodiment, for an electronic device including a front-end image processor and an application processor, the front-end image processor can determine the depth information of the difference regions, and then send the image to be processed and the depth information of the difference regions to the application processor, whereby the application processor determines the depth information of the non-difference regions. Since only the depth information of the difference regions needs to be sent, compared to a scheme that sends the depth information of the entire image to be processed, the amount of data transmission can be reduced, saving image transmission speed, and thus saving image processing speed.

[0055] In this embodiment, the image to be processed and a historical reference image are acquired; a difference comparison is performed between the image to be processed and the historical reference image to obtain a comparison result; based on the comparison result, the difference region and the non-difference region are determined from the image to be processed, and the depth information of the difference region is calculated using a depth information calculation strategy; the depth information of the region in the historical reference image corresponding to the non-difference region is determined as the depth information of the non-difference region. Thus, by calculating the depth information of the difference region of the image to be processed using a depth information calculation strategy and determining the depth information of the non-difference region of the image to be processed using the historical reference image, the depth information of the image can be determined.

[0056] In an optional embodiment, a difference comparison is performed between the image to be processed and a historical reference image to obtain a comparison result, including:

[0057] Identify identical objects in the image to be processed and historical reference images;

[0058] Identify a first region of the object in the image to be processed, and a second region in a historical reference image;

[0059] The differences between the first and second regions are compared to obtain the comparison results;

[0060] Based on the comparison results, the regions of difference and regions of non-difference are determined from the image to be processed, including:

[0061] If the comparison result shows that the area of ​​the first region is different from the area of ​​the second region and / or the position of the first region is different from the position of the second region, then the difference region and the non-difference region are determined from the image to be processed based on the first region.

[0062] It is understandable that when there is a moving object in the shooting scene, the position and / or area of ​​the moving object will change in the multiple frames of images continuously captured by the electronic device. For example, as the moving object gets closer to the electronic device, the area of ​​the moving object in the captured image will also become larger. Therefore, the area including the region where the moving object is located is the difference region of the multiple frames of images. For non-moving objects in the shooting scene, the changes in the position and area of ​​the moving object in the multiple frames of images continuously captured by the electronic device (which may be caused by the shaking of the electronic device) can be ignored. The changes in the background scene in the shooting scene can also be ignored. Therefore, in this embodiment, the electronic device can determine the region including the region where the moving object is located in the image to be processed as the difference region, and determine the region other than the difference region in the image to be processed as the non-difference region.

[0063] In other words, in this embodiment, the electronic device first identifies the same object in the image to be processed and the historical reference image; then it identifies the first region of the object in the image to be processed and the second region in the historical reference image; then it performs a difference comparison on the first region and the second region to obtain a comparison result; if the comparison result shows that the area of ​​the first region is different from the area of ​​the second region and / or the position of the first region is different from the position of the second region, it indicates that the object is a moving object. Then, the electronic device determines the difference region and the non-difference region from the image to be processed based on the first region.

[0064] For example, such as Figure 2As shown, the area of ​​the first region A21 of object B2 in the image to be processed M21 is different from the area of ​​the second region A22 of object B2 in the historical reference image M22. Therefore, the electronic device can determine the difference region and the non-difference region from the image to be processed M21 based on the first region A21.

[0065] For example, such as Figure 3 As shown, the position of object B2 in the first region A23 of the image to be processed M23 is different from the position of object B2 in the second region A24 of the historical reference image M24. Therefore, the electronic device can determine the difference region and the non-difference region from the image to be processed M23 based on the first region A23.

[0066] For example, such as Figure 4 As shown, the area of ​​the first region A25 of object B2 in the image to be processed M25 is different from the area of ​​the second region A26 of object B2 in the historical reference image M26. The position of the first region A25 of object B2 in the image to be processed M25 is also different from the position of the second region A26 of object B2 in the historical reference image M26. Therefore, the electronic device can determine the difference region and the non-difference region from the image to be processed M25 based on the first region A25.

[0067] In an optional embodiment, determining the difference region and the non-difference region from the image to be processed based on the first region includes:

[0068] In the image to be processed, the region within a preset range with the center of the first region as the origin is defined as the difference region, and the region in the image to be processed other than the difference region is defined as the non-difference region, wherein the difference region includes the first region.

[0069] Considering that the algorithm for accurately segmenting the first region is computationally expensive for electronic devices, in this embodiment, the electronic device can define the region within a preset range with the center of the first region as the origin as the difference region, and define the region in the image to be processed other than the difference region as the non-difference region. The difference region includes the first region. The preset range can be pre-set by the user or determined by the electronic device based on certain rules.

[0070] For example, such as Figure 5 As shown, assuming the first region is A21, the electronic device can identify region A31 as a difference region and identify the regions in the image M21 to be processed other than region A31 as non-difference regions.

[0071] In an optional embodiment, based on a first region, differentiating and non-differentiating regions are determined from the image to be processed, and depth information of the differentiating regions is calculated, including:

[0072] The region in the image to be processed that corresponds to the second region and forms the region of the first region is defined as the difference region, and the region in the image to be processed that is not the difference region is defined as the non-difference region.

[0073] For example, such as Figure 6 As shown, the electronic device can identify the region A32, which is composed of the region corresponding to the second region A24 and the first region A23 in the image to be processed, as the difference region, and identify the region in the image to be processed other than the difference region as the non-difference region.

[0074] In an alternative embodiment, the electronic device may also directly determine the first region as the difference region and the regions in the image to be processed other than the difference region as the non-difference region.

[0075] In an optional embodiment, before determining regions in the image to be processed other than the difference regions as non-difference regions, the method further includes:

[0076] The region to be determined is identified from the region other than the difference region in the image to be processed, and the depth information of the region to be determined is calculated using a depth information calculation strategy.

[0077] Identify the region to be compared from historical reference images that corresponds to the region to be identified, and obtain the depth information of the region to be compared.

[0078] Regions in the image to be processed other than the regions of difference are identified as non-differential regions, including:

[0079] If the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region in the image to be processed, excluding the difference region, is determined as the non-difference region.

[0080] Understandably, in order to further determine whether the image content of the non-difference region is the same as that of the region corresponding to the non-difference region in the historical reference image, the electronic device can determine at least one region to be determined from the region other than the difference region in the image to be processed, and calculate the depth information of the region to be determined through a depth information calculation strategy; determine the region to be compared with the region to be determined from the historical reference image, and obtain the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region other than the difference region in the image to be processed is determined as the non-difference region.

[0081] For example, such as Figure 7As shown, the electronic device determines regions A41 and A42 to be determined from the regions other than the difference region A31 in the image to be processed M21, and determines the comparison region A43 corresponding to the region to be determined A41 and the comparison region A44 corresponding to the region to be determined A42 from the historical reference image M22. If the depth information of the region to be determined A41 and the comparison region A43 is the same, and the depth information of the region to be determined A42 and the comparison region A44 is the same, then the electronic device can determine the regions other than the difference region in the image to be processed as non-difference regions.

[0082] It should be noted that the specific implementation of "calculating the depth information of the region to be determined through the depth information calculation strategy" can be found in the specific implementation of "calculating the depth information of the difference region through the depth information calculation strategy", which will not be repeated here.

[0083] In an optional embodiment, a difference comparison is performed between the image to be processed and a historical reference image to obtain a comparison result, including:

[0084] Determine the first focus area of ​​the image to be processed, and determine the second focus area of ​​the historical reference image;

[0085] The differences between the first focus area and the second focus area are compared to obtain the comparison results;

[0086] Based on the comparison results, the regions of difference are identified from the image to be processed, and the regions of non-difference are identified from the historical reference image, including:

[0087] If the comparison result shows that the first focus area and the second focus area are different, then the first focus area is determined as the difference area, and the area in the image to be processed other than the first focus area is determined as the non-difference area.

[0088] Considering that the focus area is the user's focus, while the focus area is not of much concern, in this embodiment, the electronic device can determine the first focus area of ​​the image to be processed and determine the second focus area of ​​the historical reference image; if the first focus area and the second focus area are different, then the electronic device can determine the first focus area as the difference area and determine the area in the image to be processed other than the first focus area as the non-difference area.

[0089] In an optional embodiment, before determining the region in the image to be processed, excluding the first focus region, as a non-discrepancy region, the method further includes:

[0090] The region to be determined is identified from the region outside the first focus region in the image to be processed, and the depth information of the region to be determined is calculated using a depth information calculation strategy.

[0091] Identify the region to be compared from historical reference images that corresponds to the region to be identified, and obtain the depth information of the region to be compared.

[0092] Regions in the image to be processed other than the first focus area are defined as non-discrepancy regions, including:

[0093] If the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region in the image to be processed, excluding the first focus region, is determined as a non-differential region.

[0094] It is understandable that "determining the region to be determined from the region other than the first focus region in the image to be processed, and calculating the depth information of the region to be determined using a depth information calculation strategy; determining the region to be compared from the historical reference image corresponding to the region to be determined, and obtaining the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region other than the first focus region in the image to be processed is determined as a non-difference region" can be found in the specific implementation of "determining the region to be determined from the region other than the difference region in the image to be processed, and calculating the depth information of the region to be determined using a depth information calculation strategy; determining the region to be compared from the historical reference image corresponding to the region to be determined, and obtaining the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region other than the difference region in the image to be processed is determined as a non-difference region", which will not be elaborated here.

[0095] In an optional embodiment, after determining the depth information of the region corresponding to the non-difference region in the historical reference image as the depth information of the non-difference region, the method further includes:

[0096] The image to be processed is blurred by using the depth information of the difference region and the depth information of the non-difference region to obtain a blurred image.

[0097] In this embodiment, after obtaining the depth information of the difference region and the depth information of the non-difference region, that is, the depth information of the image to be processed, the electronic device can use the depth information to perform blurring processing on the image to be processed to obtain a blurred image.

[0098] For example, electronic devices can use the plane containing the focus area of ​​the image to be processed as the focal plane. That is to say, in a blurred image, the focus area and the areas with the same depth information as the focus area are the clearest, while other areas in the blurred image have lower clarity as the difference in depth information from the focus area increases.

[0099] In an optional embodiment, after using the depth information of the difference region and the depth information of the non-difference region to blur the image to be processed, and obtaining the blurred image, the electronic device can also use the blurred image for subsequent image processing, such as beautification or image stabilization, etc.

[0100] In an optional embodiment, after obtaining the depth information of the difference region and the depth information of the non-difference region, the electronic device generates a depth image of the image to be processed based on the depth information of the difference region and the depth information of the non-difference region. Each pixel in the depth image represents the depth information of the corresponding pixel in the image to be processed.

[0101] In an optional embodiment, before acquiring the historical reference image, the method further includes:

[0102] Determine the number of historical candidate images;

[0103] Obtain historical reference images, including:

[0104] If the number meets the preset conditions, the historical reference image is determined from the historical candidate images.

[0105] To ensure the real-time nature of depth information, the electronic device can calculate the depth information of the image every preset frame using a depth information calculation strategy to correct errors. The preset condition can be that the number of historical reference images and the number of images to be processed is not a preset number, such as an integer multiple of 5. If the number does not meet the preset condition, a historical reference image is determined from the historical candidate images. If the number meets the preset condition, the depth information of the image to be processed is calculated using the depth information calculation strategy. For example, suppose the electronic device continuously captures images of a scene, obtaining images M11, M12, M13, M14, M15, and M16. For image M11, since it is the first frame obtained by the electronic device, the electronic device can calculate the depth information of this first frame image using the depth information calculation strategy. For images M12, M13, M14, and M16, the electronic device can obtain the depth information of each image using the image processing method provided in this application embodiment. For image M15, the electronic device can calculate the depth information of image M15 using the depth information calculation strategy. The preset frame and preset number can be set by the user or determined by the electronic device based on certain rules.

[0106] It should be noted that "calculating the depth information of the image to be processed using a depth information calculation strategy" can be found in "calculating the depth information of the difference region using a depth information calculation strategy", and will not be repeated here.

[0107] In an optional embodiment, determining a historical reference image from historical candidate images may include: using the historical candidate image whose shooting time is closest to the shooting time of the image to be processed as the historical reference image, or any historical candidate image may be determined as the historical reference image.

[0108] In an optional embodiment, in addition to comparing the depth information of the region to be compared with the depth information of the region to be determined in the historical reference image, the depth information of the region corresponding to the region to be determined in at least one historical candidate image can also be compared with the depth information of the region to be determined. If the depth information of the region to be determined is the same as the depth information of the region to be compared, and the depth information of the region to be determined is the same as the depth information of the region corresponding to the region to be determined in at least one historical candidate image, then the region in the image to be processed, excluding the difference region, is determined as a non-difference region.

[0109] In an optional embodiment, for image M15, the electronic device can calculate the depth information of image M15 simultaneously with the depth information calculation strategy provided in the image processing method of this application, in order to correct errors and facilitate the determination of the depth information of subsequent images. Since obtaining the depth information of image M15 using the image processing method provided in the embodiment of this application is less time-consuming than calculating the depth information of image M15 using the depth information calculation strategy, subsequent image processing, such as blurring, can be directly performed using the depth information of image M15 after obtaining it using the image processing method provided in the embodiment of this application.

[0110] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 200 includes: an image acquisition module 201, a difference comparison module 202, a region determination module 203, and an information determination module 204.

[0111] The image acquisition module 201 is used to acquire the image to be processed and to acquire historical reference images.

[0112] The difference comparison module 202 is used to perform difference comparison between the image to be processed and the historical reference image to obtain the comparison result.

[0113] The region determination module 203 is used to determine the difference regions and non-difference regions from the image to be processed based on the comparison results, and to calculate the depth information of the difference regions through a depth information calculation strategy.

[0114] The information determination module 204 is used to determine the depth information of the region corresponding to the non-differential region in the historical reference image as the depth information of the non-differential region.

[0115] In an optional embodiment, the difference comparison module 202 can be used to: identify the same object in the image to be processed and the historical reference image; identify a first region of the object in the image to be processed and a second region in the historical reference image; perform difference comparison on the first region and the second region to obtain a comparison result;

[0116] The region determination module 203 can be used to: if the comparison result is that the area of ​​the first region is different from the area of ​​the second region and / or the position of the first region is different from the position of the second region, then based on the first region, determine the difference region and the non-difference region from the image to be processed.

[0117] In an optional embodiment, the region determination module 203 can be used to: determine the region within a preset range in the image to be processed, with the center of the first region as the origin, as the difference region, and determine the region in the image to be processed other than the difference region as the non-difference region, wherein the difference region includes the first region.

[0118] In an optional embodiment, the region determination module 203 can be used to: determine the region to be determined from the regions other than the difference regions in the image to be processed, and calculate the depth information of the region to be determined through a depth information calculation strategy; determine the region to be compared with the region to be determined from the historical reference image, and obtain the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region other than the difference regions in the image to be processed is determined as a non-difference region.

[0119] In an optional embodiment, the difference comparison module 202 can be used to: determine a first focus area of ​​the image to be processed and determine a second focus area of ​​the historical reference image; perform a difference comparison on the first focus area and the second focus area to obtain a comparison result;

[0120] The region determination module 203 can be used to: if the comparison result shows that the first focus region and the second focus region are different, then the first focus region is determined as the difference region, and the region in the image to be processed other than the first focus region is determined as the non-difference region.

[0121] In an optional embodiment, the image acquisition module 201 can be used to: determine the number of historical candidate images; if the number meets a preset condition, determine a historical reference image from the historical candidate images.

[0122] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the image processing method provided in this embodiment.

[0123] This application also provides an electronic device, including a memory and a processor, wherein the processor executes the image processing method provided in this embodiment by calling a computer program stored in the memory.

[0124] For example, the aforementioned electronic device could be a mobile terminal such as a tablet or smartphone. See also... Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0125] The electronic device 300 may include components such as a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0126] The processor 301, also known as the central processing unit, is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing applications stored in the memory 302 and calling data stored in the memory 302, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. In an optional embodiment, the processor 301 can also be an application processor, neural network processor, or other processor with computing capabilities.

[0127] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.

[0128] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:

[0129] Acquire the image to be processed and acquire historical reference images;

[0130] The image to be processed and the historical reference image are compared to obtain the comparison results;

[0131] Based on the comparison results, the difference regions and non-difference regions are determined from the image to be processed, and the depth information of the difference regions is calculated using a depth information calculation strategy.

[0132] The depth information of the region corresponding to the non-differential region in the historical reference image is determined as the depth information of the non-differential region.

[0133] Please see Figure 10 The electronic device 300 may include components such as a processor 301, a memory 302, an input unit 303, and an output unit 304.

[0134] The processor 301, also known as the central processing unit, is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing applications stored in the memory 302 and calling data stored in the memory 302, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. In an optional embodiment, the processor 301 can also be an application processor, neural network processor, or other processor with computing capabilities.

[0135] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.

[0136] The input unit 303 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0137] The output unit 304 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The output unit may include a display screen, which may include a display area.

[0138] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:

[0139] Acquire the image to be processed and acquire historical reference images;

[0140] The image to be processed and the historical reference image are compared to obtain the comparison results;

[0141] Based on the comparison results, the difference regions and non-difference regions are determined from the image to be processed, and the depth information of the difference regions is calculated using a depth information calculation strategy.

[0142] The depth information of the region corresponding to the non-differential region in the historical reference image is determined as the depth information of the non-differential region.

[0143] In an optional embodiment, when the processor 301 performs a difference comparison between the image to be processed and a historical reference image and obtains a comparison result, it may perform the following: determine the same object in the image to be processed and the historical reference image; determine a first region of the object in the image to be processed and a second region in the historical reference image; perform a difference comparison on the first region and the second region to obtain a comparison result; when the processor 301 performs the task of determining the difference region and the non-difference region from the image to be processed based on the comparison result, it may perform the following: if the comparison result is that the area of ​​the first region is different from the area of ​​the second region and / or the position of the first region is different from the position of the second region, then determine the difference region and the non-difference region from the image to be processed based on the first region.

[0144] In an optional embodiment, when the processor 301 executes the process of determining the difference region and non-difference region from the image to be processed based on the first region, it may perform the following: determining the region within a preset range in the image to be processed with the center of the first region as the origin as the difference region, and determining the region in the image to be processed other than the difference region as the non-difference region, wherein the difference region includes the first region.

[0145] In an optional embodiment, before the processor 301 determines regions other than the difference regions in the image to be processed as non-difference regions, it may further perform the following: determine the region to be determined from the regions other than the difference regions in the image to be processed, and calculate the depth information of the region to be determined using a depth information calculation strategy; determine the region to be compared corresponding to the region to be determined from a historical reference image, and obtain the depth information of the region to be compared; when the processor 301 determines other regions other than the difference regions in the image to be processed as non-difference regions, it may perform the following: if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the regions other than the difference regions in the image to be processed are determined as non-difference regions.

[0146] In an optional embodiment, when the processor 301 performs a difference comparison between the image to be processed and a historical reference image and obtains a comparison result, it may perform the following: determine a first focus area of ​​the image to be processed and determine a second focus area of ​​the historical reference image; perform a difference comparison between the first focus area and the second focus area to obtain a comparison result; when the processor 301 performs the following: determine the difference area from the image to be processed and determine the non-difference area from the historical reference image based on the comparison result, it may perform the following: if the comparison result shows that the first focus area and the second focus area are different, then determine the first focus area as the difference area and determine the area in the image to be processed other than the first focus area as the non-difference area.

[0147] In an optional embodiment, before the processor 301 performs the acquisition of historical reference images, it may also perform: determining the number of historical candidate images; when the processor 301 performs the acquisition of historical reference images, it may perform: if the number meets a preset condition, then determine the historical reference images from the historical candidate images.

[0148] This application also provides an electronic device, please refer to Figure 11 The electronic device 300 includes a front-end image processor 305 and an application processor 306. Among them,

[0149] The front-end image processor 305 is used to acquire the image to be processed and acquire the historical reference image; perform a difference comparison between the image to be processed and the historical reference image to obtain the comparison result; determine the difference region and the non-difference region from the image to be processed based on the comparison result; and calculate the depth information of the difference region.

[0150] Application processor 306 is used to determine the depth information of the region corresponding to the non-differential region in the historical reference image as the depth information of the non-differential region.

[0151] In an optional embodiment, the front-end image processor 305 may be used to: identify identical objects in the image to be processed and a historical reference image; identify a first region of the object in the image to be processed and a second region in the historical reference image; perform a difference comparison on the first region and the second region to obtain a comparison result. If the comparison result shows that the area of ​​the first region is different from the area of ​​the second region and / or the position of the first region is different from the position of the second region, then based on the first region, the difference region and the non-difference region are determined from the image to be processed.

[0152] In an optional embodiment, the front-end image processor 305 may be used to: determine the region within a preset range with the center of the first region as the origin in the image to be processed as the difference region, and determine the region in the image to be processed other than the difference region as the non-difference region, wherein the difference region includes the first region.

[0153] In an optional embodiment, the front-end image processor 305 may be used to: determine the region to be determined from the region other than the difference region in the image to be processed, and calculate the depth information of the region to be determined through a depth information calculation strategy; determine the region to be compared with the region to be determined from the historical reference image, and obtain the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then the region other than the difference region in the image to be processed is determined as a non-difference region.

[0154] In an optional embodiment, the front image processor 305 may be used to: determine a first focus area of ​​the image to be processed and determine a second focus area of ​​a historical reference image; perform a difference comparison on the first focus area and the second focus area to obtain a comparison result; if the comparison result shows that the first focus area and the second focus area are different, then the first focus area is determined as a difference area, and the area in the image to be processed other than the first focus area is determined as a non-difference area.

[0155] In an optional embodiment, the front image processor 305 may be used to: determine a region to be determined from the region other than the first focus region in the image to be processed, and calculate the depth information of the region to be determined through a depth information calculation strategy; determine a region to be compared with the region to be determined from a historical reference image, and obtain the depth information of the region to be compared; if the depth information of the region to be determined is the same as the depth information of the region to be compared, then determine the region other than the first focus region in the image to be processed as a non-differential region.

[0156] In an optional embodiment, the front-end image processor 305 may be used to: send depth information of the difference region and the image to be processed to the application processor 306;

[0157] The application processor 305 can be used to: perform bokeh processing on the received image to be processed using the depth information of the received difference region and the depth information of the non-difference region to obtain a bokeh image.

[0158] In an alternative embodiment, the application processor 306 may be used to perform subsequent image processing on the blurred image, such as beautification or image stabilization, etc.

[0159] In an optional embodiment, the front-end image processor 305 may be used to: send depth information of the difference region and the image to be processed to the application processor 306;

[0160] The application processor 306 can be used to generate a depth image of the image to be processed based on the depth information of the difference regions and the depth information of the non-difference regions.

[0161] In an optional embodiment, the front-end image processor 305 may be used to: determine the number of historical candidate images; and if the number meets a preset condition, determine a historical reference image from the historical candidate images.

[0162] In an optional embodiment, the front-end image processor 305 may be used to: use the historical candidate image whose shooting time is closest to that of the image to be processed as a historical reference image, or determine any historical candidate image as a historical reference image.

[0163] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the image processing method above, which will not be repeated here.

[0164] The image processing apparatus provided in this application embodiment belongs to the same concept as the image processing method in the above embodiment. Any of the methods provided in the image processing method embodiment can be run on the image processing apparatus. For details of its implementation process, please refer to the image processing method embodiment, which will not be repeated here.

[0165] It should be noted that, for the image processing method of the embodiments of this application, those skilled in the art will understand that all or part of the process of implementing the image processing method of the embodiments of this application can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, it can include the process of the embodiments of the image processing method. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.

[0166] For the image processing apparatus of this application embodiment, its functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0167] The foregoing has provided a detailed description of an image processing method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Acquire the image to be processed and acquire historical reference images; The image to be processed and the historical reference image are compared to obtain the comparison result. Based on the comparison results, the difference regions and non-difference regions are determined from the image to be processed, and the depth information of the difference regions is calculated using a depth information calculation strategy. The depth information of the region in the historical reference image that corresponds to the non-differential region is determined as the depth information of the non-differential region. The step of performing a difference comparison between the image to be processed and the historical reference image to obtain the comparison result includes: A first focus area of ​​the image to be processed is determined, and a second focus area of ​​the historical reference image is determined; The differences between the first focus area and the second focus area are compared to obtain the comparison results; The step of determining the difference regions from the image to be processed and the non-difference regions from the historical reference image based on the comparison results includes: If the comparison result shows that the first focus area and the second focus area are different, then the first focus area is determined as the difference area, and the area in the image to be processed other than the first focus area is determined as the non-difference area.

2. The image processing method according to claim 1, characterized in that, Before acquiring the historical reference image, the process also includes: Determine the number of historical candidate images; The acquisition of historical reference images includes: If the quantity meets the preset conditions, a historical reference image is determined from the historical candidate images.

3. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire the image to be processed and to acquire historical reference images; The difference comparison module is used to perform a difference comparison between the image to be processed and the historical reference image to obtain the comparison result. The region determination module is used to determine the difference regions and non-difference regions from the image to be processed based on the comparison results, and to calculate the depth information of the difference regions through a depth information calculation strategy. An information determination module is used to determine the depth information of the region in the historical reference image that corresponds to the non-differential region as the depth information of the non-differential region. The step of performing a difference comparison between the image to be processed and the historical reference image to obtain the comparison result includes: A first focus area of ​​the image to be processed is determined, and a second focus area of ​​the historical reference image is determined; The differences between the first focus area and the second focus area are compared to obtain the comparison results; The step of determining the difference regions from the image to be processed and the non-difference regions from the historical reference image based on the comparison results includes: If the comparison result shows that the first focus area and the second focus area are different, then the first focus area is determined as the difference area, and the area in the image to be processed other than the first focus area is determined as the non-difference area.

4. An electronic device, characterized in that, include: A front-end image processor is used to acquire the image to be processed and to acquire historical reference images; The image to be processed and the historical reference image are compared to obtain the comparison result; based on the comparison result, the difference region and the non-difference region are determined from the image to be processed. Calculate the depth information of the difference region; The step of performing a difference comparison between the image to be processed and the historical reference image to obtain a comparison result includes: determining a first focus area of ​​the image to be processed and determining a second focus area of ​​the historical reference image; The differences between the first focus area and the second focus area are compared to obtain the comparison results; The step of determining the difference regions from the image to be processed and the non-difference regions from the historical reference image based on the comparison results includes: If the comparison result shows that the first focus area and the second focus area are different, then the first focus area is determined as the difference area, and the area in the image to be processed other than the first focus area is determined as the non-difference area. An application processor is configured to determine the depth information of the region in the historical reference image corresponding to the non-differential region as the depth information of the non-differential region.

5. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program, and the processor executing the image processing method according to any one of claims 1 to 2 by calling the computer program stored in the memory.

6. A storage medium, characterized in that, The storage medium stores a computer program that, when run on a computer, causes the computer to perform the image processing method according to any one of claims 1 to 2.

Images