Image processing method and related device

Abstract

The present application provides an image processing method and related devices, in which the electronic device obtains preview information of the shooting object from the shooting preview screen of the camera application and obtains image information of the shooting object from the image collected by the electronic device. Then the preview information and the image information are judged, and a depth calculation method suitable for the current shooting scene is selected according to the result of the judgment. The collected image is blurred according to the selected depth calculation method, so that the target shooting object is clear and the non-target shooting object is blurred in the image after blurring compared with the image before blurring. Therefore, by selecting the corresponding depth calculation method based on different shooting scenes, high-precision depth information can be obtained, thereby optimizing the blurring effect of the image.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method and related devices. Background Art

[0002] When shooting, the use of large aperture technology can achieve clear imaging of objects within a certain object distance range from the camera, while blurring the imaging of objects outside the object distance range, thereby achieving the effect of highlighting the subject and blurring other areas outside the subject in the image.

[0003] With the development of electronic devices such as mobile phones, users are accustomed to using mobile phones to capture images. Since the cameras of electronic devices such as mobile phones are limited by the aperture size of the camera, it may not be possible to directly capture large aperture images. Therefore, after the electronic device acquires the image, it can first determine the depth information of the image, and then process the image according to the depth information of the image, so that there are clear areas and blurred areas in the image, thereby simulating the blur effect of a large aperture.

[0004] However, when the image contains more texture information, the determined depth information may be inaccurate, resulting in poor blurring effect of the image processed based on the depth information, failing to achieve the blurring effect of a large aperture, and affecting the user experience. Summary of the invention

[0005] An image processing method and related device provided in an embodiment of the present application select a corresponding depth calculation method based on preview information and image information of a shooting scene, which can improve the accuracy of depth information and optimize the blurring effect of the image.

[0006] In a first aspect, the present application provides an image processing method, the method comprising:

[0007] In the blur shooting mode, displaying a first user interface, wherein the first user interface is used to display a shooting image, and the shooting image includes one or more shooting objects;

[0008] In response to a first user operation, the preview information of the first user interface and the image information of the captured image are acquired, and the captured image is blurred according to a depth calculation method to obtain a target image, wherein the depth calculation method is one selected from a calculation method based on a depth sensor and a calculation method based on an image processing algorithm after judging the preview information and / or the image information;

[0009] The target image is displayed on a second user interface, wherein a target photographic object in the target image is clear, a non-target photographic object in the target image is unclear, the target photographic object is any one or more of the one or more photographic objects, and the non-target photographic object is other than the target photographic object in the one or more photographic objects.

[0010] In the above method, in the blur mode, the electronic device can choose to use a calculation method based on a depth sensor to calculate the depth information of the image, or use a calculation method based on an image processing algorithm to calculate the depth information. Because in different shooting scenes, the accuracy of the depth information obtained by the electronic device based on different depth calculation methods may be different. Therefore, in order to enable the electronic device to obtain high-precision depth information in all shooting scenes, the electronic device can select a corresponding depth calculation method based on the shooting scene to obtain high-precision depth information. Since the preview information and image information of the image can reflect the shooting scene, the preview information and / or the image information are judged respectively, and then the depth calculation method selected according to the judgment result is more in line with the actual shooting scene. Compared with the existing solution that only calculates the depth information by a depth sensor or only by an image processing algorithm, it can meet the needs of different shooting scenes, so that images with better blurring effects can be displayed in different shooting scenes.

[0011] In a possible implementation of the first aspect, after acquiring the preview information of the first user interface and the image information of the captured image, and before blurring the captured image according to the depth calculation method to obtain the target image, the method further includes:

[0012] In a case where it is determined according to the brightness information in the preview information that the shooting scene is a highlight scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the shooting scene is a shooting scene corresponding to the electronic device displaying the first user interface, and the highlight scene is used to indicate that the intensity of the ambient light of the shooting scene is greater than or equal to the brightness threshold;

[0013] In a case where it is determined according to the brightness information in the preview information that the shooting scene is not the highlight scene, and when it is determined according to the relative speed in the preview information that the shooting scene is a high-speed motion scene, the depth calculation method is the calculation method based on the depth sensor, wherein the relative speed is the relative speed between the electronic device and the shooting object, and the high-speed motion scene is used to indicate that the relative speed is greater than a speed threshold; or,

[0014] When it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information.

[0015] In the above method, since the depth sensor is an optical device, the intensity of the ambient light will affect the accuracy of the depth information determined by the depth sensor. Therefore, the embodiment of the present application first determines whether the current shooting scene is a highlight scene. If it is a highlight scene, the depth calculation method is further selected according to the image information. If the shooting scene is not a highlight scene and the shooting scene is a high-speed motion scene, then the calculation method based on the depth sensor is selected as the depth calculation method. On the one hand, it can avoid the influence of ambient light on the depth sensor. On the other hand, it can solve the problem of low accuracy of the depth information obtained by the electronic device through the image processing algorithm in the high-speed motion scene, thereby obtaining more accurate depth information.

[0016] In a possible implementation of the first aspect, when it is determined according to brightness information in the preview information that the captured scene is a highlight scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information includes:

[0017] In a case where it is determined according to the brightness information in the preview information that the shooting scene is the highlighted scene, and when it is determined according to the scene distance detection result in the image information that the shooting scene is not a distant view, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the distant view is used to indicate that the distance between the shooting object in the shooting scene and the electronic device is greater than a distance threshold; or

[0018] When it is determined according to the scene distance detection result in the image information that the shooting scene is the distant view, the electronic device does not perform blurring processing on the shot image.

[0019] In the above method, when the shooting scene is in a high-brightness scene, the electronic device may perform image acquisition outdoors. Since the distance between the shooting object and the electronic device may be far away outdoors, that is, the scene distance detection result corresponding to the shooting scene at this time is a distant view. When the shooting scene of the image is a distant view, it means that most areas in the image are blurred, and the electronic device does not need to repeatedly blur the blurred image. Therefore, when the shooting scene is a high-brightness scene and the scene distance detection result is a distant view, the electronic device can not blur the image to save energy of the electronic device.

[0020] In a possible implementation of the first aspect, when it is determined according to the scene distance detection result in the image information that the captured scene is not a distant view, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information includes:

[0021] When it is determined according to the scene distance detection result in the image information that the captured scene is not the distant view, and when it is determined according to the clarity detection result in the image information that the captured image is not a blurred image, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the blurred image is used to indicate that the clarity of the captured image is less than or equal to a clarity threshold; or

[0022] When it is determined that the captured image is the blurred image according to the clarity detection result in the image information, the electronic device does not perform blurring processing on the captured image.

[0023] In the above method, when the shooting scene is a high-brightness scene, the camera of the electronic device may be affected by the ambient light and cause blurred imaging. Therefore, when the shooting scene is a high-brightness scene, the scene distance detection result is not a distant view, and the clarity detection result is a blurred image, the electronic device does not need to repeatedly blur the blurred image, thereby saving energy of the electronic device.

[0024] In a possible implementation of the first aspect, when it is determined according to a clarity detection result in the image information that the captured image is not a blurred image, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information includes:

[0025] In the case where it is determined according to the clarity detection result in the image information that the captured image is not the blurred image, and when it is determined according to the texture detection result in the image information that the captured image contains complex texture, the depth calculation method is the calculation method based on the depth sensor, wherein the captured image contains complex texture to indicate that the type and / or amount of texture in the captured image is greater than a texture threshold; or,

[0026] When it is determined according to the texture detection result in the image information that the captured image does not contain the complex texture, the depth calculation method is the calculation method based on the image processing algorithm.

[0027] In the above method, since the accuracy of the depth information obtained by the electronic device through the image processing algorithm is low in the scene where the image contains complex textures, more accurate depth information can be obtained through the calculation method based on the depth sensor. However, compared with the method based on the image processing algorithm, the calculation method based on the depth sensor may have the problem of high energy consumption and long time consumption. Therefore, in the scene where the image does not contain complex textures, the electronic device can choose the calculation method based on image processing to reduce energy consumption.

[0028] In a possible implementation of the first aspect, when it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information includes:

[0029] In a case where it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, when it is determined according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the shooting subject, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information, wherein the shooting subject is a shooting subject determined by the electronic device from the one or more shooting subjects in response to the second user operation, and the shooting subject is one or more of the target shooting subjects; or

[0030] When it is judged according to the shooting subject recognition result in the preview information that the one or more shooting objects include a human body and / or a human face, the depth calculation method is the calculation method based on the depth sensor; when it is judged according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the human body and the human face, the depth calculation method is the calculation method based on the image processing algorithm.

[0031] In the above method, since the human body or face in the image includes fine textures such as hair, it is necessary to determine the depth value corresponding to the fine texture by a high-precision depth calculation method (for example, a calculation method based on a depth sensor) so that the fine texture can obtain a better blur effect. And since the shooting subject is determined based on the second user operation, when the shooting subject is a human body or a face, the user usually hopes to highlight the human body or the face, so at this time the user's requirements for the blur effect of the image are also high, and the electronic device needs to determine the depth information of the image by a high-precision depth calculation method (for example, a calculation method based on a depth sensor), so that the image after blur processing has a better blur effect. When the shooting subject is not a human body or a face, since the shooting object that the electronic device needs to highlight does not include a human body or a face, the electronic device can output a high-quality blurred image even if it does not select a high-precision depth calculation method. And considering the energy consumption problem, the electronic device chooses to reduce the energy consumption generated during the blur processing by using a calculation method based on an image processing algorithm. Therefore, if the subject of the shooting is not a human body or a face, in the corresponding scene, the electronic device determines that the current shooting scene meets the conditions of the image processing algorithm.

[0032] In a possible implementation of the first aspect, when it is determined according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the shooting subject, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information includes:

[0033] In a case where it is determined according to the shooting subject recognition result in the preview information that the shooting subject is not included in the one or more shooting objects, and in a case where it is determined according to the scene subject detection result in the preview information that the one or more shooting objects do not include the scene subject, the depth calculation method is the calculation method based on the image processing algorithm; or,

[0034] When it is determined according to the scene subject detection result in the preview information that the one or more photographed objects include the scene subject, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information.

[0035] In the above method, when there is no scene subject in the shooting scene, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is relatively high. Considering the energy consumption problem, the electronic device chooses to use the calculation method based on the image processing algorithm to reduce the energy consumption generated during the blurring process.

[0036] In a possible implementation of the first aspect, when it is judged according to the scene subject detection result in the preview information that the one or more photographed objects include the scene subject, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information includes:

[0037] In a case where it is determined according to the scene subject detection result in the preview information that the one or more photographed objects include the scene subject, and when it is determined according to the number of faces in the preview information that the number of faces is greater than a number threshold, the depth calculation method is the calculation method based on the image processing algorithm; or,

[0038] When it is determined according to the number of faces in the preview information that the number of faces is less than or equal to the number threshold, the depth calculation method is the calculation method based on the depth sensor.

[0039] In the above method, when the number of faces in the image is greater than the number threshold, it means that the user needs to take pictures of multiple faces. Then, for the user, the area corresponding to each face in the image needs to be highlighted, which further indicates that the user has a low requirement for the degree of blurring of the image. Therefore, the electronic device can output a blurred image that meets the user's requirements by blurring the image through a calculation method based on an image processing algorithm or a calculation method based on a depth sensor. Taking into account the energy consumption issue, the electronic device can reduce the energy consumption generated during the blurring process by choosing a calculation method based on an image processing algorithm. When the number of faces in the image is less than or equal to the number threshold, the electronic device needs to select a higher-precision depth calculation method to better highlight the area corresponding to the face in the image.

[0040] In a possible implementation of the first aspect, after acquiring the preview information of the first user interface and the image information of the captured image, and before blurring the captured image according to the depth calculation method to obtain the target image, the method further includes:

[0041] When the depth calculation method is the calculation method based on the image processing algorithm, obtaining the temperature of the electronic device;

[0042] When the temperature is less than or equal to the first temperature threshold and greater than the second temperature threshold, determining that the calculation method based on the image processing algorithm is a calculation method based on the first image processing algorithm;

[0043] When the temperature is less than or equal to the second temperature threshold, the calculation method based on the image processing algorithm is determined to be a calculation method based on a second image processing algorithm.

[0044] In the above method, since the electronic device generates heat in the process of running the image processing algorithm, the generated heat will cause the temperature of the electronic device to rise. The electronic device will also generate energy consumption in the process of running the image processing algorithm, and the higher the energy consumption, the more heat will be generated. Therefore, in order to control the temperature of the electronic device and reduce the heat generated by the electronic device in the process of blurring the image as much as possible, the electronic device can select a method with low energy consumption to determine the depth information on the basis of ensuring the accuracy of the depth information. Usually, the energy consumption corresponding to the monocular depth estimation algorithm is different from the energy consumption corresponding to the binocular depth estimation algorithm. Therefore, the electronic device can also determine whether to use a monocular depth estimation algorithm or a binocular depth estimation algorithm based on the second temperature threshold. When the temperature of the electronic device is less than the second temperature threshold, it means that the temperature of the electronic device is low at this time. Even if the electronic device uses an image processing algorithm with high energy consumption to determine the depth information, the heat generated is not enough to cause the temperature of the electronic device to increase to a larger value, for example, it is not enough to make the temperature of the electronic device greater than the first temperature threshold and cannot be blurred. Since the accuracy of depth information obtained by an image processing algorithm with higher energy consumption may be higher than that of an image processing algorithm with lower energy consumption, when the temperature of the electronic device is less than the second temperature threshold, the electronic device selects an image processing algorithm with higher energy consumption to determine the depth information to improve the accuracy of the depth information. For example, the electronic device determines the depth information of the image through the second image processing algorithm. When the temperature of the electronic device is greater than or equal to the second temperature threshold and less than or equal to the first temperature threshold, it means that the temperature of the electronic device is close to the first temperature threshold at this time, and the electronic device needs to control the temperature of the electronic device to minimize the heat generated by the electronic device during the blurring process of the image. Therefore, the electronic device can select an image processing algorithm with lower energy consumption. For example, the electronic device determines the depth information of the image through the first image processing algorithm.

[0045] In a possible implementation of the first aspect, after the target image is displayed on the second user interface, the method further includes:

[0046] The first temperature threshold and the second temperature threshold corresponding to the N+1th blurred image are determined according to the power consumed by the electronic device when blurring the image for the Nth time and the temperature when the captured image is acquired for the Nth time, wherein N is an integer greater than 1, and the power consumed by the Nth blurred image includes the power consumed in the process from the time the electronic device starts to obtain the preview information for the Nth time to the time the electronic device displays the target image for the Nth time.

[0047] In the above method, since the temperature of the electronic device will gradually rise during the image acquisition process, and if the electronic device acquires images multiple times, the depth calculation methods used by the electronic device for the multiple acquired images may be different, and different depth calculation methods may consume different amounts of power. Different amounts of power consumed result in different amounts of heat generated, and different amounts of heat generated may cause the temperature of the electronic device to change unpredictably. Therefore, in order to control the increase in the temperature of the electronic device, the electronic device can determine the first temperature threshold and the second temperature threshold corresponding to the N+1th blurred image after blurring the image for the Nth time, based on the amount of power consumed by the Nth blurred image and the temperature when the image was acquired for the Nth time.

[0048] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0049] In a third aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0050] In a possible implementation, the chip or chip system described above in the embodiment of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0051] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a communication device, enables the communication device to execute an image processing method as described in the first aspect or any possible implementation of the first aspect.

[0053] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The following is an introduction to the drawings used in the embodiments of the present application.

[0055] Figure 1A is a scene graph of a captured image provided by an embodiment of the present application;

[0056] Figure 1B It is an image obtained by processing based on a monocular depth estimation algorithm provided in an embodiment of the present application;

[0057] Figure 2A is another scene graph of captured images provided in an embodiment of the present application;

[0058] Figure 2B It is an image obtained by processing based on a binocular depth estimation algorithm provided in an embodiment of the present application;

[0059] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0060] Figure 4 is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application;

[0061] Figure 5A It is a system desktop of an electronic device provided in an embodiment of the present application;

[0062] Figure 5B-5E A user interface of a group of camera applications provided in an embodiment of the present application;

[0063] Fig. 5F It is a user interface of a gallery application provided in an embodiment of the present application;

[0064] Fig. 6A It is a system desktop of an electronic device provided in an embodiment of the present application;

[0065] FIG. 6B to FIG. 6C is another set of camera application user interfaces provided by an embodiment of the present application;

[0066] Fig.6D It is another user interface of a gallery application provided in an embodiment of the present application;

[0067] Figure 7 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0068] Fig. 8A is a schematic diagram of preview information provided by an embodiment of the present application;

[0069] Figure 8B is a schematic diagram of image information provided by an embodiment of the present application;

[0070] Fig. 9 This is a flow chart of a scene judgment condition provided by an embodiment of the present application;

[0071] Fig.10 It is a flowchart of another method for selecting a depth calculation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0073] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0074] In order to facilitate the understanding of the embodiments of the present application, the technical problems that the present application specifically aims to solve are analyzed and proposed below.

[0075] After acquiring an image, in order to achieve a large aperture blur effect, the electronic device can first determine the depth information of the image, and then blur the image according to the depth information of the image, so that some areas in the image are clear and some areas are blurred, thereby simulating the blur effect of a large aperture lens. Among them, the depth information is used to characterize the distance of the photographed object from the camera in three-dimensional space. In an embodiment of the present application, the electronic device can blur the pixels in the image whose depth information is within a specific range to achieve a visual effect of blurring some areas in the image.

[0076] In one implementation, the electronic device typically only uses an image processing algorithm (such as a monocular depth estimation algorithm) to estimate the depth information of the image.

[0077] Exemplarily, a monocular depth estimation algorithm refers to a method for calculating the depth information of an object in an image based on an image captured by a single camera. It is mainly based on an existing depth information database or a trained depth prediction model, and performs semantic segmentation, motion analysis, texture analysis, etc. on the image to infer the depth information of the object in the image.

[0078] However, the monocular depth estimation algorithm is equivalent to inferring the corresponding three-dimensional space from a two-dimensional image. In particular, when the texture information in the image is complex, the accuracy of the depth information obtained by the electronic device based on the monocular depth estimation algorithm is low. Complex texture information can be understood as a large number of types and / or quantities of textures in the image, fine textures in the image, similar textures in the foreground and background of the image, etc.

[0079] It is understandable that when there are more types of textures in an image, it means that there are more types of objects; when there are more textures in an image, it means that the area of ​​the object is higher in the total area of ​​the image. Therefore, when there are more types and / or more textures in an image, the electronic device needs to infer the depth information corresponding to more pixels in the image based on the monocular depth estimation algorithm. Therefore, if the number of pixels to be inferred is large, it is difficult to infer high-precision depth information based only on two-dimensional images.

[0080] Among them, fine textures include but are not limited to the textures corresponding to fine objects such as hair in a portrait and twigs of a plant. Since the area of ​​fine textures in the image is also relatively small, it is difficult to infer high-precision depth information corresponding to fine textures based only on two-dimensional images.

[0081] For example, see Figure 1A and Figure 1B , Figure 1A is a scene graph of a captured image provided by an embodiment of the present application, Figure 1BThis is an image obtained by processing based on a monocular depth estimation algorithm provided in an embodiment of the present application.

[0082] like Figure 1A As shown, the electronic device 100 shoots an image containing a human body 108 and a wall 109 in a camera application. In the preview interface 111 of the electronic device 100, the electronic device 100 displays a shooting preview screen 101, a function selection area 102, a mode selection area 103, a zoom function 104, a gallery 105, a shooting button 106, and a switch button 107. Among them, the function selection area 102 includes at least one function setting button. If the electronic device 100 detects that the user touches any function setting button in the function selection area 102, the corresponding function is set according to the touched function setting button. For example, if it is detected that the user touches the flash button, the flash is set. The mode selection area 103 includes at least one shooting mode button. If the electronic device 100 detects that the user touches any shooting mode button in the mode selection area 103, the corresponding shooting mode is switched according to the touched shooting mode button. The gallery 105 is used to review the captured pictures. Specifically, the gallery 105 is used to provide the user with an entrance to review the captured pictures. The user clicks the gallery 105 to enter the interface for reviewing the captured pictures. The shooting button 106 is used to take pictures, and the switching button 107 is used to switch between the front and rear cameras during shooting.

[0083] like Figure 1AAs shown, in the preview interface 111 of the electronic device 100, the shooting mode of the electronic device 100 is in the aperture mode in the mode selection area 103. The electronic device 100 can perform blur processing on the captured image based on the image processing algorithm to achieve the blur effect of a large aperture lens. Among them, a shooting preview screen 101 is displayed in the preview interface 111, and a human body 108 and a wall 109 are displayed in the shooting preview screen 101. The human body 108 wears a hat 1081 with a striped texture on his head. Since the wall 109 contains a large number of rectangular textures, and the texture of the wall 109 is similar to the texture of the hat 1081, the image corresponding to the shooting preview screen 101 belongs to an image containing complex texture information. It can be understood that if the human body 108 is the shooting subject, then the picture behind the shooting subject (human body 108) in the shooting preview screen 101 can be called the background, that is, the background includes the wall 109. When the electronic device 100 performs blur processing on the image corresponding to the shooting preview screen 101, when the relative distance between the wall 109 and the hat 1081 is estimated based on the monocular depth estimation algorithm, the hat 1081 and the wall 109 may be regarded as objects on the same plane, that is, the hat 1081 and the wall 109 may be determined as the background, resulting in inaccurate depth information corresponding to the hat 1081. However, the electronic device still performs blur processing on the image corresponding to the shooting preview screen 101 based on the inaccurate depth information, thereby obtaining Figure 1B Image 1121 shown.

[0084] like Figure 1B As shown, the electronic device 100 displays a gallery interface 112, and the gallery interface 112 includes an image 1121 and a status bar 1122. Among them, the image 1121 is an image after the electronic device 100 performs blurring processing according to the depth information corresponding to the shooting preview screen 101. It can be seen that compared with the shooting preview screen 101, the rest of the human body 108 except the hat 1081 in the image 1121 is clearly displayed, and the hat 1081 and the wall 109 are blurred. Since the hat 1081 belongs to the human body 108, the hat 1081 belongs to the foreground and needs to be clearly displayed, but the electronic device 100 blurs the hat 1081 according to the wrong depth information, making the area that needs to be clearly displayed blurred. Therefore, when facing an image in a complex texture scene, it is difficult for the electronic device 100 to determine the depth information with higher accuracy through the monocular depth estimation algorithm, resulting in a poor blurring effect on the image.

[0085] In one implementation, the electronic device usually only uses an image processing algorithm (such as a binocular depth estimation algorithm) to estimate the depth information of the image.

[0086] Exemplarily, the binocular depth estimation algorithm refers to a method of simultaneously acquiring two images of the same scene from different perspectives through dual cameras (of a binocular camera), and calculating the depth information of each object in the image using the principle of binocular stereo vision. The binocular depth estimation algorithm is mainly based on the principle of triangulation, and the specific process may include: first, two frames of images are obtained by simultaneously photographing the object through two cameras. Then, based on the two frames of images, the pixel offset or parallax of the same object photographed by the two cameras is measured. Finally, through the geometric relationship and calibration parameters of the two cameras, the distance of the object relative to the camera, that is, the depth information, can be calculated.

[0087] Among them, parallax refers to the difference in position of the object in the two images when two cameras take pictures of the same object. The binocular depth estimation algorithm needs to perform stereo matching on the two frames of images taken by the two cameras to determine the position difference (i.e., parallax) of the object in the two frames. However, in a motion scene (a scene where there is a relative speed between the camera and the object), since there is a relative speed between the camera and the object, the shapes of the object in the two images may be different. Stereo matching of the two frames of images based on objects of different shapes will cause stereo matching errors, resulting in inaccurate parallax, and then inaccurate depth information determined based on the parallax.

[0088] For example, see Figure 2A and Figure 2B , Figure 2A is another scene graph for capturing an image provided in an embodiment of the present application. Figure 2B This is an image obtained by processing based on a binocular depth estimation algorithm provided in an embodiment of the present application.

[0089] like Figure 2A As shown, the electronic device 100 captures an image containing a human body 202 and a plant 203 in a camera application. In the preview interface 206 of the electronic device 100, the electronic device 100 displays a capture preview screen 201, a function selection area 102, a mode selection area 103, a zoom function 104, a gallery 105, a capture button 106, and a switch button 107. If the electronic device 100 detects that the user touches the aperture mode button in the mode selection area 103, it switches to the capture mode corresponding to the aperture mode button, that is, switches the capture mode of the camera application to the aperture mode.

[0090] like Figure 2AAs shown, in the preview interface 206 of the electronic device 100, the shooting mode of the electronic device 100 is in the aperture mode in the mode selection area 103. The electronic device 100 can blur the captured image through an image processing algorithm to achieve the blur effect of a large aperture lens. In the preview interface 206, a shooting preview screen 201 is displayed, and a human body 202 and a plant 203 are displayed in the shooting preview screen 201. Figure 2A In the scene shown, if the electronic device 100 simultaneously acquires the first image and the second image of different perspectives in this scene through the dual cameras (of the binocular camera), since the human body is moving forward and the movement speed is fast, the shape of the human body acquired by the first image may be different from the shape of the human body acquired by the second image. For example, if the human body moves from position one to position two, the human body acquired by the first image may correspond to the human body 204, and the human body acquired by the second image may correspond to the human body 202. It can be seen that the arm 2041 of the human body 204 is perpendicular to the ground. Since the human body is in motion, when the human body moves to position two, the arm 2021 of the human body 202 is parallel to the ground. Therefore, in the process of blurring the acquired image (for example, the image corresponding to the shooting preview screen 201) by the electronic device 100, when the electronic device 100 calculates the depth information through the binocular depth estimation algorithm, the electronic device needs to perform stereo matching on the acquired first image and the second image to determine the parallax of the human body in the first image and the second image respectively. However, since the shapes of the human body 204 in the first image and the human body 202 in the second image are different (the positions of the arms 2041 and 2021 relative to the human body are different), the stereo matching may be wrong, which may result in the inability to determine the parallax, and further the inability to determine the depth information. Therefore, when the electronic device 100 cannot determine the depth information corresponding to the shooting preview screen 201, the electronic device 100 cannot perform blur processing on the image corresponding to the shooting preview screen 201, and thus, the image stored in the gallery by the electronic device 100 is an image that has not been blurred, for example Figure 2B Image 2051 shown.

[0091] like Figure 2B As shown, the electronic device 100 displays a gallery interface 205, and the gallery interface 205 includes an image 2051 and a status bar 2052. Image 2051 is an image corresponding to the preview screen 201. It can be seen that compared with the preview screen 201, the image 2051 has the same clarity, the human body 202 and the plant 203 are both clear, and the image 2051 does not have the blur effect of a large aperture. Therefore, in a high-speed motion scene (a scene where there is a relative speed between the electronic device and the subject), it may be difficult for the electronic device 100 to determine the depth information through the binocular depth estimation algorithm, resulting in the image not having a blur effect.

[0092] In summary, in complex scenes (e.g., scenes with complex image textures or motion scenes), since image processing algorithms (e.g., monocular depth estimation algorithms or binocular depth estimation algorithms) are based on image processing technology or deep learning models to infer image depth information, the depth information of the image determined by the electronic device based on the image processing algorithm is inaccurate. When an electronic device blurs an image based on incorrect depth information, there may be a problem of false blur (e.g., the area that needs to be displayed clearly is blurred) or a problem of missed blur (e.g., the area that needs to be displayed blurred is clear), resulting in a poor blur effect, failing to achieve the blur effect of a large aperture, and affecting the user's photo-taking experience.

[0093] In order to improve the accuracy of the depth information of the image, in another implementation, the electronic device can also obtain the depth information through a depth sensor. Generally, the depth sensor is an optical sensor, and the electronic device measures the time required for the light signal of the depth sensor to be emitted and received to infer the distance information of the photographed object, thereby determining the depth information.

[0094] Specifically, the light source emitted by the transmitter of the depth sensor is reflected by the subject and then received by the receiver. By recording the time from the emission to the reception of the light source, combined with the speed of light, the distance the light source travels in space can be calculated. By taking multiple measurements of the entire shooting scene, the depth information of the image can be obtained. It can be seen that the depth sensor relies on light signals to determine the depth information, so the ambient light during shooting has a greater impact on the depth sensor. If the scene is outdoor with high light or strong backlight, it will affect the process of the depth sensor transmitting and receiving light signals to the subject, thereby affecting the accuracy of the depth information.

[0095] Therefore, although electronic devices can improve the accuracy of depth information through depth sensors, depth sensors are easily affected by ambient light. For example, the depth information obtained in outdoor high-light or strong backlight scenes is less accurate than the depth information determined by the image processing algorithm. Therefore, the method of determining depth information based on the image processing algorithm alone and the method of obtaining depth information through the depth sensor alone are both limited in usage scenarios. Therefore, how to select the corresponding depth calculation method (such as a calculation method based on an image processing algorithm or a calculation method based on a depth sensor) in combination with the shooting scene is a problem that needs to be solved.

[0096] In addition, when the time it takes for an electronic device to capture an image exceeds a time threshold, the temperature of the electronic device will rise. When the temperature reaches the temperature threshold, a photo escape will be triggered, that is, the electronic device will no longer call the algorithm to process the image after taking the image. For example, if the temperature of the electronic device is greater than the temperature threshold, the electronic device will not blur the captured image, but will directly output the captured image, which will affect the imaging effect of the image and will also affect the user's photo-taking experience to a certain extent. Therefore, in the process of the electronic device determining the depth information, how to control the temperature of the shell is also a problem that needs to be solved.

[0097] In order to solve the above problems, an embodiment of the present application provides an image processing method. In the blurred shooting mode, the electronic device obtains the preview information of the shooting object from the shooting preview screen of the camera application and the image information of the shooting object from the image captured by the electronic device in response to the user operation. Then, based on the judgment condition, the depth calculation method suitable for the current shooting scene is judged according to the preview information and the image information, and the depth information of the image is determined by the corresponding depth calculation method. Among them, the depth calculation method includes but is not limited to the calculation method based on the image processing algorithm, the calculation method based on the depth sensor, etc.

[0098] In addition, when selecting a depth calculation method, electronic devices can use the energy consumption generated by the process of determining the depth information as a judgment criterion in addition to using the accuracy of the depth information as a judgment criterion. Since the calculation method based on the depth sensor is longer and consumes more energy than the image processing method, and the calculation method based on the image processing algorithm is applicable to most shooting scenes (inapplicable shooting scenes include: shooting scenes that meet the depth sensor conditions). Therefore, when the scene information of the shooting object does not meet the depth sensor conditions, the electronic device determines the depth information of the image through a calculation method based on the image processing algorithm, which can reduce the energy consumption generated by the process of determining the depth information, thereby reducing the increase in the heat generation of the electronic device and improving the user's shooting experience.

[0099] The image processing method of the embodiment of the present application is described below in combination with the hardware structure and software structure of the electronic device.

[0100] The embodiments of the present application can be applied to any electronic device with an image capture function to execute the image processing method. The electronic device can be, for example, a mobile phone, a mobile computer, a tablet computer, a Polaroid camera, a personal digital assistant (PDA), a media player, a smart TV, a smart wearable device (such as a smart watch, smart glasses, and a smart bracelet), an e-reader, a handheld game console, a point of sales (POS), an in-vehicle electronic device (in-vehicle computer), etc. The form of the electronic device is not specifically limited in the embodiments of the present application.

[0101] In addition, the image processing method proposed in the embodiment of the present application is applied to any image capturing scenario, such as taking photos, animated images or videos.

[0102] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 3 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a depth sensor 180M, a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, etc.

[0103] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0104] The processor 110 can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0105] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0106] In some embodiments, the execution of the image processing method in the embodiment of the present application can be controlled by the processor 110 or completed by calling other components, such as calling the processing program of the embodiment of the present application stored in the internal memory 121, or calling the processing program of the embodiment of the present application stored in a third-party device through the external memory interface 120 to achieve post-blurring processing of the image.

[0107] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and a subscriber identity module (SIM) interface.

[0108] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160.

[0109] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like.

[0110] In some embodiments, the processor 110 and the display screen 194 communicate via a DSI interface to implement the display function of the electronic device 100 .

[0111] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0112] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0113] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0114] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0115] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0116] In some embodiments, the internal memory 121 may store an image processing algorithm, where the image processing algorithm is used to determine depth information of an image, and the image processing algorithm includes but is not limited to a monocular depth estimation algorithm and a binocular depth estimation algorithm.

[0117] In some embodiments, the internal memory 121 may also store an image obtained after blurring. Exemplarily, the image after blurring and the original image (i.e., the image before blurring) may be stored correspondingly. For example, after the electronic device 100 detects an instruction for opening the original image, it displays the original image, and a mark may be displayed on the original image. When the mark is triggered, the electronic device 100 opens the blurred image (the image after blurring the original image); or, after the electronic device 100 detects an instruction for opening the blurred image, it displays the blurred image, and a mark is displayed on the blurred image. When the mark is triggered, the electronic device 100 opens the original image (the original image corresponding to the blurred image).

[0118] The depth sensor 180M is used to determine the depth information of the image. The types of the depth sensor 180M include, but are not limited to, infrared sensors, laser sensors, time-of-flight (TOF) sensors, structured light sensors, etc. TOF sensors include direct time-of-flight laser radar (DTOF) and indirect time-of-flight laser radar (ITOF). The number and type of the depth sensor 180M are not limited here.

[0119] For example, the following will take the TOF sensor as an example to introduce the process of the electronic device 100 determining the depth information of the image through a calculation method based on the depth sensor. The electronic device 100 continuously sends light pulses to the photographed object (for example, each object in the photographed scene) through the TOF sensor, and then receives the light returned from the photographed object through the TOF sensor. The electronic device 100 can obtain the distance between the electronic device 100 and the photographed object based on the round-trip time of the light (the time from emitting a light pulse to the photographed object to receiving the light returned from the photographed object), and then determine the depth information of the image based on the distance between the electronic device 100 and the photographed object.

[0120] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0121] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0122] The camera 193 can be used to capture still images or videos. The object is projected onto a photosensitive element through a lens to generate an optical image. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, then converts the electrical signal into a digital image signal, and then converts the digital image signal into an image signal in a standard format.

[0123] In an embodiment of the present application, the electronic device 100 may include multiple cameras 193, for example, a first camera and a second camera for realizing a binocular camera. The types of cameras 193 include a main camera, a wide-angle camera, a telephoto camera, etc. The present application does not impose any specific restrictions on the number and types of cameras.

[0124] The software system of an electronic device (such as a mobile phone) can adopt a layered architecture, a transaction-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. This application embodiment takes the Android system with a layered architecture as an example to illustrate the software architecture of a mobile phone. Figure 4 , Figure 4 It is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application.

[0125] like Figure 4 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer (application), the application framework layer (frame), the hardware abstraction layer (HAL), the driver layer, and the hardware layer. Among them:

[0126] The application layer (application) can include a series of program packages. For example, the application can include camera, gallery and other applications. Among them, the camera application may include but is not limited to: UI module, camera module, gallery module, etc. The UI module can be a cameraUI module, which is mainly responsible for the human-computer interaction of the camera application, such as controlling the display of the preview interface and the preview screen therein, receiving and responding to user operations occurring in the preview interface. The camera module provides camera functions, focus functions, etc. The gallery module can be used to store photos taken by the user in the file system or a specific database of the electronic device for retrieval by applications such as the gallery.

[0127] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. It mainly involves the camera framework, which can include camera access interfaces such as camera extension libraries and camera services, as well as depth sensor access interfaces. It plays a role of connecting the upper and lower parts. It can interact with camera applications through application APIs and can also interact with HAL through HAL interface definition language (HIDL). The application framework layer can also include a window manager. With the support of the window manager, the camera application and the gallery application can display the photos taken to the user.

[0128] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer, a depth sensor hardware abstraction layer, and a virtualization module. Among them, the camera hardware abstraction layer may provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera), and more camera devices. The depth sensor hardware abstraction layer may provide virtual hardware for depth sensor 1, depth sensor 2, and more depth sensors.

[0129] A variety of image processing algorithms are stored in the blur module. For example, in an embodiment of the present application, the blur module may include a monocular depth estimation algorithm, a binocular depth estimation algorithm, and the like. Among them, the blur module can determine the depth calculation method in combination with the shooting parameters reported by the camera module and / or other sensors. Then, the depth information of the image is determined by the depth calculation method, and the image is blurred based on the obtained depth information. For example, the depth calculation method includes a monocular depth estimation algorithm, a binocular depth estimation algorithm, a calculation method based on a depth sensor, and the like.

[0130] In one implementation, the blur module determines the preview information and image information of the image based on the shooting parameters reported by the camera module and / or other sensors, and then determines the depth calculation method suitable for the current shooting scene based on the preview information and image information based on the judgment condition. For example, the preview information and image information are used to determine whether the shooting scene meets the depth sensor condition. If the depth sensor condition is met, the blur module determines the depth information of the image through the depth sensor. If the depth sensor condition is not met, the blur module determines the depth information of the image through an image processing algorithm (monocular depth estimation algorithm or binocular depth estimation algorithm).

[0131] Among them, the depth sensor conditions include one or more of the following: the shooting scene is in a high-speed motion scene (for example, a scene in which the relative speed between the shooting object and the electronic device is greater than a speed threshold), a scene in which the image contains complex textures (for example, a scene in which the type and / or amount of texture information in the image is greater than a texture threshold), a scene corresponding to when the shooting object (selected by the user through focusing) is a human body and / or a face, and a scene corresponding to when the user does not select a shooting subject and the number of faces in the image is less than or equal to a quantity threshold.

[0132] The driver layer is a layer between hardware and software, which includes drivers for various hardware. The driver layer may include camera device drivers, depth sensor drivers, optical signal processor drivers, digital signal processor drivers, and image processor drivers. Among them, the camera device driver is used to drive the image sensors of one or more cameras in the camera module to capture images and drive the image signal processor to pre-process the images. The depth sensor driver is used to drive one or more depth sensors in the depth sensor module to emit light signals to the subject and receive light returned by the subject. The optical signal processing driver is used to drive the optical signal processor to process light signals. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0133] The hardware layer may include a camera module, an image signal processor, a depth sensor module, an optical signal processor, a digital signal processor, an image processor, and a gyroscope sensor.

[0134] The camera module may include one or more image sensors of the camera (for example, image sensor 1, image sensor 2, etc.). Optionally, the camera module may also include a lens group (including one or more lenses), a photosensitive element, etc. The lens group and the photosensitive element may be used to cooperate with each other to realize the conversion between optical signals and electrical signals.

[0135] The depth sensor module may include one or more depth sensors, and the types of depth sensors include but are not limited to: infrared sensors, laser sensors, time-of-flight (TOF) sensors, structured light sensors, etc. TOF sensors include direct time-of-flight lidar (DTOF) and indirect time-of-flight lidar (ITOF).

[0136] The following is an example of the workflow of electronic device software and hardware, using the scenario of starting a camera application.

[0137] In the embodiment of the present application, when the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the user operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information), and identifies the control corresponding to the input event. Taking the touch operation as an example of a user operation acting on a camera application, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera capability by calling the camera device in the camera hardware abstraction layer, such as camera device 1. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer, and the camera device driver can start the sensor corresponding to the camera device, such as sensor 1, and then collect the image light signal through sensor 1, and transmit the image light signal to the image signal processor for preprocessing to obtain a preprocessed image.

[0138] In one implementation, the image signal processor transmits the preprocessed image back to the hardware abstraction layer through the camera device driver, and the hardware abstraction layer can send the preprocessed image to the blur module. Based on the support of the image signal processor and the digital signal processor, the blur module in the hardware abstraction layer can determine the depth calculation method according to the shooting parameters reported by the camera module and / or other sensors, and determine the depth information of the image through the depth calculation method, and blur the image based on the depth information to optimize the blur effect of the image.

[0139] The hardware abstraction layer returns the image data after the blurring process to the camera application through the camera interface. Subsequently, the camera application and the gallery application can display the blurred photos to the user with the support of the window manager.

[0140] Exemplarily, the blur module determines the preview information and image information of the image based on the shooting parameters reported by the camera module and / or other sensors, and then obtains the depth calculation method based on the preview information and image information based on the judgment condition. Among them, the judgment condition includes one or more of the following: depth sensor condition, image processing algorithm condition, scene judgment condition, etc. For example, taking the depth sensor condition as an example, the blur module determines whether the shooting scene meets the depth sensor condition based on the preview information and image information. If the depth sensor condition is met, the blur module determines the depth information of the image through a calculation method based on the depth sensor.

[0141] The specific implementation process of determining the depth information of an image by a calculation method based on a depth sensor may be that the camera application calls the depth sensor access interface of the application framework layer to start the depth sensor, and then sends an instruction to start the depth sensor capability by calling the depth sensor device in the depth sensor hardware abstraction layer, such as the depth sensor 1. The depth sensor hardware abstraction layer sends the instruction to the depth sensor driver of the driver layer, and the depth sensor driver can start the corresponding depth sensor, such as the depth sensor 1, and then send an optical signal to the photographed object through the depth sensor 1, and receive the optical signal returned from the photographed object through the depth sensor 1, and transmit the optical signal to the optical signal processor for processing. The optical signal processor can obtain the distance between the depth sensor 1 and the photographed object according to the flight (round trip) time of the light, and the optical signal processor transmits the distance between the depth sensor 1 and the photographed object back to the hardware abstraction layer through the depth sensor driver. The hardware abstraction layer can send the distance between the depth sensor 1 and the photographed object to the blur module, and the blur module can determine the depth information of the image based on the distance between the depth sensor 1 and the photographed object. Then the blur module blurs the image based on the depth information, thereby optimizing the blur effect of the image.

[0142] Exemplarily, if the depth sensor condition is not met, the blur module can choose to determine the depth information of the image through an image processing algorithm (monocular depth estimation algorithm or binocular depth estimation algorithm). Furthermore, the blur module can also determine whether to use a monocular depth estimation algorithm or a binocular depth estimation algorithm based on the temperature of the electronic device. Since electronic devices consume power and generate heat when running image processing algorithms. Moreover, the greater the power consumption, the more heat is generated. Therefore, in order to reduce the heat generated by electronic devices during shooting, electronic devices can select a depth calculation method with low power consumption on the basis of ensuring the accuracy of depth information to reduce the heat generated by electronic devices during shooting. It is understandable that since different image processing algorithms may generate different amounts of heat, selecting a corresponding image processing algorithm based on the temperature of the electronic device can control the temperature of the electronic device, thereby reducing the possibility that the electronic device cannot process the image due to excessive temperature.

[0143] The image processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0144] The image processing method provided in the embodiment of the present application can be applied to Figure 3 The hardware structure shown in Figure 4 In an electronic device with the software structure shown in the figure, or with more or fewer components than shown in the figure, or with some components combined, or some components separated, or with different component arrangements, etc., and with similar hardware structures and software structures.

[0145] In this embodiment, the electronic device selects a depth calculation method according to one or more of the following factors, and the one or more factors include: shooting scene, power consumption corresponding to multiple depth calculation methods, temperature of the electronic device, etc. Therefore, the electronic device can automatically select the corresponding depth calculation method according to different shooting scenes, and can reduce energy consumption on the basis of improving the accuracy of depth information, thereby improving the blurring effect of the image and reducing the heat generated by the electronic device.

[0146] The following will be combined Figure 5A-5F The present invention describes a process of blurring an image by an electronic device in a scenario where the image contains complex textures (for example, a scenario where the type and / or amount of texture information in the image is greater than a texture threshold).

[0147] Figure 5A The system desktop of an electronic device provided in an embodiment of the present application, Figure 5B-5E A user interface of a camera application provided in an embodiment of the present application, Fig. 5F It is a user interface of a gallery application provided in an embodiment of the present application.

[0148] Figure 5A This is a system desktop 501 of an electronic device provided in an embodiment of the present application. Figure 5A As shown, the system desktop 501 may include a status bar 502, a page indicator 503, and a plurality of application icons.

[0149] Among them, the status bar 502 may include one or more signal strength indicators of a mobile communication signal (also referred to as a cellular signal) (e.g., a fifth generation mobile communication technology (5th Generation Mobile Communication Technology, 5G), a wireless high fidelity (wireless fidelity, Wi-Fi) signal strength indicator, a battery status indicator, a time indicator (e.g., 8:00), etc.

[0150] The page indicator 503 may be used to indicate the positional relationship between the currently displayed page and other pages.

[0151] The plurality of application icons may include a time application icon (eg, 08:00), a date application icon (eg, January 1, Friday), a weather application icon (eg, 5 ), application market application icon, memo application icon, mall application icon, browser application icon, phone application icon, information application icon, camera application icon, setting application icon, etc. Not limited to the above icons, the system desktop 501 may also include other application icons, which are not listed here one by one. Multiple application icons may be distributed on multiple pages. Page indicator 503 may be used to indicate which page the user is currently browsing among multiple pages carrying multiple applications. The user can browse other pages by sliding left and right.

[0152] Understandably, Figure 5A The user interface described below merely illustrates a possible user interface style of an electronic device taking a mobile phone as an example and should not constitute a limitation on the embodiments of the present application.

[0153] like Figure 5A As shown, the electronic device 100 may receive a user operation to start a camera application, such as clicking an operation on a desktop icon of the camera application. In response to the operation, the electronic device may display a Figure 5B The user interface shown.

[0154] Figure 5B The user interface of the shooting and display service provided by the embodiment of the present application is exemplarily shown, also called preview interface 507. Figure 5BIn the preview interface 507 shown, the electronic device 100 displays a shooting preview screen 504, a function selection area 102, a mode selection area 103, a zoom function 104, a gallery 105, a shooting button 106 and a switch button 107. Among them, the function selection area 102 includes at least one function setting button. The function setting button includes but is not limited to: a visual button, a flash button, an artificial intelligence (AI) camera button, a color button and a setting button. If the electronic device 100 detects that the user touches any function setting button in the function selection area 102, the corresponding function is set according to the touched function setting button. For example, if it is detected that the user touches the flash button, the flash is set. The mode selection area 103 includes at least one shooting mode button. The shooting mode buttons include but are not limited to: an aperture mode button, a blur mode button, a night scene mode button, a portrait mode button, a photo mode button, a video mode button, a professional mode button, and more mode buttons. If the electronic device 100 detects that the user touches any shooting mode button in the mode selection area 103, it switches to the corresponding shooting mode according to the touched shooting mode button. The gallery 105 is used to review the taken pictures. Specifically, the gallery 105 is used to provide the user with an entry to review the taken pictures. The user clicks the gallery 105 to enter the interface for reviewing the taken pictures. The shooting button 106 is used to take pictures, and the switching button 107 is used to switch the front and rear cameras during shooting.

[0155] like Figure 5B As shown, the electronic device 100 responds to the user operation from Figure 5A The system desktop 501 shown switches to Figure 5B After the preview interface 507 shown, the shooting mode of the electronic device 100 is in the default "photographing" mode. Since the electronic device 100 in the "photographing" mode will not blur the captured image, the electronic device 100 needs to switch the shooting mode. For example, the electronic device 100 may receive a user operation to switch modes, such as receiving an operation of the user touching the blur shooting mode button in the mode selection area 103, or receiving an operation of the user dragging the mode selection area 103 to the right (right sliding operation) and causing the float to stop at the "blur" option. The electronic device 100 may switch the shooting mode from the "photographing" mode to the "blurring" mode, and display the following. Figure 5C Preview interface 508 is shown.

[0156] like Figure 5CAs shown, the float in the mode selection area 103 stops at the "blur" option, and the shooting mode of the electronic device 100 is in the blur shooting mode. The electronic device 100 can blur the captured image in this mode. In addition, a shooting preview screen 505 is displayed in the preview interface 508, and a human body 5051 and a wall 5052 are displayed in the shooting preview screen 505. The human body 5051 wears a hat 5053 with a striped texture on his head. Among them, the wall 5052 contains a large number of rectangular textures, and the texture of the wall 5052 is similar to the texture of the hat 5053.

[0157] like Figure 5D As shown, when the electronic device 100 switches to Figure 5C After the preview interface 508 shown, the electronic device 100 responds to the user's focus operation (for example, the user clicks on the human body 5051), determines that the human body 5051 is the shooting subject, and displays a focus frame in the shooting preview screen 505. Among them, the shooting subject is the shooting object corresponding to the focus of the camera, and is the shooting object that needs to be clearly displayed. In addition, the picture in front of the shooting subject in the image is the foreground, and the picture behind the shooting object is the background. The electronic device 100 can blur the foreground and / or the background so that the foreground and / or the background are blurred. For example, when the shooting subject is the human body 5051, the wall 5052 behind the human body 5051 is the background. Therefore, the electronic device 100 needs to blur the wall 5052 so that the human body 5051 is displayed clearly, while the wall 5052 is displayed blurred.

[0158] In one implementation, when the shooting mode of the electronic device is in the blur shooting mode, the electronic device usually needs to blur the captured image. Before blurring the captured image, the electronic device may also obtain the temperature of the electronic device, and then determine whether the electronic device can blur the image based on the temperature. For example, if the temperature of the electronic device is less than or equal to the first temperature threshold, the electronic device may blur the image. If the temperature of the electronic device is greater than the first temperature threshold, the electronic device does not blur the image.

[0159] The following describes the electronic device displaying a blurred image by taking the temperature of the electronic device being less than or equal to the first temperature threshold as an example. Figure 5D In the embodiment, the electronic device 100 can obtain preview information based on the shooting preview screen 505 in the preview interface 508. The specific method of obtaining the preview information can refer to step S701 in the following embodiment, which will not be repeated here.

[0160] like Figure 5EAs shown, the electronic device 100 responds to the user's photo operation, such as clicking the shooting button 106, collects the image reported by the camera corresponding to the timestamp, and then determines the image information according to the collected image. The specific method of determining the image information can be referred to step S702 in the following embodiment, which will not be repeated here.

[0161] In one implementation, the electronic device 100 selects a depth calculation method based on the acquired preview information and image information, wherein the specific selection process can be found in step S703 in the following embodiment and will not be repeated here. For example, in the shooting preview screen 505, the wall 5052 contains a large number of rectangular textures, and the texture of the wall 5052 is similar to the texture of the hat 5053, so the electronic device 100 can determine that the texture scene corresponding to the current image is a complex texture scene, which meets the depth sensor condition. When the captured image meets the depth sensor condition, the electronic device 100 can determine the depth information through a calculation method based on the depth sensor, and then blur the image based on the depth information, and save it as a blurred image after blurring the image, for example Fig. 5F The blurred image 5061 is shown.

[0162] like Fig. 5F As shown, the electronic device 100 displays a gallery interface 506, which includes a blurred image 5061 and a status bar 502. The blurred image 5061 is an image obtained by blurring the image corresponding to the preview screen 505 based on the depth information after the electronic device 100 determines the depth information of the image corresponding to the preview screen 505 by a calculation method based on a depth sensor. Since the hat 5053 is part of the human body 5051 area and the human body 5051 is the subject of the shooting, the human body 5051 and the hat 5053 need to be clearly displayed. Fig. 5F It can be seen that, compared with the shooting preview screen 505, the human body 5051 and the hat 5053 are clearly displayed in the blurred image 5061, while the wall 5052 is blurred. Therefore, the areas that need to be clearly displayed in the blurred image 5061 (the human body 5051 and the hat 5053) are clearly displayed, and the areas that need to be blurred (the wall 5052) are blurred. The electronic device 100 has a better blurring effect on the blurred image 5061.

[0163] When the texture scene corresponding to the image is a complex texture scene, the embodiment of the present application can obtain more accurate depth information compared to directly determining the depth information by calculation based on the image processing algorithm, thereby obtaining a blurred image with better blurring effect. Figure 1B For example, Figure 1BIt is an image after the electronic device 100 performs blur rendering based on the depth information obtained by the calculation method of the image processing algorithm. It can be seen that the hat 1081 in the human body 108 is blurred, indicating that the depth information of the hat 1081 obtained by the electronic device 100 based on the calculation method of the image processing algorithm is inaccurate, resulting in the blurring of the hat 1081 that needs to be clearly displayed. The embodiment of the present application selects the depth calculation method based on the preview information and the image information, so that the electronic device 100 can determine the depth information in a manner suitable for the shooting scene to improve the accuracy of the depth information, thereby optimizing the blurring effect of the image.

[0164] The following will be combined Figure 6A-6D The process of blurring an image by an electronic device when the shooting scene is in high-speed motion (for example, a scene in which the relative speed between the shooting object and the electronic device is greater than a speed threshold) is described.

[0165] Fig. 6A The system desktop of an electronic device provided in an embodiment of the present application, FIG. 6B to FIG. 6C is another set of camera application user interfaces provided by the embodiment of the present application, Fig.6D This is another user interface of a gallery application provided in an embodiment of the present application.

[0166] Fig. 6A This is a system desktop 501 of an electronic device provided in an embodiment of the present application. Fig. 6A As shown, the system desktop 501 may include a status bar 502, a page indicator 503, and a plurality of application icons. The status bar 502, the page indicator 503, and the plurality of application icons may be seen in Figure 5A The relevant introduction will not be repeated here.

[0167] Understandably, Fig. 6A The user interface described below merely illustrates a possible user interface style of an electronic device taking a mobile phone as an example and should not constitute a limitation on the embodiments of the present application.

[0168] like Fig. 6A As shown, the electronic device 100 may receive a user operation of opening a camera application, such as clicking a desktop icon of the camera application. In response to the operation, the electronic device may display a Figure 6B The user interface shown.

[0169] Figure 6B The user interface of the shooting and display service provided by the embodiment of the present application is exemplarily shown, also called preview interface 604. Figure 6BIn the preview interface 604 shown, the electronic device 100 displays a shooting preview screen 601, a function selection area 102, a mode selection area 103, a zoom function 104, a gallery 105, a shooting button 106, and a switch button 107. Among them, the function selection area 102, the mode selection area 103, the zoom function 104, the gallery 105, the shooting button 106, and the switch button 107 can be seen in FIG. Figure 5B The relevant introduction will not be repeated here.

[0170] like Figure 6B As shown, the electronic device 100 responds to the user operation from Fig. 6A The system desktop 501 shown switches to Figure 6B After the preview interface 604 is shown, the shooting mode of the electronic device 100 is in the "blur" mode. At this time, the electronic device 100 can blur the captured image without switching the shooting mode.

[0171] A human body 6011 and a plant 6012 are displayed in the shooting preview screen 601, and an arm 6013 of the human body 6011 is perpendicular to the ground.

[0172] like Figure 6C As shown, a shooting preview screen 602 is displayed in the preview interface 605 , a human body 6021 and a plant 6012 are displayed in the shooting preview screen 602 , and an arm 6022 of the human body 6021 is parallel to the ground.

[0173] In one implementation, when the shooting mode of the electronic device is in the blur shooting mode, the electronic device usually needs to blur the captured image. Before blurring the captured image, the electronic device may also obtain the temperature of the electronic device, and then determine whether the electronic device can blur the image based on the temperature. For example, if the temperature of the electronic device is less than or equal to the first temperature threshold, the electronic device may blur the image. If the temperature of the electronic device is greater than the first temperature threshold, the electronic device does not blur the image.

[0174] The following describes the display of a blurred image by an electronic device by taking the temperature of the electronic device being less than or equal to the first temperature threshold as an example. Figure 6B In the embodiment, the electronic device 100 can obtain the first preview information based on the shooting preview screen 601 in the preview interface 604. Figure 6C In the embodiment, the electronic device 100 can obtain the second preview information based on the shooting preview screen 602 in the preview interface 605. The specific method of obtaining the first preview information and the second preview information can be referred to step S701 in the following embodiment, which will not be repeated here.

[0175] like Figure 6CAs shown, the electronic device 100 responds to the user's photo operation, such as clicking the shooting button 106, collects the image reported by the camera corresponding to the timestamp, and then determines the image information according to the collected image. The specific method of determining the image information can be referred to step S702 in the following embodiment, which will not be repeated here.

[0176] In one implementation, the electronic device 100 selects a depth calculation method based on the acquired preview information and image information, wherein the specific selection process can be referred to step S703 in the following embodiment, which will not be described in detail here. For example, if the moment when the electronic device 100 acquires the shooting preview screen 601 is different from the moment when the shooting preview screen 602 is acquired, for example, the electronic device 100 first acquires the shooting preview screen 601, and then acquires the shooting preview screen 602 after a certain interval. Moreover, by comparing the shooting preview screen 601 with the shooting preview screen 602, it can be seen that the position of the arm of the same human body has changed (the arm 6013 is perpendicular to the ground, and the arm 6022 is parallel to the ground), then it can be explained that in the process of the electronic device 100 acquiring the shooting preview screen 601 and the shooting preview screen 602, the electronic device 100 and the shooting object (such as the human body) have relative motion. The electronic device can further determine the motion state of the shooting scene according to the first preview information and the second preview information, and then determine whether the motion state of the shooting scene meets the depth sensor condition. For example, the electronic device 100 determines that the motion state of the shooting scene is a high-speed motion state according to the first preview information and the second preview information. Then, the current shooting scene is a high-speed motion scene, which meets the depth sensor condition. The electronic device 100 can determine the depth information by a calculation method based on the depth sensor, and then blur the image based on the depth information. After blurring the image, it is saved as a blurred image, for example Fig.6D The blurred image 6031 is shown.

[0177] It should be noted that, since the electronic device 100 did not receive the user's focus operation before receiving the user's photo operation, the user did not select the subject to be photographed, that is, the user did not select the subject to be photographed that needs to be highlighted. When the electronic device 100 does not receive the user's focus operation, it usually highlights the scene subject in the photographed object. For example, the scene subject includes human body, animals, plants, etc. Since there is a human body 6021 in the shooting preview screen 602, the electronic device 100 can highlight the human body 6021. Since the screen behind the human body 6021 includes the plant 6012, the human body 6021 is the foreground in the image shooting preview screen 602, and the plant 6012 is the background in the image shooting preview screen 602. Then, the electronic device 100 needs to blur the background (plant 6012) so that the foreground (human body 6021) is displayed clearly, while the plant 6012 is displayed blurred.

[0178] like Fig.6D As shown, the electronic device 100 displays a gallery interface 603, which includes a blurred image 6031 and a status bar 502. The blurred image 6031 is an image obtained by blurring the image corresponding to the preview screen 602 based on the depth information after the electronic device 100 determines the depth information of the image corresponding to the preview screen 602 by calculating the depth sensor. Fig.6D It can be seen that, compared with the shooting preview screen 602, the human body 6021 appears clear and the plant 6012 appears blurred in the blurred image 6031. Therefore, the area that needs to be clearly displayed in the blurred image 6031 (human body 6021) appears clear, and the area that needs to be blurred (plant 6012) appears blurred. The electronic device 100 has a better blurring effect on the blurred image 6031.

[0179] When the shooting scene corresponding to the image is a high-speed motion scene, the embodiment of the present application can obtain more accurate depth information compared to directly determining the depth information by calculation based on the image processing algorithm, thereby obtaining a blurred image with better blurring effect. Figure 2B For example, Figure 2B It is an image rendered by the electronic device 100 after blurring based on the depth information obtained by the calculation method based on the image processing algorithm. It can be seen that both the human body 202 and the plant 203 are displayed clearly, indicating that the depth information determined by the electronic device 100 based on the calculation method of the image processing algorithm is incorrect, resulting in the plant 203 that needs to be displayed blurred being displayed clearly. The embodiment of the present application selects the depth calculation method based on the preview information and the image information, so that the electronic device 100 can determine the depth information in a manner suitable for the shooting scene to improve the accuracy of the depth information, thereby optimizing the blurring effect on the image.

[0180] Above FIG. 5A to FIG. 5F ,as well as FIG. 6A to FIG. 6D This paper introduces the schematic diagram of the interface of the electronic device displaying the blurred image. Figure 7 , Fig.10 The process of an electronic device determining depth information of an image and then blurring the image based on the depth information is specifically described.

[0181] See also Figure 7 , Figure 7 is a flow chart of an image processing method provided in an embodiment of the present application. Figure 7 As shown, the flow chart includes S701 to S703. Figure 7 The exemplary flow chart shown illustrates the specific implementation method of the electronic device selecting the depth calculation method through the decision module.

[0182] S701, in a blur shooting mode, displaying a first user interface.

[0183] The first user interface is used to display the captured image, which includes one or more captured objects. Figure 5C , Figure 5D and Figure 5E The preview interface 508 in Figure 6B The preview interface 604, Figure 6C A preview interface 605 is provided.

[0184] Exemplarily, when the camera of the electronic device is focused, one or more photographed objects displayed in the first user interface are clear; when the camera of the electronic device is not focused, one or more photographed objects displayed in the first user interface may be unclear.

[0185] In an embodiment of the present application, the camera application of the electronic device provides a variety of shooting modes, such as aperture shooting mode, night scene shooting mode, professional shooting mode, blur shooting mode, etc. Exemplarily, in the blur shooting mode, the electronic device can choose whether to determine the depth information of the image by software (such as image processing algorithm) or by hardware (such as by depth sensor) according to the contents contained in steps S701 to S703, so as to achieve a large aperture effect of the image according to the depth information. Since the existing electronic devices mainly process the image by software in the aperture shooting mode to achieve the large aperture effect. Therefore, in order to distinguish the implementation method of the aperture shooting mode from the implementation method provided in the embodiment of the present application, the shooting mode corresponding to the implementation method provided in the embodiment of the present application is called blur shooting mode.

[0186] In one implementation, in a camera application, when the electronic device switches the shooting mode of the electronic device from a shooting mode other than the blur shooting mode to the blur shooting mode in response to a user operation for switching the mode, preview information can be obtained from the shooting preview screen. Figure 5B As shown, the electronic device displays a user interface corresponding to the blur shooting mode in response to a user operation on the blur shooting mode control in the mode selection area.

[0187] In one implementation, within a preset time, the electronic device responds to a user operation of clicking the camera application to start the camera application and the blur shooting mode. For example, after the electronic device switches the camera application in the blur shooting mode from the foreground to the background, within a preset time, the electronic device can respond to a click operation on the desktop icon of the camera application to display a user interface corresponding to the blur shooting mode. The preview information includes information determined by the electronic device from the shooting preview screen. For example, from Figure 5D The shooting preview screen 505 shown, or, from Figure 6B The shooting preview screen 601 shown or from Figure 6C The information determined in the shooting preview screen 602 is shown.

[0188] For details, please refer to Fig. 8A , Fig. 8A It is a schematic diagram of preview information provided in an embodiment of the present application, wherein the preview information includes one or more of the following: motion status of the shooting scene, brightness information of the shooting scene, scene subject detection results, number of faces, etc.

[0189] Among them, the motion state of the shooting scene includes but is not limited to a stationary state, a low-speed motion state, a high-speed motion state, etc. Since the electronic device is used to capture the image of the shooting object, the motion state of the shooting scene is the motion state of the shooting object relative to the electronic device. For example, in a scene where there is a relative speed between the electronic device and the shooting object, such as the shooting object is moving and the electronic device is stationary, or the shooting object is stationary and the electronic device is moving, it can be said that the shooting scene is in motion. Furthermore, the electronic device can determine whether the shooting scene is in a stationary scene, a low-speed motion scene, or a high-speed motion scene based on the relative speed between the electronic device and the shooting object.

[0190] For example, the electronic device can determine the relative speed between the electronic device and the subject according to the shooting preview screen in the camera application, thereby determining the motion state of the shooting scene according to the relative speed. In one implementation, the electronic device compares the relative speed with a speed threshold to determine the motion state of the shooting scene.

[0191] For example, the electronic device can obtain multiple frames of images in the shooting preview screen, and then determine the relative speed of the electronic device and the shooting object according to the displacement of the shooting object in the multiple frames of images. Among them, the multiple frames of images can be images collected by the electronic device at different times based on the shooting preview screen. In one implementation, the first frame image and the second frame image are images collected by the electronic device at different times based on the shooting preview screen. The electronic device performs key point detection and matching on the first frame image and the second frame image to obtain a key point correspondence relationship, which is used to represent the correspondence between the first key point in the first frame image and the second key point in the second frame image. Then, the displacement between the first key point and its corresponding second key point is determined, and the relative speed between the electronic device and the shooting object can be determined according to the time difference and displacement between the first frame image and the second frame image.

[0192] It can be understood that, since the multiple frames of images are images collected by the electronic device at different times based on the shooting preview screen, the relative speed determined based on the multiple frames of images corresponds to the time when the electronic device collects the images. For example, the relative speed determined by the electronic device based on the first frame of image and the second frame of image can correspond to the relative speed between the electronic device and the photographed object when the electronic device collects the second frame of image. Therefore, the electronic device can determine the relative speed corresponding to each frame of image of the shooting preview screen according to the above method.

[0193] For another example, when the relative speed is equal to 0, it can be determined that the motion state of the current shooting scene is a static state and the current shooting scene is a static scene. When the relative speed is greater than 0 and less than the speed threshold, it can be determined that the motion state of the current shooting scene is a low-speed motion state and the current shooting scene is a low-speed motion scene. When the relative speed is greater than or equal to the speed threshold, it can be determined that the motion state of the current shooting scene is a high-speed motion state and the current shooting scene is a high-speed motion scene.

[0194] It should be noted that when the shooting scene is in a high-speed motion scene, the accuracy of the depth information obtained by the electronic device through the image processing algorithm is low, and it is necessary to determine the depth information of the image through a calculation method based on a depth sensor. Therefore, the embodiment of the present application uses the motion state of the shooting scene as a judgment condition for selecting the depth calculation method, which can improve the accuracy of the depth information when the shooting scene is in a high-speed motion scene.

[0195] Among them, the brightness information of the shooting scene is used to characterize the intensity of the ambient light in the shooting scene. Ambient light includes natural light such as sunlight and artificial light such as incandescent light. Since the depth sensor is an optical device, the intensity of the ambient light will affect the accuracy of the depth information determined by the depth sensor. Therefore, the embodiment of the present application uses the intensity of the ambient light as a judgment condition in selecting the depth calculation method, which can avoid the problem of obtaining erroneous depth information due to lighting problems.

[0196] Exemplarily, the electronic device can process each frame of the preview screen captured in the camera application to obtain brightness information corresponding to each frame of the image, and then compare the brightness information corresponding to each frame of the image with the brightness threshold to obtain the brightness scene corresponding to each frame of the preview screen.

[0197] For example, when the brightness information is greater than or equal to the brightness threshold, it can be determined that the current shooting scene is a high-brightness scene. When the brightness information is less than the brightness threshold, it can be determined that the current shooting scene is a low-brightness scene.

[0198] The scene subject detection result is used to indicate whether there is a pre-stored scene subject type in the types of the photographed objects. For example, the scene subject types include but are not limited to human body, animal, plant, etc.

[0199] Exemplarily, the electronic device may process each frame of the preview image captured in the camera application to obtain a scene subject detection result corresponding to each frame of the image.

[0200] For example, when the photographed object does not contain the type of the pre-stored scene subject, it can be determined that the current photographed scene has no scene subject. When the photographed object contains the type of the pre-stored scene subject, it can be determined that the current photographed scene has a scene subject.

[0201] The number of faces is used to characterize the number of faces in the scene subject. When the electronic device determines that there is no scene subject in the current shooting scene based on the scene subject detection result, the electronic device does not need to determine the number of faces. When the electronic device determines that there is a scene subject in the current shooting scene based on the scene subject detection result, the electronic device can further determine whether there are faces in the shooting scene, and the number of faces if there are faces.

[0202] Exemplarily, the electronic device may process each frame of the preview image captured in the camera application to obtain the number of faces corresponding to each frame of the image.

[0203] For example, when the number of faces in the photographed object is 0, it can be determined that the current shooting scene is a non-portrait scene. When the number of faces in the photographed object is greater than or equal to the number threshold, it can be determined that the current shooting scene is a multi-person scene. When the number of faces in the photographed object is greater than 0 and less than the number threshold, it can be determined that the current shooting scene is a non-multi-person scene.

[0204] S702, in response to a first user operation, obtain preview information of a first user interface and image information of a captured image, and blur the captured image according to a depth calculation method to obtain a target image.

[0205] The first user operation is used to control the electronic device to capture images. For example, the first user operation includes but is not limited to a touch operation, a somatosensory operation, a voice operation, etc. The touch operation includes a user's click operation (for example Figure 5E The operation of taking a photo is triggered by contacting the electronic device, such as the user clicking the shooting button in the preview interface in the video) or key operation. Somatosensory operation includes the operation of the user triggering the photo taking through gestures or facial expressions. Voice operation includes the operation of triggering the photo taking through pre-stored voice commands (such as "take a photo", etc.). The embodiment of the present application does not limit the type of the first user operation, and the operation for triggering the electronic device to capture an image can be referred to as the first user operation.

[0206] It should be noted that, in step S701, the electronic device processes each frame of the preview screen in the blur shooting mode before receiving the first user operation, so as to determine the preview information corresponding to each frame of the preview screen. The electronic device needs to perform blur processing on the image collected at the moment of receiving the first user operation, so the electronic device can obtain the preview information corresponding to the image at the moment of receiving the first user operation from the preview information corresponding to the multiple frames of images, and then determine the depth calculation method according to the preview information.

[0207] In a possible implementation, the electronic device collects an image corresponding to the currently shot preview screen in response to the first user operation, and obtains image information from the collected image.

[0208] For details, please refer to Figure 8B , Figure 8B is a schematic diagram of image information provided by an embodiment of the present application. Figure 8B As shown, the image information includes one or more of the following: texture detection result, clarity detection result, and scene distance detection result.

[0209] Among them, the texture detection result in the image information is used to indicate the complexity of the texture in the image. Among them, the image texture refers to the visual texture effect presented in the image, which is composed of regular changes in pixel color and brightness. Specifically, the image texture is composed of a set of recurring texture units, which can be lines, points, shapes, etc., and their appearance can be regular, random or semi-regular. Since the accuracy of the depth information obtained by the electronic device through the image processing algorithm is low in the scene where the image contains complex textures, it is necessary to determine the depth information of the image through a calculation method based on a depth sensor. Therefore, the embodiment of the present application uses the texture detection result as a judgment condition for selecting the depth calculation method, which can improve the accuracy of the depth information in complex texture scenes.

[0210] For example, the electronic device can determine the texture detection result from the collected image through image processing technology or artificial intelligence (AI) model. Then, the electronic device compares the texture detection result with the texture threshold to determine the texture scene corresponding to the image.

[0211] For example, when the texture detection result is greater than the texture threshold, it can be determined that the texture scene corresponding to the current image is a complex texture scene. Further, when the texture scene corresponding to the image is a complex texture scene, it can also be explained that the image contains more texture types and / or contains a large number of textures. When the texture detection result is less than or equal to the texture threshold, it can be determined that the texture scene corresponding to the current image is a common texture scene.

[0212] The clarity detection result is used to characterize the clarity of the image. For example, the electronic device can determine the clarity detection result from the collected image through image processing technology or AI model. Then, the electronic device compares the clarity detection result with the clarity threshold to determine the clarity quality of the image.

[0213] For example, when the clarity detection result is greater than the clarity threshold, it can be determined that the clarity quality of the image is better and it is a clear image. When the clarity detection result is less than or equal to the clarity threshold, it can be determined that the clarity quality of the image is poor and it is a blurred image.

[0214] Among them, the scene distance detection result in the image information is used to characterize the relative distance between the photographed object and the electronic device, so as to determine the distance type of the shooting scene. For example, the distance type of the shooting scene includes but is not limited to close-up and long-range. Exemplarily, the electronic device can determine the scene distance detection result from the collected image through image processing technology or AI model. Then, the electronic device compares the scene distance detection result with the distance threshold to determine the distance type of the shooting scene (for example, determine whether the shooting scene is a close-up or a long-range).

[0215] For example, when the scene distance detection result is greater than the distance threshold, it can be determined that the current shooting scene is a distant view. When the scene distance detection result is less than or equal to the distance threshold, it can be determined that the current shooting scene is a near view.

[0216] In a possible implementation, first, the electronic device can determine whether the captured image contains a scene subject. When the captured image does not contain a scene subject, for example, the captured image does not contain a scene subject such as a human body, plant, or animal, the electronic device can determine that the shooting scene of the captured image is a distant view. When the captured image contains a scene subject, the electronic device can further determine the distance type corresponding to the shooting scene based on the distance from the scene subject to the electronic device. For example, when the distance between the scene subject in the captured image and the electronic device is less than or equal to a distance threshold, it can be determined that the shooting scene of the captured image is a close view. When the distance between the scene subject in the captured image and the electronic device is greater than a distance threshold, it can be determined that the shooting scene of the captured image is a distant view.

[0217] In a possible implementation, the electronic device determines the depth calculation method according to the preview information and the image information based on the judgment condition.

[0218] Among them, the depth calculation method includes but is not limited to the calculation method based on image processing algorithm and the calculation method based on depth sensor. The calculation method based on image processing algorithm is mainly that the electronic device determines the depth information based on the information contained in the image by software, and the calculation method based on depth sensor is mainly that the electronic device measures the distance between the electronic device and the photographed object through hardware such as depth sensor to determine the depth information.

[0219] Exemplarily, the judgment conditions provided in the embodiments of the present application include one or more of the following: depth sensor conditions, image processing algorithm conditions, blur escape conditions, scene judgment conditions, etc. It should be noted that when the electronic device determines that the current shooting scene meets any of the above judgment conditions based on the preview information and the image information, the depth information can be determined in a manner corresponding to the judgment condition.

[0220] In a possible implementation, after the electronic device obtains the preview information and the image information according to step S701 and step S702, the electronic device first determines whether the shooting scene meets the blur escape condition according to the preview information and the image information. Among them, in the camera application, "escape" generally refers to the processing mechanism when the electronic device encounters an error or anomaly during the photo or video recording process. It can include functions such as automatically saving the current shooting content, restoring to a stable state, or preventing data loss. This processing mechanism ensures that even if a problem occurs, the user's data and shooting content will not be lost. Therefore, if it is determined that the blur escape condition is met, the electronic device does not blur the collected image to perform blur escape. If it is determined that the blur escape condition is not met, the electronic device needs to blur the image. Further, the electronic device again determines whether the current shooting scene meets the depth sensor condition and / or the image processing algorithm condition according to the preview information and the image information. Among them, the specific implementation method of determining whether the current shooting scene meets the depth sensor condition and / or the image processing algorithm condition can refer to the following embodiment.

[0221] Exemplarily, the blur escape condition includes one or more of the following: the shooting scene is a highlight scene and the scene distance detection result is a distant view, the shooting scene is a highlight scene, the scene distance detection result is a close view, and the clarity detection result is a blurred image.

[0222] Among them, when the shooting scene is a highlight scene and the scene distance detection result is a distant view, since the depth sensor is an optical device and is easily affected by ambient light, it is necessary to specifically analyze the situation when the intensity of the ambient light is large (such as a highlight scene). For example, when the shooting scene is in a highlight scene, the electronic device may perform image acquisition outdoors. Since the distance between the shooting object and the electronic device may be far away outdoors, the scene distance detection result corresponding to the shooting scene is a distant view. When the shooting scene of the image is a distant view, it means that most areas in the image are blurred, and the electronic device does not need to repeatedly blur the blurred image. Therefore, when the shooting scene is a highlight scene and the scene distance detection result is a distant view, the blur escape condition is met and the electronic device does not need to process the image.

[0223] Among them, when the shooting scene is a high-brightness scene, the camera of the electronic device may be affected by the ambient light and cause blurred imaging. Therefore, when the shooting scene is a high-brightness scene, the scene distance detection result is a close view and the clarity detection result is a blurred image, the electronic device does not need to repeatedly blur the blurred image. Therefore, the shooting scene at this time meets the blur escape conditions and the electronic device does not need to process the image.

[0224] In a possible implementation, the electronic device may determine whether the current shooting scene meets the depth sensor condition based on the preview information obtained in S701 and the image information obtained in S702. When it is determined that the depth sensor condition is met, the electronic device determines the depth information of the image by means of the depth sensor. When it is determined that the depth sensor condition is not met, the electronic device determines the depth information of the image by means of an image processing algorithm.

[0225] Among them, the depth sensor conditions include one or more of the following: the shooting scene is in a high-speed motion scene (for example, a scene in which the relative speed between the shooting object and the electronic device is greater than a speed threshold), a scene in which the image contains complex textures (for example, a scene in which the type and / or amount of texture information in the image is greater than a texture threshold), a scene corresponding to when the shooting subject is a human body and / or a face, and a scene corresponding to when the user has not selected a shooting subject and the number of faces in the image is less than or equal to a quantity threshold.

[0226] In one implementation, when the electronic device determines that the current shooting scene is in a high-speed motion scene based on the motion state of the shooting scene in the preview information, the electronic device can determine that the current shooting scene meets the depth sensor conditions, and the electronic device determines the depth information of the image through a calculation method based on the depth sensor.

[0227] In one implementation, when the electronic device determines that the image contains complex textures based on the texture detection results in the image information, the electronic device determines that the current shooting scene is a complex texture scene. The electronic device can determine that the current shooting scene meets the depth sensor conditions, and the electronic device determines the depth information of the image through a calculation method based on the depth sensor.

[0228] In one implementation, before the electronic device receives the first user operation, if the electronic device also receives the second user operation, the electronic device can also determine whether the current shooting scene meets the depth sensor condition according to the shooting subject indicated by the second user operation. The second user operation is used to determine the shooting subject. The shooting subject is the shooting object corresponding to the focus of the camera. For example, the shooting subject can be a person, an animal, a plant, or a landscape.

[0229] For example, Figure 5D In the process, the user clicks on the human body displayed in the preview screen, and the electronic device responds to the user's click operation (i.e., the second user operation) by Figure 5DThe human body in the image is taken as the subject of the shooting. Since the electronic device can use the subject of the shooting as the part that is clearly displayed when blurring the image, and blur the rest of the parts other than the subject of the shooting. Therefore, the electronic device can select the corresponding depth calculation method according to the type of the shooting subject. For example, since the human body or face in the image includes fine textures such as hair, it is necessary to determine the depth value corresponding to the fine texture by a high-precision depth calculation method (for example, a calculation method based on a depth sensor) so that the fine texture can obtain a better blurring effect. And when the shooting subject is a human body or a face, the user usually hopes to highlight the human body or the face, so at this time the user's requirements for the blurring effect of the image are also high, and the electronic device needs to determine the depth information of the image by a high-precision depth calculation method (for example, a calculation method based on a depth sensor). That is, when the shooting object is a human body and / or a face, the electronic device can determine that the current shooting scene meets the depth sensor condition, and the electronic device determines the depth information of the image by a calculation method based on a depth sensor.

[0230] In one implementation, if the electronic device does not receive the second user operation instruction before the electronic device receives the first user operation, it means that the user has not selected the shooting subject at this time, then the electronic device will usually clearly display the scene subject and blur the parts other than the scene subject. In addition, since the user uses the blurred shooting mode of the camera application, it is usually to highlight the scene subject in the image and blur other parts. When the number of faces in the image is less than or equal to the number threshold, the electronic device needs to select a higher-precision depth calculation method to better highlight the area corresponding to the face in the image. Therefore, when the electronic device does not receive the second user operation and the number of faces in the preview information is less than or equal to the number threshold, the electronic device can determine that the current shooting scene meets the depth sensor conditions, and the electronic device determines the depth information of the image through a calculation method based on the depth sensor.

[0231] In a possible implementation, the electronic device may determine whether the current shooting scene satisfies the image processing algorithm conditions based on the preview information obtained in S701 and the image information obtained in S702. When it is determined that the image processing algorithm is obtained, the electronic device determines the depth information of the image by the calculation method of the image processing algorithm. When it is determined that the image processing algorithm is not satisfied, the electronic device determines the depth information of the image by the calculation method based on the depth sensor.

[0232] Exemplarily, the image processing algorithm conditions include one or more of the following: the shooting scene is a still scene or a low-speed motion scene and there is no scene subject in the shooting scene, the shooting scene is a still scene or a low-speed motion scene and there is a scene subject in the shooting scene but the scene subject does not include a human body and / or a human face, the scene distance detection result is a close shot, the clarity detection result is a clear image and the texture detection result is an ordinary texture scene, the corresponding scene when the shooting subject is not a human body or a human face, and the corresponding scene when the user has not selected a shooting subject and the number of faces in the image is greater than a quantity threshold.

[0233] In one implementation, when the shooting scene is a static scene or a slow-motion scene and there is no scene subject in the shooting scene, when the shooting scene is in a static scene or a slow-motion scene, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is high, but considering that the energy consumption generated by the calculation method based on the depth sensor is greater than the energy consumption generated by the calculation method based on the image processing algorithm, when the shooting scene is in a static scene or a slow-motion scene, the electronic device can choose to determine the depth information through the calculation method based on the image processing algorithm. Further, when there is no scene subject in the shooting scene, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is high. Similarly, considering the energy consumption problem, the electronic device chooses to reduce the energy consumption generated during the blurring process by the calculation method based on the image processing algorithm. Therefore, if the shooting scene is a static scene or a slow-motion scene and there is no scene subject in the shooting scene, the electronic device determines that the current shooting scene meets the image processing algorithm conditions, and the electronic device blurs the image based on the calculation method of the image processing algorithm.

[0234] In one implementation, when the shooting scene is a static scene or a slow-motion scene and there is a scene subject in the shooting scene but the scene subject does not include a human body and / or a human face, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is relatively high, and considering the energy consumption problem, the electronic device chooses to use the calculation method based on the image processing algorithm to reduce the energy consumption generated during the blurring process. Therefore, if the shooting scene is a static scene or a slow-motion scene and there is a scene subject in the shooting scene but the scene subject does not include a human body and / or a human face, the electronic device determines that the current shooting scene meets the image processing algorithm conditions, and the electronic device blurs the image based on the calculation method of the image processing algorithm. In one implementation, when the scene distance detection result is a close view and the clarity detection result is a clear image, the shooting scene does not meet the blurring escape condition, and the electronic device needs to blur the image. Furthermore, when the texture detection result is a common texture, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is relatively high, and considering the energy consumption problem, the electronic device chooses to use the calculation method based on the image processing algorithm to reduce the energy consumption generated during the blurring process. Therefore, if the scene distance detection result is a close view and the clarity detection result is a clear image, the electronic device determines that the current shooting scene meets the image processing algorithm conditions, and the electronic device blurs the image based on the calculation method of the image processing algorithm.

[0235] In one implementation, in the corresponding scene when the subject is not a human body or a face, since the subject that the electronic device needs to highlight does not include a human body or a face, the electronic device can output a high-quality blurred image even if it does not select a high-precision depth calculation method. And considering the energy consumption problem, the electronic device chooses to reduce the energy consumption generated during the blurring process by using a calculation method based on an image processing algorithm. Therefore, if in the corresponding scene when the subject is not a human body or a face, the electronic device determines that the current shooting scene meets the conditions of the image processing algorithm, and the electronic device blurs the image based on the calculation method of the image processing algorithm.

[0236] In one implementation, in the corresponding scene when the user does not select the subject to be photographed and the number of faces in the image is greater than the number threshold, when the electronic device does not receive the second user operation instruction, it means that the user has not selected the subject to be photographed at this time, then the electronic device usually displays the subject of the scene clearly, and blurs the parts other than the subject of the scene. When the number of faces in the image is greater than the number threshold, it means that the user needs to take pictures of multiple faces, then for the user, the area corresponding to each face in the image needs to be highlighted, which further shows that the user has a low requirement for the degree of blurring of the image. Therefore, the electronic device blurs the image by a calculation method based on an image processing algorithm or a calculation method based on a depth sensor, and can output a blurred image that meets the user's requirements. Considering the energy consumption problem, the electronic device chooses to reduce the energy consumption generated during the blurring process by a calculation method based on an image processing algorithm. Therefore, if the electronic device determines that the current shooting scene meets the conditions of the image processing algorithm in the corresponding scene when the user does not select the subject to be photographed and the number of faces in the image is greater than the number threshold, the electronic device blurs the image based on the calculation method of the image processing algorithm.

[0237] In a possible implementation, the electronic device determines the depth information of the image by means of a calculation method based on a depth sensor.

[0238] Among them, the types of depth sensors include but are not limited to: infrared sensors, laser sensors, time-of-flight (TOF) sensors, structured light sensors, etc. TOF sensors include direct time-of-flight lidar (DTOF) and indirect time-of-flight lidar (ITOF).

[0239] For example, the following will take the TOF sensor as an example to introduce the process of an electronic device determining the depth information of an image through a calculation method based on a depth sensor. The electronic device continuously sends light pulses to the photographed object (e.g., each object in the photographed scene) through the TOF sensor, and then receives the light returned from the photographed object through the TOF sensor. The electronic device can obtain the distance between the electronic device and the photographed object based on the flight (round-trip) time of the light, and then determine the depth information of the image based on the distance between the electronic device and the photographed object. Among them, the depth information of the image can be represented in the form of a depth map.

[0240] In a possible implementation, the electronic device determines the depth information of the image by calculating based on an image processing algorithm.

[0241] Among them, image processing algorithms include monocular depth estimation algorithms and binocular depth estimation algorithms. Monocular depth estimation algorithms refer to methods for calculating the depth information of the object in an image based on an image captured by a single camera. It is mainly based on an existing depth information database or a trained depth prediction model, and performs semantic segmentation, motion analysis, texture analysis and other processing on the image to infer the depth information of the object in the image.

[0242] The binocular depth estimation algorithm refers to a method of obtaining two images of the same scene from different perspectives through two cameras (binocular cameras) at the same time, and calculating the depth information of each object in the image using the principle of binocular stereo vision. The binocular depth estimation algorithm is mainly based on the principle of triangulation, and the specific process may include: first, two frames of images are obtained by shooting the object with two cameras at the same time. Then, based on the two frames of images, the pixel offset or parallax of the same object shot by the two cameras is measured. Finally, through the geometric relationship and calibration parameters of the two cameras, the distance of the object relative to the camera, that is, the depth information, can be calculated based on the trained depth prediction model.

[0243] The following will be combined Fig. 9 The process of selecting a depth calculation method based on scene judgment conditions by an electronic device is described in detail.

[0244] See also Fig. 9 , Fig. 9 is a flow chart of a scene judgment condition provided by an embodiment of the present application, such as Fig. 9 As shown, the flowchart includes S901 to S909. Fig. 9 The exemplary flowchart shown illustrates the process of an electronic device selecting a depth calculation method based on scene judgment conditions.

[0245] S901: Whether it is a highlight scene.

[0246] Specifically, since the depth sensor is an optical device and is easily affected by ambient light, it is necessary to specifically analyze the situation when the intensity of the ambient light is large (such as a highlight scene). Therefore, the electronic device can first determine whether the current shooting scene is a highlight scene based on the brightness information of the shooting scene. If the current shooting scene is determined to be a highlight scene based on the brightness information of the shooting scene in the preview information, the electronic device executes S907. If the current shooting scene is determined to be a low-brightness scene based on the brightness information of the shooting scene in the preview information, the electronic device executes S902.

[0247] S902: Whether it is a high-speed motion scene.

[0248] Specifically, since the shooting scene is in a high-speed motion scene, the accuracy of the depth information obtained by the electronic device through the image processing algorithm is low, and it is necessary to determine the depth information of the image through a calculation method based on a depth sensor. Therefore, if the current shooting scene is judged to be a high-speed motion scene based on the motion state of the shooting scene in the preview information, the electronic device determines the depth information through a calculation method based on a depth sensor. If the current shooting scene is judged to be a static scene or a low-speed motion scene based on the motion state of the shooting scene in the preview information, the electronic device executes S903.

[0249] S903: Whether a second user operation is received.

[0250] The second user operation is used to determine the subject of the shooting. If the second user operation is received before the image is captured, the image needs to be blurred based on the subject of the shooting indicated by the second user operation, that is, S904 is executed. If the second user operation is not received before the image is captured, the image needs to be blurred based on the scene subject in the shooting object, that is, S905 is executed.

[0251] S904: Whether human body and / or human face are included.

[0252] Since the electronic device can use the subject of the shooting as the part that is clearly displayed when blurring the image, and blur the rest of the parts other than the subject of the shooting. Therefore, the electronic device can select the corresponding depth calculation method according to the type of the shooting subject. For example, since the human body or face in the image includes fine textures such as hair, it is necessary to determine the depth value corresponding to the fine texture by a high-precision depth calculation method (for example, a calculation method based on a depth sensor) so that the fine texture can obtain a better blurring effect. And when the shooting subject is a human body or a face, the user usually hopes to highlight the human body or the face, so at this time the user's requirements for the blurring effect of the image are also high, and the electronic device needs to determine the depth information of the image by a high-precision depth calculation method (for example, a calculation method based on a depth sensor). That is, when the shooting subject is a human body and / or a face, the electronic device determines the depth information of the image by a calculation method based on a depth sensor. When the shooting subject does not include a human body or a face, execute S905.

[0253] S905, whether there is a scene subject.

[0254] Specifically, when there is no scene subject in the shooting scene, the accuracy of the depth information obtained by the electronic device through the calculation method based on the image processing algorithm and the calculation method based on the depth sensor is relatively high. Taking into account the energy consumption issue, the electronic device chooses to use the calculation method based on the image processing algorithm to reduce the energy consumption generated during the blurring process. Therefore, when it is determined based on the scene subject detection result in the preview information that there is no scene subject in the current shooting scene, the electronic device determines the depth information of the image through the calculation method based on the image processing algorithm. When it is determined based on the scene subject detection result in the preview information that there is a scene subject in the current shooting scene, the electronic device executes S906 and further selects the depth calculation method according to the type of scene subject.

[0255] S906: Whether the number of faces is greater than a threshold number.

[0256] Specifically, when the user has not selected a subject to be photographed and the number of faces in the image is greater than the number threshold, the electronic device will usually clearly display the scene subject (face) and blur the parts other than the scene subject (face). When the number of faces in the image is greater than the number threshold, it means that the user needs to take pictures of multiple faces. Then, for the user, the area corresponding to each face in the image needs to be highlighted, which further shows that the user has a low requirement for the degree of blurring of the image. Therefore, the electronic device can output a blurred image that meets the user's requirements by blurring the image through a calculation method based on an image processing algorithm or a calculation method based on a depth sensor. Considering the energy consumption issue, the electronic device can reduce the energy consumption generated during the blurring process by choosing a calculation method based on an image processing algorithm. Therefore, when the number of faces is greater than the number threshold, the electronic device chooses to determine the depth information through a calculation method based on an image processing algorithm.

[0257] When the number of faces in the image is less than or equal to the number threshold, the electronic device needs to select a depth calculation method with higher accuracy to better highlight the area corresponding to the faces in the image. Therefore, when the number of faces is less than or equal to the number threshold, the electronic device chooses to determine the depth information through a calculation method based on a depth sensor.

[0258] S907, whether it is a long-term vision.

[0259] Specifically, when the shooting scene is a highlight scene, the electronic device can further determine whether the shooting scene is a distant view. This is because when the shooting scene is in a highlight scene, the electronic device may be performing image acquisition outdoors. Since the distance between the shooting object and the electronic device may be far outdoors, the scene distance detection result corresponding to the shooting scene at this time is a distant view. When the shooting scene of the image is a distant view, it means that most areas in the image are blurred, and the electronic device does not need to repeatedly blur the blurred image. Therefore, when the shooting scene is a highlight scene and the scene distance detection result is a distant view, the electronic device may not process the image. If it is determined that the current shooting scene is a close view, execute S908.

[0260] S908: Is it a blurred image?

[0261] Specifically, when the definition detection result is a blurred image, the electronic device does not need to perform repeated blurring processing on the blurred image, so the electronic device may not process the image at this time. When the definition detection result is a clear image, the electronic device executes S909.

[0262] S909: Whether it is a complex texture scene.

[0263] Since the accuracy of the depth information obtained by the electronic device through the image processing algorithm is low in scenes where the image contains complex textures, it is necessary to determine the depth information of the image through a calculation method based on a depth sensor. Therefore, when the image contains complex textures, that is, when the shooting scene is a complex texture scene, the electronic device chooses to determine the depth information through a calculation method based on a depth sensor. When the texture detection result is an ordinary texture scene, the electronic device determines the depth information through a calculation method based on an image processing algorithm.

[0264] Step S703: display the target image on the second user interface.

[0265] The target shooting object in the target image is clear, the non-target shooting object in the target image is unclear, the target shooting object is any one or more of the one or more shooting objects, and the non-target shooting object is other than the target shooting object in the one or more shooting objects.

[0266] For example, the second user interface is Fig. 5F The gallery interface 506 shown in FIG. Fig. 5F As shown, the target shooting objects are a human body 5051 and a hat 5053, and the non-shooting object is a wall 5052. It can be seen that the human body 5051 and the hat 5053 are clear, and the wall 5052 is not clear.

[0267] For example, the second user interface is Fig.6DThe gallery interface 603 shown in FIG. Fig.6D As shown, the target photographing object is a human body 6021, and the non-target photographing object is a plant 6012. It can be seen that the human body 6021 is clear, and the plant 6012 is not clear.

[0268] See also Fig.10 , Fig.10 FIG. 1 is another flowchart of selecting a depth calculation method provided by an embodiment of the present application. Fig.10 As shown, the flowchart includes S1001 to S1004. Fig.10 The exemplary flow chart shown illustrates a specific implementation method when an electronic device selects a depth calculation method based on temperature.

[0269] S1001, obtaining the temperature of the electronic device.

[0270] Specifically, when the shooting mode of the electronic device is in the blur shooting mode, the electronic device needs to blur the captured image. However, when the temperature of the electronic device is high, for example, higher than the first temperature threshold, the electronic device may not process the image after capturing it, but directly output the captured image. Therefore, the electronic device may first obtain the current temperature of the electronic device, and then execute S1002 to determine whether to blur the image according to the temperature.

[0271] S1002, determining whether the temperature is greater than a first temperature threshold.

[0272] Specifically, after the electronic device obtains the temperature according to step S1001, it can compare the temperature with the first temperature threshold. When the temperature of the electronic device is greater than the first temperature threshold, it means that the temperature of the electronic device is high at this time, and the electronic device cannot process the collected image, so the electronic device does not blur the image at this time, and directly outputs the collected image. When the temperature of the electronic device is less than or equal to the first temperature threshold, it means that the temperature of the electronic device is normal at this time, and the electronic device can execute S1003 to blur the image.

[0273] S1003, the electronic device determines a depth calculation method through a decision module.

[0274] When the temperature of the electronic device is less than or equal to the first temperature threshold, the electronic device can select a depth calculation method through the decision module. Figure 7 Steps S701 to S703 are shown and will not be described in detail here.

[0275] Among them, the depth calculation method that the electronic device can obtain through the decision module includes one or more of the following: a calculation method based on a depth sensor, a calculation method based on an image processing algorithm, and a method of not blurring the image. Since the calculation method based on the image processing algorithm includes but is not limited to a monocular depth estimation algorithm and a binocular depth estimation algorithm, etc., when the electronic device determines according to step S703 that the depth information of the image is determined by the image processing algorithm, it may include the following situations:

[0276] When the electronic device includes only one camera or the electronic device captures images only through one camera, the electronic device determines the depth information of the image through a monocular depth estimation algorithm.

[0277] When the electronic device includes two or more cameras and can simultaneously capture two or more images, since the electronic device can determine the depth information based on one of the two or more images captured, or can determine the depth information based on two of the two or more images captured, the electronic device can further execute S1004 to determine whether to determine the depth of the image through one image (through a monocular depth estimation algorithm) or through two images (through a binocular depth estimation algorithm).

[0278] S1004, determining whether the temperature is greater than a second temperature threshold.

[0279] Since electronic devices generate heat in the process of running image processing algorithms, the generated heat will cause the temperature of the electronic devices to rise. Electronic devices also generate energy consumption in the process of running image processing algorithms, and the higher the energy consumption, the more heat will be generated. Therefore, in order to control the temperature of electronic devices and reduce the heat generated by electronic devices in the process of blurring images as much as possible, electronic devices can choose a method with lower energy consumption to determine depth information on the basis of ensuring the accuracy of depth information. Usually, the energy consumption corresponding to a monocular depth estimation algorithm is different from the energy consumption corresponding to a binocular depth estimation algorithm. Therefore, the electronic device can also determine whether to use a monocular depth estimation algorithm or a binocular depth estimation algorithm based on a second temperature threshold.

[0280] Specifically, when the temperature of the electronic device is less than the second temperature threshold, it means that the temperature of the electronic device is relatively low at this time. Even if the electronic device uses an image processing algorithm with higher energy consumption to determine the depth information, the heat generated is not enough to cause the temperature of the electronic device to increase to a larger value, for example, it is not enough to make the temperature of the electronic device greater than the first temperature threshold and unable to perform blurring. Since the accuracy of the depth information that may be obtained by the image processing algorithm with higher energy consumption is higher than that of the image processing algorithm with lower energy consumption, when the temperature of the electronic device is less than the second temperature threshold, the electronic device selects the image processing algorithm with higher energy consumption to determine the depth information to improve the accuracy of the depth information. For example, the electronic device determines the depth information of the image through the second image processing algorithm.

[0281] When the temperature of the electronic device is greater than or equal to the second temperature threshold and less than or equal to the first temperature threshold, it means that the temperature of the electronic device is close to the first temperature threshold, and the electronic device needs to control the temperature of the electronic device to reduce the heat generated by the electronic device in the process of blurring the image as much as possible. Therefore, the electronic device can select an image processing algorithm with lower energy consumption. For example, the electronic device determines the depth information of the image through the first image processing algorithm.

[0282] Exemplarily, since different electronic devices consume different amounts of power (energy consumption) when running a monocular depth estimation algorithm and a binocular depth estimation algorithm, the electronic device may first obtain a first amount of power consumed when running the monocular depth estimation algorithm and a second amount of power consumed when running the binocular depth estimation algorithm. Then, based on the magnitude of the first amount of power and the second amount of power, one of the monocular depth estimation algorithm and the binocular depth estimation algorithm is used as the second image processing algorithm, and the other algorithm is used as the first image processing algorithm.

[0283] Among them, the electronic device consumes power and generates heat when running the image processing algorithm. And, the greater the power consumption, the more heat is generated. Therefore, the electronic device can characterize the heat generated when running the image processing algorithm according to the power consumed when running the image processing algorithm. In order to reduce the heat generated by the electronic device during the shooting process, the electronic device can select a depth calculation method with less power consumption on the basis of ensuring the accuracy of the depth information to reduce the heat generated by the electronic device during the shooting process. Since when the heat of the electronic device is greater than the first temperature threshold, the electronic device no longer processes the collected image, so when the electronic device continuously shoots, reducing the heat generated by the electronic device during the shooting process can extend the time when the temperature of the electronic device reaches the first temperature threshold, so that the time for the electronic device to process the image is extended, thereby meeting the user's photography needs.

[0284] Specifically, the electronic device first obtains a first amount of electricity consumed by blurring an image using a monocular depth estimation algorithm and a second amount of electricity consumed by blurring an image using a binocular depth estimation algorithm, and then determines the image processing algorithm corresponding to the larger amount of electricity between the first amount of electricity and the second amount of electricity as the second image processing algorithm.

[0285] For example, when the first power level is greater than the second power level, the first image processing algorithm is a binocular depth estimation algorithm, and the second image processing algorithm is a monocular depth estimation algorithm.

[0286] For another example, when the first power level is less than the second power level, the first image processing algorithm is a monocular depth estimation algorithm, and the second image processing algorithm is a binocular depth estimation algorithm.

[0287] For another example, when the first power is equal to the second power, the electronic device can choose to determine the depth information through a monocular depth estimation algorithm or a binocular depth estimation algorithm. Therefore, the electronic device can further choose which image processing algorithm to use according to the quality of the collected image. For example, take the example of the electronic device acquiring the first image and the second image. If the clarity detection result of the first image determined by the electronic device according to step S702 is consistent with the second image, it means that the clarity of the first image and the second image is consistent, and the electronic device can choose to determine the depth information through a binocular depth estimation algorithm to improve the accuracy of the depth information. If the clarity detection result of the first image is inconsistent with the second image, it means that one of the first image and the second image is a clear image and the other image is a blurred image. Therefore, in order to ensure the accuracy of the depth information, the electronic device can estimate the depth information of the clear image in the first image and the second image through a monocular depth estimation algorithm.

[0288] like Fig.10 As shown, the first temperature threshold in step S1002 and the second temperature threshold in step S1004 can be determined by the electronic device according to the power consumed by the electronic device for blurring the image and the temperature of the electronic device. Since the temperature of the electronic device will gradually rise in the process of collecting images, and if the electronic device collects images multiple times, the depth calculation methods used by the electronic device for the multiple collected images may be different, and different depth calculation methods may consume different amounts of power. Different amounts of power consumption lead to different amounts of heat generated, and different amounts of heat generated may lead to unpredictable changes in the temperature of the electronic device.

[0289] Therefore, in order to control the temperature increase of the electronic device, the electronic device can determine the first temperature threshold and the second temperature threshold corresponding to the N+1th blurred image according to the power consumed by the Nth blurred image and the temperature when the image was captured Nth time after blurring the image for the Nth time. Wherein, N is an integer greater than 1. When N is 1, it means that the electronic device is blurring the image for the first time at this time, and the electronic device can obtain the pre-stored first temperature threshold and the second temperature threshold. When N is greater than 1, the electronic device can obtain the first temperature threshold and the second temperature threshold determined when the image is blurred for the N-1th time.

[0290] The power consumed by blurring the image for the Nth time includes the power consumed in the process from when the electronic device starts to obtain the preview information to when the electronic device outputs the blurred image for the Nth time.

[0291] In a possible implementation, if the Nth blurred image consumes a large amount of power, for example, greater than the power threshold, then the Nth blurred image generates a large amount of heat, causing the temperature of the electronic device to rise faster. In order to extend the time when the temperature of the electronic device reaches the first temperature threshold, so that the time for the electronic device to process the image is extended, thereby meeting the user's photography needs, the electronic device can reduce the first temperature threshold and / or the second temperature threshold so that the electronic device can choose a lower energy consumption method to determine the depth information. For example, the electronic device can use the difference between the first temperature threshold corresponding to the Nth blurred image and the first difference as the first temperature threshold corresponding to the N+1th blurred image, and the electronic device can use the difference between the second temperature threshold corresponding to the Nth blurred image and the second difference as the second temperature threshold corresponding to the N+1th blurred image.

[0292] The above embodiment is described by taking the blurring process of a single frame image as an example. In the specific implementation process, the embodiment of the present application is also applicable to blurring a continuous multi-frame image (such as a video or a moving picture). For example, the electronic device can perform the above steps S701-S703 and steps S1001-S1004 on each frame of the continuous multi-frame image to select a depth calculation method suitable for the current shooting scene, thereby obtaining depth information through the depth calculation method. After obtaining the depth information of each frame image, blurring is performed on each frame image according to the depth information, so that the blurring effect of each frame image can be optimized, and the energy utilization rate of the electronic device can be improved.

[0293] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0294] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0295] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0296] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0297] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0298] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0299] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0300] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0301] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0302] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0303] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0304] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk), etc.

[0305] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0306] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: In the blur shooting mode, displaying a first user interface, wherein the first user interface is used to display a shooting preview screen, and the shooting preview screen includes one or more shooting objects; Processing multiple frames of images displayed on the shooting preview screen to obtain preview information corresponding to the multiple frames of images respectively; In response to the first user operation, an image corresponding to the preview screen is captured as a captured image, wherein the captured image includes the one or more captured objects; Preview information of the image corresponding to the first user operation and image information of the captured image in the multiple frames of images are obtained, and the captured image is blurred according to a depth calculation method to obtain a target image, wherein the depth calculation method is one selected from a calculation method based on a depth sensor and a calculation method based on an image processing algorithm according to one or more results after judging the preview information and / or the image information, and the one or more results after judging the preview information and / or the image information include one or more of the following: a result of judging whether the captured scene is a highlight scene based on brightness information in the preview information, a result of judging whether the captured scene is a high-speed motion scene based on a relative speed in the preview information, a result of judging whether the captured scene is a distant scene based on a scene distance detection result in the image information, and a result of judging whether the captured scene is a high-speed motion scene based on a clarity detection result in the image information. a result of determining whether the captured image is a blurred image, a result of determining whether the captured image contains a complex texture based on a texture detection result in the image information, a result of determining whether the captured image contains a captured subject based on a captured subject recognition result in the preview information, a result of determining whether the captured image contains a scene subject based on a scene subject detection result in the preview information, and a result of determining whether the number of faces in the preview information is greater than a number threshold based on the number of faces in the preview information; the calculation method based on the depth sensor is used to measure the distance between the electronic device and the one or more captured objects through a depth sensor, the distance between the electronic device and the one or more captured objects is used to determine the depth information of the captured image, and the calculation method based on the image processing algorithm is used to determine the depth information of the captured image based on information contained in the captured image; The target image is displayed on a second user interface, wherein a target photographic object in the target image is clear, a non-target photographic object in the target image is unclear, the target photographic object is any one or more of the one or more photographic objects, and the non-target photographic object is other than the target photographic object in the one or more photographic objects.

2. The method according to claim 1, characterized in that After obtaining the preview information of the image corresponding to the first user operation in the multiple frames of images and the image information of the captured image, and before blurring the captured image according to the depth calculation method to obtain the target image, the method further includes: In a case where it is determined according to the brightness information in the preview information that the shooting scene is the highlight scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the shooting scene is a shooting scene corresponding to the electronic device displaying the first user interface, and the highlight scene is used to indicate that the intensity of the ambient light of the shooting scene is greater than or equal to a brightness threshold; When it is determined according to the brightness information in the preview information that the shooting scene is not the highlight scene, when it is determined according to the relative speed in the preview information that the shooting scene is the high-speed motion scene, the depth calculation method is the calculation method based on the depth sensor, wherein the relative speed is the relative speed between the electronic device and the photographed object, and the high-speed motion scene is used to indicate that the relative speed is greater than a speed threshold; or, when it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information.

3. The method according to claim 2, characterized in that In the case where it is determined according to the brightness information in the preview information that the shooting scene is the highlight scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information includes: In a case where it is judged according to the brightness information in the preview information that the shooting scene is the highlighted scene, and when it is judged according to the scene distance detection result in the image information that the shooting scene is not the distant view, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the distant view is used to indicate that the distance between the shooting object in the shooting scene and the electronic device is greater than a distance threshold; or, when it is judged according to the scene distance detection result in the image information that the shooting scene is the distant view, the electronic device does not blur the captured image.

4. The method according to claim 3, characterized in that: When it is determined according to the scene distance detection result in the image information that the shooting scene is not the distant view, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, including: When it is judged according to the scene distance detection result in the image information that the captured scene is not the distant view, and when it is judged according to the clarity detection result in the image information that the captured image is not the blurred image, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information, wherein the blurred image is used to indicate that the clarity of the captured image is less than or equal to the clarity threshold; or, when it is judged according to the clarity detection result in the image information that the captured image is the blurred image, the electronic device does not blur the captured image.

5. The method according to claim 4, characterized in that When it is determined according to the clarity detection result in the image information that the captured image is not the blurred image, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the image information includes: When it is judged according to the clarity detection result in the image information that the captured image is not the blurred image, and when it is judged according to the texture detection result in the image information that the captured image contains the complex texture, the depth calculation method is the calculation method based on the depth sensor, wherein the captured image contains complex textures to indicate that the type and / or quantity of textures in the captured image is greater than the texture threshold; or, when it is judged according to the texture detection result in the image information that the captured image does not contain the complex texture, the depth calculation method is the calculation method based on the image processing algorithm.

6. The method according to any one of claims 2 to 5, characterized in that: When it is determined according to the brightness information in the preview information that the shooting scene is not the highlight scene, and when it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information, including: In a case where it is determined according to the brightness information in the preview information that the shooting scene is not the highlight scene, and in a case where it is determined according to the relative speed in the preview information that the shooting scene is not the high-speed motion scene, and when it is determined according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the shooting subject, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information, wherein the shooting subject is a shooting subject determined by the electronic device from the one or more shooting subjects in response to the second user operation, and the shooting subject is one or more of the target shooting subjects; or, when it is determined according to the shooting subject recognition result in the preview information that the one or more shooting objects include a human body and / or a human face, the depth calculation method is the calculation method based on the depth sensor; when it is determined according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the human body and the human face, the depth calculation method is the calculation method based on the image processing algorithm.

7. The method according to claim 6, characterized in that When it is determined according to the subject recognition result in the preview information that the one or more photographed objects do not include the subject, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information, including: When it is judged according to the shooting subject recognition result in the preview information that the one or more shooting objects do not include the shooting subject, when it is judged according to the scene subject detection result in the preview information that the one or more shooting objects do not include the scene subject, the depth calculation method is the calculation method based on the image processing algorithm; or, when it is judged according to the scene subject detection result in the preview information that the one or more shooting objects include the scene subject, the depth calculation method is determined from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information.

8. The method according to claim 7, characterized in that When it is determined according to the scene subject detection result in the preview information that the one or more photographed objects include the scene subject, determining the depth calculation method from the calculation method based on the depth sensor and the calculation method based on the image processing algorithm according to the preview information, including: In a case where it is judged according to the scene subject detection result in the preview information that the one or more photographed objects include the scene subject, and when it is judged according to the number of faces in the preview information that the number of faces is greater than the number threshold, the depth calculation method is the calculation method based on the image processing algorithm; or, when it is judged according to the number of faces in the preview information that the number of faces is less than or equal to the number threshold, the depth calculation method is the calculation method based on the depth sensor.

9. The method according to claim 2, characterized in that: After obtaining the preview information of the image corresponding to the first user operation in the multiple frames of images and the image information of the captured image, and before blurring the captured image according to the depth calculation method to obtain the target image, the method further includes: When the depth calculation method is the calculation method based on the image processing algorithm, obtaining the temperature of the electronic device; When the temperature is less than or equal to a first temperature threshold corresponding to blurring the captured image and greater than or equal to a second temperature threshold, determining that the calculation method based on the image processing algorithm is a calculation method based on the first image processing algorithm, the first temperature threshold is used to indicate that when the temperature of the electronic device is less than or equal to the first temperature threshold, the electronic device blurs the captured image, and the second temperature threshold is used to determine the image processing algorithm used by the electronic device to blur the captured image; When the temperature is less than the second temperature threshold, the calculation method based on the image processing algorithm is determined to be a calculation method based on a second image processing algorithm.

10. The method according to claim 9, characterized in that After the target image is displayed on the second user interface, the method further includes: The first temperature threshold and the second temperature threshold corresponding to blurring the captured image for the N+1th time are determined according to the power consumed by the electronic device when blurring the captured image for the Nth time and the temperature when the captured image is acquired for the Nth time, wherein N is an integer greater than 1, and the power consumed by blurring the captured image for the Nth time includes the power consumed in the process from the time the electronic device starts to obtain the preview information for the Nth time to the time the electronic device displays the target image for the Nth time.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

13. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 10.

14. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 10.

Images