Electronic device, method of controlling electronic device, and computer-readable storage medium

Abstract

An electronic device according to an embodiment of this application includes: an imaging module for taking a photograph of a subject and acquiring a camera image; and a processor for controlling the imaging module and acquiring the camera image.

Description

Technical Field

[0001] This invention relates to an electronic device, a method for controlling the electronic device, and a computer-readable storage medium. Background Technology

[0002] Traditionally, there have been electronic devices equipped with digital cameras, such as smartphones, which capture objects such as people.

[0003] Currently, a technique is widely used in which photographs with bokeh images of objects located in the foreground or background of the subject are artificially generated from images taken by cameras with deep depth of field (e.g., smartphone cameras), such as those taken with digital single-lens reflex (DSLR) cameras.

[0004] If a photo is taken using a camera with a deep depth of field (e.g., a smartphone camera), an image can be obtained that focuses from the near part to the far part. Therefore, image processing produces a bokeh image in which the part that needs attention becomes sharper, while the foreground and background of that part become blurred.

[0005] A problem in image processing for such bokeh images is that poor-quality frames, including, for example, images with strong motion blur or fragmented images, cannot yield correct results no matter how well they are processed. Therefore, if the algorithm uses a rolling mean, it consumes energy and stores garbage data. Summary of the Invention

[0006] This disclosure aims to solve at least one of the aforementioned technical problems. Therefore, this disclosure provides an electronic device and a method for controlling the electronic device.

[0007] This disclosure provides an electronic device including:

[0008] An imaging module is used to take photographs of the subject and acquire camera images; and

[0009] The processor is used to control the imaging module to acquire camera images and depth images, process the acquired camera images based on the depth images, and output the processed camera images based on the processed camera images.

[0010] Specifically, after acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, which includes an image of the subject; and

[0011] The processor determines whether the new frame camera image is good or bad; and

[0012] Specifically, when the processor determines that the new frame camera image is good, the processor calculates the depth information corresponding to the new frame camera image, and obtains the depth image corresponding to the camera image based on the calculated depth information;

[0013] On the other hand, when the processor determines that the new frame camera image is bad, the processor obtains the depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

[0014] In this electronic device,

[0015] The image processing of the camera image includes generating a bokeh image on the camera image.

[0016] In this electronic device,

[0017] The electronic device also includes an inertial measurement unit for detecting the acceleration of the electronic device;

[0018] The inertial measurement unit detects the acceleration of the electronic device during the time interval between a first moment and a second moment, wherein the first moment is the moment when the previous frame of the camera image is captured, and the second moment is the moment when the new frame of the camera image is captured; and

[0019] Specifically, when the maximum value of the acceleration of the electronic device detected by the inertial measurement unit is less than a preset first threshold, the processor determines that the new frame camera image is good;

[0020] On the other hand, when the maximum value of the acceleration of the electronic device detected by the inertial measurement unit is equal to or greater than the first threshold, the processor determines that the new frame camera image is bad.

[0021] In electronic devices,

[0022] The inertial measurement unit detects the acceleration along the x-axis, y-axis, and z-axis of the electronic device, respectively.

[0023] Specifically, when the maximum value of all accelerations of the electronic device detected by the inertial measurement unit in the x-axis, y-axis and z-axis directions is less than the first threshold, the processor determines that the new frame camera image is good;

[0024] On the other hand, when the maximum value of the acceleration of the electronic device detected by the inertial measurement unit in any of the x-axis, y-axis and z-axis directions is equal to or greater than the first threshold, the processor determines that the new frame camera image is bad.

[0025] In electronic devices,

[0026] The imaging module also includes an additional camera module for capturing the subject and acquiring a differential image with pixel values ​​that vary over time according to the captured image.

[0027] The additional camera module acquires the differential image during the time period from the first moment of capturing the previous frame camera image to the second moment of capturing the new frame camera image;

[0028] The processor obtains the sum of the maximum values ​​of each pixel in the differential image acquired by the additional camera module; and

[0029] Specifically, when the total maximum value of all pixels in the difference image is less than a preset second threshold, the processor determines that the new frame camera image is good;

[0030] On the other hand, when the sum of the maximum values ​​of all pixels in the differential image is equal to or greater than the second threshold, the processor determines that the new frame camera image is bad.

[0031] In electronic devices,

[0032] In this process, the imaging module captures the subject at the first moment and acquires the previous frame camera image; then, the imaging module captures the subject at the second moment and acquires the new frame camera image.

[0033] Specifically, for each pixel, the processor obtains the difference between the pixel value of that pixel in the previous frame camera image and the pixel value of the corresponding pixel in the new frame camera image.

[0034] Specifically, the processor obtains the sum of the differences between all pixels in the camera image; and

[0035] Specifically, when the total difference between all pixels is less than a preset third threshold, the processor determines that the new frame camera image is good.

[0036] On the other hand, when the total difference between all pixels is equal to or greater than the third threshold, the processor determines that the new frame camera image is bad.

[0037] In electronic devices,

[0038] The imaging module includes:

[0039] A first camera module, used to capture the subject and acquire first camera images; and

[0040] A second camera module is used to capture the subject and acquire images from the second camera; and

[0041] The processor acquires the camera image based on the first camera image and the second camera image; and

[0042] The processor calculates the depth information corresponding to the camera image and obtains the depth image corresponding to the camera image based on the calculated depth information.

[0043] In electronic devices,

[0044] The imaging module includes a first camera module for capturing the subject and acquiring camera images; and

[0045] The processor calculates the depth information corresponding to the camera image and obtains a depth image corresponding to the camera image based on the calculated depth information.

[0046] This disclosure provides a method for controlling an electronic device, comprising: an imaging module for capturing a photograph of a subject and acquiring a camera image; and a processor for controlling the imaging module to acquire the camera image and a depth image, processing the acquired camera image based on the depth image, and outputting a processed camera image based on the camera image acquired through the depth image processing.

[0047] The method includes:

[0048] After acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, which includes an image of the subject being photographed, and...

[0049] The processor determines whether the new frame camera image is good or bad, and

[0050] Specifically, the processor calculates the depth information corresponding to the new frame camera image, and when the processor determines that the new frame camera image is good, the processor obtains a depth image corresponding to the camera image based on the calculated depth information.

[0051] On the other hand, when the processor determines that the new frame camera image is bad, the processor obtains the depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

[0052] This disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein, when executed by a processor, the computer program implements a method for controlling an electronic device. The electronic device includes: an imaging module for capturing a photograph of a subject and acquiring a camera image; and a processor for controlling the imaging module to acquire the camera image and a depth image, processing the acquired camera image based on the depth image, and outputting a processed camera image based on the camera image acquired through the depth image processing.

[0053] The method includes:

[0054] After acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, which includes an image of the subject being photographed, and...

[0055] The processor determines whether the new frame camera image is good or bad, and

[0056] Specifically, the processor calculates the depth information corresponding to the new frame camera image, and when the processor determines that the new frame camera image is good, it obtains a depth image corresponding to the camera image based on the calculated depth information.

[0057] On the other hand, when the processor determines that the new frame camera image is bad, it acquires a depth image for the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image. Attached Figure Description

[0058] These and / or other aspects and advantages of the embodiments of this disclosure will become apparent and more readily understood from the following description taken with reference to the accompanying drawings, in which:

[0059] Figure 1 This is a diagram illustrating an example arrangement of an electronic device 100 and a subject 101 according to an embodiment of the present invention.

[0060] Figure 2 It is shown Figure 1 A diagram showing an example configuration of the electronic device 100.

[0061] Figure 3 It is shown Figure 1 and Figure 2 A diagram of another example of the imaging module 102 of the electronic device 100 shown.

[0062] Figure 4 It is shown Figure 1 and Figure 2 A diagram showing another example of the imaging module 102 of the electronic device 100 shown.

[0063] Figure 5 It is shown Figure 1 and Figure 2 The diagram illustrates an example of the processing flow of the electronic device 100 that captures a subject and outputs a processed camera image.

[0064] Figure 6 This is a diagram illustrating an example of the relationship between acquired camera image frames and time.

[0065] Figure 7 This indicates sequential execution. Figure 5The diagram shows the relationship between the multiple frames processed and the determined result.

[0066] Figure 8 It shows the... Figure 7 The execution of frames (M+1), (M+2), and (M+3) shown is as follows. Figure 5 A diagram showing a specific example of the processing. Detailed Implementation

[0067] Embodiments of this disclosure will now be described in detail, and examples of these embodiments will be illustrated in the accompanying drawings. Throughout this specification, the same or similar elements and elements having the same or similar functions are denoted by the same reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate this disclosure, and should not be construed as limiting this disclosure.

[0068] Figure 1 This is a diagram illustrating an example arrangement of an electronic device 100 and a subject 101 according to an embodiment of the present invention. Figure 2 It is shown Figure 1 A diagram showing an example configuration of the electronic device 100.

[0069] like Figure 1 and Figure 2 As shown, for example, electronic device 100 includes a first camera module 10, a second camera module 20, and an image signal processor 30. The image signal processor 30 controls the first camera module 10 and the second camera module 20, and processes camera image data acquired from the camera module 10.

[0070] exist Figure 1 and Figure 2 In the example, the imaging module 102 consists of a first camera module 10 and a second module 20. The imaging module 102 is defined as a module that captures at least one subject 101 and acquires camera images.

[0071] Therefore, as Figure 2 As shown, the imaging module 102 includes a first camera module 10 that captures a subject 101 and acquires a first camera image, and a second camera module 20 that captures a subject 101 and acquires a second camera image.

[0072] exist Figure 2 In the example, the depth information of the camera image is calculated based on the parallax of the first camera module 10 and the second module 20, and the depth image corresponding to the camera image is obtained based on the calculated depth information.

[0073] like Figure 2As shown, the first camera module 10 includes, for example, a main lens 10a capable of focusing on a subject, a main image sensor 10b for detecting images input via the main lens 10a, and a main image sensor driver 10c for driving the main image sensor 10b.

[0074] In addition, such as Figure 2 As shown, the first camera module 10 includes, for example, a focusing & OIS actuator 10f for actuating the main lens 10a, and a focusing & OIS driver 10e for driving the focusing & OIS actuator 10f.

[0075] For example( Figure 2 As shown), the first camera module 10 acquires the first camera image of the subject 101.

[0076] like Figure 2 As shown, the second camera module 20 includes, for example, a main lens 20a capable of focusing on a subject, a main image sensor 20b for detecting images input via the main lens 20a, and a main image sensor driver 20c for driving the main image sensor 20b.

[0077] In addition, such as Figure 2 As shown, the second camera module 20 includes, for example, a focusing & OIS actuator 20f for actuating the main lens 20a, and a focusing & OIS driver 20e for driving the focusing & OIS actuator 20f.

[0078] For example( Figure 2 As shown), the second camera module 20 acquires the second camera image of the subject 101.

[0079] In addition, such as Figure 2 As shown, for example, electronic device 100 includes a global navigation satellite system (GNSS) module 40, a wireless communication module 41, a codec 42, a speaker 43, a microphone 44, a display module 45, an input module 46, an inertial measurement unit (IMU) 47, a main processor 48, and a memory 49.

[0080] For example, GNSS module 40 measures the current position of electronic device 100.

[0081] For example, such as Figure 2 As shown, codec 42 performs encoding and decoding bidirectionally using a predetermined encoding / decoding method.

[0082] For example, speaker 43 outputs sound based on sound data decoded by codec 42.

[0083] For example, microphone 44 outputs sound data to codec 42 based on the input sound.

[0084] Display module 45 displays predefined information. Display module 45 is, for example, a touch panel.

[0085] Input module 46 receives user input (user operation). Input module 46 is included, for example, in a touch panel.

[0086] The IMU 47 detects, for example, the angular velocity and acceleration of electronic device 100.

[0087] The main processor 48 controls the Global Navigation Satellite System (GNSS) module 40, the wireless communication module 41, the codec 42, the speaker 43, the microphone 44, the display module 45, the input module 46, and the IMU 47.

[0088] exist Figure 2 In this example, processor 103 consists of image signal processor 30 and main processor 48. Processor 103 is defined as a controller that controls imaging module 102 and acquires camera images.

[0089] For example, processor 103 controls imaging module 102 to acquire camera images and depth images. Then, processor 103 outputs processed camera images (the camera image data includes bokeh image data) based on the camera images acquired through depth image processing.

[0090] The memory 49 stores the programs and data required by the image processor 30 to control the first camera module 10 and the second camera module 20, the acquired image data, and the programs and data required by the main processor 48 to control the electronic device 100.

[0091] For example, memory 49 includes a computer-readable storage medium thereon storing a computer program that, when executed by processor 103, implements a method for controlling electronic device 100. For example, the method includes: after acquiring a previous frame camera image, processor 103 controls imaging module 102 to acquire a new frame camera image containing an image of a subject; and processor 103 determines whether the new frame camera image is good or bad, wherein, when processor 103 determines the new frame camera image is good, processor 103 calculates depth information corresponding to the new frame camera image and acquires a depth image corresponding to the camera image based on the calculated depth information; on the other hand, when processor 103 determines the new frame camera image is bad, processor 103 acquires a depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

[0092] In this embodiment, the electronic device 100 with the above configuration is a mobile phone such as a smartphone, but it may also be other types of electronic devices (e.g., tablet computers and PDAs) that include an imaging module 102.

[0093] As mentioned above, in Figure 1 and Figure 2 In the example shown, the imaging module 102 includes a first camera module 10 and a second camera module 20. The first camera module 10 captures the subject 101 to obtain a first camera image. The second camera module 20 captures the subject 101 to obtain a second camera image.

[0094] exist Figure 1 and Figure 2 In the example shown, processor 103 acquires a camera image based on a first camera image and a second camera image. Then, processor 103 calculates depth information corresponding to the camera image, and acquires a depth image corresponding to the camera image based on the calculated depth information.

[0095] However, the configuration of the imaging module 102 of the electronic device 100 is not limited to... Figure 1 and Figure 2 The configuration shown.

[0096] For example, Figure 3 It is shown Figure 1 and Figure 2 A diagram of another example of the imaging module 102 of the electronic device 100 shown.

[0097] Therefore, instead Figure 1 and Figure 2 The example shown, for example, is as follows Figure 3 As shown, the imaging module 102 may consist of only the first camera module 10.

[0098] exist Figure 3 In the example shown, processor 103 calculates depth information corresponding to the camera image, and processor 103 obtains a depth image corresponding to the camera image based on the calculated depth information.

[0099] Next, Figure 4 It is shown Figure 1 and Figure 2 A diagram showing another example of the imaging module 102 of the electronic device 100.

[0100] like Figure 4 As shown, the imaging module 102 may also include an additional camera module 50.

[0101] Figure 4The attached camera module 50 shown captures the subject 101 and acquires a differential image with pixel values ​​resulting from changes in the captured image over time.

[0102] The additional camera module 50 is, for example, an event camera with shooting timing and FPS (frames per second) performance close to that of the imaging module 102.

[0103] exist Figure 4 In the example shown, as described below, based on the differential image acquired by the additional camera module 50, the processor 103 determines whether the new frame camera image is a good camera image (valid camera image) or a bad camera image (invalid camera image).

[0104] Bad frames with such poor camera images include images with strong motion blur or fragmented images. On the other hand, good frames with good camera images do not include images with strong motion blur or fragmented images.

[0105] [Examples of methods for controlling electronic devices]

[0106] Next, an example of a method for controlling an electronic device 100 having the above configuration and functions will be described.

[0107] Figure 5 It is shown Figure 1 and Figure 2 The diagram illustrates an example of the processing flow of the electronic device 100 capturing a subject and outputting a processed camera image. Figure 6 This is a diagram illustrating an example of the relationship between acquired camera image frames and time.

[0108] Here, as Figure 6 As shown, the new frame (denoted as N) is the latest frame captured at the time of shooting. In addition, the previous frame (denoted as N-1) is the frame that immediately precedes the new frame (N).

[0109] exist Figure 5 In the example shown, the following will describe the method used for... Figure 6 The method for controlling image processing of the new frame camera image (N) shown.

[0110] Here, for example, such as Figure 5 As shown in step S1, the imaging module 102 at least captures the subject 101 and acquires a camera image.

[0111] Next, as Figure 5 As shown in step S2, the processor 103 controls the imaging module 102 to acquire a new frame camera image (N) including an image of the subject 101.

[0112] Next, as Figure 5As shown in step S3, the processor 103 determines whether the acquired new frame camera image (N) is good or bad.

[0113] As mentioned above, bad frames with bad camera images include images with strong motion blur or fragmented images. On the other hand, good frames with good camera images do not include images with strong motion blur or fragmented images.

[0114] Next, when processor 103 is in Figure 5 When the processor 103 determines that the new frame camera image (N) is good in step S3, Figure 5 In step S4, depth information corresponding to the camera image is calculated. Furthermore, in... Figure 5 In step S5, the processor 103 acquires a depth image corresponding to the camera image based on the calculated depth information. Then, in Figure 5 In step S6, the processor 103 processes the camera image based on the acquired depth image.

[0115] On the other hand, when processor 103 is in Figure 5 When it is determined in step S3 that the new frame camera image (N) is bad, the processor 103... Figure 5 In step S4, depth information corresponding to the camera image is not calculated. Then, in Figure 5 In step S5, the processor 103 acquires a depth image corresponding to the previous frame (N-1) camera image before the new frame (N). Then, in Figure 5 In step S6, the processor 103 processes the camera image based on the acquired depth image.

[0116] The image processing of the camera image includes: generating a bokeh image on the camera image.

[0117] Then, in Figure 5 In step S7, the processor 103 outputs the processed camera image (the camera image data includes bokeh image data).

[0118] Methods for generating depth images generally employ monocular, stereo, or TOF (time-of-flight) data. However, this invention can also be applied to other depth image generation methods.

[0119] Here, the following will describe in Figure 5 Some specific examples of the above-described good / bad determination method for the frame shown in step S3.

[0120] [First example of a method for determining the outcome]

[0121] As mentioned above, in Figure 2In the example shown, electronic device 100 includes an inertial measurement unit 47 for detecting the acceleration of electronic device 100.

[0122] Therefore, in the first example of this determination method, firstly, the inertial measurement unit 47 detects the acceleration of the electronic device 100 during the time period from the first moment ta of capturing the previous frame (N-1) camera image to the second moment tb of capturing the new frame camera image (N).

[0123] Next, the processor 103 determines whether the acquired new frame camera image is good or bad.

[0124] That is, more specifically, when the maximum value of the acceleration of the electronic device 100 detected by the inertial measurement unit 47 is less than a preset first threshold, the processor 103 determines that a new frame camera image (N) is desirable.

[0125] On the other hand, when the maximum value of the acceleration of the electronic device 100 detected by the inertial measurement unit 47 is equal to or greater than the first threshold, the processor 103 determines that the new frame camera image (N) is bad.

[0126] Additionally, it should be noted that when the new frame (N) camera image is a camera image of the frame first acquired by the subject 101, the processor 103 determines that the new frame (N) camera image is preferable, regardless of the maximum value of the acceleration of the electronic device 100 detected by the inertial measurement unit 47.

[0127] Additionally, it should be noted that the maximum value of acceleration is an absolute value. This method of determination focuses only on the amount of change in acceleration, not the direction of that change.

[0128] Additionally, it should be noted that the total number of pixels in the previous frame (N-1) of the camera image is the same as the total number of pixels in the new frame (N) of the camera image.

[0129] Here, in the above determination method, more specifically, the inertial measurement unit 47 can detect the acceleration of the electronic device 100 in the x-axis direction, y-axis direction and z-axis direction respectively.

[0130] In this case, when the maximum value of all accelerations of the electronic device detected by the inertial measurement unit 47 in the x-axis direction, y-axis direction and z-axis direction is less than the first threshold, the processor 103 determines that a new frame (N) camera image is desirable.

[0131] On the other hand, when the maximum value of the acceleration of the electronic device 100 detected by the inertial measurement unit 47 in any of the x-axis, y-axis and z-axis directions is equal to or greater than the first threshold, the processor 103 determines that the new frame (N) camera image is bad.

[0132] [Second example of a determination method]

[0133] Optionally, in the above Figure 4 In the example shown, the imaging module 102 includes the additional camera module 50, which captures the subject 101 and acquires a differential image having pixel values ​​corresponding to changes in the captured image over time.

[0134] In a second example of this determining method, the additional camera module 50 acquires differential images during a time period from a first moment ta of capturing the previous frame (N-1) camera image to a second moment tb of capturing the new frame (N) camera image.

[0135] Then, the processor 103 obtains the sum of the maximum values ​​of each pixel in the differential image acquired by the additional camera module 50.

[0136] When the total value of the maximum value of all pixels in the differential image is less than a preset second threshold (when the image fluctuation is small), the processor 103 determines that the new frame (N) camera image is good.

[0137] On the other hand, when the total value of the maximum value of all pixels in the differential image is equal to or greater than the second threshold (when the image fluctuates greatly), the processor 103 determines that the new frame (N) camera image is bad.

[0138] Furthermore, it should be noted that when the new frame (N) camera image is obtained by capturing the camera image of the frame first acquired by the subject 101, the processor 103 determines that the new frame (N) camera image is good, regardless of the total value of the maximum value of all pixels in the differential image.

[0139] Additionally, it should be noted that the maximum value of each pixel in the difference image is an absolute value. This determination method focuses only on the amount of change in the captured image, not the direction of that change.

[0140] [Third example of a method for determining the outcome]

[0141] Optionally, the difference between the pixel values ​​of the pixels in the previous frame (N-1) camera image and the pixel values ​​of the pixels in the new frame (N) camera image can be used to determine whether the frame is good or bad.

[0142] In a third example of this determination method, firstly, the imaging module 102 acquires the previous frame (N-1) camera image by capturing the subject 101 at a first moment ta.

[0143] Subsequently, the imaging module 102 acquires a new frame (N) camera image by capturing the subject 101 at the second time tb.

[0144] Next, the processor 103 determines whether the acquired new frame (N) camera image is good or bad.

[0145] More specifically, for each corresponding pixel, the processor 103 obtains the difference between the pixel value of the pixel in the previous frame (N-1) camera image and the pixel value of the corresponding pixel in the new frame (N) camera image.

[0146] Next, the processor 103 obtains the total difference of all pixels in the camera image.

[0147] Next, when the total difference of all pixels is less than a preset third threshold, the processor 103 determines that the new frame (N) camera image is acceptable.

[0148] On the other hand, when the total difference of all pixels is equal to or greater than the third threshold, the processor 103 determines that the new frame (N) camera image is bad.

[0149] Additionally, it should be noted that when the new frame (N) camera image is a camera image obtained by first capturing a frame of the subject 100, the processor 103 determines that the new frame (N) camera image is good, regardless of the total difference of all pixels.

[0150] Furthermore, it should be noted that the difference is an absolute value. In this method of determination, only the amount of change in the difference is considered, not the direction of that change.

[0151] Next, an example will be described to illustrate the relationship between the determination results of multiple frames determined by the above determination method and the frame image processing of multiple frames.

[0152] Figure 7 This indicates sequential execution. Figure 5 The diagram shows the relationship between the multiple frames processed and the determined result. Figure 8 It shows the... Figure 7 The execution of frames (M+1), (M+2), and (M+3) shown is as follows. Figure 5 A diagram showing a specific example of the processing.

[0153] exist Figure 7 and Figure 8 In this context, frame (M), frame (M+1), frame (M+2), and frame (M+3) represent any consecutive frames.

[0154] like Figure 7 and Figure 8 As shown, frame (M+2) is identified as a bad frame, and the depth calculation process is skipped during the processing of frame (M+2). Then, the depth image of the frame (M+1) preceding frame (M+2) is used as the depth image of frame (M+1).

[0155] In this way, for a bad new frame, some processing is skipped, and the depth image of the previous frame is used. Therefore, calculating the depth image for the bad new frame is skipped. However, the input image for this bad new frame is used as is. Thus, no frame dropping occurs.

[0156] As described above, the electronic device according to the present invention, by skipping some processing for bad frames, can obtain a margin in the processor's (chip's) computing power and allowable temperature. Therefore, when a good frame is input, the processor's processing can be executed at a higher speed.

[0157] In the description of embodiments of this disclosure, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to the directions or positions described or shown in the accompanying drawings. These relative terms are used only to simplify the description of this disclosure and do not indicate or imply that the mentioned devices or elements must have a particular orientation, or must be constructed or operated in a particular orientation. Therefore, these terms should not be construed as limiting this disclosure.

[0158] Furthermore, terms such as “first” and “second” are used in the description herein not to indicate or imply relative importance or significance, or to imply the number of technical features indicated. Therefore, a feature defined as “first” and “second” may include one or more of those features. In the description of this disclosure, “multiple” means “two or more” unless otherwise stated.

[0159] In the description of embodiments of this disclosure, unless otherwise specified or limited, the terms “installation,” “connection,” “coupling,” etc., are broad and can be, for example, a fixed connection, a detachable connection, or an integral connection; they can also be a mechanical connection or an electrical connection; they can also be a direct connection or an indirect connection through an intermediate structure; or they can be an internal connection between two elements that can be understood by those skilled in the art based on the specific circumstances.

[0160] In embodiments of this disclosure, unless otherwise specified or limited, the structure of the first feature "on" or "below" the second feature may include embodiments in which the first feature and the second feature are in direct contact, and may also include embodiments in which the first feature and the second feature are not in direct contact with each other, but are in contact through an additional feature formed between them. Furthermore, the first feature "above," "on top of," or "on the second feature" may include embodiments in which the first feature is orthogonal or obliquely positioned "above," "on top of," or simply means that the height of the first feature is greater than the height of the second feature; while the first feature "below," "under," or "at the bottom of," the second feature may include embodiments in which the first feature is orthogonal or obliquely positioned "below," "under," or "at the bottom of," or simply means that the height of the first feature is less than the height of the second feature.

[0161] Various embodiments and examples have been provided in the foregoing description to implement different structures of this disclosure. To simplify this disclosure, certain elements and arrangements have been described above. However, these elements and arrangements are merely examples and are not intended to limit this disclosure. Furthermore, reference numerals and / or reference letters may be repeated in different examples of this disclosure. Such repetition is for simplification and clarity purposes and does not indicate a relationship between different embodiments and / or arrangements. Additionally, examples of different processes and materials are provided in this disclosure. However, those skilled in the art will understand that other processes and / or materials may also be applied.

[0162] Throughout this specification, references to "embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" mean that a particular feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this disclosure. Therefore, the appearance of these phrases throughout this specification does not necessarily refer to the same embodiment or example of this disclosure. Furthermore, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0163] Any process or method described in the flowchart or otherwise herein can be understood as including one or more modules, code segments, or portions of code comprising executable instructions for implementing specific logical functions or steps in the process, and the scope of the preferred embodiments of this disclosure includes other implementations, which those skilled in the art will understand may be implemented in a different order than that shown or discussed, including in substantially the same order or in the reverse order.

[0164] The logic and / or steps otherwise described herein or shown in the flowcharts, such as a specific list of executable instructions for implementing logical functions, may be embodied in, or used in conjunction with, any computer-readable medium intended for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a processor-included system, or other system capable of obtaining instructions from an instruction execution system, apparatus, or device that executes instructions). For the purposes of this specification, "computer-readable medium" can be any means that adaptively includes, stores, transmits, propagates, or transports programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer accessories (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because, for example, when a program needs to be obtained electronically, it can be optically scanned onto paper or other suitable media, then edited, decrypted or processed by other suitable methods, and then the program can be stored in computer memory.

[0165] It should be understood that each part of this disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory and executed by an appropriate instruction execution system. For example, if implemented in hardware, similarly in another embodiment, these steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing data signals, application-specific integrated circuits (ASICs) having appropriate combinations of logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0166] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented using program instructions and related hardware. These programs can be stored in a computer-readable storage medium, and when run on a computer, they include one or a combination of the steps in the method embodiments of this disclosure.

[0167] Furthermore, the various functional units of the embodiments of this disclosure can be integrated into a processing module, or these units can be separate physical entities, or two or more units can be integrated into a processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as an independent product, the integrated module can be stored in a computer-readable storage medium.

[0168] The aforementioned storage media can be read-only memory, disk, CD, etc.

[0169] Although embodiments of the present disclosure have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the present disclosure. Changes, modifications, substitutions, and variations may be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. An electronic device, comprising: The imaging module is used to take photos of the subject and acquire camera images; as well as The processor is used to control the imaging module to acquire camera images and depth images, process the acquired camera images based on the depth images, and output the processed camera images based on the processed camera images. Wherein, after acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, the new frame camera image containing an image of the subject; and The processor determines whether the new frame camera image is good or bad; and When the processor determines that the new frame camera image is good, the processor calculates the depth information corresponding to the new frame camera image, and obtains the depth image corresponding to the camera image based on the calculated depth information; On the other hand, when the processor determines that the new frame camera image is bad, the processor obtains the depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

2. The electronic device according to claim 1, wherein, Image processing of the camera image includes generating a bokeh image on the camera image.

3. The electronic device according to claim 1, further comprising: An inertial measurement unit is used to detect the acceleration of the electronic device; The inertial measurement unit detects the acceleration of the electronic device during the time interval between a first moment and a second moment, wherein the first moment is the moment when the previous frame of the camera image is captured, and the second moment is the moment when the new frame of the camera image is captured; and Wherein, when the maximum value of the acceleration of the electronic device detected by the inertial measurement unit is less than a preset first threshold, the processor determines that the new frame camera image is good; On the other hand, when the maximum value of the acceleration of the electronic device detected by the inertial measurement unit is equal to or greater than the first threshold, the processor determines that the new frame camera image is bad.

4. The electronic device according to claim 3, in, The inertial measurement unit detects the acceleration of the electronic device in the x-axis, y-axis and z-axis directions, respectively. and Wherein, when the maximum value of all accelerations of the electronic device detected by the inertial measurement unit in the x-axis direction, the y-axis direction, and the z-axis direction is less than the first threshold, the processor determines that the new frame camera image is good; On the other hand, when the maximum value of the acceleration of the electronic device in any of the x-axis, y-axis and z-axis directions detected by the inertial measurement unit is equal to or greater than the first threshold, the processor determines that the new frame camera image is bad.

5. The electronic device according to claim 1, in, The imaging module also includes an additional camera module for capturing the subject and acquiring a differential image having pixel values ​​that change over time according to the captured image. The additional camera module acquires the differential image during a time period from the first moment of capturing the previous frame camera image to the second moment of capturing the new frame camera image; The processor acquires the sum of the maximum values ​​of each pixel in the differential image acquired by the additional camera module; and Wherein, when the total value of the maximum value of all pixels in the difference image is less than a preset second threshold, the processor determines that the new frame camera image is good; On the other hand, when the sum of the maximum values ​​of all pixels in the difference image is equal to or greater than the second threshold, the processor determines that the new frame camera image is bad.

6. The electronic device according to claim 1, wherein, The imaging module captures the subject at a first moment and acquires the previous frame camera image; then, the imaging module captures the subject at a second moment and acquires the new frame camera image. Specifically, for each pixel, the processor obtains the difference between the pixel value of that pixel in the previous frame camera image and the pixel value of the corresponding pixel in the new frame camera image; The processor obtains the sum of the differences between all pixels in the camera image; and Wherein, when the total difference between all the pixels is less than a preset third threshold, the processor determines that the new frame camera image is good; On the other hand, when the total difference between all the pixels is equal to or greater than the third threshold, the processor determines that the new frame camera image is bad.

7. The electronic device according to claim 1, in, The imaging module includes: A first camera module, used to capture the subject and acquire first camera images; and A second camera module is used to capture the subject and acquire images from the second camera; and The processor acquires the camera image based on the first camera image and the second camera image; and The processor calculates the depth information corresponding to the camera image, and obtains a depth image corresponding to the camera image based on the calculated depth information.

8. The electronic device according to claim 1, wherein, The imaging module includes a first camera module for capturing the subject and acquiring the camera image; as well as The processor calculates depth information corresponding to the camera image and obtains a depth image corresponding to the camera image based on the calculated depth information.

9. A method for controlling an electronic device, the electronic device comprising: The imaging module is used to take photos of the subject and acquire camera images; And a processor, used to control the imaging module to acquire camera images and depth images, process the acquired camera images based on the depth images, and output the processed camera images based on the camera images acquired through the depth image processing; The method includes: After acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, the new frame camera image containing the image of the subject; and The processor determines whether the new frame camera image is good or bad, and The processor calculates depth information corresponding to the new frame camera image, and when the processor determines that the new frame camera image is good, the processor obtains a depth image corresponding to the camera image based on the calculated depth information. On the other hand, when the processor determines that the new frame camera image is bad, the processor obtains the depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, the computer program implements a method for controlling an electronic device, the electronic device comprising: an imaging module for capturing a photograph of a subject and acquiring a camera image; and a processor for controlling the imaging module to acquire the camera image and a depth image, processing the acquired camera image based on the depth image, and outputting a processed camera image based on the camera image acquired through the depth image processing; and The method includes: After acquiring the previous frame camera image, the processor controls the imaging module to acquire a new frame camera image, the new frame camera image containing the image of the subject; and The processor determines whether the new frame camera image is good or bad; and The processor calculates depth information corresponding to the new frame camera image, and when the processor determines that the new frame camera image is good, it obtains a depth image corresponding to the camera image based on the calculated depth information. On the other hand, when the processor determines that the new frame camera image is bad, the processor obtains the depth image of the new frame camera image based on the depth image corresponding to the previous frame camera image, without calculating the depth information corresponding to the new frame camera image.

Images