Motion snapshot method and electronic device
By detecting moving objects in real time in electronic devices and reducing exposure time, combining HDR and multi-frame image fusion technology, the problems of slow imaging speed and poor image quality when shooting moving objects are solved, and water ripples are avoided through flicker light source detection technology. Achieve more efficient shooting effects.
Patent Information
- Application Number
- PCT/CN2024/083747
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-03-26
- Publication Date
- 2024-11-28
AI Technical Summary
When shooting moving objects, existing electronic devices have slow imaging speed, slow shutter hysteresis and poor image quality, especially when water ripple problems are easily occurring under flickering light sources.
By detecting moving objects during preview and reducing exposure time, combining HDR and multi-frame image fusion technology to improve image quality; in scintillation light source detection, the exposure time of the first camera is detected and adjusted to avoid water ripple.
It improves the imaging speed and image quality when shooting moving objects, reduces shutter hysteresis, and effectively avoids water ripple under the flickering light source.
Smart Images

Figure CN2024083747_28112024_PF_FP_ABST
Abstract
Description
Motion capture method and electronic equipment
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310590214.8 and application name “Motion capture method and electronic device”, and the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310595470.6 and application name “Flickering light source detection method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of electronic technology, and in particular to a motion capture method and electronic equipment. Background Art
[0003] With the advancement of electronic technology, electronic devices such as mobile phones and tablets are generally equipped with cameras. Users can use these devices to capture various objects in motion and record their wonderful moments in motion. However, capturing moving objects effectively, especially clear ones, often requires complex exposure settings. Most users have little understanding of camera exposure, making this operation difficult.
[0004] Summary of the Invention
[0005] The present application provides a motion capture method and electronic equipment.
[0006] In the first aspect, the present application provides a motion capture method, which may include: an electronic device starts a first camera, displays a preview image on a display screen, and the preview image comes from an image captured by the first camera, and the electronic device caches the latest captured multiple frames of images (also called cached frames). During the preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, the exposure time of the first camera is reduced during the preview, for example, the exposure time is reduced to shorter than the light energy cycle. The electronic device can detect a photo operation, such as a shutter button being pressed. In response to the photo operation, the electronic device saves the photo, which may come from the cached frame.
[0007] In the first aspect, activation of the first camera may be triggered by the user opening a camera application. Upon activation, the first camera's exposure time may be a first duration, which is relatively long and unsuitable for capturing moving objects, as it can easily produce motion artifacts. For example, the first duration may be greater than the light energy period and an integer multiple of the light energy period, such as 50ms.
[0008] With the method of the first aspect, the electronic device reduces the exposure time and enters short-exposure mode when it detects a moving object during preview, rather than reducing the exposure time only after the user presses the shutter button. This cached preview frame has the reduced exposure effect. Once a photo operation is subsequently detected, the electronic device can obtain the photo to be saved based on the cached preview frame, without having to wait for the short exposure time to take effect before outputting the image. This alleviates issues such as slow imaging speed and shutter lag.
[0009] In conjunction with the first aspect, in some embodiments, when reducing the exposure time of the first camera, the electronic device may further detect the motion speed (also a relative speed) of the moving object in the preview image and control the exposure reduction based on the motion speed: the faster the motion speed, the greater the exposure time is reduced, and the shorter the reduced exposure time. The motion speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the motion speed.
[0010] In conjunction with the first aspect, in some embodiments, if the preview is a high dynamic range (HDR) preview, the method may further include: applying different exposure values (EV) at different frame times during the preview to cache multiple frames of images with different brightnesses. Applying different exposure values (EV) at different frame times may specifically include: using the same exposure time but different exposure gains at different frame times, or using different exposure times and different exposure gains at different frame times, or using the same exposure gain but different exposure times at different frame times.
[0011] In combination with the first aspect, in some embodiments, if the preview is an HDR preview, then before the electronic device saves the photo, the method may further include: the electronic device uses a reference frame and multiple frames of images with different exposure values EV to perform HDR fusion processing to obtain an HDR-fused image frame; the reference frame is the image frame with the highest clarity in the preview buffer, and the multiple frames of images with different exposure values EV are selected from cached image frames other than the reference frame. In this way, the electronic device saving the photo may specifically include: saving the HDR-fused image frame as a photo. The photo comes from the latest multi-frame image captured in the cache, which specifically means here that the photo comes from the reference frame and multiple frames of images with different exposure values EV, and is generated by the fusion of the reference frame and multiple frames of images with different exposure values EV.
[0012] In combination with the first aspect, in some embodiments, if the preview is not an HDR preview, before the electronic device saves the photo, the method may further include: the electronic device uses the reference frame and the multi-frame preferred frame to perform multi-frame fusion noise reduction processing to obtain the image frame after the multi-frame fusion noise reduction processing; the reference frame is the image frame with the highest clarity in the preview buffer area, and the multi-frame preferred frame is selected from the cached image frames other than the reference frame. In this way, the electronic device saves the photo, which may specifically include: saving the image frame after the multi-frame fusion noise reduction processing as a photo. The photo comes from the latest multi-frame image captured in the cache, which specifically means here: the photo comes from the reference frame and the multi-frame preferred frame, and is generated by the fusion of the reference frame and the multi-frame preferred frame.
[0013] In this way, when the user intends to take a photo, the electronic device can quickly take out multiple frames of images used to generate the photo from the cache frame and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality; moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented.
[0014] In conjunction with the first aspect, in some embodiments, during preview, the electronic device may further activate a flickering light source detection device to detect a flickering light source. Reducing the exposure time of the first camera during preview may specifically include: if a flickering light source is detected during preview, the electronic device controls the reduced exposure time to be an integer multiple of the light energy cycle, with the minimum exposure time set to one light energy cycle; if no flickering light source is detected during preview, the electronic device reduces the exposure time to less than the light energy cycle.
[0015] The flicker light source detection device can include an ambient light sensor (ALS) and a multispectral sensor. The ALS operates at a high frequency F. According to Shannon's theorem, when F is greater than twice the light energy period F', the energy variation of the flicker light source can be determined by reading the ALS data, thereby confirming the presence of a flicker light source in the current environment. The multispectral sensor can directly read the energy variation information of the flicker light source.
[0016] In combination with the first aspect, in some embodiments, the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.
[0017] The second camera is not specifically introduced for flickering light source detection, and the images it captures will not be displayed. It is usually used as a depth of field camera, a wide-angle camera, or a macro camera. Its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms. As a result, the images captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of working frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple working frequencies. Taking the two working frequencies of 50Hz and 60Hz as an example, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).
[0018] In combination with the first aspect, in some embodiments, when the flickering light source detection device is a second camera, the electronic device starts the flickering light source detection device to perform flickering light source detection, which may specifically include: the electronic device performs image recognition on the image captured by the second camera, and determines whether the image content contains water ripple features. If so, it is determined that a flickering light source is detected; the detection result is used to control the exposure time of the first camera.
[0019] In combination with the first aspect, in some embodiments, before the electronic device starts the flickering light source detection device to perform flickering light source detection, the electronic device can also first determine that the ambient brightness is neither an overly bright environment nor an overly dark environment, wherein the brightness of the overly bright environment is higher than the first high brightness, such as 10,000 lux, and the brightness of the overly dark environment is lower than the first low brightness, such as 50 Lux.
[0020] In conjunction with the first aspect, in some embodiments, the exposure duration of the first camera may be set by an automatic exposure control (AEC); the electronic device determining that the ambient brightness is neither too bright nor too dark may specifically include: determining that the ambient brightness is not too bright based on the exposure duration of the first camera being greater than a third duration (e.g., 20ms), and determining that the ambient brightness is not too dark based on the exposure duration of the first camera being less than a fourth duration (200ms), where the third duration is less than the fourth duration. The third duration and the fourth duration are recorded in an exposure table of the first camera, for example, the third duration belongs to several shortest exposure times corresponding to high EV in the exposure table, and the fourth duration belongs to several longest exposure times corresponding to low EV in the exposure table.
[0021] In a second aspect, the present application provides a method for detecting a flickering light source, which may include: an electronic device activating a first camera and displaying an image captured by the first camera on a display screen; the electronic device also activating a second camera and setting the exposure time of the second camera to a second duration, where the second duration is shorter than a light energy period. The electronic device may perform image recognition on the image captured by the second camera to determine whether the image content contains water ripple features. If so, it is considered that a flickering light source exists in the shooting environment, and the exposure time of the first camera is controlled to be an integer multiple of the light energy period. The image captured by the second camera will not be displayed and is only used for flickering light source detection.
[0022] Through the method of the second aspect, the electronic device can control the exposure of the first camera based on the flickering light source detection result, and control the exposure time of the first camera to be an integer multiple of the light energy cycle to avoid water ripples in the preview image provided by the first camera.
[0023] For example, when the shooting mode is switched from the normal shooting mode to the motion capture mode, when reducing the exposure time to alleviate the motion ghosting problem, the electronic device can specifically reduce the exposure time of the first camera to a small integer multiple such as one or two times the light energy period, that is, the reduced exposure time is controlled to be an integer multiple of the light energy period, so as to simultaneously avoid the problem of water ripples in the imaging picture under the flickering light source.
[0024] In combination with the second aspect, in some embodiments, if the image captured by the second camera contains water ripple features, when reducing the exposure time of the first camera, the electronic device can specifically control the reduced exposure time of the first camera to one light energy period.
[0025] In combination with the second aspect, in some embodiments, when starting the first camera, the electronic device can set the exposure time of the first camera to a first duration, where the first duration is greater than the light energy period and is an integer multiple of the light energy period, for example 50ms. This can prevent the user from seeing the generation of water ripples at the beginning of the camera startup.
[0026] In conjunction with the second aspect, in some embodiments, before the electronic device activates the first camera, the process further includes: detecting a user operation of opening a camera application. That is, the activation of the first camera may be triggered by the user opening the camera application. The first camera may be a primary camera, and the second camera may not be a primary camera.
[0027] In combination with the second aspect, in some embodiments, the first camera can be multiple cameras. In this case, the preview image displayed by the camera application on the display screen is composed of images simultaneously captured by these multiple cameras, so as to provide users with the function of taking photos or recording videos with multiple cameras.
[0028] In conjunction with the second aspect, in some embodiments, the second duration can be further set to meet the exposure time required to detect flickering light sources at various power frequencies, i.e., to be shorter than the minimum light energy period among the light energy periods of these various power frequencies. For example, using 50Hz and 60Hz power frequencies, the second duration can be set to 7ms, which is shorter than the light energy period of a 60Hz light source (approximately 8.3 milliseconds).
[0029] In combination with the second aspect, in some embodiments, before the electronic device starts the second camera, it also includes: the electronic device determines that the ambient brightness is neither too bright nor too dark, wherein the brightness of the overbright environment is higher than the first high brightness, such as 10,000 lux, and the brightness of the overdark environment is lower than the first low brightness, such as 50 Lux.
[0030] In conjunction with the second aspect, in some embodiments, the exposure duration of the first camera may be set by an automatic exposure control (AEC); the electronic device determining that the ambient brightness is neither too bright nor too dark may specifically include: determining that the ambient brightness is not too bright based on the exposure duration of the first camera being greater than a third duration (e.g., 20ms), and determining that the ambient brightness is not too dark based on the exposure duration of the first camera being less than a fourth duration (200ms), where the third duration is less than the fourth duration. The third duration and the fourth duration are recorded in an exposure table for the first camera, for example, the third duration is one of several shortest exposure times corresponding to high EV in the exposure table, and the fourth duration is one of several longest exposure times corresponding to low EV in the exposure table.
[0031] In conjunction with the second aspect, in some embodiments, the first camera and the second camera are not fixed. The following uses the first camera as the primary camera as an example to illustrate various switching situations of the first camera.
[0032] The electronic device may also include a telephoto camera. In this case, after the electronic device displays the preview image on the display screen, the method may further include: the electronic device detects a user operation of increasing the zoom ratio to the zoom range of the telephoto camera, and switches the first camera from a main camera to a telephoto camera.
[0033] The electronic device may also include a wide-angle camera. In this case, after the electronic device displays the preview image on the display screen, the method may further include: the electronic device detects a user operation of reducing the zoom magnification to the zoom range of the wide-angle camera, and switches the first camera from a main camera to the wide-angle camera.
[0034] The main camera may specifically include a rear main camera and a front main camera. In this case, after the electronic device displays the preview image on the display screen, the method may further include: the electronic device detecting a user operation of flipping the camera, switching the first camera from the rear main camera to the front main camera, or switching the first camera from the front main camera to the rear main camera.
[0035] After the electronic device displays the preview image on the display screen, the method further includes: the electronic device detects a user operation of switching the shooting mode to multi-channel photo taking or video recording, starts one or more cameras other than the main camera, and also adds them as the first camera.
[0036] In conjunction with the second aspect, in some embodiments, when the first camera is switched, the second camera can be selected from the remaining cameras. The second camera can choose to switch to the original first camera, especially for the case where the electronic device only has two cameras, there is no need to start a new camera, and the response speed is faster. Of course, in actual applications, it is also possible to consider switching the second camera to a camera with lower power consumption based on issues such as power consumption, and the embodiments of the present application are not limited to this. The second camera can also be fixed, such as a depth of field camera. The image data collected by this type of camera does not participate in the preview image display, but is only used as auxiliary data (such as depth of field data) for other cameras, such as depth data of a person's face, and therefore will not be occupied by the first camera due to changes in the shooting mode.
[0037] In a third aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
[0038] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
[0039] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
[0040] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when the above-mentioned computer program product is run on an electronic device, enables the above-mentioned electronic device to perform the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
[0041] It is understandable that the electronic device provided in the third aspect, the chip system provided in the fourth aspect, the computer storage medium provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to perform the methods provided in this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] FIG1 shows a motion capture method provided in Example 1 of the present application;
[0043] FIG2 exemplarily shows the change in image quality before and after exposure reduction;
[0044] FIG3 shows a motion capture method provided in Example 2 of the present application;
[0045] FIG4 shows a motion capture method provided in Example 3 of the present application;
[0046] FIG5 exemplarily shows the working principle of a rolling shutter;
[0047] FIG6 exemplarily shows that a preview image provided by a rolling shutter camera has a water ripple problem;
[0048] FIG7 exemplarily illustrates setting the exposure time of a rolling shutter camera to twice the light energy period to solve the water ripple problem;
[0049] FIG8 shows the overall process of the flashing light source method provided by an embodiment of the present application;
[0050] FIG9 exemplarily shows the hardware structure of an electronic device according to an embodiment of the present application;
[0051] FIG10 exemplarily shows the software architecture of the electronic device according to an embodiment of the present application;
[0052] FIG11 shows a detailed implementation process of the flashing light source method provided in an embodiment of the present application based on the internal architecture of an electronic device. DETAILED DESCRIPTION
[0053] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.
[0054] The shutter is a crucial component of a camera, controlling the exposure time of the photosensitive film. When capturing moving objects, a fast shutter speed is crucial for successful capture. The faster the object moves, the higher the shutter speed should be. A fast shutter speed means a short exposure time. If a faster shutter speed (shorter exposure time) is used while keeping the aperture constant, the exposure gain must also be increased to improve the camera's sensitivity and avoid reducing the brightness of the image. Parameters such as aperture, exposure time, and exposure gain are collectively referred to as exposure parameters. Multiple sets of exposure parameters can exist to achieve the same exposure value (EV).
[0055] To make it easier for users to capture moving objects, many smart devices have designed motion capture modes (also known as motion photo modes). When the user presses the shutter button, an appropriate shutter speed is automatically calculated and applied to the photo frame, eliminating the need for the user to worry about controlling exposure. The photo frame is then saved as a photo. In theory, the photo frame is the image frame at the moment the shutter button is pressed. However, in practice, due to issues such as shutter lag, the photo frame is usually released slightly later than the moment the shutter button is pressed. However, this motion capture function still has some problems, as follows:
[0056] 1. Imaging speed is slower than still-life photography. You must wait for the new exposure value automatically calculated by the smart device to take effect before applying the new exposure parameters to the captured frame or re-exposing it before outputting the image. This can result in slower imaging speeds. This effect can involve reconfiguring the camera and enabling related components to adopt the new exposure parameters. For example, reconfiguring and enabling the shutter and its controller to implement the new exposure time, and reconfiguring and enabling the exposure gain circuit (such as the sensor gain circuit and the signal processor gain circuit) and its controller to implement the new exposure gain.
[0057] 2. Shutter lag is more severe than in still photography. Shutter lag refers to the delay between the moment the user presses the shutter button and the actual moment the image is captured. As mentioned in 1 above, the time from when the shutter button is pressed to when the shutter is activated to capture the image is longer due to the need to wait for the new exposure value to take effect.
[0058] 3. Image quality is worse than still photography. Because smart devices automatically calculate a faster shutter speed for motion capture, you need to increase the exposure gain to avoid a decrease in image brightness. However, increasing the exposure gain introduces more color noise and brightness noise, reducing image quality.
[0059] 4. Banding: When using a rolling shutter camera under a flickering light source, if the rolling shutter exposure time is less than the light energy cycle, uneven exposure will appear in the image, resulting in the appearance of "banding."
[0060] Shutters can be categorized as global shutters or rolling shutters, each with different exposure methods. With a global shutter, all pixels on the film begin to be exposed to light simultaneously, and after the same exposure time, all pixels end their exposure at the same time. With a rolling shutter, each row of pixels on the film is exposed row by row, with each row exposed for the same amount of time until all pixels are fully exposed. The exposure start and end times for different rows vary.
[0061] Shooting moving objects with a rolling shutter camera can cause the moving object to appear in the image (referred to as motion smear). This distortion is particularly severe when the object is moving very quickly. Furthermore, the slower the shutter speed (the longer the exposure time), the more severe the motion smear problem. The solution is to increase the shutter speed.
[0062] Taking photos under certain light sources with a rolling shutter camera may also cause "water ripples" to appear in the image. These light sources are often driven by alternating current (AC). The energy transmitted by AC is not constant, but varies with a fixed frequency (such as 50Hz or 60Hz), causing periodic fluctuations in the light intensity of the light source. The period followed is the light energy cycle corresponding to this fixed frequency (also known as the power frequency). For example, the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz, and the light energy cycle is 10 milliseconds, meaning the light source flickers 100 times per second. In this article, this type of light source is also referred to as a flickering light source.
[0063] For a rolling shutter camera, when the exposure time is less than the light energy cycle of the light source, since the exposure start and end times of each row of pixels on the photosensitive film are different, the cumulative light energy received by each row of pixels within the same exposure time is different, and light and dark stripes will appear on the image.
[0064] The various embodiments of the present application are intended to solve some or all of the above-mentioned problems, which are described below with reference to the accompanying drawings.
[0065] Figure 1 shows a motion capture method provided by embodiment 1. This method can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more severe than that of still photography.
[0066] As shown in FIG1 , the method may include:
[0067] S401: The electronic device starts a first camera.
[0068] S402: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
[0069] As time passes, S402 may continue to be executed, and the preview stream from the first camera may continue to be displayed, so as to continuously provide a preview display to the user.
[0070] In addition, electronic devices can also provide a preview frame buffering function to cache the latest few frames of images (hereinafter referred to as cached frames). In this way, when the user intends to take a photo, the image frames used to generate the photo can be quickly retrieved from the cached frames. In this article, the latest cached multiple frames of images can refer to the multiple frames of preview images that are earlier than the photo time but closest to the photo time. The photo time can refer to the time when the electronic device detects the photo operation, such as the time when the user presses the shutter button.
[0071] S403: During the preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S405; if there is no moving object, execute S404.
[0072] How to determine whether there is a moving object in the preview image will be introduced later, so I will not expand on it here.
[0073] S404: The electronic device maintains normal exposure of the first camera during preview.
[0074] Here, for controlling the exposure time, S404 is not an additional action to be performed. The electronic device can maintain normal exposure of the first camera without changing the exposure time of the first camera (maintaining the first duration).
[0075] S405: The electronic device reduces the exposure time of the first camera during preview, for example, reducing the exposure time to a value shorter than the light energy period.
[0076] S406: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.
[0077] The photo-taking operation is not limited to the shutter button being pressed. The photo-taking operation can also include other user operations, such as voice commands with specific content (such as the voice command "YOYO, please take a photo!"), specific actions (such as jumping actions), specific expressions (such as laughing), etc.
[0078] S407: In response to the photo-taking operation, the electronic device saves the photo, which may be from a cache frame.
[0079] As can be seen, in Example 1, the electronic device reduces the exposure time and enters short-exposure mode when it detects a moving object during preview, rather than reducing the exposure time only after the user presses the shutter button. This allows the cached preview frame to have the reduced exposure effect. Once a photo action is subsequently detected, the electronic device can obtain the photo to be saved based on the cached preview frame, without having to wait for the short exposure time to take effect before outputting the image. This alleviates the issues of slow imaging speed and shutter lag.
[0080] Figure 2 illustrates the change in image quality before and after exposure reduction. As shown in Figure 2, frames 1 and 2 are images captured during the initial normal exposure, with an exposure time of 30ms. The rotating windmill in the preview image exhibits motion smear, and the image is blurry. Frames 51 and 52 are images captured after the exposure time is reduced during preview. The exposure time is reduced to 10ms, and the rotating windmill in the preview image no longer exhibits motion smear, and the image is clear. Frames 53 and 54 are images captured when the user presses the shutter button. Due to noticeable shaking of the phone caused by the shutter-pressing operation, the image appears overall blurry. Frame 55 is an image captured after the photo operation, at which point the device returns to a stable state, with an exposure time of 10ms, and the image is overall clear. In this embodiment, in response to the user clicking the shutter button to take a photo, the electronic device can generate a photo based on the image frames with the exposure reduction effect cached during preview (e.g., frames 51 and 52). This not only avoids the problem of motion smear but also allows for rapid photo generation.
[0081] In the first embodiment, determining whether there is a moving object in the preview image (S403) may be specifically implemented by using optical flow between adjacent preview frames to detect moving objects. If the pixel movement area reaches a certain area threshold, the preview image is considered to contain a moving object. The motion detected in this manner is relative motion between the camera and the object being photographed, which can be divided into three specific cases: Case 1, the camera remains stationary and the object is in motion; Case 2, the object remains stationary and the camera is in motion; Case 3, both the object and the camera are in motion, but at different speeds. The motion mentioned in these three cases refers to motion relative to the ground, which can be referred to as absolute motion.
[0082] The optical flow method is not limited to detecting motion. Electronic devices can also use motion sensors such as gyroscopes and accelerometers to detect whether the electronic device itself (camera) is in motion. Once motion is detected, it can be considered that there is a moving object in the preview image. This is because when the camera is in motion, the object being photographed in the preview image is likely to be moving. This excludes the special case where both the object being photographed and the camera are moving at the same speed.
[0083] In Example 1, when reducing the exposure time of the first camera, the electronic device may further detect the motion speed (also a relative speed) of the moving object in the preview image and control the exposure reduction based on the motion speed: the faster the motion speed, the greater the exposure time is reduced, and the shorter the reduced exposure time. The motion speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the motion speed.
[0084] Figure 3 shows a motion capture method provided by Example 2. Based on the technical problem solved by Example 1, the method provided by Example 2 can further improve the problem that the imaging quality is worse than that of still photography.
[0085] As shown in FIG3 , the method may include:
[0086] Reduce exposure time when previewing
[0087] S501: The electronic device starts a first camera.
[0088] S502: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
[0089] S503: During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S505; if not, execute S504.
[0090] S504: The electronic device maintains normal exposure of the first camera during preview.
[0091] S505: The electronic device reduces the exposure time of the first camera during preview.
[0092] For the above S501-S505, reference may be made to steps S401-S405 in the first embodiment, which will not be repeated here.
[0093] In addition, the preview can specifically be a high dynamic range (HDR) preview. In this case, the electronic device can also apply different EVs at different frame moments to obtain multiple cached frames with different brightness (different EVs). Specifically, the electronic device can obtain multiple frames of preview images with different EVs by changing the exposure gain and exposure time. For example, the same exposure time but different exposure gains are used at different frame moments to achieve different EVs, or different exposure times and different exposure gains are used at different frame moments to achieve different EVs, or the same exposure gain but different exposure times are used at different frame moments to achieve different EVs.
[0094] Multi-frame fusion noise reduction
[0095] S506: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.
[0096] S507 . The electronic device determines whether the preview is an HDR preview. If not, execute S508 - S511 . If so, execute S512 - S515 .
[0097] S508: The electronic device may select a reference frame from the cached preview frames.
[0098] As can be seen from the first embodiment, the cached preview frames refer to the latest cached multi-frame images, which are multi-frame images that are earlier than the photographing time but closest to the photographing time.
[0099] The reference frame may be the one with the highest definition among the multiple preview frames cached. The multiple frames selected in S509 are aligned according to the reference frame to eliminate the impact of motion on definition and enhance details. The algorithm used for the multi-frame image alignment may be a motion estimation and motion compensation algorithm commonly used in video coding and decoding technology, that is, the entire image is divided into a series of non-overlapping blocks, and the minimum sum of squared differences (SSD) or sum of absolute differences (SAD) between each block and each block in the reference frame is calculated to estimate the position or motion vector corresponding to the current block in the reference frame, and then compensation is performed based on this.
[0100] Specifically, the electronic device can analyze the image clarity of these preview frames using a clarity operator (such as the Sobel operator) and further consider the device jitter factor and shutter time at these preview frames to select a reference frame. The device jitter factor can be determined based on motion data collected by motion sensors such as gyroscopes and accelerometers. It essentially reflects the degree of device (camera) jitter. A larger factor indicates greater device jitter, resulting in more blurred images and poor image clarity. A shorter shutter time reduces motion artifacts and improves image clarity.
[0101] The electronic device can specifically run an algorithm model to consider the impact of one or more of the clarity operator, device jitter factor, and shutter time on image clarity. The algorithm model can be a neural network model.
[0102] S509 : The electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames excluding the reference frame.
[0103] S510: The electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.
[0104] There are many methods for multi-frame fusion noise reduction, such as summing and averaging, which are not limited in this embodiment.
[0105] S511. The electronic device saves the fused image frame as a photo.
[0106] That is, the saved photos mentioned in S407 in the first embodiment come from the cache frame, which specifically means that the saved photos come from the reference frame selected in S508 and the multiple frames of images selected in S509, and are generated by the fusion of them.
[0107] S512: The electronic device may select a reference frame from the cached preview frames.
[0108] S513 : The electronic device may select multiple frames of images with different exposure values EV from the remaining cache frames, where the remaining cache frames refer to cache frames excluding the reference frame.
[0109] The premise for implementing S512 is that, in order to implement HDR fusion, the electronic device reduces the exposure time during the preview and also changes the exposure gain and exposure time to obtain multiple preview frames with different EVs. For example, under the premise of shortening the exposure time relative to the first duration, the same exposure time but different exposure gains are used at different frame moments to achieve different EVs, or different exposure times and different exposure gains are used at different frame moments to achieve different EVs, or the same exposure gain but different exposure times are used at different frame moments to achieve different EVs.
[0110] S514: The electronic device performs HDR fusion using the selected reference frame and multiple frames of images with different EVs to obtain a fused image frame.
[0111] Like multi-frame fusion noise reduction, HDR fusion also involves aligning multiple frames of images with different EVs based on a reference frame, and then performing multi-frame fusion noise reduction on these multiple frames. The difference between the two is that HDR fusion can also perform brightness fusion (also known as exposure fusion) on multiple frames of images with different brightness (different EVs).
[0112] S515: The electronic device saves the image frame after HDR fusion processing as a photo.
[0113] That is, the saved photos mentioned in S407 of the first embodiment come from the cached frames, which specifically means that the saved photos come from the reference frame selected in S512 and the multiple frames of images with different EVs selected in S513, and are generated by the fusion of them.
[0114] It can be seen that when the user intends to take a photo, the electronic device in Example 2 can quickly take out multiple frames of images used to generate photos from the cache frame and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality. Moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented. At the same time, for the case of motion capture, the algorithm effect parameters and ISP parameters of the HDR algorithm or multi-frame fusion are specially adapted, and the effect is debugged for high exposure gain and short exposure time. Among them, the algorithm effect parameters can be, for example, noise reduction intensity, contrast intensity, sharpening intensity, fusion ratio, ghost threshold, motion threshold, matching threshold, fusion pyramid size and other common fusion, noise reduction, and dynamic range algorithm adjustable parameters (the parameters will vary depending on the algorithm prototype). ISP debugging parameters can include image debugging parameters such as auto exposure, auto focus, auto white balance parameters (also known as 3A parameters), lens shading correction (LSC), gamma (GAMMA), dynamic range correction (DRC), YUV noise reduction (YNR), etc. For example, YNR is a common YUV noise reduction parameter of ISP. It reduces the noise impact caused by high exposure gain during motion by targeted adjustment. In this way, frame selection, image fusion algorithm and parameter separation can compensate for the noise caused by increased exposure gain, while enhancing the image effect.
[0115] FIG4 illustrates a motion capture method provided in Example 3. In this embodiment, the first camera may be a rolling shutter camera, which may produce ripples in the preview image provided by the camera under a flickering light source. Based on the technical problem solved in Example 2, the method provided in Example 3 can further solve the banding problem.
[0116] As shown in FIG4 , the method may include:
[0117] Reduce exposure time when previewing
[0118] S601: The electronic device activates a first camera.
[0119] S602: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
[0120] S603: During the preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, the electronic device executes S604; if there is no moving object, the electronic device executes S605.
[0121] S604: The electronic device maintains normal exposure of the first camera during preview.
[0122] The above S601-S604 can refer to steps S501-S504 in the second embodiment and will not be repeated here.
[0123] S605: The electronic device reduces the exposure time of the first camera during preview.
[0124] In this embodiment, the electronic device can reduce the exposure time while also taking into account the water ripple problem. The implementation of S605 may specifically include S6051-S6055:
[0125] S6051. Determine whether to perform flicker detection.
[0126] S6052: Start the flashing light source detection device.
[0127] A flicker light source detection device can include an ambient light sensor (ALS) and a multispectral sensor. The ALS operates at a high frequency F. According to Shannon's theorem, when F exceeds twice the light energy period F', the energy variation of the flicker light source can be determined by reading the ALS data, thereby confirming the presence of a flicker light source in the current environment. The multispectral sensor can directly read the energy variation information of the flicker light source.
[0128] In other words, dedicated flicker detectors, such as high-sampling ambient light sensors and multispectral flicker detectors, can be used to detect the presence of flickering light sources in the shooting environment and, in conjunction with the camera, resolve the water ripple problem. However, such devices require additional cost and design space.
[0129] To this end, the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.
[0130] The second camera is not specifically introduced for flickering light source detection. It is usually used as a depth of field camera, a wide-angle camera, or a macro camera. Its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms. As a result, the image captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of working frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple working frequencies. Taking the two working frequencies of 50Hz and 60Hz as an example, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).
[0131] In addition, the exposure time referred to by normal exposure can be set to an integer multiple of the light energy cycle, which can avoid the user seeing water ripples during preview during normal exposure.
[0132] How to use the second camera to detect flickering light sources will be described in detail in the embodiments shown in Figures 8 and 11 below.
[0133] S6053: Perform a flashing light source detection through the flashing light source detection device. If no flashing light source is detected, execute S6054; otherwise, execute S6055.
[0134] Specifically, when the flicker light source detection device is a second camera, the specific implementation of S6053 may include: the electronic device performing image recognition on the image captured by the second camera to determine whether the image content contains water ripple characteristics. If so, it is determined that a flicker light source is present in the shooting environment. The image captured by the second camera is not sent for display and is only used for flicker detection, so the user will not perceive it.
[0135] S6054: Ignore the water ripple problem and reduce the exposure time as expected, for example, reduce the exposure time to less than the light energy period, for example, 9 milliseconds.
[0136] Here, expectation may mean: the exposure time needs to be reduced according to the moving speed of the moving object, the image quality in HDR and other scenarios. For example, when the object moves very fast, the exposure time is usually reduced to a very short time, such as 5 milliseconds, which is shorter than the light energy period.
[0137] S6055. If a flickering light source is detected, the reduced exposure time can be controlled to be an integer multiple of the light energy cycle (such as 1 times, 2 times, etc.). The shortest exposure time can be set to 1 times the light energy cycle, such as 10ms.
[0138] As shown in S6051, before initiating flicker detection, the electronic device may also determine that the ambient brightness is appropriate, such as the indoor lighting environment, based on the exposure time set by the automatic exposure control (AEC) for the first camera, and further determine that flicker detection is necessary. This is because the exposure time set by the AEC for the light sensor can, to a certain extent, reflect the brightness of the shooting environment.
[0139] Flicker detection is generally not required in shooting environments that are too bright or too dark. This is because in too bright environments, the primary concern is overexposure, requiring a significant reduction in exposure time, making it difficult to address moiré. In too dark environments, there's generally no need to reduce exposure time to below the light energy period, as this can easily lead to underexposure. Here, "too bright" or "too dark" can be defined with reference to the exposure table on electronic device 100. For example, the shortest exposure times corresponding to high EV in the exposure table can be used as exposure times for too bright environments. If the AEC exposure time falls within these shortest exposure times, the environment is considered too bright. Another example is the longest exposure times corresponding to low EV in the exposure table can be used as exposure times for too dark environments. If the AEC exposure time falls within these longest exposure times, the environment is considered too dark. In other words, the reference exposure value for too bright or too dark can be derived from the exposure table. Therefore, if the AEC exposure time falls within the remaining exposure times in the exposure table, it indicates that the environment is neither too bright nor too dark. The electronic device is not limited to analyzing the ambient brightness by the exposure time set by the AEC for the first camera. The electronic device can also collect ambient light through the ambient light brightness sensor to determine whether the shooting environment is too bright or too dark. This application does not impose any restrictions on its technical implementation.
[0140] Multi-frame fusion noise reduction
[0141] S606: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.
[0142] S607: The electronic device determines whether the preview is an HDR preview. If not, execute S608-S611; if so, execute S612-S615.
[0143] S608: The electronic device may select a reference frame from the cached preview frames.
[0144] S609: The electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames excluding the reference frame.
[0145] S610: The electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.
[0146] S611: The electronic device saves the fused image frame as a photo.
[0147] S612: The electronic device may select a reference frame from the cached preview frames.
[0148] S613: The electronic device may select multiple frames of images with different exposure values EV from the remaining cache frames, where the remaining cache frames refer to cache frames excluding the reference frame.
[0149] S614: The electronic device performs HDR fusion using the selected reference frame and multiple frames of images with different EVs to obtain a fused image frame.
[0150] S615: The electronic device saves the image frame after HDR fusion processing as a photo.
[0151] The above S606-S615 can refer to steps S506-S515 in the second embodiment, and will not be repeated here.
[0152] In addition, the specific implementation of reducing the exposure time described in S6051-S6055 can also be combined with the first embodiment as the implementation method of step S405 in the first embodiment.
[0153] Taking photos under certain light sources using a rolling shutter camera may cause "banding" to appear in the image. These light sources are often driven by alternating current (AC). The energy transmitted by AC is not constant, but varies with a fixed frequency (such as 50Hz or 60Hz), resulting in periodic fluctuations in the light intensity of the light source. The period followed is the light energy cycle corresponding to this fixed frequency (also known as the power frequency). For example, the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz, and the light energy cycle is 10 milliseconds, meaning the light source flickers 100 times per second. In this article, this type of light source is also referred to as a flickering light source.
[0154] For a rolling shutter camera, when the exposure time is less than the light energy cycle of the light source, each row of pixels on the photosensitive film receives different amounts of accumulated light energy within the same exposure time, due to the different start and end times of exposure. This results in alternating light and dark stripes appearing on the image. For example, as shown in Figure 5, when imaging the first frame, the light energy received by the first row of pixels on the photosensitive film is the light energy accumulated from time 0 to time t2, while the light energy received by the nth row of pixels on the photosensitive film is the light energy accumulated from time t1 to 5 milliseconds. Therefore, the light energy received by the nth row is less than that received by the first row. For another example, as shown in Figure 5, when imaging the N+1th frame, the light energy received by the first row of pixels on the photosensitive film is the light energy accumulated from time t3 to time t5, while the light energy received by the nth row of pixels on the photosensitive film is the light energy accumulated from time t4 to time t6. Therefore, the light energy received by the first row is less than that received by the nth row. In an image frame, rows with higher light energy appear as bright stripes, while rows with lower light energy appear as dark stripes, as shown in Figure 6. The alternating light and dark stripes are water ripples, also known as flicker. Moreover, users can also observe the rolling water ripples when shooting preview or recording.
[0155] In shooting scenarios with fast shutter speeds, such as motion capture and high dynamic range (HDR) modes, water ripples are more likely to appear in the image. After detecting that the light source in the shooting environment is a flickering light source, one way to avoid this problem is to set the exposure time to an integer multiple of the light energy period. As shown in Figure 7, the light energy period is 10 milliseconds, so the exposure time can be set to 10 milliseconds, which is 1 times the light energy period.
[0156] Detecting flickering light sources is key to resolving the water ripple problem. One detection method uses image recognition algorithms to identify water ripple features in image content. Once the presence of water ripples is confirmed, the exposure time is adjusted to mitigate the ripples. However, until the ripples are eliminated, the user will still see them, resulting in a poor user experience. Another detection method uses specialized flicker detectors, such as high-sampling ambient light sensors or multispectral flicker detectors, to detect the presence of flickering light sources in the shooting environment. These sensors, in conjunction with the camera, can address the water ripple problem. However, these devices incur additional cost and design space.
[0157] The embodiments of the present application provide a flickering light source detection method, which can detect flickering light sources without introducing additional dedicated detection devices.
[0158] The electronic device implementing this method has two or more rolling shutter cameras. When one or more cameras are used for image preview display, a camera is selected from the remaining cameras to detect flickering light sources. The camera used for preview display is called the first camera, and the camera used for flicker light detection (also known as flicker detection) is called the second camera. The second camera is not specifically introduced for flicker light detection and is typically used as a depth of field camera, wide-angle camera, or macro camera. The shutters of the first and second cameras are both rolling shutters.
[0159] FIG8 shows the overall process of the flickering light source detection method provided by an embodiment of the present application.
[0160] S701: The electronic device activates a first camera.
[0161] The activation of the first camera can be triggered by the user opening the camera app. In this case, the camera app's shooting mode can be normal shooting mode with a zoom ratio of 1x. The first camera can be the rear main camera, and the second camera is selected from the remaining cameras. The rear main camera can be the one with the highest pixel count on the electronic device, and its focal length is typically between that of a wide-angle camera and a telephoto camera.
[0162] Specifically, when starting the first camera, the exposure time of the first camera can be set to a first duration, which is greater than the light energy cycle and is an integer multiple of the light energy cycle, such as 50ms. This can prevent users from seeing the generation of water ripples at the beginning of the camera startup. Unlike the motion capture mode and HDR photo mode, which require reducing the exposure time, the exposure time of the first camera in the normal shooting mode can be relatively long, such as 50ms, 100ms and other high multiples of the light energy cycle. The first duration of exposure time can be set by the ISP for the first camera through the AEC. The light energy cycle is not limited to the light energy cycle under the 50Hz working frequency, but can also be the light energy cycle under the 60Hz working frequency, or the light energy cycle under other working frequencies.
[0163] The first camera is not limited to the scenario where the user opens the camera application. The scenario in which the first camera is activated can also be other scenarios that require the camera to be activated to capture images, such as video call scenarios, scan code payment scenarios, identity verification scenarios, etc. For example, in the scan code payment scenario, when it is detected that the user has activated the scan code payment, the electronic device can activate the rear main camera to scan the payment QR code. In this case, the first camera is the rear main camera. For another example, in the identity verification scenario, the electronic device can activate the front main camera and the front depth of field camera to scan and obtain facial feature data for user identity verification. In this case, the first camera can be the front main camera. For another example, in the video call scenario, the electronic device can activate the front main camera to capture a portrait of the user in front of the screen.
[0164] The image captured by the first camera is sent to the display and used as the preview image displayed on the display by the camera application.
[0165] S702: The electronic device displays the image captured by the first camera on a display screen.
[0166] The first camera can be a single camera, in which case the preview image displayed on the display by the camera application only comes from that camera. The first camera can also be multiple cameras, in which case the preview image displayed on the display by the camera application is composed of images simultaneously captured by these multiple cameras, providing users with the ability to take photos or record videos with multiple cameras.
[0167] As time passes, S702 may continue to be executed, and the preview stream from the first camera may continue to be displayed, so as to continuously provide a preview display to the user.
[0168] S703, the electronic device starts the second camera and sets the exposure time of the second camera to a second duration, which is shorter than the light energy period, for example 9ms, so that the image captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
[0169] Before executing S703 , the electronic device may also determine that the ambient brightness is appropriate, such as the indoor lighting environment, based on the exposure time set by the AEC for the first camera, and further determine that flicker detection is required. This is because the exposure time of the light sensor set by the AEC can reflect the brightness of the shooting environment to a certain extent.
[0170] Flicker detection is generally not required in shooting environments that are too bright or too dark. This is because in too bright environments, the primary concern is overexposure, requiring a significant reduction in exposure time, making it difficult to address moiré. In too dark environments, there's generally no need to reduce exposure time to below the light energy period, as this can easily lead to underexposure. Here, "too bright" or "too dark" can be defined with reference to the exposure table on electronic device 100. For example, the shortest exposure times corresponding to high EV in the exposure table can be used as exposure times for too bright environments. If the AEC exposure time falls within these shortest exposure times, the environment is considered too bright. Another example is the longest exposure times corresponding to low EV in the exposure table can be used as exposure times for too dark environments. If the AEC exposure time falls within these longest exposure times, the environment is considered too dark. In other words, the reference exposure value for too bright or too dark can be derived from the exposure table. Therefore, if the AEC exposure time falls within the remaining exposure times in the exposure table, it indicates that the environment is neither too bright nor too dark. The electronic device is not limited to analyzing the ambient brightness by the exposure time set by the AEC for the first camera. The electronic device can also collect ambient light through the ambient light brightness sensor to determine whether the shooting environment is too bright or too dark. This application does not impose any restrictions on its technical implementation.
[0171] S704, the electronic device performs image recognition on the image captured by the second camera (not displayed), and determines whether the image content contains water ripple features. If so, the exposure time of the first camera is controlled to be an integer multiple of the light energy cycle. The image captured by the second camera will not be displayed and is only used for flicker detection, so the user will not perceive it. The flicker detection result may include: a flickering light source is detected, and a flickering light source is not detected. If the image content captured by the second camera contains water ripple features, it is considered that there is a flickering light source in the shooting environment. Thereafter, the electronic device can control the exposure of the first camera according to the flicker detection result, and control the exposure time of the first camera to be an integer multiple of the light energy cycle to avoid water ripples in the preview image provided by the first camera.
[0172] For example, when the shooting mode is switched from the normal shooting mode to the motion capture mode, when reducing the exposure time to alleviate the motion ghosting problem, the electronic device can specifically reduce the exposure time of the first camera to a small integer multiple such as one or two times the light energy period, that is, the reduced exposure time is controlled to be an integer multiple of the light energy period, so as to simultaneously avoid the problem of water ripples in the imaging picture under the flickering light source.
[0173] Which camera is the first camera or the second camera is not fixed.
[0174] When the user first opens the camera application, the electronic device can default to the normal photo mode with a zoom ratio of 1. At this time, the first camera can be the main camera, and the second camera can be selected from the remaining cameras.
[0175] However, when the shooting mode changes or the zoom ratio changes, the first camera can switch from the main camera to other cameras.
[0176] For example, during photo preview, video preview, or video recording, if it is detected that the user increases the zoom ratio to the zoom range of the telephoto camera, the electronic device may switch the first camera from the main camera to the telephoto camera. For another example, during photo preview, video preview, or video recording, if it is detected that the user decreases the zoom ratio to the zoom range of the wide-angle camera, the electronic device may switch the first camera from the main camera to the wide-angle camera. For another example, if it is detected that the photo mode is switched from normal mode to macro mode, the electronic device may switch the first camera from the main camera to the macro camera.
[0177] The second camera can be switched to the original first camera, especially for the case where the electronic device has only two cameras, there is no need to start a new camera, and the response speed is faster. Of course, in actual applications, it is also possible to consider switching the second camera to a camera with lower power consumption based on issues such as power consumption, and the embodiments of the present application are not limited to this. The second camera can also be fixed, such as a depth of field camera. The image data collected by this type of camera does not participate in the preview image display, but is only used as auxiliary data (such as depth of field data) for other cameras, such as depth data of the face, and therefore will not be occupied by the first camera due to changes in the shooting mode.
[0178] Next, the software and hardware architecture of the electronic device provided in the embodiments of the present application is introduced.
[0179] Electronic equipment can be equipped or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.
[0180] 9 exemplarily shows an electronic device 100 provided by an embodiment of the present application. The electronic device 100 can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more severe than that of still photography.
[0181] As shown in FIG9 , the electronic device 100 may include: a processor 110, a memory 120, a camera 130, a display screen 140, and a flickering light source detection device (not shown).
[0182] The processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface. These interfaces are used for data exchange between the processor 110 and peripheral devices.
[0183] The memory 120 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) for the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct reading and writing by the processor 110. The processor 110 may also be provided with a storage unit, which may be a cache memory unit, for storing instructions or data that has just been used or is recycled by the processor 110. The implementation code of the flickering light source detection method provided in the embodiments of the present application may be stored in the NVM. When the camera application is launched, this code may be loaded into the RAM. In this way, the processor 110 can directly read the program code from the RAM to implement the flickering light source detection method provided in the embodiments of the present application. In addition, image files such as photos and videos captured by the user using the camera application can be written to the NVM for storage and viewing by the user.
[0184] Images captured by the camera 130 can be displayed to provide a preview image. The camera 130 may include a lens, a photosensor, and a flexible printed circuit board (FPCB). The FPCB is responsible for connecting the other components of the camera 130 to the processor 110, for example, transmitting the raw data output by the photosensor to the processor 110. When taking a photo, the shutter of the camera 130 is opened, allowing light to enter and illuminate the photosensor. The photosensor converts the light signal into an electrical signal, which is then further converted into a digital signal through analog-to-digital conversion (ADC) for transmission to the ISP for processing. The ISP can perform the following processing on the photosensor output data: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, and bad pixel removal. The ISP can also be integrated into the camera 130.
[0185] The display screen 140 can be used to display the image captured by the camera 130. The image processed by the ISP will be sent to the display screen 140 to show the user a preview of the image captured by the camera. Here, sending to the display screen means pushing the image captured by the camera to the frame buffer (FB) for storage. The frame buffer is a storage space that can be located in the video memory or the internal memory, and is used to store the rendering data processed by the graphics card chip or to be extracted. The content of the frame buffer corresponds to the interface display on the display screen 140. It can be simply understood as a cache corresponding to the content displayed on the display screen 140. Modifying the content in the frame buffer is to modify the content on the display screen 140.
[0186] The flicker light source detection device can be used for flicker detection, and specifically may include an ambient light sensor (ALS) and a multispectral sensor. Among them, the ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', the energy change of the flicker light source can be restored by reading the data of the ALS, thereby confirming whether there is a flicker light source in the current environment. The multispectral sensor can directly read the energy change information of the flicker light source. The flicker light source detection device can also be a rolling shutter camera (called a second camera) on an electronic device. The image captured by the second camera will not be sent for display and is only used for flicker detection, so the user will not perceive it. The second camera is not specially introduced for flicker detection. It can usually be a depth of field camera, a wide-angle camera, a macro camera, etc., with low resolution and low power consumption. It is a non-primary camera, and the embodiments of the present application give it new functions. The second camera can also be one or more cameras.
[0187] In addition to the flicker detection results provided by the second camera, the exposure time of the camera that provides the preview display (referred to as the first camera) can also be controlled by the automatic exposure control (AEC) function. In short, AEC automatically adjusts the exposure time according to the light intensity. In different environments, the intensity of light varies greatly, and the exposure time of the photosensor in the camera 130 also needs to adapt accordingly to ensure normal exposure of the image. The automatic exposure control (AEC) provided by the ISP provides this adaptive capability. Therefore, the exposure time of the photosensor can reflect the brightness of the shooting environment to a certain extent.
[0188] In addition, the electronic device 100 may also store an exposure table to be used by the AEC. A camera's exposure table specifies the exposure parameters (such as exposure time, exposure gain, and aperture size) to be used by the camera at different exposure values (EV) and can be debugged and recorded by engineers.
[0189] As shown in FIG9 , the electronic device 100 may further include an audio module 150 , a speaker 150A, a receiver 150B, a microphone 150C, and an earphone jack 150D.
[0190] Among them, the electronic device can implement audio functions through the audio module 150, speaker 150A, receiver 150B, microphone 150C, headphone jack 150D, and application processor (AP). For example, music playback, recording, etc. The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be set in the processor 110, or some functional modules of the audio module 150 can be set in the processor 110. The speaker 150A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The electronic device can listen to music or listen to hands-free calls through the speaker 150A. The receiver 150B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device answers a call or voice message, the voice can be heard by placing the receiver 150B close to the human ear. Microphone 150C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 150C to input the sound signal into the microphone 150C. The electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 150C to realize sound signal collection, noise reduction, sound source identification, directional recording function, etc. The headphone jack 150D is used to connect wired headphones. The headphone jack 150D can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0191] As shown in FIG9 , the electronic device 100 may further include a sensor module 160 , which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.
[0192] The structure shown in FIG9 does not constitute a specific limitation on the electronic device. The electronic device may include more or fewer components than shown, or some components may be combined or separated, or the components may be arranged differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0193] FIG10 shows the main software architecture involved in detecting flickering light sources using another rolling shutter camera on the electronic device 100 .
[0194] As shown in Figure 10, the software architecture may include the following functions or modules: camera application (camera apk), camera service (camera service), camera hardware abstraction layer (camera HAL), flicker detection (CameraFlickerDetector), camera VNDK and Kernel part.
[0195] The camera app (camera APK) can be located in the application layer. Besides the camera app, the application layer can include a variety of other apps, such as the gallery app. The camera app is primarily responsible for user interaction, such as listening for and responding to user actions like selecting a shooting mode, adjusting the zoom factor, and selecting a focus point.
[0196] The camera service can reside in the application framework layer. It includes interfaces for camera management and camera devices, serving as a link between the upper and lower layers. It can interact with camera applications through the application programming interface (API) and with the camera HAL through the HAL interface definition language (HIDL). Furthermore, the camera service can interact with the vendor native development kit (VNDK).
[0197] The camera hardware abstraction layer (HAL) represents the virtualized hardware functionality related to the camera within the hardware abstraction layer (HAL). The HAL, located as an interface between the application framework and driver layers, provides a virtual hardware platform for the operating system, freeing upper layers from the underlying hardware implementation. The camera HAL communicates with the camera service via a HIDL interface and issues camera control operations via standard HAL interfaces. The camera HAL includes a camera entry (CamEntry) module and an automatic exposure control processor (AECProcessor) module. The CamEntry module serves as the entry point to the HAL, receiving commands such as open, close, stream configuration, and request from the camera service. It is a universal entry point for camera HAL functionality. The AECProcessor module provides ISP-related AEC functions, such as providing the first camera's AEC parameters (such as exposure time) to the CameraFlickerDetector module, enabling it to determine whether flicker detection is required in the current shooting environment.
[0198] The flicker detection (CameraFlickerDetector) module can be located in the application framework layer (framework) or the application layer. The CameraFlickerDetector module can obtain the AEC parameters (such as exposure time) of the first camera from the camera HAL through the HIDL interface, so as to decide whether to start the second camera for flicker detection. When it is determined that flicker detection is to be performed, the CameraFlickerDetector module can issue the control operation of starting the second camera and setting the exposure time through the interface of the camera VNDK. The CameraFlickerDetector module may include a CameraHelper module and an ImageReader module, wherein the CameraHelper module can be used to implement the call of functions such as opening (open), closing (close), stream configuration (configurestream), and request of the camera service of the second camera through the call of the VNDK interface, and the ImageReader module is mainly used for receiving and further processing the images sent after the second camera is started.
[0199] The Vendor Native Development Kit (VNDK) is a collection of libraries that allows vendors to develop their own HALs (vendors). The camera VNDK is the "own camera HAL" developed by the vendor of the second camera. The camera VNDK can be used to receive the control operation issued by the CameraFlickerDetector to start the second camera and forward the control operation to the camera service. The camera VNDK can include the ACameraManager module, the ACameraDevice module, the ACameraReader module, the ACaptureSession module, and the ACaptureRequest module. Among them, the ACameraManager module provides the function of accessing the camera service from the vendor layer, the ACameraDevice module provides the function of accessing the camera device from the vendor layer, the ACameraReader module is responsible for reading image data, such as data captured by the second camera, the ACaptureSession module allows the vendor to manage the frames captured by the camera, and the ACaptureRequest module is responsible for the settings and output targets required for the camera to capture a single image.
[0200] The kernel part may include various hardware driver control modules, such as ISP, photosensor, DSP, AP and other driver control.
[0201] FIG10 only shows the main software architecture involved in the flickering light source detection method provided in the embodiment of the present application, which is essentially only a part of the software architecture of the electronic device 100 .
[0202] FIG10 also shows a process for detecting flickering light sources using another rolling shutter camera, including:
[0203] 1. First camera startup process
[0204] The camera app calls the camera service, which authenticates the app. After authentication, the camera service calls the camera HAL, which activates the primary camera and controls the photosensor and image processing interface (ISP). Image frames are then displayed for functions like image preview and photo capture.
[0205] On this basis, to perform flicker detection without the user noticing, the electronic device 100 can activate a second camera for detection. The second camera is not specifically introduced for flicker detection. Typically, it can be a depth of field camera, a wide-angle camera, or a macro camera, etc., with lower resolution and lower power consumption. It is a non-primary camera. The embodiments of the present application give it new functions.
[0206] 2. Flicker detection process
[0207] First, when preview is enabled for the first camera, the AECProcessor module in the camera HAL transmits parameters such as the first camera's exposure time to the CameraFlickerDetector module via a HIDL interface. The exposure time set by the AEC reflects the brightness of the shooting environment. Therefore, the CameraFlickerDetector module can determine whether to perform flicker detection based on the exposure time. Flicker detection is typically performed indoors. Flicker detection is generally not necessary in shooting environments that are too bright or too dark. In bright environments, overexposure is the primary concern, requiring a significant reduction in exposure time, making it difficult to minimize water ripples. In dark environments, there's generally no need to reduce the exposure time to below the light energy cycle, as this can easily lead to underexposure. Here, "too bright" or "too dark" can be defined with reference to the exposure table on electronic device 100. For example, the shortest exposure times corresponding to high EV in the exposure table can be determined as exposure times for too bright environments. If the actual exposure time falls within these shortest exposure times, the environment is considered too bright. For another example, the longest exposure times corresponding to low EV in the exposure table can be determined as exposure times for too dark environments. Once the actual exposure time falls within these longest exposure times, it is considered that the environment is too dark.
[0208] Then, after determining that flicker detection is to be performed, the CameraFlickerDetector module transmits a control command to start the second camera to the camera VNDK.
[0209] The camera VNDK then passes the CameraFlickerDetector module to the camera service to start controlling the second camera.
[0210] After authenticating the CameraFlickerDetector module, the camera service calls the camera HAL again to start the second camera through the camera HAL, so that the second camera can be started from the camera HAL side without involving the application layer, so the user is not aware of it. At the same time, the CameraFlickerDetector module controls the exposure of the second camera, making the exposure time of the second camera less than the light energy cycle. For example, if the exposure time is set to 9ms, the image captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The CameraFlickerDetector module can also call the camera HAL to read the output image of the second camera, detect whether the image content has water ripple features through image recognition technology, and pass the detection results to the first camera. The output image of the second camera is not displayed and is only used for flicker detection.
[0211] After the first camera learns the flicker detection result, it can reduce the exposure time of the first camera to twice the light energy cycle, such as 10ms, after detecting a moving object in the preview image, so as to solve the problem of water ripples in the imaging image under the flickering light source.
[0212] The above software architecture newly introduces the CameraFlickerDetector module and implements interaction between the CameraFlickerDetector module, the camera HAL, and the camera service through VNDK and HIDL. This does not involve modifications to the camera application or the architecture layers, and has good portability, scalability, and maintainability.
[0213] The following, combined with Figure 11, further illustrates the flickering light source detection method according to an embodiment of the present application, focusing on the internal architecture of electronic device 100. Figure 11 illustrates this method using the example of a photo or video preview provided by a camera application to facilitate a deeper understanding of the present embodiment by those skilled in the art. The details herein should not limit the scope of protection of this application. The following expands upon this.
[0214] Normal photo mode
[0215] S11. Upon detecting a user operation of opening a camera application, the application processor (AP) may activate the rear main camera.
[0216] In this case, the rear main camera is used as the primary camera. The photo mode can be normal photo mode, and the exposure time of the rear main camera is longer than the light energy cycle, for example, 50ms, or a multiple of the light energy cycle. The exposure time of the rear main camera can be set by the AEC of the ISP.
[0217] Prior to the S11, the display screen may display a system desktop, which includes a desktop icon for a camera application. The user operation of opening the camera application may refer to the user clicking the desktop icon for the camera application.
[0218] S12. The rear main camera transmits the captured raw image frames (RAW) to the ISP.
[0219] The ISP can perform the following processing on the RAW image output by the rear main camera: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, bad pixel removal, etc. The image processed by the ISP can be a color image, such as a YUV image.
[0220] S13.ISP can transmit the output YUV image frame of the rear main camera to the display.
[0221] Here, S13 only illustrates the flow path of image data and does not mean that the ISP directly sends data to the display. Its implementation includes the display process. The YUV image frames of the rear main camera can be cached in the frame buffer. When rendering the photo preview image, the GPU reads the YUV image frames of the rear main camera from the frame buffer and displays them on the display.
[0222] S14. The display screen displays the YUV image frame of the rear main camera.
[0223] From then on, the rear main camera began to provide preview images to realize the preview function in normal photo mode.
[0224] In normal photo mode, even in the presence of flickering light, the rear main camera's exposure time is longer than an integer multiple of the light energy period, so no ripples appear in the preview image. However, early detection of flickering light sources is essential because users may switch to motion capture mode or HDR shooting mode at any time, which require a reduced exposure time. How to reduce the exposure time of the first camera in these shooting modes while preventing ripples from appearing is the problem that the embodiments of this application focus on solving.
[0225] S15. The AP can obtain the exposure time set by the AEC for the rear main camera from the ISP. The exposure time set by the AEC can reflect the brightness of the shooting environment.
[0226] S16.AP can determine whether flicker detection is required based on the exposure time set by AEC for the rear main camera.
[0227] The environment for flicker detection is usually an indoor scene. Flicker detection is generally not required in shooting environments that are too bright or too dark. For specific instructions, please refer to the previous article and will not be repeated here.
[0228] Flicker detection
[0229] S17. After determining to perform flicker detection, the application processor (AP) may start the depth-of-field camera as a second camera and set the exposure time of the depth-of-field camera to be less than the light energy cycle, such as 9 milliseconds.
[0230] Since the exposure time of the second camera is less than the light energy cycle, the image captured by the second camera under the flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
[0231] After starting the first camera, the AP may also directly start the second camera without determining whether to perform flicker detection, that is, S15-S16 may not be executed.
[0232] After being activated, the depth-of-field camera may transmit the captured raw image frames (RAW) to the ISP.
[0233] The ISP can perform a series of processing on the RAW image and output a YUV image frame. As a second camera, the YUV image frame of the depth of field camera is not sent to the display, that is, it will not be transmitted to the display screen for display.
[0234] S19.AP can obtain the YUV image frame of the depth camera from the ISP.
[0235] S20.AP performs image recognition on the YUV image frames from the depth-of-field camera, detecting whether the image content contains water ripple features to obtain flicker detection results. If the image content contains water ripple features, it is considered that there is a flickering light source in the shooting environment. Otherwise, it is considered that there is no flickering light source in the shooting environment.
[0236] The AP can run the software architecture shown in FIG10 , particularly the CameraFlickerDetector module, to implement flicker detection. Flicker detection can continue to be performed over time, particularly while the first camera is on.
[0237] Thereafter, such as S21-S22, the photo mode may be switched to the motion capture mode or the HDR mode, and the electronic device may set the exposure time of the first camera based on the flicker detection result to avoid the water ripple problem.
[0238] S21. The AP learns that the camera mode has been switched to motion capture mode or HDR mode.
[0239] While S22.AP reduces the exposure time of the rear main camera to meet the needs of motion capture mode or HDR mode, it can specifically reduce the exposure time to 1 times the light energy cycle to take into account the problem of avoiding water ripples.
[0240] The exposure time is not limited to 1 times the light energy cycle. The exposure time can also be reduced to 2 times, 3 times, or other low multiples of the light energy cycle, depending on the exposure reduction requirements of the motion capture mode or HDR mode.
[0241] The first camera is not static. The following steps illustrate how to switch the first camera in several typical scenarios (zooming, flipping the camera, and multi-channel photography or video recording).
[0242] Zoom
[0243] In the zoom scenario, increasing the zoom ratio to the zoom range of the telephoto camera, or decreasing the zoom ratio to the zoom range of the wide-angle camera, can change the first camera that provides the preview image.
[0244] S23. The application processor (AP) detects that the zoom ratio is increased to the zoom range of the telephoto camera.
[0245] S24. The application processor (AP) can turn off the rear main camera.
[0246] S25. The application processor (AP) may activate the telephoto camera and use the telephoto camera as the first camera.
[0247] At this time, if you are in a shooting mode that requires a low exposure time, such as motion capture or HDR, when starting the telephoto camera, you can set its exposure time to 1 times the light energy cycle. Please refer to S26.
[0248] S27. The telephoto camera transmits the captured raw image frames (RAW) to the ISP.
[0249] The ISP can perform a series of processing on the RAW image and output a YUV image frame.
[0250] The S28.ISP can transmit the YUV image frames from the telephoto camera to the display.
[0251] Here, S27 only illustrates the flow path of image data, and does not mean that the ISP directly sends data to the display screen. Its implementation includes the process of sending and displaying.
[0252] S29. The display screen displays the YUV image frame of the telephoto camera.
[0253] From this point on, the telephoto camera begins providing preview images, and the primary camera used for preview display switches from the main rear camera to the telephoto camera. Reducing the zoom ratio to the wide-angle camera's zoom range is similar to changing the primary camera's zoom range. Refer to the adaptive adjustments in S23-S28 for details, and will not be elaborated here.
[0254] Setting the telephoto camera exposure time based on the flicker detection results can also avoid the appearance of water ripples in its imaging image.
[0255] Flip camera
[0256] In a scenario where the user flips the camera, the first camera can be switched from the rear main camera to the front main camera.
[0257] S30. The application processor (AP) detects that the camera is flipped.
[0258] S31. The application processor (AP) can turn off the rear main camera.
[0259] S32. The application processor (AP) may activate the front main camera and use the front main camera as the first camera.
[0260] At this time, if the front camera is in a shooting mode that requires a low exposure time, such as motion capture or HDR, by default, the exposure time can be set to 1 times the light energy cycle when the front main camera is started. Please refer to S33.
[0261] S34. The front main camera transmits the captured raw image frame (RAW) to the ISP.
[0262] The ISP can perform a series of processing on the RAW image and output a YUV image frame.
[0263] The S35.ISP can transmit the YUV image frames of the front main camera to the display.
[0264] Here, S35 only illustrates the flow path of image data, and does not mean that the ISP directly sends the YUV image frame of the front main camera to the display screen. Its implementation includes the process of sending and displaying.
[0265] S36. The display screen displays the YUV image frame of the front main camera.
[0266] From then on, the front-facing main camera begins providing preview images, and setting its exposure time based on flicker detection results can also prevent water ripples in the image. In scenarios where the front camera is the default, such as video calls, flipping the camera can also trigger the first camera to switch from the front main camera to the rear main camera. The implementation process is similar and will not be elaborated here.
[0267] Multi-channel photo or video recording
[0268] When the shooting mode is switched to the multi-channel photo or video mode, one or more cameras other than the main camera are started and also added as the first camera.
[0269] S37. The application processor (AP) detects that the camera mode is switched to the front and rear dual-channel camera mode.
[0270] Dual-camera photography is just one type of multi-camera photography, which can also include simultaneous photography with multiple rear cameras, including the main, telephoto, and wide-angle cameras. Multi-camera recording is similar.
[0271] S38. The application processor (AP) may activate the front main camera and add the front main camera as the first camera.
[0272] In this case, the first camera includes two cameras: a rear main camera and a front main camera. The application processor (AP) does not shut down the rear main camera, and the YUV image frames from the rear main camera are not sent to the display independently. Instead, they need to be spliced with the YUV image frames from the front main camera and sent together for display.
[0273] S39. Like the rear main camera, the front main camera can also transmit the captured raw image frames (RAW) to the ISP.
[0274] The ISP can also perform a series of processing on the RAW image and output the YUV image frame of the front main camera. The YUV image frame of the front main camera needs to be combined with the YUV image frame of the rear main camera to form a stitched preview image to achieve the function of simultaneous front and rear camera photo preview.
[0275] The S40.ISP can transmit the stitched image frames from the front and rear main cameras to the display.
[0276] Here, S40 only illustrates the flow path of image data, and does not mean that the ISP directly sends the spliced image frame to the display screen. Its implementation includes the process of sending it for display.
[0277] S41. The display screen displays the spliced image frame.
[0278] If the first camera needs to reduce the exposure time, the exposure time of the first camera can be set based on the flicker detection result to avoid water ripples in the imaging picture.
[0279] The above examples of "zoom," "flip camera," and "multi-channel photo or video" illustrate that the first camera can be changed. These scenarios are not limited to these; the electronic device can also switch the first camera in other scenarios. For example, when the photo mode is switched to "macro mode," the first camera can be switched to the macro camera.
[0280] In the embodiment of Figure 11, when the first camera switches, the second camera always uses the depth of field camera. This is not limited to the second camera, and the second camera can switch, and the embodiment of the application does not limit its switching strategy. For example, for an electronic device with only two cameras (a front camera and a rear camera), when a camera flip is detected, the first camera switches to the front camera, and the second camera can switch to the rear camera.
[0281] In the embodiment of FIG11 , each camera is coupled to one ISP. However, the present invention is not limited thereto, and an electronic device may be configured with multiple ISPs, and multiple cameras may be coupled to different ISPs.
[0282] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.
[0283] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.
[0284] Each step in the above method embodiments provided in this application can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as hardware processor execution, or can be implemented by a combination of hardware and software modules in the processor.
[0285] The present application also provides an electronic device, which may include a memory and a processor, wherein the memory may be used to store a computer program, and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.
[0286] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.
[0287] In one possible design, the chip system also includes a memory, which is used to store program instructions and data. The memory is located inside or outside the processor.
[0288] The chip system can be composed of chips, or can include chips and other discrete devices.
[0289] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.
[0290] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.
[0291] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.
[0292] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.
[0293] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.
[0294] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive Solid State Disk), etc.
[0295] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.
[0296] In short, the above are only embodiments of the technical solution of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>
Claims
1. A motion capture method, characterized in that: include: The electronic device starts a first camera; The electronic device displays a preview image on a display screen, the preview image being from an image captured by the first camera, and caches the latest captured multiple frames of images; During preview, the electronic device determines whether there is a moving object in the preview image, and if there is a moving object, reduces the exposure time of the first camera during preview; The electronic device detects a photo-taking operation; In response to the photo-taking operation, the electronic device saves photos, where the photos come from the cached multiple frames of images that have been most recently captured.
2. The method according to claim 1, characterized in that The reducing the exposure time of the first camera during preview specifically includes: During preview, the exposure time of the first camera is reduced according to the moving speed of the moving object. The faster the moving speed of the moving object is, the more the exposure time of the first camera is reduced, and the reduced exposure time of the first camera is shorter.
3. The method according to claim 1 or 2, characterized in that If the preview is a high dynamic range HDR preview, the method further includes: during the preview, applying different exposure values EV at different frame times to cache multiple frames of images with different brightness.
4. The method according to claim 3, characterized in that The applying different exposure values EV at different frame moments specifically includes: using the same exposure time but different exposure gains at different frame moments, or using different exposure times and different exposure gains at different frame moments, or using the same exposure gain but different exposure times at different frame moments.
5. The method according to any one of claims 1 to 4, characterized in that If the preview is not an HDR preview, before the electronic device saves the photo, the method further includes: The electronic device uses a reference frame and a multi-frame preferred frame to perform multi-frame fusion noise reduction processing to obtain an image frame after the multi-frame fusion noise reduction processing; the reference frame is an image frame with the highest clarity in a preview buffer area, and the multi-frame preferred frame is selected from the cached image frames other than the reference frame.
6. The method according to claim 5, characterized in that The electronic device saving the photo specifically includes: saving the image frames after the multi-frame fusion noise reduction processing as the photo; The photo comes from the cached multiple frames of the most recently acquired images, specifically including: the photo comes from the reference frame and the multiple preferred frames, and is generated by the fusion of the reference frame and the multiple preferred frames.
7. The method according to any one of claims 1 to 6, characterized in that If the preview is an HDR preview, before the electronic device saves the photo, the method further includes: The electronic device uses a reference frame and multiple image frames with different exposure values EV to perform HDR fusion processing to obtain an HDR fused image frame; the reference frame is the image frame with the highest clarity in the preview buffer area, and the multiple image frames with different exposure values EV are selected from the cached image frames other than the reference frame.
8. The method according to claim 7, characterized in that The electronic device saving the photo specifically includes: saving the HDR fused image frame as the photo; The photo comes from the cached multi-frame images that are most recently captured, specifically including: the photo comes from the reference frame and the multi-frame images with different exposure values EV, and is generated by the fusion of the reference frame and the multi-frame images with different exposure values EV.
9. The method according to any one of claims 1 to 8, characterized in that Also includes: During preview, the electronic device starts a flickering light source detection device to perform flickering light source detection.
10. The method according to claim 9, characterized in that The reducing the exposure time of the first camera during preview specifically includes: If a flickering light source is detected during preview, the electronic device controls the reduced exposure time to be an integer multiple of the light energy cycle, and the shortest exposure time is set to one times the light energy cycle.
11. The method according to claim 9 or 10, characterized in that The reducing the exposure time of the first camera during preview specifically includes: If no flickering light source is detected during preview, the electronic device reduces the exposure time to less than the light energy period.
12. The method according to any one of claims 9 to 11, characterized in that The flickering light source detection device includes: a second camera, the second camera is a rolling shutter camera, the exposure time of the second camera is less than the light energy cycle, and the image collected by the second camera will not be displayed.
13. The method according to claim 12, characterized in that The electronic device starts a flickering light source detection device to perform flickering light source detection, specifically including: The electronic device performs image recognition on the image captured by the second camera to determine whether the image content contains water ripple features. If so, it determines that a flickering light source is detected; the detection result is used to control the exposure time of the first camera.
14. The method according to any one of claims 1 to 5, characterized in that Before the electronic device starts the flashing light source detection device to perform flashing light source detection, it also includes: the electronic device determines that the ambient brightness is neither an overly bright environment nor an overly dark environment, wherein the brightness of the overly bright environment is higher than a first high brightness, and the brightness of the overly dark environment is lower than a first low brightness.
15. The method according to claim 14, characterized in that The electronic device determines that the ambient brightness is neither too bright nor too dark, specifically including: determining that the ambient brightness is not too bright according to the exposure time set for the first camera by the automatic exposure control AEC is greater than a third time duration; determining that the ambient brightness is not too dark according to the exposure time set for the first camera by the automatic exposure control AEC is less than a fourth time duration, and the third time duration is less than the fourth time duration.
16. The method according to claim 15, characterized in that The third duration and the fourth duration are recorded in an exposure table of the first camera.
17. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 16 is executed.
18. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 16.
19. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 16 is executed.
Citation Information
Patent Citations
Generation method of high-dynamic range image, mobile terminal and storage medium
CN109286758A
Shooting method and related device
CN115529409A
Exposure parameter determination method and device
CN116074634A
Motion snapshot method and electronic equipment
CN117692761A
Flickering light source detection method and electronic equipment
CN117692781A
