Motion capture method and electronic equipment

Abstract

The embodiment of the present application provides a motion capture method and an electronic device, wherein the electronic device reduces the exposure time and enters a short exposure mode when a moving object is detected during preview, rather than reducing the exposure time after the user presses the shutter. In this way, the cached preview frame has an exposure reduction effect, and once a photo operation is detected, the electronic device can obtain the photo to be saved based on the cached preview frame without waiting for the short exposure time to take effect before the photo is output, thereby improving the problems of slow imaging speed and shutter lag during motion capture.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a motion capture method and an electronic device. Background Art

[0002] With the development of electronic technology, electronic devices such as mobile phones and tablets are generally equipped with cameras. Users can use such electronic devices to capture various objects in motion to record their wonderful moments in motion. However, taking good photos of moving objects, especially taking clear photos of moving objects, usually requires more complex exposure settings, and most users have little knowledge of camera exposure, so the operation is difficult. Summary of the invention

[0003] The present application provides a motion capture method and an electronic device.

[0004] In the first aspect, the present application provides a motion capture method, which may include: an electronic device starts a first camera, displays a preview image on a display screen, the preview image comes from an image captured by the first camera, and the electronic device caches the latest captured multiple frames of images (also called cached frames). During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, the exposure time of the first camera is reduced during preview, for example, the exposure time is reduced to a shorter than the light energy cycle. The electronic device can detect a photo operation, such as a shutter button being pressed, and in response to the photo operation, the electronic device saves the photo, which may come from the cached frame.

[0005] In the first aspect, the activation of the first camera may be triggered by an event that the user opens a camera application. When the first camera is activated, the exposure time of the first camera may be a first duration, which is relatively long and not suitable for capturing moving objects, and prone to motion smear. For example, the first duration may be greater than the light energy period and an integer multiple of the light energy period, such as 50ms.

[0006] By using the method of the first aspect, the electronic device reduces the exposure time and enters the short exposure mode when it detects a moving object during preview, rather than reducing the exposure time after the user presses the shutter. In this way, the cached preview frame has the effect of reducing exposure, and once a photo operation is detected, the electronic device can obtain the photo to be saved based on the cached preview frame without waiting for the short exposure time to take effect before the photo can be output, so the problems of slow imaging speed and shutter lag are improved.

[0007] In combination with the first aspect, in some embodiments, when reducing the exposure time of the first camera, the electronic device may further detect the movement speed (also a relative speed) of the moving object in the preview image, and control the exposure reduction according to the movement speed: the faster the movement speed, the more the exposure time is reduced, and the shorter the reduced exposure time. The movement speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames of images with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the movement speed.

[0008] In combination with the first aspect, in some embodiments, if the preview is a high dynamic range HDR preview, the method may further include: during the preview, applying different exposure values ​​EV at different frame moments to cache multiple frames of images with different brightness. Wherein, applying different exposure values ​​EV at different frame moments may specifically include: using the same exposure time but different exposure gains at different frame moments, or using different exposure times and different exposure gains at different frame moments, or using the same exposure gain but different exposure times at different frame moments.

[0009] In combination with the first aspect, in some embodiments, if the preview is an HDR preview, before the electronic device saves the photo, the method may further include: the electronic device uses a reference frame and multiple frames of images with different exposure values ​​EV to perform HDR fusion processing to obtain an HDR fused image frame; the reference frame is the image frame with the highest clarity in the preview buffer, and the multiple frames of images with different exposure values ​​EV are selected from cached image frames outside the reference frame. In this way, the electronic device saves the photo, which may specifically include: saving the HDR fused image frame as a photo. The photo comes from the latest captured multiple frames of the cache, which specifically means here that the photo comes from the reference frame and multiple frames of images with different exposure values ​​EV, and is generated by the fusion of the reference frame and multiple frames of images with different exposure values ​​EV.

[0010] In combination with the first aspect, in some embodiments, if the preview is not an HDR preview, before the electronic device saves the photo, the method may further include: the electronic device uses the reference frame and the multi-frame preferred frame to perform multi-frame fusion noise reduction processing to obtain an image frame after the multi-frame fusion noise reduction processing; the reference frame is the image frame with the highest clarity in the preview buffer, and the multi-frame preferred frame is selected from the cached image frames other than the reference frame. In this way, the electronic device saves the photo, which may specifically include: saving the image frame after the multi-frame fusion noise reduction processing as a photo. The photo comes from the latest multi-frame image captured in the cache, which specifically means here: the photo comes from the reference frame and the multi-frame preferred frame, and is generated by the fusion of the reference frame and the multi-frame preferred frame.

[0011] In this way, when the user intends to take a photo, the electronic device can quickly take out multiple frames of images used to generate the photo from the cache frames, and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality; moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented.

[0012] In combination with the first aspect, in some embodiments, during preview, the electronic device may also start a flickering light source detection device to detect flickering light sources. Reducing the exposure time of the first camera during preview may specifically include: if a flickering light source is detected during preview, the electronic device controls the reduced exposure time to be an integer multiple of the light energy cycle, and the shortest exposure time is set to one light energy cycle; if no flickering light source is detected during preview, the electronic device sets the exposure time to less than the light energy cycle.

[0013] The flicker light source detection device may include: an ambient light sensor (ALS) and a multispectral sensor. The ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', the energy change of the flicker light source can be restored by reading the data of the ALS, thereby confirming whether there is a flicker light source in the current environment. The multispectral sensor can directly read the energy change information of the flicker light source.

[0014] In combination with the first aspect, in some embodiments, the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.

[0015] The second camera is not specially introduced for flickering light source detection, and the images it collects will not be displayed. It is usually used as a depth of field camera, a wide-angle camera or a macro camera, and its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms, so that the images collected by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of operating frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple operating frequencies. Taking the two operating frequencies of 50Hz and 60Hz as examples, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).

[0016] In combination with the first aspect, in some embodiments, when the flickering light source detection device is a second camera, the electronic device starts the flickering light source detection device to perform flickering light source detection, which may specifically include: the electronic device performs image recognition on the image captured by the second camera, and determines whether the image content contains water ripple features. If so, it is determined that a flickering light source is detected; the detection result is used to control the exposure time of the first camera.

[0017] In combination with the first aspect, in some embodiments, before the electronic device starts the flashing light source detection device to perform flashing light source detection, the electronic device can also first determine that the ambient brightness is neither too bright nor too dark, wherein the brightness of the overbright environment is higher than a first high brightness, such as 10,000 lux, and the brightness of the overdark environment is lower than a first low brightness, such as 50 Lux.

[0018] In combination with the first aspect, in some embodiments, the exposure duration of the first camera may be set by an automatic exposure control AEC; the electronic device determines that the ambient brightness is neither too bright nor too dark, which may specifically include: determining that the ambient brightness is not too bright based on the exposure duration of the first camera being greater than a third duration (such as 20ms), determining that the ambient brightness is not too dark based on the exposure duration of the first camera being less than a fourth duration (200ms), and the third duration being less than the fourth duration. The third duration and the fourth duration are recorded in the exposure table of the first camera, for example, the third duration belongs to several shortest exposure times corresponding to high EV in the exposure table, and the fourth duration belongs to several longest exposure times corresponding to low EV in the exposure table.

[0019] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0020] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] It can be understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The motion capture method provided in the first embodiment of the present application is shown;

[0025] Figure 2 The change of image quality before and after exposure reduction is shown as an example;

[0026] Figure 3 The motion capture method provided in the second embodiment of the present application is shown;

[0027] Figure 4 The motion capture method provided in the third embodiment of the present application is shown;

[0028] Figure 5 The hardware structure of the electronic device of the embodiment of the present application is exemplarily shown;

[0029] Figure 6 The software architecture of the electronic device according to the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0030] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0031] The shutter is an important part of the camera, which is used to control the exposure time of the photosensitive film. When capturing a moving object, setting a faster shutter speed is the guarantee of successful capture, and the faster the object moves, the higher the shutter speed should be set. A fast shutter speed means a short exposure time. If a faster shutter speed (shorter exposure time) is used with the aperture unchanged, the exposure gain needs to be increased at the same time to increase the camera's sensitivity to avoid a decrease in the brightness of the image. Parameters such as aperture size, exposure time, and exposure gain can be called exposure parameters. There can be multiple sets of exposure parameters to achieve the same exposure value (EV).

[0032] In order to facilitate users to shoot moving objects, many smart devices have designed motion capture mode (or motion photo mode). When the user presses the shutter button, it will automatically calculate a suitable shutter speed for the photo frame, without the user having to worry about how to control the exposure. The photo frame will be saved as a photo. In theory, the photo frame is the image frame at the moment the shutter button is pressed, but in reality, considering issues such as shutter lag, the photo frame is generally slightly later than the moment the shutter button is pressed. However, this motion capture function still has some problems, as follows:

[0033] 1. The imaging speed is slower than still photography. You need to wait for the new exposure value automatically calculated by the smart device to take effect before applying new exposure parameters to the photo frame or re-exposing and reproducing the image, which will lead to the problem of slower imaging speed. This effect can mean that the camera needs to reconfigure and enable related components to adopt new exposure parameters, such as reconfiguring and enabling the shutter and its controller to take effect of the new exposure time, reconfiguring and enabling the exposure gain circuit (such as the sensor gain circuit, the signal processor gain circuit) and its controller to take effect of the new exposure gain.

[0034] 2. Shutter lag is more serious than still photography. Shutter lag refers to the delay from the moment the user presses the shutter button to the actual imaging moment. As mentioned in 1 above, since it is necessary to wait for the new exposure value to take effect, the time from the shutter button being pressed to the shutter starting to capture the image is longer.

[0035] 3. The image quality is worse than that of still photography. Since the smart device automatically calculates a new faster shutter speed for motion capture, it is necessary to increase the exposure gain at the same time to avoid a decrease in the brightness of the image. However, increasing the exposure gain will introduce more color noise and brightness noise, reducing the image quality.

[0036] 4. Banding problem: When using a rolling shutter camera to take pictures under a flickering light source, if the exposure time of the rolling shutter is less than the light energy cycle, uneven exposure will appear in the image, showing "banding".

[0037] Shutters can be divided into global shutter and rolling shutter, and the two have different exposure methods. Among them, the exposure method of the global shutter is: all pixels on the photosensitive film start to be exposed to light at the same time, and after experiencing the same exposure time, they end the exposure at the same time; the exposure method of the rolling shutter is: each row of pixels on the photosensitive film is exposed row by row in a rolling manner, and the exposure time of each row is the same until all pixels are exposed. The exposure start time and end time of different rows are different.

[0038] Shooting moving objects with a rolling shutter camera may cause the moving objects in the image to have a ghost image (referred to as motion ghosting), especially when the moving object is too fast, this deformation problem is more serious. Moreover, the slower the shutter speed (the longer the exposure time), the more serious the motion ghosting problem. The solution to this problem is to increase the shutter speed.

[0039] Taking pictures with a rolling shutter camera under certain light sources may also cause "water ripples" to appear in the image. These light sources are often driven by alternating current. The energy transmitted by alternating current is not constant, but changes with a fixed frequency (such as 50Hz or 60Hz), which causes periodic fluctuations in the light intensity of the light source. The period followed is the light energy cycle corresponding to the fixed frequency (also called the industrial frequency). For example, the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz, and the light energy cycle is 10 milliseconds, that is, the light source flickers 100 times in 1 second. In this article, this light source is also called a flickering light source.

[0040] For a rolling shutter camera, when the exposure time is less than the light energy cycle of the light source, since the exposure start and end times of each row of pixels on the photosensitive film are different, the cumulative light energy received by each row of pixels within the same exposure time is different, and light and dark stripes will appear on the image.

[0041] The various embodiments of the present application are intended to solve some or all of the above-mentioned problems, which are described below in conjunction with the accompanying drawings.

[0042] Figure 1 The motion capture method provided by the first embodiment is shown. The method can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more serious than that of still photography.

[0043] like Figure 1 As shown, the method may include:

[0044] S401: The electronic device starts a first camera.

[0045] S402: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.

[0046] As time passes, S402 may continue to be executed, and the preview stream from the first camera may continue to be displayed, so as to continuously provide a preview display to the user.

[0047] In addition, the electronic device may also provide a preview frame cache function to cache the latest few frames of images (hereinafter referred to as cache frames), so that when the user intends to take a photo, the image frames used to generate the photo can be quickly taken from the cache frames. In this article, the latest cached multi-frame images may refer to the multi-frame preview images that are earlier than the photo time but closest to the photo time. The photo time may refer to the time when the electronic device detects the photo operation, such as the time when the user presses the shutter button.

[0048] S403: During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S405; if there is no moving object, execute S404.

[0049] How to determine whether there is a moving object in the preview image will be introduced later, so I will not expand on it here.

[0050] S404: The electronic device maintains normal exposure of the first camera during preview.

[0051] Here, for controlling the exposure time, S404 is not an additional action to be performed. The electronic device can achieve normal exposure of the first camera without changing the exposure time of the first camera (maintaining the first duration).

[0052] S405: The electronic device reduces the exposure time of the first camera during preview, for example, reducing the exposure time to a value shorter than the light energy period.

[0053] S406: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.

[0054] Not limited to the shutter button being pressed, the photo operation may also include other user operations, such as voice commands with specific content (such as the voice command "YOYO, please take a photo!"), specific actions (such as jumping actions), specific expressions (such as laughing), etc.

[0055] S407: In response to the photo-taking operation, the electronic device saves the photo, which may be from a cache frame.

[0056] It can be seen that in the first embodiment, the electronic device reduces the exposure time and enters the short exposure mode when it detects a moving object during preview, rather than reducing the exposure time after the user presses the shutter. In this way, the cached preview frame has the effect of reducing exposure, and once the photo operation is detected, the electronic device can obtain the photo to be saved based on the cached preview frame, without having to wait for the short exposure time to take effect before the photo is output, so the problems of slow imaging speed and shutter lag are improved.

[0057] Figure 2 The following is an example showing the change in image quality before and after exposure reduction. Figure 2As shown, the 1st and 2nd frames are images of the initial normal exposure. At this time, the exposure time is 30ms. The rotating windmill in the preview screen has motion smear, and the image is blurred. The 51st and 52nd frames are images captured after reducing the exposure time during preview. The exposure time is reduced to 10ms. The rotating windmill in the preview screen no longer has motion smear, and the image is clear. The 53rd and 54th frames are images when the user presses the shutter button. Due to the obvious shaking of the mobile phone caused by the shutter pressing operation, the image appears blurred as a whole. The 55th frame is an image captured after the photo operation. At this time, the device returns to stability, the exposure time is 10ms, and the image is clear as a whole. In this embodiment, in response to the user's photo operation of clicking the shutter, the electronic device can generate photos based on the image frames with reduced exposure effect cached during preview (such as the 51st and 52nd frames), which can not only avoid the problem of motion smear, but also quickly generate photos.

[0058] In the first embodiment, the specific implementation of determining whether there is a moving object in the preview image (S403) may include: detecting the moving object by using the optical flow method between adjacent preview frames, and if the pixel movement area reaches a certain area threshold, it is considered that there is a moving object in the preview image. The motion detected in this way is the relative motion between the camera and the photographed object, which can be divided into three cases: case one, the camera remains stationary and the photographed object is in motion; case two, the photographed object remains stationary and the camera is in motion; case three, the photographed object and the camera are both moving, but the speeds of the two are different. The motion mentioned in these three cases refers to the motion relative to the ground, which can be called absolute motion.

[0059] Not limited to the optical flow method, the electronic device can also detect whether the electronic device itself (camera) is in motion through motion sensors such as gyroscopes and accelerometers. Once the electronic device is detected to be in motion, it can be considered that there is a moving object in the preview image. Because when the camera is in motion, the object being photographed is likely to be moving in the preview image, and the special case that both the object being photographed and the camera are moving at the same speed is excluded.

[0060] In the first embodiment, when the electronic device reduces the exposure time of the first camera, it can further detect the movement speed (also a relative speed) of the moving object in the preview image, and control the exposure reduction according to the movement speed: the faster the movement speed, the more the exposure time is reduced, and the shorter the exposure time after reduction. The movement speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames of images with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the movement speed.

[0061] Figure 3The motion capture method provided by the second embodiment is shown. Based on the technical problem solved by the first embodiment, the method provided by the second embodiment can further improve the problem that the imaging quality is worse than that of still photography.

[0062] like Figure 3 As shown, the method may include:

[0063] Reduce exposure time when previewing

[0064] S501: The electronic device starts a first camera.

[0065] S502: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.

[0066] S503: During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S505; if there is no moving object, execute S504.

[0067] S504: The electronic device maintains normal exposure of the first camera during preview.

[0068] S505: The electronic device reduces the exposure time of the first camera during preview.

[0069] The above S501-S505 can refer to steps S401-S405 in the first embodiment, and will not be repeated here.

[0070] In addition, the preview can specifically be a high dynamic range (HDR) preview, in which case the electronic device can also apply different EVs at different frame times to obtain multiple cache frames with different brightness (different EVs). Specifically, the electronic device can obtain multiple frames of preview images with different EVs by changing the exposure gain and exposure time. For example, the same exposure time but different exposure gains are used at different frame times to achieve different EVs, or different exposure times and different exposure gains are used at different frame times to achieve different EVs, or the same exposure gain but different exposure times are used at different frame times to achieve different EVs.

[0071] Multi-frame fusion noise reduction

[0072] S506: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.

[0073] S507. The electronic device determines whether the preview is an HDR preview. If not, execute S508-S511. If so, execute S512-S515.

[0074] S508: The electronic device may select a reference frame from the cached preview frames.

[0075] With reference to the first embodiment, it can be known that the cached preview frames refer to the latest cached multi-frame images, which are multi-frame images that are earlier than the photographing time but closest to the photographing time.

[0076] The reference frame may be the one with the highest definition among the multiple preview frames cached. The multiple frames selected in S509 are aligned according to the reference frame to eliminate the influence of motion on definition and enhance details. The algorithm used for the alignment of multiple frames may be a motion estimation and motion compensation algorithm commonly used in video coding and decoding technology, that is, the entire image is divided into a series of non-overlapping blocks, and the minimum value of the sum of squared difference (SSD) or the sum of absolute difference (SAD) between each block and each block of the reference frame is calculated to estimate the position or motion vector corresponding to the current block in the reference frame, and then compensation is performed based on this.

[0077] Specifically, the electronic device can analyze the image clarity of the multiple preview frames based on a clarity operator (such as a sobel operator), and can further consider the device jitter factor, shutter time, etc. at the multiple preview frame moments to select a reference frame. The device jitter factor can be determined based on motion data collected by motion sensors such as gyroscopes and accelerometers. It is essentially a factor that reflects the degree of device (camera) jitter. The larger it is, the more severe the device jitter is, which will cause image blur and be detrimental to image clarity. The shorter the shutter time, the less severe the motion smear problem is, and the more conducive to image clarity.

[0078] The electronic device can specifically run an algorithm model to consider the impact of one or more of a clarity operator, a device jitter factor, and a shutter time on image clarity. The algorithm model can be a neural network model.

[0079] S509: The electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.

[0080] S510: The electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.

[0081] There are many methods for multi-frame fusion noise reduction, such as adding and averaging, which is not limited in this embodiment.

[0082] S511. The electronic device saves the fused image frames as photos.

[0083] That is, the saved photos mentioned in S407 in the first embodiment come from the cache frame, which specifically means that the saved photos come from the reference frame selected in S508 and the multiple frames of images selected in S509, and are generated by the fusion of them.

[0084] S512: The electronic device may select a reference frame from the cached preview frames.

[0085] S513: The electronic device may select multiple image frames with different exposure values ​​EV from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.

[0086] The premise for implementing S512 is that, in order to implement HDR fusion, the electronic device reduces the exposure time in the preview while also changing the exposure gain and exposure time to obtain multiple preview frames with different EVs. For example, under the premise of shortening the exposure time relative to the first duration, the same exposure time but different exposure gains are used at different frame moments to achieve different EVs, or different exposure times and different exposure gains are used at different frame moments to achieve different EVs, or the same exposure gain but different exposure times are used at different frame moments to achieve different EVs.

[0087] S514: The electronic device uses the selected reference frame and multiple frames of images with different EVs to perform HDR fusion to obtain fused image frames.

[0088] Like multi-frame fusion noise reduction, HDR fusion also includes multi-frame image alignment based on the reference frame for multiple frames of images with different EVs, and then multi-frame fusion noise reduction for the multiple frames. The difference between the two is that HDR fusion can also perform brightness fusion (also known as exposure fusion) based on multiple frames of images with different brightness (different EVs).

[0089] S515: The electronic device saves the image frame after HDR fusion processing as a photo.

[0090] That is, the saved photos mentioned in S407 in the first embodiment come from the cache frame, which specifically means that the saved photos come from the reference frame selected in S512 and the multi-frame images with different EVs selected in S513, and are generated by their fusion.

[0091] It can be seen that when the user intends to take a photo, the electronic device in Example 2 can quickly take out multiple frames of images used to generate photos from the cache frame, and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality; moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented. At the same time, for the case of motion capture, the algorithm effect parameters and ISP parameters of the HDR algorithm or multi-frame fusion are specially adapted, and the effect is debugged for high exposure gain and short exposure time. Among them, the algorithm effect parameters can be, for example, noise reduction intensity, contrast intensity, sharpening intensity, fusion ratio, ghost threshold, motion threshold, matching threshold, fusion pyramid size and other common fusion, noise reduction, and dynamic range algorithm adjustable parameters (parameters may vary depending on the algorithm prototype). ISP debugging parameters can include automatic exposure, automatic focus, automatic white balance parameters (also known as 3A parameters), lens shading correction (LSC), gamma (GAMMA), dynamic range correction (DRC), YUV noise reduction (YNR) and other image debugging effect parameters. For example, YNR is a common YUV noise reduction parameter of ISP, which reduces the noise impact caused by high exposure gain during motion by targeted adjustment. In this way, frame selection, image fusion algorithm and parameter separation are used to compensate for the noise caused by the increase in exposure gain, while enhancing the image effect.

[0092] Figure 4 The motion capture method provided by the third embodiment is shown. In this embodiment, the first camera may be a rolling shutter camera, and under a flickering light source, water ripples may appear in the preview image provided by it. Based on the technical problem solved by the second embodiment, the method provided by the third embodiment can further solve the water ripple (banding) problem.

[0093] like Figure 4 As shown, the method may include:

[0094] Reduce exposure time when previewing

[0095] S601: The electronic device starts a first camera.

[0096] S602: The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.

[0097] S603: During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S604; if there is no moving object, execute S605.

[0098] S604: The electronic device maintains normal exposure of the first camera during preview.

[0099] The above S601 - S604 may refer to steps S501 - S504 in the second embodiment, which will not be described in detail here.

[0100] S605: The electronic device reduces the exposure time of the first camera during preview.

[0101] In this embodiment, the electronic device can reduce the exposure time while taking into account the water ripple problem. The implementation of S605 may specifically include S6051-S6055:

[0102] S6051. Determine whether to perform flicker detection.

[0103] S6052, start the flashing light source detection device.

[0104] The flicker light source detection device may include: an ambient light sensor (ALS) and a multispectral sensor. The ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', the energy change of the flicker light source can be restored by reading the data of the ALS, thereby confirming whether there is a flicker light source in the current environment. The multispectral sensor can directly read the energy change information of the flicker light source.

[0105] That is, dedicated flicker detectors such as high-sampling-rate ambient light sensors and multi-spectral flicker detectors can be used to detect whether there is a flickering light source in the shooting environment, and cooperate with the camera to solve the water ripple problem. However, such devices require additional costs and design space.

[0106] To this end, the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.

[0107] The second camera is not specially introduced for flickering light source detection. It is usually used as a depth of field camera, a wide-angle camera or a macro camera. Its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms. Therefore, the image captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of operating frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple operating frequencies. Taking 50Hz and 60Hz as examples, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).

[0108] In addition, the exposure time referred to by normal exposure can be set to an integer multiple of the light energy cycle, which can avoid the user seeing water ripples when previewing during normal exposure.

[0109] S6053, perform flickering light source detection through the flickering light source detection device, if no flickering light source is detected, execute S6054, otherwise execute S6055.

[0110] Specifically, when the flicker light source detection device is a second camera, the specific implementation of S6053 may include: the electronic device performs image recognition on the image captured by the second camera to determine whether the image content contains water ripple features, and if so, it is considered that there is a flicker light source in the shooting environment. The image captured by the second camera will not be sent for display and is only used for flicker detection, so the user will not perceive it.

[0111] S6054. Ignore the water ripple problem and reduce the exposure time as expected, for example, reduce the exposure time to less than the light energy period, for example, 9 milliseconds.

[0112] Here, expectation may mean: reducing the exposure time according to the moving speed of the moving object, the image quality in HDR and other scenarios. For example, when the object moves very fast, the exposure time is usually reduced to a very short time, such as 5 milliseconds, which is shorter than the light energy cycle.

[0113] S6055. If a flickering light source is detected, the reduced exposure time can be controlled to be an integer multiple of the light energy cycle (such as 1 times, 2 times, etc.). The shortest exposure time can be set to 1 times the light energy cycle, such as 10ms.

[0114] As shown in S6051, before starting flicker detection, the electronic device can also determine that the ambient brightness is appropriate, such as the indoor lighting environment, based on the exposure time set by the automatic exposure control (AEC) for the first camera, and then determine that flicker detection is required. This is because the exposure time of the photosensitive sensor set by the AEC can reflect the brightness of the shooting environment to a certain extent.

[0115] Flicker detection is generally not required in a shooting environment that is too bright or too dark, because in a too bright environment, the main consideration is the overexposure problem, and the exposure time needs to be significantly reduced, and it is difficult to take into account the water ripple problem; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise the problem of underexposure may easily occur. Here, too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the AEC exposure time belongs to these shortest exposure times, it is considered that the environment is too bright. For another example, the longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the AEC exposure time belongs to these longest exposure times, it is considered that the environment is too dark. That is, the reference exposure value for too bright or too dark can be derived from the exposure table. Then, if the AEC exposure time belongs to the remaining exposure time in the exposure table, it can reflect that the ambient brightness is neither too bright nor too dark. The electronic device is not limited to analyzing the ambient brightness by the exposure time set by the AEC for the first camera. The electronic device can also collect ambient light through an ambient light brightness sensor to determine whether the shooting environment is too bright or too dark. This application does not impose any restrictions on its technical implementation.

[0116] Multi-frame fusion noise reduction

[0117] S606: The electronic device may detect a photo-taking operation, such as a shutter button being pressed.

[0118] S607. The electronic device determines whether the preview is an HDR preview. If not, execute S608-S611. If so, execute S612-S615.

[0119] S608: The electronic device may select a reference frame from the cached preview frames.

[0120] S609: The electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.

[0121] S610: The electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.

[0122] S611: The electronic device saves the fused image frame as a photo.

[0123] S612: The electronic device may select a reference frame from the cached preview frames.

[0124] S613: The electronic device may select multiple image frames with different exposure values ​​EV from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.

[0125] S614: The electronic device performs HDR fusion using the selected reference frame and multiple image frames with different EVs to obtain fused image frames.

[0126] S615: The electronic device saves the image frame after HDR fusion processing as a photo.

[0127] The above S606-S615 can refer to steps S506-S515 in the second embodiment, which will not be repeated here.

[0128] In addition, the specific implementation of reducing the exposure time described in S6051-S6055 can also be combined with the first embodiment as an implementation method of step S405 in the first embodiment.

[0129] Next, the software and hardware architecture of the electronic device provided in the embodiment of the present application is introduced.

[0130] Electronic equipment can be equipped Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.

[0131] Figure 5 The electronic device 100 provided by the embodiment of the present application is exemplarily shown. The electronic device 100 can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more serious than that of still photography.

[0132] like Figure 5 As shown, the electronic device 100 may include: a processor 110, a memory 120, a camera 130, a display screen 140, and a flickering light source detection device (not shown).

[0133] The processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. These interfaces are used for data exchange between the processor 110 and peripherals.

[0134] The memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store data of users and applications. The non-volatile memory can also store executable programs and store data of users and applications, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110. A storage unit can also be set in the processor 110, and the storage unit can be a high-speed cache storage unit, which can be used to save instructions or data just used or cyclically used by the processor 110. The implementation code of the flickering light source detection method provided in the embodiment of the present application can be stored in the NVM. When the camera application is started, the code can be loaded into the RAM. In this way, the processor 110 can directly read the program code from the RAM to implement the flickering light source detection method provided in the embodiment of the present application. In addition, image files such as photos and videos obtained by the user using the camera application can be written into the NVM for storage for the user to browse.

[0135] The image captured by the camera 130 can be displayed to provide a preview image. The camera 130 may include: a lens, a photosensitive sensor and a flexible printed circuit board (FPCB). The FPCB is responsible for connecting other components of the camera 130 with the processor 110, for example, transmitting the raw data output by the photosensitive sensor to the processor 110. When taking a picture, the shutter of the camera 130 is opened, and light enters and shines on the photosensitive sensor. The photosensitive sensor converts the light signal into an electrical signal, and then further converts the electrical signal into a digital signal through analog digital convert (ADC) to be transmitted to the ISP for processing. The ISP can perform the following processing on the output data of the photosensitive sensor: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, removal of bad pixels, etc. The ISP can also be integrated into the camera 130.

[0136] The display screen 140 can be used to display the image captured by the camera 130. The image processed by the ISP will be sent to the display screen 140 to show the user a preview of the image captured by the camera. Here, sending to the display screen means pushing the image captured by the camera to the frame buffer (FB) for storage. The frame buffer is a storage space that can be located in the video memory or the internal memory, and is used to store the rendering data processed by the graphics card chip or to be extracted. The content of the frame buffer corresponds to the interface display on the display screen 140, and it can be simply understood as the cache corresponding to the content displayed on the display screen 140. Modifying the content in the frame buffer is to modify the content on the display screen 140.

[0137] The flicker light source detection device can be used for flicker detection, and specifically may include an ambient light sensor (ALS) and a multispectral sensor. Among them, the ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', the energy change of the flicker light source can be restored by reading the data of the ALS, thereby confirming whether there is a flicker light source in the current environment. The multispectral sensor can directly read the energy change information of the flicker light source. The flicker light source detection device can also be a rolling shutter camera (called the second camera) on an electronic device. The image captured by the second camera will not be sent for display and is only used for flicker detection, so the user does not perceive it. The second camera is not specially introduced for flicker detection. It can usually be a depth of field camera, a wide-angle camera, a macro camera, etc. It has a low resolution and low power consumption. It is a non-primary camera. The embodiment of the present application gives it a new function. The second camera can also be one or more cameras.

[0138] In addition to the flicker detection results provided by the second camera, the exposure time of the camera that provides the preview display (referred to as the first camera) can also be controlled by the automatic exposure control (AEC) function. In short, AEC automatically adjusts the exposure time according to the light intensity. In different environments, the intensity of light varies greatly, and the exposure time of the photosensor in the camera 130 also needs to be adapted accordingly to ensure normal exposure of the image. The automatic exposure control (AEC) provided by the ISP provides this adaptive capability. Therefore, the exposure time of the photosensor can reflect the brightness of the shooting environment to a certain extent.

[0139] In addition, the electronic device 100 may also store an exposure table to be used by the AEC. The exposure table of a camera sets the exposure parameters (such as exposure time, exposure gain, aperture size) to be used by the camera at different exposure values ​​(EV), which may be debugged and recorded by engineers.

[0140] like Figure 5As shown, the electronic device 100 may further include: an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, and an earphone interface 150D.

[0141] Among them, the electronic device can realize audio functions through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, the headphone interface 150D, and the application processor (AP). For example, music playing, recording, etc. The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 are arranged in the processor 110. The speaker 150A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The electronic device can listen to music or listen to hands-free calls through the speaker 150A. The receiver 150B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device answers a call or voice message, the voice can be answered by placing the receiver 150B close to the human ear. Microphone 150C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 150C to input the sound signal into the microphone 150C. The electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 150C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording functions, etc. The headphone jack 150D is used to connect wired headphones. The headphone jack 150D can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0142] like Figure 5 As shown, the electronic device 100 may further include a sensor module 160, which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.

[0143] Figure 5The illustrated structure does not constitute a specific limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0144] Figure 6 The main software architecture involved in detecting flickering light sources using another rolling shutter camera on the electronic device 100 is shown.

[0145] like Figure 6 As shown, the software architecture may include the following functions or modules: camera application (camera apk), camera service (camera service), camera hardware abstraction layer (camera HAL), flicker detection (CameraFlickerDetector), camera VNDK and Kernel. Among them,

[0146] The camera application (camera apk) can be located in the application layer. In addition to the camera application, the application layer can include a series of other applications, such as the gallery application. The camera application can be mainly responsible for human-computer interaction, such as monitoring and responding to user operations such as selecting a shooting mode, adjusting the zoom ratio, and selecting a focus.

[0147] The camera service can be located in the application framework layer. The camera service can include interfaces such as camera management and camera devices, which can interact with the camera application through the application program interface (API) and with the camera HAL through the HAL interface definition language (HIDL). In addition, the camera service can also interact with the vendor native development kit (VNDK).

[0148] The camera hardware abstraction layer (camera HAL) can be the part of the virtual hardware function about the camera in the hardware abstraction layer (HAL). The hardware abstraction layer is located in the interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system, and the upper layer does not need to care about the underlying hardware implementation. The camera HAL can communicate with the camera service through the HIDL interface, and can issue control operations for the camera through the standard hardware abstraction layer interface. The camera HAL can include a camera startup (CamEntry) module and an automatic exposure control processor (AECProcessor) module, wherein the CamEntry module can be used as the entrance of the HAL, receiving instructions such as open, close, stream configuration (configurestream), and request from the camera service, and is a general camera HAL function entrance. The AECProcessor module can provide ISP's AEC-related functions, such as providing the AEC parameters (such as exposure time) of the first camera to the CameraFlickerDetector, so that the CameraFlickerDetector can determine whether the current shooting environment requires flicker detection.

[0149] The flicker detection (CameraFlickerDetector) module can be located in the application framework layer (framework) or the application layer. The CameraFlickerDetector module can obtain the AEC parameters (such as exposure time) of the first camera from the camera HAL through the HIDL interface, so as to decide whether to start the second camera for flicker detection. When it is determined that flicker detection is to be performed, the CameraFlickerDetector module can issue control operations to start the second camera and set the exposure time through the interface of the camera VNDK. The CameraFlickerDetector module may include a CameraHelper module and an ImageReader module, wherein the CameraHelper module can be used to implement the call of functions such as opening (open), closing (close), stream configuration (configurestream), and request (request) of the camera service of the second camera through the call of the VNDK interface, and the ImageReader module is mainly used for receiving and further processing the images sent after the second camera is started.

[0150] The Vendor Native Development Kit (VNDK) is a collection of libraries for vendors to develop their own HALs (vendors). The camera VNDK is the "own camera HAL" developed by the vendor of the second camera. The camera VNDK can be used to receive the control operation of starting the second camera issued by the CameraFlickerDetector, and forward the control operation to the cameraservice. The camera VNDK may include the ACameraManager module, the ACameraDevice module, the ACameraReader module, the ACaptureSession module, and the ACaptureRequest module. Among them, the ACameraManager module provides the function of accessing the camera service from the vendor layer, the ACameraDevice module provides the function of accessing the camera device from the vendor layer, the ACameraReader module is responsible for reading image data, such as the data captured by the second camera, the ACaptureSession module provides the function of allowing the vendor to manage the frames captured by the camera, and the ACaptureRequest module is responsible for the settings and output targets required for the camera to capture a single image.

[0151] The kernel part may include various hardware driver control modules, such as ISP, photosensor, DSP, AP and other driver controls.

[0152] Figure 6 Only the main software architecture involved in the flickering light source detection method provided in the embodiment of the present application is shown, which is essentially only a part of the software architecture of the electronic device 100.

[0153] Figure 6 The process of using another rolling shutter camera to detect flickering light sources is also shown, including:

[0154] 1. First camera startup process

[0155] The camera application calls the camera service, which authenticates the camera application. After authentication, the camera service calls the camera HAL to start the first camera through the camera HAL, and control the photosensor and ISP to output images. The image frame will be sent to the display to realize image preview, photo taking and other functions.

[0156] On this basis, in order to perform flicker detection without letting the user know, the electronic device 100 can start the second camera to perform detection. The second camera is not specially introduced for flicker detection. It can usually be a depth of field camera, a wide-angle camera, or a macro camera, etc. It has a low resolution and low power consumption. It is a non-primary camera. The embodiment of the present application gives it a new function.

[0157] 2. Flicker detection process

[0158] First, when the preview of the first camera is turned on, the AECProcessor module in the camera HAL can pass the exposure time and other parameters of the first camera to the CameraFlickerDetector module through the HIDL interface. The exposure time set by the AEC can reflect the brightness of the shooting environment. Therefore, the CameraFlickerDetector module can determine whether to perform flicker detection based on the exposure time. The environment for flicker detection is usually an indoor scene. Flicker detection is generally not required in a shooting environment that is too bright or too dark, because in a too bright environment, the main consideration is the overexposure problem and the exposure time needs to be significantly reduced, which makes it difficult to take into account the problem of avoiding water ripples; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise it is easy to cause underexposure. Here, too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the actual exposure time belongs to these shortest exposure times, it is considered that the environment is too bright. For another example, the longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the actual exposure time falls within these longest exposure times, it is considered that the environment is too dark.

[0159] Then, after determining that flicker detection is to be performed, the CameraFlickerDetector module transmits a control command to start the second camera to cameraVNDK.

[0160] The camera VNDK then passes the CameraFlickerDetector module to the camera service to start the control operation of the second camera.

[0161] After authenticating the CameraFlickerDetector module, the camera service calls the camera HAL again to start the second camera through the camera HAL, so that the second camera can be started from the camera HAL side without involving the application layer, so the user does not perceive it. At the same time, the CameraFlickerDetector module controls the exposure of the second camera so that the exposure time of the second camera is less than the light energy cycle. For example, if the exposure time is set to 9ms, the image collected by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected. The CameraFlickerDetector module can also call the camera HAL to read the output image of the second camera, detect whether the image content has water ripple features through image recognition technology, and pass the detection results to the first camera. The output image of the second camera is not displayed and is only used for flicker detection.

[0162] After the first camera learns the flicker detection result, it can reduce the exposure time of the first camera to twice the light energy cycle, such as 10ms, after detecting a moving object in the preview image, so as to solve the problem of water ripples in the imaging image under the flickering light source.

[0163] The above software architecture newly introduces the CameraFlickerDetector module, and implements the interaction between the CameraFlickerDetector module and the camera HAL and camera service through VNDK and HIDL. It does not involve modifications to the camera application and various layers of the architecture, and has good portability, scalability, and maintainability.

[0164] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、

[0165] <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface. After being parsed and rendered, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0166] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0167] Each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0168] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0169] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0170] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0171] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0172] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0173] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0174] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0175] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0176] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0177] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive Solid State Disk), etc.

[0178] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0179] In short, the above are only embodiments of the technical solution of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A motion capture method, characterized in that: include: The electronic device starts a first camera, where the first camera is a main camera; The electronic device displays a preview image on a display screen, the preview image comes from the image captured by the first camera, and caches the latest captured multiple frames of images; wherein, if the preview is a high dynamic range (HDR) preview, different exposure values ​​EV are applied to the multiple frames of images; During preview, the electronic device starts the second camera to detect the flickering light source; the exposure time of the second camera is less than the light energy cycle, the second camera is a depth of field camera, and the image collected by the second camera will not be sent for display; During preview, the electronic device determines whether there is a moving object in the preview image, and if there is a moving object, reduces the exposure time of the first camera during preview; wherein, reducing the exposure time of the first camera during preview specifically includes: if a flickering light source is detected during preview, the electronic device controls the reduced exposure time to be an integer multiple of the light energy cycle, and the exposure time is set to be one light energy cycle at the shortest; if no flickering light source is detected during preview, the electronic device sets the exposure time to less than the light energy cycle; the electronic device detects a photo taking operation; The electronic device determines whether the preview is an HDR preview; If it is not an HDR preview, a multi-frame fusion noise reduction process is performed using the reference frame and the multi-frame preferred frame to obtain an image frame after the multi-frame fusion noise reduction process, and the image frame after the fusion process is saved as a photo; the reference frame is the image frame with the highest definition in the preview buffer area, and the multi-frame preferred frame is selected from the cached image frames other than the reference frame; If it is an HDR preview, the reference frame and multiple frames of images with different exposure values ​​EV are used to perform HDR fusion processing to obtain an HDR fused image frame; the reference frame is the image frame with the highest definition in the preview buffer area, and the multiple frames of images with different exposure values ​​EV are selected from the cached image frames other than the reference frame; Wherein, the shutters of the first camera and the second camera are both rolling shutters.

2. The method according to claim 1, characterized in that The reducing the exposure time of the first camera during preview specifically includes: During preview, the exposure time of the first camera is reduced according to the moving speed of the moving object. The faster the moving speed of the moving object is, the more the exposure time of the first camera is reduced, and the reduced exposure time of the first camera is shorter.

3. The method according to claim 1, characterized in that Applying different exposure values ​​EV at different frame moments specifically includes: using the same exposure time but different exposure gains at different frame moments, or using different exposure times and different exposure gains at different frame moments, or using the same exposure gain but different exposure times at different frame moments.

4. The method according to any one of claims 1 to 3, characterized in that The electronic device starts the second camera to detect the flickering light source, specifically including: The electronic device performs image recognition on the image captured by the second camera to determine whether the image content contains water ripple features. If so, it determines that a flickering light source is detected; the detection result is used to control the exposure time of the first camera.

5. The method according to any one of claims 1 to 4, characterized in that Before the electronic device starts the second camera to detect the flashing light source, it also includes: the electronic device determines that the ambient brightness is neither too bright nor too dark, wherein the brightness of the overbright environment is higher than the first high brightness, and the brightness of the overdark environment is lower than the first low brightness.

6. The method according to claim 5, characterized in that The electronic device determines that the ambient brightness is neither too bright nor too dark, specifically including: determining that the ambient brightness is not too bright according to the exposure time set for the first camera by the automatic exposure control AEC is greater than a third time duration; determining that the ambient brightness is not too dark according to the exposure time set for the first camera by the automatic exposure control AEC is less than a fourth time duration, and the third time duration is less than the fourth time duration.

7. The method according to claim 6, characterized in that The third duration and the fourth duration are recorded in an exposure table of the first camera.

8. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method according to any one of claims 1 to 7 is executed.

9. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 7 is executed.

Images