Snapshot processing method and electronic equipment

Abstract

The embodiment of the invention provides a snapshot processing method and electronic equipment, and relates to the technical field of terminals. The problem that the electronic device occupies a large amount of memory resources in the snapshot process is solved. According to the specific scheme, a first image frame is collected, and the first image frame is stored in a first queue; the first queue further comprises a second image frame; the acquisition time of the second image frame is earlier than that of the first image frame; acquiring a third image frame; wherein the acquisition time of the first image frame is earlier than that of the third image frame; the first image frame, the second image frame and the third image frame all comprise a target shooting object; when the first queue meets a first condition, not storing the third image frame; the first condition is a condition indicating that a target image frame is stored in the first queue, and the target image frame is an image frame showing that a target shooting object is located at a vertex or a bottom point of a motion track; and in response to a first operation of snapshot indicated by the user, generating a corresponding photographed image based on the target image frame.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a snapshot processing method and electronic device. Background Technology

[0002] With the development of electronic technology, camera functionality has become an essential feature of electronic devices. However, activating camera functions consumes a significant amount of memory resources. In particular, when using camera functions based on complex algorithms (such as snapshot capabilities), the memory usage can reach 1-2GB. Since the memory resources of electronic devices are ultimately limited, excessive memory usage can lead to problems such as lag and stuttering. Summary of the Invention

[0003] This application provides a snapshot processing method and electronic device to reduce the memory resources occupied during the activation of the snapshot function.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] Firstly, this application provides a snapshot processing method. This method is applied to an electronic device. Before actually performing a snapshot, the electronic device can acquire and display a preview stream. The preview stream consists of multiple consecutively acquired image frames. The electronic device can store the consecutively acquired image frames into a first queue. The first queue is a cache queue used to store the raw image data acquired during the snapshot process. For example, when a second image frame is acquired, it is stored in the first queue. Then, when a first image frame is acquired, it is also stored in the first queue. During the acquisition of the preview stream, the electronic device can acquire target image frames, that is, image frames that record the target point of the subject's motion trajectory. The target point is the point among the trajectory points corresponding to the motion trajectory that is farthest or closest to a preset reference line. These target image frames are also the exciting frames that the user expects to capture.

[0006] After the target image frame is stored in the first queue, the first queue satisfies a first condition. Once the first queue satisfies the first condition, the electronic device can discard newly acquired image frames. For example, if a third image frame is acquired, and the acquisition time of the third image frame is later than that of the first image frame, after the first image frame is stored in the first queue, and the first condition is met, the third image frame can be discarded. Then, in response to the user's instruction to capture the image, the electronic device can retrieve the target image frame from the first queue and generate the corresponding photographed image based on the target image frame.

[0007] In the above embodiments, before detecting the first operation of user-instructed image capture, a decision is made on which image frames need to be stored in the first queue and which image frames can be discarded, based on whether the first queue meets a first condition. While saving the target image frame, it is not necessary to store all image frames in the preview stream, reducing the memory resource consumption of image capture.

[0008] In some examples, if the first queue meets the first condition and is full of image frames, and a third image frame is acquired, the third image frame can be discarded. Understandably, if the first queue does not meet the first condition and is full of image frames, after adding the newly acquired image frame to the first queue, the image frame with the earliest entry time in the first queue is deleted. By not adding any new image frames to the first queue when it meets the first condition and is full, the method of preventing the target image frame in the first queue from being replaced by other image frames is adopted.

[0009] Accordingly, in some embodiments, the electronic device also needs to determine that the first queue is full of image frames before discarding the third image frame.

[0010] In this example, even if the first queue is short, the problem of the target image frame being deleted when the user instructs to capture the image will not occur. Therefore, when configuring the first queue, the electronic device can minimize its queue length; for example, the first queue can be configured to store only 3 or 4 image frames, significantly reducing memory resource usage compared to existing technologies.

[0011] In other examples, if the first queue is not full, even if the first queue meets the first condition, newly acquired image frames can still be stored in the first queue normally. For example, after the first image frame is stored in the first queue, and before the third image frame is acquired, the electronic device acquires the fifth image frame. When the first queue meets the first condition, it is determined that the first queue is not full, and the fifth image frame is stored in the first queue.

[0012] Understandably, after the first queue meets the first condition, if the first queue is not full of image frames, the newly acquired image frames are stored in the first queue without affecting the image frames already stored in the first queue. This increases the number of image frames actually stored in the first queue, increases the probability that the user's desired capture of the best frames has been stored in the first queue, and improves the fault tolerance of the method.

[0013] In other possible examples, if the first queue meets the first condition, the third image frame can be discarded regardless of whether the first queue is full of image frames. This way, there's no need to check if the first queue is full before discarding the third image frame. In this example, if the first queue is not full of image frames, in response to the first queue meeting the first condition, no more image frames are added to the first queue, and the buffer containing unstored image frames is released, further reducing memory resource usage.

[0014] In some embodiments, the first queue further includes a fourth image frame, which is acquired earlier than the second image frame. The acquisition times of the second and first image frames are adjacent and consecutive. After acquiring an image frame, the electronic device can obtain the confidence distance value corresponding to the image frame. The confidence distance value is the distance between the target object and a preset reference line within the image frame. For example, after acquiring the fourth image frame, a third confidence distance corresponding to the fourth image frame can be obtained; after acquiring the second image frame, a second confidence distance corresponding to the second image frame can be obtained; and after acquiring the first image frame, a first confidence distance corresponding to the first image frame can be obtained. Thus, after the electronic device actually obtains the first confidence distance, it can compare the confidence distance values ​​of multiple image frames to determine whether the first queue meets the first condition.

[0015] For example, in a scenario where the target point is the point farthest from the preset reference line among the trajectory points corresponding to the motion trajectory, if the second confidence distance corresponding to the second image frame is greater than the first confidence distance and the third confidence distance corresponding to the fourth image frame, the electronic device can determine that the second image frame stored in the first queue is the target image frame, and accordingly, it can determine that the first queue satisfies the first condition.

[0016] For example, in the scenario where the target point is the point closest to the preset reference line among the trajectory points corresponding to the motion trajectory, if the second confidence distance is less than the first confidence distance and the third confidence distance, the electronic device can determine that the second image frame stored in the first queue is the target image frame, and accordingly, it can determine that the first queue meets the first condition.

[0017] Understandably, there is a time interval between when an electronic device captures an image frame and when it stores that image frame. There is also a time interval between when it stores an image frame and when it displays it. Furthermore, there is a time interval between when the user sees the image frame and when they instruct the device to capture it. That is, when the user sees the second image frame and instructs the device to capture it, the electronic device has already captured multiple image frames whose capture time is after the second image frame. In this scenario, after storing the second image frame in the first queue, the faster the electronic device can identify that the second image frame is the target image frame, the sooner it can stop storing image frames whose capture time is after the second image frame into the first queue.

[0018] In the above embodiments, the time required to obtain the confidence distance of an image frame and compare the confidence distances of multiple image frames is relatively short. That is, by obtaining and comparing the confidence distances, the second image frame in the first queue can be quickly identified as the target image frame. In this way, the storage of newly acquired image frames into the first queue can be paused earlier. For example, after acquiring the next image frame after the second image frame (i.e., the first image frame), the second image frame can be quickly identified as the target image frame, thereby reducing the occupation of memory resources.

[0019] In some embodiments, before acquiring the first image frame, the electronic device may fit a first predicted parabolic curve. The first predicted parabolic curve is a simulated curve of the target object's motion trajectory in a preset fitted coordinate system. The first axis of the fitted coordinate system indicates the acquisition time. The second axis of the fitted coordinate system indicates the confidence distance. During the capture of the target object, in the acquired preview stream, each image frame corresponds to a trajectory point of the target object on its motion trajectory, and each image frame corresponds to an acquisition time and a confidence distance value.

[0020] For example, the value of the first predicted parabolic curve on the first axis is at least related to the acquisition time corresponding to the second and fourth image frames, and the value on the second axis is at least related to the confidence distance corresponding to the second and fourth image frames.

[0021] Based on the aforementioned first predicted parabolic curve, before the electronic device actually acquires the first image frame, it can predict the confidence distance value corresponding to the first image frame, i.e., the first predicted confidence distance value, based on the image frame acquisition time interval. When the first predicted confidence distance value is less than the second confidence distance and the second confidence distance is greater than the third confidence distance, or when the first predicted confidence distance value is greater than the second confidence distance and the second confidence distance is less than the third confidence distance, the second image frame is predicted to be the target image frame. In this way, after the first image frame is actually acquired, only a simple verification is needed, such as comparing the first confidence distance (actual confidence distance value) of the first image frame with the second confidence distance, to determine whether the first queue meets the first condition.

[0022] In the above embodiments, by using a predictive approach, the time required to identify the second image frame in the first queue as the target image frame can be further shortened, the number of image frames that need to be stored in the first queue can be reduced, and the occupation of memory resources can be reduced.

[0023] In some embodiments, the first queue further stores a sixth image frame, which was acquired earlier than the fourth image frame. Before acquiring the second image frame, the electronic device can fit a second predictive parabolic curve. The value corresponding to the second predictive parabolic curve on the first axis is related at least to the acquisition times of the sixth and fourth image frames, and the value corresponding to the second axis is related at least to the confidence distance values ​​of the sixth and fourth image frames. Based on the second predictive parabolic curve, the second predictive confidence distance value corresponding to the second image frame is estimated before the second image frame is actually acquired.

[0024] For example, when the second predicted confidence distance value is greater than the third confidence distance, and the third confidence distance is greater than the fourth confidence distance of the sixth image frame, it is predicted that the current first queue does not meet the first condition; or, when the second predicted confidence distance value is less than the third confidence distance, and the third confidence distance is less than the fourth confidence distance, it can also be predicted that the current first queue does not meet the first condition. After it has been predicted that the current first queue does not meet the first condition, the actually acquired second image frame can be directly stored in the first queue without additional verification.

[0025] Additionally, after acquiring the second image frame, if the first queue is full, the second image frame is stored in the first queue, and the image frame with the earliest storage time in the first queue is deleted. If the first queue is not full, the second image frame is stored in the first queue.

[0026] In some embodiments, generating a corresponding photographed image based on a target image frame can be achieved by performing one or more image processing steps on the target image frame to obtain the corresponding photographed image.

[0027] In other embodiments, a corresponding photographed image is generated based on the target image frame. Alternatively, one or more image processing operations can be performed based on multiple image frames in the first queue to generate a dynamic photograph as the photographed image, with the target image frame as the cover image.

[0028] In the above embodiments, the diverse needs of users for snapshot capture can be met, the user's satisfaction with the snapshot capture results can be improved, the re-capture rate can be reduced, and the human-computer interaction efficiency of snapshot capture can be improved.

[0029] In some embodiments, after generating the corresponding photographed image, the electronic device may release the buffer occupied by the image frames in the first queue.

[0030] In the above embodiments, after generating the captured image, the occupied buffer is released to reduce the long-term occupation of memory resources.

[0031] In some embodiments, after releasing the buffer occupied by image frames in the first queue, or before the first queue is stored with image frames, the electronic device first acquires a seventh image frame, and then acquires an eighth image frame. Specifically, when the seventh image frame is acquired, it is not stored in the first queue. When the eighth image frame is acquired, if the confidence distances corresponding to the seventh and eighth image frames are different, the eighth image frame is stored in the first queue, and a first number of buffers are marked according to the queue length of the first queue. Afterward, the electronic device can execute the steps of the snapshot processing method for the first aspect of the protection request. Subsequently, the electronic device acquires a ninth image frame, and stores the ninth image frame in the first queue, with the original image data of the ninth image frame stored in the first number of buffers.

[0032] The first number of buffers includes a buffer for storing the raw image data of the eighth image frame. Furthermore, the first number can be 3 frames or 4 frames.

[0033] In other embodiments, after releasing the buffer occupied by image frames in the first queue, or before the first queue is stored with image frames, the electronic device first acquires the tenth image frame, and then acquires the eleventh image frame. When the tenth image frame is acquired, it is not stored in the first queue. When the eleventh image frame is acquired, if the confidence distances corresponding to the tenth and eleventh image frames are the same, the eleventh image frame is also not stored in the first queue. Then, the twelfth image frame is acquired. If the confidence distances corresponding to the eleventh and twelfth image frames are different, the twelfth image frame can be stored in the first queue, and a first number of buffers are marked according to the queue length of the first queue. Afterwards, the electronic device can execute the steps of the snapshot processing method for the first aspect of the protection request. The first number of buffers includes a buffer for storing the original image data of the eleventh image frame.

[0034] In the above embodiments, by determining whether the confidence distance between the acquired image frames changes, it is possible to predict whether to trigger the capture processing method provided in the first aspect, thereby reducing system overhead.

[0035] In some embodiments, after determining that the first queue meets the first condition again, if the first queue is full of image frames, the thirteenth image frame is discarded after being acquired. If the first queue is not full of image frames, the thirteenth image frame is stored in the first queue after being acquired.

[0036] In addition, when the duration for which the first condition is met in the first queue exceeds a preset duration threshold, the buffer occupied by the image frames in the first queue is released.

[0037] For example, the duration for which the first queue satisfies the first condition can be the time length between the time point when the second image frame is stored and the current time point. As another example, the duration for which the first queue satisfies the first condition can also be the time length between the time point when the first queue satisfies the first condition and the current time point.

[0038] In the above embodiments, in the case of shooting timeout, the buffer occupied in the first queue can be released, which can not only reduce the time occupied by memory resources, but also prepare for the next shooting scene, ensuring that the user gets the desired photo image while reducing memory usage.

[0039] In some embodiments, after acquiring a first image frame, the electronic device downsamples the first image frame to obtain a first smaller image. The first smaller image is then stored in a second queue. The second queue stores smaller images of the acquired image frames, and also includes a smaller image corresponding to the second image frame. Similarly, after acquiring a third image frame, the third image frame is downsampled to obtain a second smaller image; this second smaller image is also stored in the second queue. Before generating the corresponding captured image based on the target image frame, the electronic device can use the smaller images in the second queue, leveraging a pre-configured precision algorithm, to identify the target image frame among the acquired image frames and retrieve it from the first queue.

[0040] In the above embodiments, the electronic device identifies the target image frame during the acquisition of the preview stream in two ways: The first method is based on confidence distance to quickly identify the image frame as the target image frame from the first queue. The second method utilizes a pre-configured precision algorithm to accurately identify the target image frame from all acquired image frames.

[0041] Understandably, the target image frames determined by the two methods may be exactly the same or they may differ. Of course, even if there are differences, the acquisition times of the two methods are adjacent. The first method requires less time to determine the target image frame, while the second method determines the target image frame more accurately.

[0042] The electronic device can first quickly identify the target image frame in the first queue using the first method. The identification result of the first method triggers the locking of the buffer occupied by the image frame already stored in the first queue. After the buffer is locked, the image frames stored in the buffer are not replaced by other data. At this time, the first queue not only stores the identified target image frame, but also image frames adjacent to its acquisition time.

[0043] Next, the target image frame is identified from all the acquired image frames using a second method. Based on the target image frame identified using the second method, the corresponding raw image data is retrieved from the first queue and subjected to appropriate image processing, resulting in a captured image that more closely matches the user's expectations.

[0044] Furthermore, when both methods are used together, even if the target image frames determined by the two methods differ, the probability that the target image frame determined by the second method will be included in the locked first queue is very high. Thus, by using both methods simultaneously, not only is memory resource consumption reduced, but accuracy is also guaranteed.

[0045] In some embodiments, before the first image frame is acquired, the electronic device displays a shooting preview interface in response to a second user operation. After the first image frame is acquired, the first image frame can be displayed on the shooting preview interface. After the third image frame is acquired, the third image frame can be displayed on the shooting preview interface.

[0046] In a second aspect, an electronic device is provided in the embodiments of this application. The electronic device includes one or more processors and a memory. The memory is coupled to the processor and is used to store computer program code, which includes computer instructions. When one or more processors execute the computer instructions, the one or more processors are used to perform the methods described in the first aspect and its possible embodiments.

[0047] Thirdly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect and its possible embodiments.

[0048] Fourthly, this application provides a computer program product that, when run on the aforementioned electronic device, causes the electronic device to perform the methods described in the first aspect and its possible embodiments.

[0049] Understandably, the electronic devices, computer storage media, and computer program products provided in the above aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0050] Figure 1 A schematic diagram illustrating the preview functionality of an electronic device provided in an embodiment of this application in zero shutter lag (ZSL) mode;

[0051] Figure 2A schematic diagram illustrating the principle of the electronic device provided in this application for capturing images in ZSL mode;

[0052] Figure 3 One of the example images of a scene where an electronic device, according to an embodiment of this application, identifies a highlight frame;

[0053] Figure 4 Example diagram of a scene where the electronic device provided in the embodiments of this application identifies a highlight frame;

[0054] Figure 5 Example diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0055] Figure 6 Example diagram of the software structure of the electronic device provided in the embodiments of this application;

[0056] Figure 7 A schematic diagram illustrating the principle of how an electronic device, as provided in this application embodiment, maps an image frame to a fitted coordinate system;

[0057] Figure 8 A schematic diagram illustrating the principle of fitting and predicting parabolas for electronic devices provided in embodiments of this application;

[0058] Figure 9 An example diagram of the sliding window in the memory buffer queue during the acquisition of image frames by the electronic device provided in this application embodiment;

[0059] Figure 10 One of the example diagrams illustrating a scenario where an electronic device performs a snapshot processing method according to an embodiment of this application;

[0060] Figure 11 A second example of a scenario illustrating the electronic device performing the snapshot processing method provided in this application embodiment;

[0061] Figure 12 The third example diagram illustrates a scenario where an electronic device performs a snapshot processing method according to an embodiment of this application.

[0062] Figure 13 Fourth example of a scenario illustrating the electronic device performing the snapshot processing method provided in this application embodiment;

[0063] Figure 14 Fifth example of a scenario illustrating the electronic device performing the snapshot processing method provided in this application embodiment;

[0064] Figure 15 A flowchart illustrating the steps of the snapshot processing method provided in this application embodiment. Detailed Implementation

[0065] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0066] For ease of understanding, the technical terms involved in the embodiments of this application will be introduced below.

[0067] (1) Zero shutter lag (ZSL) is a camera mode that reduces the delay time of the camera shutter, making shooting more immediate and responsive, thereby reducing the difference between the preview image and the captured image.

[0068] In an exemplary scenario, when an application with shooting capabilities (e.g., a camera app) is running in the foreground of an electronic device and ZSL camera mode is enabled, a shooting preview interface provided by the camera app can be displayed. This shooting preview interface is used to display a preview stream from the target camera sensor, which consists of multiple continuously acquired preview frames (or preview images, image frames).

[0069] Figure 1 This illustrates the process of generating and displaying a preview stream in ZSL camera mode. An example is provided where the preview stream includes image frames 1, 2, 3, 4, 5, 6, and 7. Figure 1 As shown, the target camera sensor of the electronic device sequentially acquires raw image data (e.g., raw images) from image frames 1 to 7. Each time the target camera sensor acquires a raw image (e.g., the raw image of image frame 1), it can transmit the raw image (e.g., the raw image of image frame 1) to the image signal processor (ISP). After processing by the ISP, a displayable image frame (e.g., image frame 1) is obtained.

[0070] Understandably, the electronic device can transmit the raw images of image frames 1 through 7 to the ISP frame by frame, according to the order of acquisition. After processing by the ISP, the corresponding image frames are obtained. In this way, the electronic device can display image frames 1 through 7 one by one on the shooting preview interface.

[0071] The aforementioned ISP can include an ISP first module, an ISP second module, and an ISP third module. Different modules can perform different processing on image data. For example, the ISP first module may include one or more of the following processing: binning, HDR fusion, etc. The ISP second module may include one or more of the following processing: bad pixel correction (BPC), black level correction (BLC), lens shade correction (LSC), automatic white balance (AWB), Bayer domain noise reduction (NR), Demosaic, etc. The ISP third module may include one or more of the following processing: color correction (CC), YUV domain noise reduction (NR), color enhancer (CE), sharpening, tone mapping, etc.

[0072] For example, after the raw image is input into the ISP, the first module of the ISP performs remosaic processing on the first image data. The second module of the ISP performs Bayer domain processing and outputs data in RGB format. The third module of the ISP performs either RGB domain or YUV domain processing and outputs data in YUV format.

[0073] The above descriptions of the ISP first module, ISP second module, and ISP third module are merely illustrative and the embodiments of this application are not limited thereto.

[0074] Each time the target camera sensor acquires a raw image frame, it can store that raw image in a raw image cache queue. This cache queue can be a data structure used to query and retrieve stored raw images; in subsequent embodiments, it will be simply referred to as a cache queue. For example, such as... Figure 1 As shown, the target camera sensor acquires raw images of image frames 1 to 7, which can be stored sequentially in the buffer queue.

[0075] (2) Capture the Best: This is a shooting function implemented based on the ZSL mode.

[0076] exist Figure 1 On the basis of, such as Figure 2 As shown, the electronic device also includes a sensing module. The sensing module can determine the highlight frames from the image frames already acquired by the target camera sensor (e.g., image frames 1 to 7).

[0077] The "highlight frame" can be an image frame that matches the user's shooting expectations. For example, when a user uses an electronic device to take a photo of person A jumping, the highlight frame could be an image frame showing person A at the highest point of the jump. Similarly, when a user uses an electronic device to take a photo of an athlete hurdling, the highlight frame could be an image frame showing the athlete at the highest point of the jump, or an image frame showing the athlete landing after completing the hurdle. The implementation details of the perception module's prediction of the highlight frame are related to the actual capture algorithm configured in the electronic device. For example, based on small images of multiple captured image frames, the motion trajectory of the subject is identified. Then, the highlight frame is determined by combining this motion trajectory. The subject can be a person, object, animal, etc., and this embodiment does not specifically limit this.

[0078] like Figure 3 As shown, after the target camera sensor acquires the raw image of image frame 1, it stores the raw image of image frame 1 in a buffer queue and then passes it to the ISP. The ISP can downsample the raw image of image frame 1 to obtain a smaller image of image frame 1. The smaller image contains the same content as the original image data, but its size is smaller. The ISP can send the smaller image of image frame 1 to the perception module, triggering the perception module to begin evaluating key frames. Subsequently, the target camera sensor acquires raw images of other image frames (e.g., image frame 7). Similarly, the target camera sensor can store the raw images of other image frames (e.g., image frame 7) in a buffer queue and then pass them to the ISP. The ISP can downsample the raw images of other image frames (e.g., image frame 7) to obtain corresponding smaller images, which are then passed to the perception module. In this way, the perception module can continue to evaluate key frames based on image frame 1 and other image frames.

[0079] Taking image frame 5 as an example, which was evaluated as a highlight by the perception module. Continuing... Figure 2 As shown, upon detecting a user-instructed shooting action and having determined that image frame 5 is a highlight frame, the electronic device can retrieve the raw image of image frame 5 from the buffer queue. Then, the raw image of image frame 5 is processed using an image processing algorithm module (e.g., a post-processing algorithm module) in the ISP and / or camera algorithm library to obtain the captured photo (e.g., the photographed image) corresponding to image frame 5, and stored. It is understandable that, in addition to storing the captured photo, it can also be displayed to present the shooting result to the user.

[0080] exist Figure 3 In the scenario shown, at the time point when the perception module receives the small image of image frame 8, it identifies the highlight frame as image frame 5 based on image frames 1 to 7. Figure 3In the scenario shown, to ensure that the perception module can retrieve the raw image of a highlighted frame from the cache queue after identifying it, the cache queue has at least eight buffers for storing the raw images of the image frames captured by the target camera sensor.

[0081] If the cache queue is full, for example, if it already contains 8 frames of raw images, and a new raw image needs to be added, the oldest raw image in the cache queue can be deleted. Afterward, the cache queue will always be full of raw images, meaning a significant amount of memory will be constantly occupied.

[0082] In other embodiments, when the electronic device enables the HDR mode capture function, compared to Figure 3 In the snapshot scenario shown, the number of buffers corresponding to the cache queue is doubled, which means that the memory resources occupied during the snapshot process are also doubled.

[0083] like Figure 4 As shown, the raw image of the image frame (such as image frame 8) acquired by the target camera sensor can be: one raw image acquired under a long exposure time and one raw image acquired under a short exposure time.

[0084] Next, both the raw images captured with long exposure times and those captured with short exposure times are stored in a buffer queue. Thus, each image frame requires two buffers to store the raw images corresponding to different exposure times. Figure 4 In the scenario shown, similarly, at the time point when the perception module receives the small image of image frame 8, it identifies the highlight frame as image frame 5 based on image frames 1 to 7. Figure 4 In the scenario shown, to ensure that the perception module can retrieve the raw image of a highlighted frame from the cache queue after identifying it, the cache queue must have at least 16 buffers. Compared to Figure 3 The scenario shown occupies twice the amount of memory.

[0085] To address the aforementioned issues, this application provides a snapshot processing method applied to an electronic device equipped with a target camera sensor. In a snapshot scenario, image frames captured by the target camera sensor can be stored in a buffer queue (also known as a first queue). When the movement of the subject is identified by combining multiple consecutive image frames, if it is determined that image frame a, representing the highest point of the subject's trajectory, has been captured and is already stored in the buffer queue, the buffer occupied by the stored image frame is locked. This prevents the image frames stored in the buffer from being replaced by other data.

[0086] For example, after the electronic device acquires an image frame and stores it in a buffer queue, it can fit a predictive parabolic curve based on the acquired image frame. This predictive parabolic curve can be used to predict the confidence distance value of the object to be photographed in the next image frame (e.g., image frame b) to be acquired by the target camera sensor. The confidence distance value refers to the height of the object relative to a specified baseline (also called a preset reference line) in the image frame. For example, the specified baseline can be the line composed of the last row of pixels in the image frame. The process of fitting the predictive parabolic curve is described in detail in subsequent embodiments and will not be repeated here.

[0087] Optionally, the predicted parabolic curve is a parabola in a fitted coordinate system, where the horizontal axis is the time axis and the vertical axis is the confidence distance axis. The electronic device can determine whether the confidence distance value of the captured object in image frame b meets the preset conditions.

[0088] The image frame b satisfies the preset condition in the following ways: the predicted coordinates corresponding to image frame b are located at the position where the peak of the predicted parabolic curve is decreasing or the trough is increasing. The horizontal axis of the predicted coordinates represents the predicted acquisition time point, and the vertical axis represents the confidence distance value predicted by the predicted parabolic curve. The predicted acquisition time point can be the sum of the actual acquisition time point of the adjacent previous image frame and the acquisition time interval. Furthermore, the predicted parabolic curve can include peaks or troughs. The confidence distance value corresponding to the peak is higher than the confidence distance value of other points in the predicted parabolic curve, and the confidence distance value corresponding to the trough is lower than the confidence distance value of other points in the predicted parabolic curve. Optionally, if the acquisition time of image frame b is after the acquisition time of the peak and adjacent to the peak, it can be determined that image frame b satisfies the preset condition. Optionally, if the acquisition time of image frame b is after the acquisition time of the trough and adjacent to the trough, it can also be determined that image frame b satisfies the preset condition.

[0089] If the predicted confidence distance value of the object in image frame b meets a preset condition, the currently acquired image frame is pre-determined to be image frame a, that is, the image frame where the object is located at the target point of the motion trajectory. For example, the target point is the point farthest from the specified baseline; that is, the pixel corresponding to the target point in the image frame is farthest from the preset reference line relative to the pixels corresponding to other points in the motion trajectory. Alternatively, the target point can be the point closest to the specified baseline; that is, the pixel corresponding to the target point in the image frame is closest to the preset reference line relative to the pixels corresponding to other points in the motion trajectory.

[0090] Subsequently, when the raw image of image frame b is actually acquired, the buffer used to store the raw image of image frame a, and the buffer used to store image frame b, are locked. After locking the buffers occupied by image frames a and b, even if there is no free buffer to store the newly acquired image frame, the raw images of image frames a and b will not be deleted, so that the raw images of image frames a and b can be effectively preserved for a long time.

[0091] Understandably, when the confidence distance value of the subject in image frame b meets the preset condition, there will be an image frame in both image frames a and b that shows the subject jumping to its highest point. The highlight frame evaluated by the perception module is usually also the image frame showing the subject jumping to its highest point. The time required to fit the predicted parabolic curve and determine image frames a and b is shorter than the time required for the perception module to evaluate the highlight frame. Before the perception module evaluates the highlight frame, by retaining only the raw images of image frames b and a, the amount of buffer occupied during the capture process can be reduced, that is, the memory resource usage can be reduced.

[0092] For example, electronic devices can be desktops, laptops, tablets, handheld computers, mobile phones, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), televisions, VR devices, AR devices, and other devices with cameras.

[0093] like Figure 5 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0094] The aforementioned sensor module 180 may include sensors such as pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, and bone conduction sensors.

[0095] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include... Figure 5 It can show more or fewer parts, or combine some parts, or split some parts, or arrange different parts. Figure 5 The components shown can be implemented in hardware, software, or a combination of both.

[0096] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0097] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0098] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0099] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0100] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a touch layer and a display panel. The touch layer is used to sense user interaction with display screen 194. The display panel can be a liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), flexible light-emitting diode (FLED), minimized, microLED, micro-OLED, quantum dot light-emitting diode (QLED), etc. Electronic device 100 can realize shooting functions through ISP, camera 193, video codec, GPU, display screen 194, and application processor.

[0101] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (image sensor). The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0102] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor, such as a camera sensor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0103] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0104] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0105] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0106] Figure 6 This is a schematic diagram of the architecture (including software system and some hardware) used in the embodiments of this application. Figure 6 As shown, the application architecture is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the application architecture can be divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0107] like Figure 6 As shown, the application layer includes camera applications and gallery applications. This is understandable. Figure 6 The examples shown are only a portion of the applications; in fact, the application layer can include other applications, which this application does not limit. For example, the application layer also includes applications such as messaging, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, and Alipay. Furthermore, the application layer also includes other applications with camera functions, such as video recording applications and photo editing applications.

[0108] like Figure 6As shown, the application framework layer includes a camera access interface. The camera access interface includes camera management and camera devices. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer includes multiple camera nodes. The camera algorithm library includes a perception module, a fitting and judgment module, and a memory control module, etc. Of course, the camera algorithm library may also include post-processing algorithm modules, decision modules, and electronic antishake (EIS) algorithm modules, which are not shown in the figure.

[0109] It is understood that the aforementioned perception module, fitting judgment module, and memory control module are algorithm modules required to implement the method provided in the embodiments of this application. Their operating principles and logic will be explained in detail in subsequent embodiments, and will not be repeated here. In addition, the perception module, fitting judgment module, and memory control module can also be located in the application layer or application framework layer, and the embodiments of this application do not specifically limit them in this regard.

[0110] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. For example... Figure 6 As shown, the driver layer includes camera device drivers, digital signal processor drivers, and graphics processor drivers, etc.

[0111] The hardware layer includes multiple camera sensors, an ISP, a digital signal processor, a graphics processor, and a gyroscope sensor. Figure 6 (Not shown). Understandably, the hardware layer may also include other hardware modules, without specific limitations.

[0112] In this system, each camera node in the camera hardware abstraction layer corresponds to a camera sensor in the hardware layer. For example, an application with shooting capabilities (e.g., a camera application) in the application layer can access the camera node through the camera access interface. This camera node then manages the corresponding camera sensor, such as setting its configuration parameters and controlling image acquisition.

[0113] In an exemplary scenario, a user can tap the camera application, allowing the electronic device to run the camera application in the foreground. The camera application sends control commands to the camera hardware abstraction layer (HIB) via the camera access interface. These commands may carry information such as the enabled camera mode (e.g., photo mode, video mode, or portrait mode) and zoom level. In response to these control commands, the HIB invokes the camera algorithm library. The decision module within the camera algorithm library determines the target camera sensor (e.g., the target camera sensor) and its configuration parameters (including sensor output method, parameter configuration for the ISP module, and parameter configuration for the post-processing algorithm module) based on the zoom level, camera mode, and ambient light. The decision module can then pass the target camera sensor's identifier and configuration parameters to the HIB.

[0114] The camera hardware abstraction layer can create a target camera node corresponding to the target camera sensor. Through the target camera node, configuration parameters are sent to the camera device driver. The camera device driver then sends these configuration parameters to the hardware layer; for example, it sends the sensor image output method to the target camera sensor and the parameter configurations for each ISP module to the ISP. Subsequently, the target camera sensor can output images based on the sensor image output method. The ISP can then perform corresponding processing based on the parameter configurations for the ISP modules. It is understood that the configuration parameters passed to the target camera sensor can be called camera parameters, and in addition to the sensor image output method exemplified in the previous embodiments, they can also include automatic exposure (AE) strategies, output image size, etc., without specific limitations here.

[0115] The camera algorithm library is also used to send digital signals to the digital signal processor (DSP) driver in the driver layer, so that the DSP driver can call the DSP in the hardware layer to perform digital signal processing. The DSP can then return the processed digital signal to the camera algorithm library through its driver.

[0116] The camera algorithm library is also used to send digital signals to the graphics signal processor (GSPM) driver in the driver layer, so that the GSPM driver can call the graphics processor in the hardware layer to perform digital signal processing. The graphics processor can then return the processed image data to the camera algorithm library through the graphics processor driver.

[0117] Additionally, when the camera application is running in the foreground, it can send preview or capture commands to the camera hardware abstraction layer via the camera access interface. The capture command is generated by the camera application in response to a user's instruction to take a picture. The preview command is periodically generated by the camera application while displaying the camera preview interface and before any user-instructed shooting is detected.

[0118] After receiving a preview or capture command, the camera hardware abstraction layer (HAL) can transmit the command to the target camera sensor via the camera device driver. The target camera sensor, responding to the command, acquires a raw image and transmits it to the ISP (Image Signal Processor), which converts it into an image frame visible to the user. The image frame output by the ISP can be sent to the camera device driver. The camera device driver can then send the image frame back to the HAL. The HAL can then send the image frame to the post-processing algorithm module for further processing, or it can send it to the camera access interface. The camera access interface can then send the image frame returned by the HAL to the camera application. The camera application can then draw the image frame and trigger the display screen to show it.

[0119] Understandably, after opening the camera app and displaying the shooting preview interface, the camera app can trigger the display screen to show a preview stream from the target camera sensor. The operation of triggering the display of the shooting preview interface can be referred to as the second operation.

[0120] During the display of the preview stream, the fitting judgment module can analyze each image frame of the preview stream to determine whether to execute the snapshot processing method provided in the embodiments of this application.

[0121] As one implementation, the capture processing method provided in this application embodiment is executed when the target camera sensor acquires the preview stream, or after one capture processing method is executed, and the fitting judgment module detects that the confidence distance values ​​corresponding to the same subject in two adjacent image frames are different. As another implementation, if the confidence distance values ​​corresponding to the same subject in the image frames detected by the fitting judgment module are all the same, the capture processing method is not executed when the target camera sensor acquires the preview stream, or after one capture processing method is executed.

[0122] like Figure 7As shown, starting from the acquisition of the preview stream by the target camera sensor, or after completing one capture processing method, the target camera sensor sequentially acquires raw images of image frames 701, 704, 705, 706, 707, and 708. Each image frame 701 to 708 includes the subject 702 (also referred to as the target object) and a specified baseline 703. Each image frame corresponds to an acquisition time (also referred to as the acquisition moment), and the distance between the reference point corresponding to the subject 702 and the specified baseline 703 in each image frame is a confidence distance value. For example, the reference point can be any identifiable point on the subject 702; for instance, if the subject 702 is a person, the reference point could be a pixel displaying the feet. Alternatively, the reference point can be a specific point on the subject 702, such as a pixel displaying the centroid or center point.

[0123] like Figure 7 As shown, the acquisition time of image frame 701 is t1, and the confidence distance value corresponding to the captured object 702 in image frame 701 is h1. That is, the actual coordinates of image frame 701 in the fitted coordinate system are (t1, h1). Similarly, the actual coordinates of image frame 704 in the fitted coordinate system are (t2, h2), image frame 705 is (t3, h3), image frame 706 is (t4, h4), image frame 707 is (t5, h5), and image frame 708 is (t6, h6).

[0124] For example, image frame 704 and image frame 701 are acquired at adjacent times. After image frame 704 is acquired, the fitting and judgment module determines that the confidence distance values ​​of the captured object 702 in image frame 701 and image frame 704 are the same or similar (the difference between the confidence distance values ​​of the two image frames is less than a preset threshold), and does not trigger the start of the snapshot processing method provided in this application embodiment.

[0125] For example, image frame 704 and image frame 705 are acquired at adjacent times. After image frame 705 is acquired, the fitting and judgment module determines that the confidence distance values ​​of the captured object 702 in image frame 704 and image frame 705 are different, triggering the execution of the snapshot processing method provided in this application embodiment.

[0126] Through the above implementation, when the subject is not jumping, the electronic device acquires and displays a preview stream according to conventional strategies. When the subject is jumping, the electronic device can execute the snapshot processing method while displaying the preview stream showing the subject's jumping process. This reduces memory resource usage and unnecessary power consumption.

[0127] In addition, such as Figure 7 As described above, during the jumping process of the subject 702, the actual coordinates of the continuous multiple image frames acquired by the electronic device in the fitted coordinate system approximate a parabolic curve. Understandably, the expression for the parabolic curve includes at least: y = ax 2 +bx+c. Where a, b, and c are constants to be fitted, x is the value of the horizontal axis in the fitted coordinate system (i.e., the acquisition time of the image frame), and y is the value of the vertical axis in the fitted coordinate system (i.e., the confidence distance value of the image frame). The horizontal axis of the fitted coordinate system can be called the first axis, and the vertical axis can be called the second axis. Thus, by using the actual coordinates of at least three different image frames, a set of values ​​for a, b, and c can be fitted, thereby determining a parabolic curve, which is the predicted parabolic curve mentioned in the previous embodiment. The more image frames involved in the fitting, the closer the fitted predicted parabolic curve will be to the actual parabolic curve. Figure 7 The parabolic curve shown is closer to the actual situation.

[0128] In some embodiments, after the electronic device performs the snapshot processing method and acquires three image frames, the fitting judgment module in the electronic device can fit a predictive parabolic curve based on the three image frames. This predictive parabolic curve can be used to predict the confidence distance value of the image frames that have not yet been acquired.

[0129] like Figure 8 As shown, when image frames 1, 2, and 3 (referred to as the sixth image frame) have been acquired, but image frame 4 has not been acquired, the actual coordinates (T) of image frame 1 in the fitted coordinate system are used as the basis for determining the coordinates. a1 H a1 The actual coordinates (T) of image frame 2 in the fitted coordinate system a2 H a2 ) and the actual coordinates (T) of image frame 3 in the fitted coordinate system a3 H a3 ), and fitted the corresponding prediction curve 801. Where, T a1 H represents the actual acquisition time of image frame 1. a1 T represents the actual confidence distance value for image frame 1. a2 H represents the actual acquisition time of image frame 2. a2 T represents the actual confidence distance value for image frame 2. a3 H represents the actual acquisition time of image frame 3. a3 This represents the actual confidence distance value for image frame 3.

[0130] According to T a3 The acquisition time interval, and the acquisition time corresponding to the predicted image frame 4, can also be denoted as T. b4 Based on the predicted Tb4 Based on the prediction curve 801, the confidence distance value corresponding to image frame 4 can be predicted, which can also be denoted as H. b4 It is understandable that there may be a difference between the predicted confidence distance value and the actual confidence distance value, and the embodiments of this application do not limit this.

[0131] After acquiring image frame 4 (referred to as the fourth image frame), the prediction curve is refitted based on the received first four image frames. The refitted prediction parabolic curve can be used to predict the confidence distance value corresponding to the next image frame (the fifth frame), and so on.

[0132] In this embodiment, before the next image frame is acquired, the confidence distance value of the subject in the next image frame is predicted. Based on the predicted confidence distance value, it is determined whether a highlight frame has been acquired (specific implementation details can be found in subsequent embodiments, and will not be elaborated here).

[0133] In some embodiments, the cache queue is in an unlocked state when the snapshot processing method is started. The cache queue is also in an unlocked state until a highlight frame is determined to have been captured. After a highlight frame is determined to have been captured, the cache queue is locked. After the cache queue enters the locked state, it can be unlocked again under any of the following conditions: (1) the locked state lasts for a specified duration, and no instruction to capture is received. (2) after receiving a user instruction to capture, the snapshot processing method is started again.

[0134] For example, when the cache queue is locked, the corresponding buffer remains fixed, and the data already stored in the buffer will not be replaced. However, while the cache queue is locked, if it contains an empty buffer (i.e., a buffer where no data has been written), newly acquired raw images can be stored in the empty buffer. Furthermore, raw images already stored in the buffer cannot be overwritten. If the cache queue does not contain an empty buffer while it is locked, newly acquired raw images will not be stored temporarily.

[0135] For example, when the buffer queue is in an unlocked state, the corresponding buffer can change. For instance, after a new raw image frame is acquired, a new buffer is added to the buffer queue to store that raw image. Of course, to maintain the same number of buffers in the buffer queue, the target buffer corresponding to the buffer queue can be released after adding the new buffer. The raw image stored in the target buffer was acquired earlier than the raw images in other buffers in the buffer queue.

[0136] In some embodiments, the cache queue can be conceptually viewed as having a corresponding memory sliding window in the storage space, and the memory sliding window can point to the buffer corresponding to the cache queue. That is, the buffer corresponding to the cache queue can be found in the storage space of the electronic device through the memory sliding window. The buffer indicated by the memory sliding window is also the memory resource that can be occupied during the snapshot process.

[0137] For example, when the buffer corresponding to the cache queue changes, the memory sliding window will also change accordingly. The change of the memory sliding window can also be called the sliding of the memory sliding window.

[0138] Let's take a buffer queue that provides 4 buffers as an example. Figure 9 As shown, with image frames 1, 2, 3, and 4 already acquired, the memory sliding window 901 of the buffer queue points to buffers 902, 903, 904, and 905. Buffer 902 stores the raw image of image frame 1, buffer 903 stores the raw image of image frame 2, buffer 904 stores the raw image of image frame 3, and buffer 905 stores the raw image of image frame 4. Furthermore, the buffer queue is currently in an unlocked state.

[0139] The electronic device can fit a corresponding predictive parabolic curve based on the acquired image frames. Using this predictive parabolic curve, the confidence distance value corresponding to the yet-to-be-acquired image frame 5 (referred to as the second image frame) can be predicted; this can also be called the second predicted confidence distance value. If, based on the predicted confidence distance value corresponding to image frame 5, it is determined that no exciting frame has been acquired yet, then after image frame 5 is actually acquired, the memory sliding window 901 is slid.

[0140] like Figure 9 As shown, the sliding memory window 908 points to buffer 903, buffer 904, buffer 905, and buffer 906. Among them, buffer 906 is used to store the raw image of image frame 5.

[0141] Understandably, during the sliding of the memory window 901, the memory control module can mark buffer 906 as occupied. After marking buffer 906 as occupied, a buffer needs to be released. For example, the memory control module can release the target buffer. Among the buffers corresponding to the cache queue, the data stored in the target buffer was acquired earliest. For instance, if image frame 1 was acquired earlier than image frames 2, 3, and 4, the memory control module can determine buffer 902 as the target buffer and release it. Thus, while the cache queue is in an unlocked state, only four buffers are always occupied to store the raw images acquired during the capture process.

[0142] After image frame 5 is actually acquired, the electronic device refits the corresponding predicted parabolic curve based on image frames 1, 2, 3, 4, and 5. Using this predicted parabolic curve, the electronic device can predict the confidence distance value corresponding to the yet-to-be-acquired image frame 6. If, based on the predicted confidence distance value corresponding to image frame 6, it is determined that no exciting frame has been acquired, then after image frame 6 is actually acquired, the memory sliding window 908 is slid again.

[0143] like Figure 9 As shown, the memory sliding window 909, after sliding again, points to buffer 904, buffer 905, buffer 906, and buffer 907. Among them, buffer 907 can be used to store the raw image of image frame 6.

[0144] After image frame 6 is actually acquired, the electronic device refits the corresponding predicted parabolic curve based on image frames 1, 2, 3, 4, 5, and 6. Using this predicted parabolic curve, the electronic device can predict the confidence distance value corresponding to the yet-to-be-acquired image frame 7. If, based on the predicted confidence distance value corresponding to image frame 7, it is determined that a highlight frame has been acquired, the buffer queue is locked. After the buffer queue enters the locked state, the memory sliding window 909 stops sliding until the buffer queue is unlocked again. Thus, while the buffer queue is locked, only 4 buffers are used to store the raw images acquired during the capture process.

[0145] The following, with reference to the accompanying diagram, describes one method for electronic devices to determine whether a highlight frame has been captured:

[0146] like Figure 10 As shown, after the target camera sensor of the electronic device acquires the raw image of image frame 1, it stores the raw image of image frame 1 into buffer 1002.

[0147] Additionally, the target camera sensor can also transmit the raw image of image frame 1 to the ISP. After downsampling the raw image of image frame 1, the ISP generates a smaller... Figure 1 Understandably, the resolution of the smaller image is smaller than that of the raw image, and correspondingly, the data size of the smaller image is smaller than that of the raw image. In addition, the image content contained in the smaller image is the same as that in the raw image.

[0148] like Figure 10 As shown, the ISP will small Figure 1 It is passed to the perception module and the fitting judgment module.

[0149] Optionally, the sensing module receives small Figure 1 Then, based on the acquired image frames (e.g., image frame 1), the evaluation of highlight frames can be initiated. As shown in the aforementioned embodiments, the implementation details of the perception module's prediction of highlight frames are related to the actual capture algorithm configured in the electronic device, and the time required is relatively long. For example, highlight frames can only be identified after multiple image frames have been acquired.

[0150] Optionally, the fitting judgment module receives a small Figure 1 After that, the small Figure 1 Stored in the fitting queue (e.g., called the second queue), and based on small... Figure 1 Calculate the confidence distance value (H) of image frame 1. a1 ), determine the actual coordinates of image frame 1 in the fitted coordinate system.

[0151] When the number of small plots in the fitting queue is 1, the fitting judgment module instructs the memory control module to determine the buffer corresponding to the cache queue, which can also be called determining the buffer indicated by the memory sliding window 1001.

[0152] like Figure 10 As shown, the buffer indicated by the memory sliding window 1001 includes buffer 1002, which contains the raw image of image frame 1, and may also include buffer 1003, buffer 1004, and buffer 1005. Among them, buffer 1003, buffer 1004, and buffer 1005 are blank buffers that have not contained data.

[0153] In addition, the memory control module can also mark the buffer indicated by the memory sliding window 1001 as occupied, for example, by incrementing the reference count of buffer 1002, buffer 1003, buffer 1004 and buffer 1005 by 1.

[0154] Continue as Figure 10As shown, the target camera sensor acquires the raw image of image frame 2. The target camera sensor can store the raw image of image frame 2 in buffer 1003. Additionally, the target camera sensor can also transmit the raw image of image frame 2 to the ISP. After downsampling the raw image of image frame 2, the ISP generates a smaller... Figure 2 ISPs will small Figure 2 It is passed to the perception module and the fitting judgment module.

[0155] Optionally, the sensing module receives small Figure 2 Subsequently, the perception module continues to evaluate highlight frames based on the acquired image frames (e.g., image frame 1 and image frame 2). The fitting and judgment module receives small... Figure 2 After that, the small Figure 2 Stored in the fitting queue, and based on small Figure 2 Calculate the confidence distance (H) of image frame 2. a2 ), determine the actual coordinates of image frame 2 in the fitted coordinate system.

[0156] In some embodiments, when the number of small plots in the fitting queue is 2, the fitting judgment module may instruct the memory control module to slide the memory sliding window 1001.

[0157] like Figure 10 As shown, the sliding mechanism of the memory sliding window 1001 includes: the memory control module can increment the reference count of an empty buffer 1006 by 1, indicating that buffer 1006 is occupied. Additionally, the memory control module can decrement the reference count of buffer 1002 by 1, indicating that buffer 1002 is being released. Thus, if memory cleanup is triggered, the data stored in buffer 1002 can be deleted.

[0158] like Figure 10 As shown, the sliding memory window 1001 can indicate buffer 1003, buffer 1004, buffer 1005, and buffer 1006 after sliding. Among them, buffer 1006 can be an empty buffer.

[0159] In a possible embodiment, when the number of small images in the fitting queue is 2, if the confidence distance values ​​of all image frames whose acquisition time points are earlier than image frame 1 are not greater than image frame 1, and the confidence distance value of image frame 1 is not greater than the confidence distance value of image frame 2, the fitting judgment module can instruct the memory control module to slide the memory sliding window 1001. If the confidence distance values ​​of all image frames whose acquisition time points are earlier than image frame 1 are not greater than image frame 1, and the confidence distance value of image frame 1 is greater than the confidence distance value of image frame 2, the fitting judgment module can instruct the memory control module to lock the memory sliding window 1001, that is, to lock the cache queue. After the cache queue is locked, the buffers in the buffers corresponding to the memory sliding window 1001 that have stored image frames are locked. Buffers that have not stored image frames are locked after an image frame is stored. Alternatively, buffers that have not stored image frames can be released.

[0160] In this way, when the number of small images in the fitting queue is insufficient to fit the predicted parabolic curve, and the real highlight frame is image frame 1, the raw image of image frame 1 can be saved to avoid the problem of accidentally deleting the highlight frame.

[0161] In other embodiments, when the number of small plots in the fitting queue is 2, the memory sliding window 1001 may not need to be slid.

[0162] like Figure 11 As shown, the target camera sensor acquires the raw image of image frame 3. The target camera sensor can store the raw image of image frame 3 in buffer 1004. Additionally, the target camera sensor can also transmit the raw image of image frame 3 to the ISP. After downsampling the raw image of image frame 3, the ISP generates a smaller... Figure 3 ISPs will small Figure 3 It is passed to the perception module and the fitting judgment module.

[0163] Optionally, the sensing module receives small Figure 3 Subsequently, the perception module continues to evaluate highlight frames based on the acquired image frames (e.g., image frame 1, image frame 2, and image frame 3). The fitting and judgment module receives small... Figure 3 After that, the small Figure 3 Stored in the fitting queue, and based on small Figure 3 Calculate the confidence distance value (H) of image frame 3. a3 ), determine the actual coordinates of image frame 3 in the fitted coordinate system.

[0164] In some embodiments, when the number of small plots in the fitting queue is greater than 2, the fitting judgment module can make a judgment based on the small plots in the fitting queue (small... Figure 1 ,Small Figure 2 and small Figure 3 ), and fitted the predicted parabolic curve 1101. Among them, Figure 11 The middle coordinate point 1102 indicates the small Figure 1 The actual coordinates (T) of the corresponding image frame 1 a1 H a1 Coordinate point 1103 indicates a small... Figure 2 The actual coordinates (T) of the corresponding image frame 2 a2 H a2 Coordinate point 1104 indicates a small... Figure 3 The actual coordinates (T) of the corresponding image frame 3 a3 H a3 The process of fitting the predicted parabolic curve 1101 can be referred to the aforementioned embodiments, and will not be repeated here.

[0165] After obtaining the predicted parabolic curve 1101, the fitting judgment module can predict the predicted coordinates of the yet-to-be-acquired image frame 4 in the fitting coordinate system based on this predicted parabolic curve 1101. The predicted coordinates of image frame 4 coincide with coordinate point 1105 in the predicted parabolic curve 1101. The x-coordinate of coordinate point 1105 is T. a3 The sum of the time intervals between data collection and acquisition can be denoted as T. b4 That is, the predicted acquisition time of image frame 4. The ordinate of point 1105 is the confidence distance value of the predicted image frame 4, which can be denoted as H. b4 .

[0166] like Figure 11 As shown, H a1 H a2 H a3 All less than H b4 This indicates that the predicted coordinates corresponding to image frame 4 are not at the descent position of the peak of the predicted parabolic curve 1101. In other embodiments, if H a1 H a2 H a3 All are greater than H b4 This indicates that the predicted coordinates corresponding to image frame 4 are not at the rising position of the trough of the predicted parabolic curve 1101. Thus, before image frame 4 is acquired, it can be pre-determined that the confidence distance value of image frame 4 does not meet the preset conditions.

[0167] like Figure 11 As shown, the target camera sensor actually acquires the raw image of image frame 4. The target camera sensor can store the raw image of image frame 4 in buffer 1005. Additionally, the target camera sensor can also pass the raw image of image frame 4 to the ISP. After downsampling the raw image of image frame 4, the ISP generates a smaller... Figure 4 ISPs will small Figure 4It is passed to the perception module and the fitting judgment module.

[0168] Optionally, the sensing module receives small Figure 4 Subsequently, the perception module continues to evaluate highlight frames based on the acquired image frames (e.g., image frame 1, image frame 2, image frame 3, and image frame 4). The fitting and judgment module receives small... Figure 4 Then, it is stored in the fitting queue.

[0169] If the confidence distance value of image frame 4 does not meet the preset conditions, then the small Figure 4 After being stored in the fitting queue, the fitting judgment module instructs the memory control module to slide the memory sliding window 1001.

[0170] like Figure 11 As shown, the sliding mechanism of the memory sliding window 1001 includes: the memory control module can increment the reference count of an empty buffer 1106 by 1, indicating that buffer 1106 is occupied. Additionally, the memory control module can decrement the reference count of buffer 1003 by 1, indicating that buffer 1003 is being released. Thus, if memory cleanup is triggered, the data stored in buffer 1003 can be deleted.

[0171] In other possible embodiments, before storing the image frame into the buffer queue, if the buffer queue contains a blank buffer, then after storing the raw image of the image frame into the blank buffer of the buffer queue, the memory sliding window 1001 is not slid. If the buffer queue does not contain a blank buffer, then the raw image of the image frame is stored into a blank buffer in the storage space, then the buffer is added to the buffer queue, and then the target buffer in the buffer queue is released.

[0172] In addition, such as Figure 12 As shown, the fitting judgment module receives small Figure 4 Then, it is also possible to base the fitting on the small plots in the queue (small plots) Figure 1 ,Small Figure 2 ,Small Figure 3 and small Figure 4 ), and fitted the predicted parabolic curve 1201. Among them, Figure 12 The middle coordinate point 1202 indicates the small Figure 4 The actual coordinates (T) of the corresponding image frame 4 a4 H a4 ). T a4 H represents the actual acquisition time of image frame 4. a4 This is the actual confidence distance value for image frame 4 (e.g., referred to as the third confidence distance).

[0173] After obtaining the predicted parabolic curve 1201 (referred to as the second predicted parabolic curve), the fitting judgment module can predict the predicted coordinates of the yet-to-be-acquired image frame 5 in the fitting coordinate system based on the predicted parabolic curve 1201. The predicted coordinates of the image frame 5 coincide with the coordinate point 1203 in the predicted parabolic curve 1201. The abscissa of coordinate point 1203 is T. a4 The sum of the time intervals between data collection and acquisition can be denoted as T. b5 That is, the predicted acquisition time of image frame 5. The ordinate of coordinate point 1203 is the confidence distance value of the predicted image frame 5, which can be denoted as H. b5 .

[0174] like Figure 12 As shown, H a1 H a2 H a3 H a4 All less than H b5 This indicates that the predicted coordinates corresponding to image frame 5 are not located at the peak-falling position or trough-rising position of the predicted parabolic curve 1201. Thus, before image frame 5 is acquired, it can be pre-determined that the confidence distance value of image frame 5 does not meet the preset conditions.

[0175] like Figure 12 As shown, the target camera sensor actually acquires the raw image of image frame 5. The target camera sensor can store the raw image of image frame 5 in buffer 1006. Additionally, the target camera sensor can also pass the raw image of image frame 5 to the ISP. After downsampling the raw image of image frame 5, the ISP generates a smaller... Figure 5 ISPs will small Figure 5 It is passed to the perception module and the fitting judgment module.

[0176] Similarly, if the confidence distance value of the pre-judged image frame 5 does not meet the preset conditions, the fitting judgment module will reduce the value by a small amount. Figure 5 After being stored in the fitting queue, the fitting judgment module instructs the memory control module to slide the memory sliding window 1001.

[0177] like Figure 12 As shown, the sliding mechanism of the memory sliding window 1001 includes: the memory control module can increment the reference count of an empty buffer 1204 by 1, indicating that buffer 1204 is occupied. Additionally, the memory control module can decrement the reference count of buffer 1004 by 1, indicating that buffer 1004 is being released. Thus, if memory cleanup is triggered, the data stored in buffer 1004 can be deleted.

[0178] like Figure 13 As shown, the fitting judgment module receives small Figure 5Then, it is also possible to base the fitting on the small plots in the queue (small plots) Figure 1 ,Small Figure 2 ,Small Figure 3 ,Small Figure 4 and small Figure 5 ), and fitted the predicted parabolic curve 1301. Among them, Figure 13 The middle coordinate point 1302 indicates the small Figure 5 The actual coordinates (T) of the corresponding image frame 5 a5 H a5 ). T a5 H represents the actual acquisition time of image frame 5. a5 This is the actual confidence distance value of image frame 5, also known as the second confidence distance.

[0179] After obtaining the predicted parabolic curve 1301 (also referred to as the first predicted parabolic curve), the fitting judgment module can predict the predicted coordinates of the yet-to-be-acquired image frame 6 in the fitting coordinate system based on the predicted parabolic curve 1301. The predicted coordinates of the image frame 6 (also referred to as the first image frame) coincide with the coordinate point 1303 in the predicted parabolic curve 1301. The abscissa of coordinate point 1303 is T. a5 The sum of the time intervals between data collection and acquisition can be denoted as T. b6 That is, the predicted acquisition time of image frame 6. The ordinate of coordinate point 1303 is the confidence distance value of the predicted image frame 6, which can be called the first prediction confidence distance value, or denoted as H. b6 .

[0180] like Figure 13 As shown, H b6 Less than H a5 H a1 H a2 H a3 H a4 All less than H a5 This indicates that the predicted coordinates corresponding to image frame 6 are at the descending position of the peak of the predicted parabolic curve 1301. Thus, even before image frame 6 is acquired, it can be pre-determined that the confidence distance value of image frame 6 meets the preset conditions.

[0181] Additionally, if H b6 Greater than H a5 H a1 H a2 H a3 H a4 All are greater than H a5 This indicates that the predicted coordinates corresponding to image frame 6 are at the rising position of the trough of the predicted parabolic curve 1301, and it can also be determined in advance that the confidence distance value of image frame 6 meets the preset conditions.

[0182] like Figure 13As shown, the target camera sensor actually acquires the raw image of image frame 6. The target camera sensor can store the raw image of image frame 6 in buffer 1106. Additionally, the target camera sensor can also pass the raw image of image frame 6 to the ISP. After downsampling the raw image of image frame 6, the ISP generates a smaller... Figure 6 (First small image). ISP will small Figure 6 It is passed to the perception module and the fitting judgment module.

[0183] If the confidence distance value of the pre-judged image frame 6 meets the preset conditions, the fitting judgment module can also be based on small... Figure 6 The confidence distance value corresponding to image frame 6 is calculated, which can be called the first confidence distance, or H. a6 Thus, by combining the actual acquisition time of image frame 6, the actual coordinates (T) of image frame 6 in the fitted coordinate system can be obtained. a6 H a6 ). T a6 This is the actual acquisition time for image frame 6.

[0184] If H a6 Less than H a5 H a1 H a2 H a3 H a4 All less than H a5 This indicates that the actual coordinates corresponding to image frame 6 are at the descent position of the predicted parabolic curve 1301. If H a6 Greater than H a5 H a1 H a2 H a3 H a4 All are greater than H a5 This indicates that the actual coordinates corresponding to image frame 6 are at the rising position of the trough of the predicted parabolic curve 1301. In both of these cases, the fitting judgment module can determine that image frame 5 is a captured highlight frame, which may be called the target image frame. After the highlight frame is stored in the cache queue, it can be said that the cache queue satisfies the first condition.

[0185] In some embodiments, after determining that a highlight frame has been captured, the fitting judgment module can lock the cache queue, also known as locking the memory sliding window 1001. Thus, as... Figure 13As shown, the buffers indicated by the memory sliding window 1001 include buffer 1005, buffer 1006, buffer 1106, and buffer 1204. Data has been stored in buffer 1005, buffer 1006, and buffer 1106, and accordingly, buffer 1005, buffer 1006, and buffer 1106 are locked.

[0186] After the buffer queue is locked, if the memory sliding window 1001 still contains an empty buffer, the newly acquired raw image can be stored in the empty buffer. After the raw image is stored in the empty buffer, the buffer can be locked. If the memory sliding window 1001 does not contain an empty buffer, the newly acquired raw image is not stored temporarily.

[0187] like Figure 14 As shown, buffer 1204 in the memory sliding window 1001 is a blank buffer. The target camera sensor actually acquires the raw image of image frame 7 (which can be called the fifth image frame). The target camera sensor can store the raw image of image frame 7 into buffer 1204, and then lock buffer 1204. In addition, the target camera sensor can also pass the raw image of image frame 7 to the ISP. After downsampling the raw image of image frame 7, the ISP generates a small... Figure 7 ISPs will small Figure 7 It is passed to the perception module.

[0188] Optionally, the sensing module receives small Figure 7 Subsequently, the perception module continues to evaluate highlight frames based on the already acquired image frames (e.g., image frames 1-7). In other embodiments, the ISP can also... Figure 7 Send it to the fitting evaluation module. The fitting evaluation module can then... Figure 7 Stored in the fitting queue, and based on small Figure 7 Calculate the actual coordinates of image frame 7 in the fitted coordinate system.

[0189] Continue as Figure 14 As shown, the target camera sensor actually acquires the raw image of image frame 8 (which can be referred to as the third image frame). The buffer indicated by the memory sliding window 1001 stores raw images, but not the raw image of image frame 8, thus discarding image frame 8. Additionally, the target camera sensor can also pass the raw image of image frame 8 to the ISP. After downsampling the raw image of image frame 8, the ISP generates a smaller... Figure 8 (Second small image). ISP will small Figure 8 It is passed to the sensing module. Optionally, the sensing module receives a small... Figure 8 Afterwards, the perception module continues to evaluate the highlights based on the acquired image frames (e.g., image frames 1-8).

[0190] Figure 14 The scene shown and Figure 3 In the scenarios shown, what they have in common is that the perception modules all receive small... Figure 8 At that point in time, the highlight frame was identified as image frame 5. The difference is that, while ensuring the raw image of the highlight frame was saved, Figure 14 In this scenario, it only requires 4 buffers. Figure 3 However, in the scenario, it requires 8 buffers. Obviously, after applying the method provided in the embodiments of this application, the memory resources occupied can be effectively reduced in the snapshot scenario.

[0191] In other embodiments, after the cache queue enters a locked state, image frames acquired by the target camera sensor are no longer stored, reducing memory usage. In this scenario, both image frame 7 and image frame 8 can be referred to as the third image frame.

[0192] Additionally, after the buffer queue enters a locked state, the electronic device can time the process to determine the duration of the locked state. If the specified duration is reached and no user-instructed shooting operation is detected, the memory control module can release the buffer corresponding to the memory sliding window 1001. For example, ... Figure 14 The reference counts of buffer1005, buffer1006, buffer1106, and buffer1204 shown are decremented by 1.

[0193] If the specified duration is not reached, and a user-instructed shooting action is detected, in response to the shooting action, the perception module can read the raw image of the highlight frame (i.e., image frame 5) from the buffer queue and pass it to the ISP for processing to obtain the captured photo corresponding to the highlight frame. Afterwards, the memory control module can release the buffer corresponding to the memory sliding window 1001.

[0194] In summary, as an implementation method, such as Figure 15 As shown, the electronic device performs the image capture process as follows:

[0195] S101, the electronic device acquires an image frame c and stores it in the buffer queue.

[0196] Here, image frame c can refer to any image frame acquired after the electronic device starts executing the capture processing method.

[0197] In some embodiments, after the target camera sensor acquires the raw image of image frame c, the raw image of image frame c can be stored in the buffer corresponding to the buffer queue.

[0198] S102, the electronic device acquires the confidence distance value 1 corresponding to image frame c.

[0199] In some embodiments, after the target camera sensor acquires the raw image of image frame c, it can transmit the raw image of image frame c to the ISP, where the ISP performs downsampling processing to obtain the corresponding smaller image c. Then, based on the smaller image c, the confidence distance value 1 corresponding to image frame c can be calculated. If the confidence distance value 1 corresponding to image frame c is calculated, the process proceeds to S103. If the confidence distance value 1 is not calculated based on the smaller image c, the process ends.

[0200] S103, the electronic device acquires the acquisition time 1 corresponding to image frame c.

[0201] Among them, the acquisition time 1 and the confidence distance value 1 can form the actual coordinates of image frame c in the fitted coordinate system.

[0202] S104, the electronic device stores the small image c of image frame c into the fitting queue.

[0203] In some embodiments, after obtaining the actual coordinates of image frame c, step S104 is executed. For example, the small image c can be stored in a fitting queue, and the actual coordinates of image frame c can be associated with the small image c. In this way, the electronic device can obtain not only the small image c from the fitting queue, but also the actual coordinates of image frame c through the small image c.

[0204] In other embodiments, S104 can also be replaced by storing the actual coordinates of image frame c in the fitting queue. Compared to S104, the actual coordinates of image frame c are used to replace small image c.

[0205] In other embodiments, the electronic device executes S104 after executing S101. After executing S104, it then executes S102 and S103. After obtaining the actual coordinates of image frame c, the actual coordinates of image frame c are then associated with the smaller image c in the fitting queue.

[0206] S105, the electronic device determines whether the number of small images stored in the fitting queue is equal to 1.

[0207] In some embodiments, if the number of small plots in the fitting queue is equal to 1, the process proceeds to S106 to determine the memory sliding window corresponding to the cache queue. If the number of small plots in the fitting queue is not equal to 1, the process proceeds to S107.

[0208] Understandably, the small image is an example of an object already stored in the fitting queue. The objects already stored in the fitting queue can also be small images associated with actual coordinates or the actual coordinates of an image frame. Accordingly, in other possible embodiments, if the objects stored in the fitting queue are the actual coordinates of an image frame, the above S105 can be replaced by: determining whether the number of actual coordinates of image frames in the fitting queue is equal to 1. If the objects stored in the fitting queue are small images associated with actual coordinates, or if the objects stored in the fitting queue include small images associated with actual coordinates and small images not associated with actual coordinates, the above S105 can be replaced by: determining whether the number of small images associated with actual coordinates in the fitting queue is equal to 1.

[0209] Alternatively, in a possible embodiment, the electronic device executes S104 and S105 after executing S101. After executing S105, if it is not necessary to fit and predict the parabolic curve, for example, if the process needs to proceed to S106, S102 and S103 may not be executed.

[0210] S106, the electronic device marks a preset number of buffers in the occupied buffer queue. Among them, the marked buffers include the buffers storing the raw image of the image frame c.

[0211] In some examples, the preset number can be 3 frames or a positive integer greater than 4 frames; this application embodiment does not limit this. Additionally, marking occupied buffers can be done by incrementing the reference count corresponding to the buffer by 1. All occupied buffers are used to store raw images captured during the snapshot process. After marking the occupied preset number of buffers, a memory sliding window pointing to the occupied buffer can be determined. Specific implementation details can be found in the foregoing embodiments. Figure 10 The scenario shown will not be elaborated upon here.

[0212] S107, the electronic device determines whether the number of small images stored in the fitting queue is equal to 2.

[0213] In some embodiments, if the number of small images in the fitting queue is equal to 2, the magnitude of the confidence distance between the corresponding confidence distance values ​​of two small images can be used to determine whether a highlight frame has been captured, i.e., the process proceeds to S108. If the number of small images in the fitting queue is not equal to 2, it indicates that the process can be used to determine whether a highlight frame has been captured by fitting a predicted parabolic curve, i.e., the process proceeds to S109.

[0214] S108, the electronic device determines whether a highlight frame has been captured based on the confidence distance value of 1.

[0215] In some embodiments, the fitting queue includes small image c and small image d. Small image d can be a smaller version of image frame d, and image frame d was acquired earlier than image frame c. The predicted parabolic curve cannot be fitted based solely on small images c and d. In this scenario, it can be determined whether a highlight frame has been captured based on the confidence distance value and confidence distance 1 of image frame d.

[0216] In some scenarios, the highlight frame is the image frame with the highest confidence distance. In this scenario, if the confidence distance value of image frame d is greater than the confidence distance value 1, image frame d is determined to be a highlight frame. That is, it can be determined that a highlight frame has been captured, and then the process proceeds to S113. If the confidence distance value of image frame d is not greater than the confidence distance value 1, then the number of captured image frames is insufficient to determine whether a highlight frame has been captured, and the process proceeds to S114, or the process ends.

[0217] In other scenarios, the highlight frame is the image frame with the smallest confidence distance. In this scenario, if the confidence distance value of image frame d is less than the confidence distance value 1, image frame d is determined to be a highlight frame. That is, it can be determined that a highlight frame has been captured, and then the process proceeds to S113. If the confidence distance value of image frame d is not less than the confidence distance value 1, then the captured image frames are insufficient to determine whether a highlight frame has been captured, and the process proceeds to S114.

[0218] In some scenarios, a highlight frame can be either an image frame showing the subject at the highest point (or near the highest point) of its motion trajectory, or an image frame showing the subject at the lowest point (or near the lowest point) of its motion trajectory. In this scenario, if the number of small images in the fitting queue is equal to 2, S108 can be skipped and S114 can be executed. Further details can be found in the documentation. Figure 10 The scene shown.

[0219] S109, the electronic device fits a predicted parabolic curve based on the small plots in the fitting queue.

[0220] In some embodiments, the number of small images in the fitting queue is greater than two. When the number of small images in the fitting queue is greater than two, the predicted parabolic curve can be fitted based on the actual coordinates of the image frames corresponding to the small images in the fitting queue. Implementation details can be found in the foregoing embodiments and will not be repeated here.

[0221] S110, the electronic device predicts the confidence distance value 2 of the next frame image frame e based on the predicted parabolic curve.

[0222] In some embodiments, the implementation details of the electronic device predicting the confidence distance value of the next image frame are as described in the foregoing embodiments and will not be repeated here.

[0223] S111, the electronic device pre-determines whether image frame e meets the preset conditions based on the confidence distance value 2.

[0224] In some embodiments, the method for pre-determining whether an image frame meets preset conditions can refer to the foregoing embodiments, and will not be repeated here. For example, Figure 11 and Figure 12 In the scenario shown, it can be predicted that the next image frame will not meet the preset conditions. For example, Figure 13 In the scenario shown, it can be predicted that the next image frame will meet the preset conditions.

[0225] If the pre-judged image frame e meets the preset conditions, the process proceeds to S112. If the pre-judged image frame e does not meet the preset conditions, the process proceeds to S114.

[0226] S112, after actually acquiring image frame e, the electronic device determines whether a highlight frame has been acquired based on the confidence distance value 1 of image frame c and the actual confidence distance value 3 of image frame e.

[0227] For example, a highlight frame is an image frame showing the subject at the highest point (or near the highest point) of its motion trajectory. In this scenario, if confidence distance 1 is greater than confidence distance 3, and also greater than the confidence distance values ​​of other image frames acquired earlier than image frame c, it is determined that a highlight frame has been acquired, and the process proceeds to S113. If confidence distance 1 is not greater than confidence distance 3, then the acquired image frames are insufficient to determine whether a highlight frame has been acquired, and the process proceeds to S114. For another example, a highlight frame is an image frame showing the subject at the lowest point (or near the lowest point) of its motion trajectory. In this scenario, if confidence distance 1 is less than confidence distance 3, and also less than the confidence distance values ​​of other image frames acquired earlier than image frame c, it is determined that a highlight frame has been acquired, and the process proceeds to S113. If confidence distance 1 is not less than confidence distance 3, then the acquired image frames are insufficient to determine whether a highlight frame has been acquired, and the process proceeds to S114.

[0228] For another example, a highlight frame can be either an image frame showing the subject at the highest point (or near the highest point) of its motion trajectory, or an image frame showing the subject at the lowest point (or near the lowest point) of its motion trajectory. In this scenario, if confidence distance 1 is greater than confidence distance 3, and also greater than the confidence distance values ​​of other image frames acquired earlier than image frame c, it is determined that a highlight frame has been acquired, and the process proceeds to S113. If confidence distance 1 is less than confidence distance 3, but greater than the confidence distance values ​​of other image frames acquired earlier than image frame c, the acquired image frames are insufficient to determine whether a highlight frame has been acquired, and the process proceeds to S114.

[0229] If confidence distance 1 is less than confidence distance 3, and also less than the confidence distance values ​​of other image frames whose acquisition time is earlier than image frame c, it is determined that a highlight frame has been acquired, and the process proceeds to S113. If confidence distance 1 is greater than confidence distance 3, and less than the confidence distance values ​​of other image frames whose acquisition time is earlier than image frame c, the acquired image frames are insufficient to determine whether a highlight frame has been acquired, and the process proceeds to S114.

[0230] S113, Electronic device lock is marked as occupied buffer.

[0231] In some embodiments, the electronic device can lock the buffer queue. For example, it can lock the buffer containing data already stored in the buffer queue. After locking the buffer, refer to... Figure 14 It processes newly acquired image frames.

[0232] S114, the electronic device marks that it occupies a new buffer, and releases the target buffer in the buffer queue.

[0233] In some embodiments, the new buffer can be a frame buffer in storage space that is not marked as occupied. The electronic device can add the new buffer to the buffer queue and then release the target buffer in the buffer queue. The target buffer is a buffer in the buffer queue that already contains data and is marked as occupied. Furthermore, the raw image stored in the target buffer was acquired earlier than other raw images already stored in the buffer queue. For specific implementation details, please refer to the foregoing embodiments. Figure 11 and Figure 12 The scene shown.

[0234] In other embodiments, in S101, if the buffer in the buffer queue is full of data, the raw image of image frame c is stored in an empty buffer in the storage space and marked as occupied. Correspondingly, S114 can be replaced by: adding the buffer containing image frame c to the buffer queue and releasing the target buffer in the buffer queue.

[0235] S115, the electronic device can determine whether it has received a user instruction to take a picture.

[0236] Understandably, during the process of acquiring image frames, in addition to executing S101 to S114 as described above, the electronic device can also display the acquired image frames. During the process of displaying the image frames, the user can trigger the electronic device to generate a snapshot photo corresponding to the featured frame by instructing a shooting operation (e.g., the first operation).

[0237] In some embodiments, after executing S113 or S114, S115 is executed. If it is determined that a user instruction to take a picture has been received, the process proceeds to S116. If it is determined that no user instruction to take a picture has been received, the process proceeds to S119.

[0238] In other embodiments, if the electronic device receives a user instruction to take a picture, it can skip S115 and execute S116.

[0239] S116, the electronic device reads the raw image of the best frame from the buffer queue.

[0240] In some embodiments, after the sensing module identifies a highlight frame, the electronic device reads the raw image of the highlight frame from the buffer marked as occupied in the buffer queue. For example, Figure 14 In the scenario shown, the perception module identifies image frame 5 as the highlight frame. During execution S116, the raw image can be read from buffer 1006 as the raw image of the highlight frame. Buffer 1006 is a buffer marked as occupied in the buffer queue, and it stores the raw image of image frame 5.

[0241] In a possible scenario, when executing S116, the sensing module fails to recognize a highlight frame. The electronic device can use the last raw image stored in the cache queue as the raw image of the highlight frame and read that raw image.

[0242] S117, the electronic device generates a snapshot photo of the highlight frame based on the raw image of the highlight frame.

[0243] In some embodiments, the implementation details of S116 and S117 described above can be found in [reference]. Figure 2 The scenario shown will not be elaborated upon here.

[0244] S118, the electronic device releases all buffers marked as occupied in the buffer queue.

[0245] S119, the electronic device determines whether a shooting timeout has occurred.

[0246] For example, after executing S113, the cache queue is locked. If the cache queue remains locked for a duration exceeding a specified duration (a preset duration threshold), a shooting timeout can be determined. In other embodiments, a shooting timeout can also be determined if the duration for which the cache queue meets a first condition (e.g., the duration between the time when a highlight frame is stored in the cache queue and the current time, or the duration between the time when a highlight frame is determined to have been stored in the cache queue and the current time) exceeds a specified duration.

[0247] For example, after executing S114, the cache queue remains unlocked. In this scenario, if the capture duration exceeds a specified duration, a capture timeout can be determined. The aforementioned capture duration can be the time interval between the time point when the first image frame is written to the cache queue and the time point when S119 is executed.

[0248] If the electronic device determines that a shooting timeout has occurred, the process proceeds to S118. If the electronic device determines that no shooting timeout has occurred, the process ends and waits for the electronic device to execute S115 again.

[0249] In some embodiments, after releasing the occupied buffer, the electronic device continues to acquire image frames and compares whether the confidence distance values ​​corresponding to two adjacent acquired image frames are the same. If they are the same, the acquired image frame is discarded. If they are different, the latest acquired image frame is saved, that is, S101 is executed again.

[0250] For example, after releasing the occupied buffer, the electronic device sequentially acquires the seventh and eighth image frames. The electronic device determines that the confidence distances corresponding to the seventh and eighth image frames are different. Based on the eighth image frame, S101 is executed again to store the eighth image frame into the buffer queue, and to mark the occupied buffers according to the queue length of the buffer queue as a first number (or a preset number). The first number of buffers includes the buffer used to store the raw image data of the eighth image frame. Afterwards, the ninth image frame is acquired, and the ninth image frame can be stored into the buffer queue.

[0251] For example, after releasing the occupied buffer, the electronic device sequentially acquires the tenth and eleventh image frames. The electronic device determines that the confidence distances corresponding to the tenth and eleventh image frames are different, and discards both. Then, it acquires the twelfth image frame. It is determined that the confidence distances corresponding to the eleventh and twelfth image frames are different. Based on the twelfth image frame, S101 is executed again to store the twelfth image frame in the buffer queue, and to mark the occupied buffers according to the queue length of the buffer queue as a first number (or a preset number). The first number of buffers includes the buffer used to store the original image data of the twelfth image frame.

[0252] Subsequently, if another exciting frame is stored in the buffer queue, the buffer occupied by the already stored image frame in the buffer queue can be locked again; this can be called locking the buffer queue. After locking the buffer queue, if the buffer queue is full of image frames, the thirteenth image frame is discarded after being acquired. If the buffer queue is not full of image frames, the thirteenth image frame is stored in the buffer queue after being acquired.

[0253] After applying the above-mentioned image capture method, electronic devices can reduce memory resource usage when capturing images in response to user operations, such as when the subject is jumping or hurdling.

[0254] Some embodiments of this application also provide an electronic device, which may include a memory and one or more processors. The memory and processors are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the electronic device in the above method embodiments.

[0255] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0256] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0257] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0259] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0260] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0261] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0262] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for capturing and processing images, characterized in that, The method includes: A first image frame is acquired and stored in a first queue; the first queue also includes a second image frame; the acquisition time of the second image frame is earlier than that of the first image frame. A third image frame is acquired; the acquisition time of the third image frame is later than the acquisition time of the first image frame; the first image frame, the second image frame, and the third image frame all contain the target object; when the first queue meets a first condition, the third image frame is discarded; the first condition indicates that a target image frame has been stored in the first queue, and the target object in the target image frame is located at the target point of the motion trajectory, and the target point is the point farthest or closest to the preset reference line among the trajectory points corresponding to the motion trajectory; In response to the user's instruction to capture the first image, a corresponding photographic image is generated based on the target image frame.

2. The method according to claim 1, characterized in that, The first queue also includes a fourth image frame, the fourth image frame being acquired earlier than the second image frame, and the second image frame and the first image frame being acquired at adjacent and consecutive times. After storing the first image frame in the first queue, the method further includes: Obtain the first confidence distance of the first image frame; When the second confidence distance corresponding to the second image frame is greater than the first confidence distance and the third confidence distance corresponding to the fourth image frame, or when the second confidence distance is less than the first confidence distance and the third confidence distance, the second image frame that has been stored in the first queue is determined to be the target image frame; the confidence distance value is the distance between the target object and the preset reference line in the image frame.

3. The method according to claim 2, characterized in that, Before discarding the third image frame, the method further includes: It is determined that the first queue is full of image frames.

4. The method according to any one of claims 1-3, characterized in that, After the first image frame is stored in the first queue, and before the third image frame is acquired, the method further includes: The fifth image frame was acquired; When the first queue meets the first condition, it is determined that the first queue is not full of image frames; The fifth image frame is stored in the first queue.

5. The method according to claim 2, characterized in that, Before the first image frame is acquired, the method includes: Fit a first predicted parabolic curve; wherein the value of the first predicted parabolic curve on the first axis is at least related to the acquisition time corresponding to the second image frame and the fourth image frame, and the value on the second axis is at least related to the confidence distance value corresponding to the second image frame and the fourth image frame; Based on the first predicted parabolic curve, estimate the first predicted confidence distance value corresponding to the first image frame; When the first predicted confidence distance value is less than the second confidence distance and the second confidence distance is greater than the third confidence distance, or when the first predicted confidence distance value is greater than the second confidence distance and the second confidence distance is less than the third confidence distance, the second image frame is predicted to be the target image frame.

6. The method according to claim 5, characterized in that, The first queue also stores a sixth image frame, which was acquired earlier than the fourth image frame. Before the second image frame is acquired, the method includes: A second predictive parabolic curve is fitted; wherein the value of the second predictive parabolic curve on the first axis is at least related to the acquisition time of the sixth image frame and the fourth image frame, and the value on the second axis is at least related to the confidence distance value of the sixth image frame and the fourth image frame; Based on the second predicted parabolic curve, estimate the second predicted confidence distance value corresponding to the second image frame; When the second predicted confidence distance value is greater than the third confidence distance, and the third confidence distance is greater than the fourth confidence distance of the sixth image frame, it is predicted that the first queue does not meet the first condition; Alternatively, if the second predicted confidence distance value is less than the third confidence distance, and the third confidence distance is less than the fourth confidence distance, it is predicted that the first queue does not meet the first condition; After acquiring the second image frame, the method further includes: if the first queue is full of image frames, storing the second image frame into the first queue, and deleting the image frame with the earliest storage time in the first queue; if the first queue is not full of image frames, storing the second image frame into the first queue.

7. The method according to claim 1, characterized in that, After generating the corresponding photographed image, the method further includes: Release the buffer occupied by the image frames in the first queue.

8. The method according to claim 7, characterized in that, After releasing the buffer occupied by the image frames in the first queue, the method further includes: The seventh and eighth image frames were acquired sequentially. It is determined that the confidence distances corresponding to the seventh and eighth image frames are different; The eighth image frame is stored in the first queue, and a first number of buffers are marked according to the queue length of the first queue; wherein the first number of buffers includes a buffer for storing the original image data of the eighth image frame. A ninth image frame is acquired, and the ninth image frame is stored in the first queue, wherein the original image data of the ninth image frame is stored in the first number of buffers.

9. The method according to claim 7, characterized in that, The method further includes: The tenth and eleventh image frames were acquired sequentially. It is determined that the confidence distances corresponding to the tenth and eleventh image frames are the same; The twelfth image frame was acquired; It was determined that the confidence distances corresponding to the eleventh and twelfth image frames were different; The twelfth image frame is stored in the first queue, and a first number of buffers are marked according to the queue length of the first queue; wherein the first number of buffers includes a buffer for storing the original image data of the eleventh image frame.

10. The method according to claim 8 or 9, characterized in that, The method further includes: After determining that the first queue meets the first condition again, if the first queue is full of image frames, the thirteenth image frame is discarded after being acquired; if the first queue is not full of image frames, the thirteenth image frame is stored in the first queue after being acquired. When the duration for which the first condition is met in the first queue exceeds a preset duration threshold, the buffer occupied by the image frames in the first queue is released.

11. The method according to claim 1, characterized in that, After acquiring the first image frame, the method further includes: The first image frame is downsampled to obtain the first smaller image; The first small image is stored in the second queue, wherein the second queue is used to store the small images of the acquired image frames, and the second queue also includes the small image corresponding to the second image frame; After acquiring the third image frame, the method further includes: downsampling the third image frame to obtain a second smaller image; and storing the second smaller image in a second queue. Before generating the corresponding captured image based on the target image frame, the method further includes: Based on the small images in the second queue, the target image frame is identified among the acquired image frames; The target image frame is retrieved from the first queue.

12. The method according to claim 1, characterized in that, Before the first image frame is acquired, the method further includes: In response to the user's second action, the shooting preview interface is displayed; After the first image frame is acquired, the method further includes: displaying the first image frame on the shooting preview interface; After the third image frame is acquired, the method further includes displaying the third image frame on the shooting preview interface.

13. An electronic device, characterized in that, The electronic device includes a camera sensor, a processor, and a memory. The camera sensor is used to acquire raw image data of image frames, and the memory is used to store computer instructions and the raw image data of the image frames. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-12.

15. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-12.

Images