Shooting method and electronic equipment

Abstract

A shooting method and an electronic device, the method being applied to the electronic device, the method comprising: in response to a first operation acting on a shooting control, starting a first video encoder; encoding the first video frame through a first video encoder; in the process of encoding the first video frame by the first video encoder, in response to a second operation acting on the shooting control, encoding a second video frame by the first video encoder; generating a first dynamic image, wherein the first dynamic image corresponds to the first video frame; generating a second dynamic image, wherein the second dynamic image corresponds to the second video frame; and controlling the first video encoder to stop encoding. The embodiment of the invention is used for realizing continuous shooting of dynamic images, improving the coding efficiency and saving processing and storage resources.

Description

Shooting method and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 29, 2023, with application number 202310623035.X and invention name “A Shooting Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a shooting method and electronic equipment. Background Art

[0003] When an electronic device takes a photo, the user can click to shoot multiple times in succession, and the electronic device can take multiple frames of images continuously. When an electronic device takes dynamic images (referred to as "motion pictures"), each motion picture taken includes a short period of video. Therefore, if a user takes multiple shots in a short period of time, and the electronic device receives a new shooting request while the previous shooting request is being processed, the electronic device cannot respond to the new shooting request and cannot achieve continuous shooting of dynamic images. The above will result in the electronic device being unable to take multiple motion pictures in a short period of time like taking photos, and the user's shooting experience is poor.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a shooting method and an electronic device for realizing a continuous shooting function of moving pictures in a short period of time, improving encoding efficiency, and saving processing resources.

[0006] In a first aspect, an embodiment of the present application provides a shooting method, which is applied to an electronic device, including: starting a first video encoder in response to a first operation applied to a shooting control; encoding a first video frame through the first video encoder; in the process of encoding the first video frame by the first video encoder, encoding a second video frame through the first video encoder in response to a second operation applied to the shooting control; generating a first dynamic image, the first dynamic image corresponding to the first video frame; generating a second dynamic image, the second dynamic image corresponding to the second video frame; and controlling the first video encoder to stop encoding.

[0007] The first video frame and the second video frame are both images to be encoded. The first video frame and the second video frame may include multiple frames of images to be encoded. The images to be encoded in the first video frame and the second video frame are non-repeating. The first dynamic image includes all images to be encoded in the first video frame, and the first dynamic image also includes part of the second video frame. The second dynamic image includes all second video frames, and the second dynamic image may also include part of the first video frame. In other words, it can be understood that the images to be encoded used to generate the first dynamic image and the second dynamic image may be partially repeated. The first operation is described with reference to FIG. 1C and the related description of S801, and will not be repeated here. The second operation is described with reference to FIG. 1F and the related operations of S834, and will not be repeated here. The first video encoder may refer to the video encoders in FIG. 6A to FIG. 6D, and FIG. 8A to FIG. 8D. The first video encoder may obtain the first video frame and the second video frame from the first video cache queue.

[0008] In an embodiment of the present application, when multiple requests for shooting dynamic images are received continuously, the electronic device can keep a group of encoders from stopping encoding, provide encoding to multiple mixers, and stop encoding after the videos of all dynamic images are synthesized. This can enable the user to shoot dynamic images continuously and improve the user's shooting experience.

[0009] In one possible embodiment, after controlling the first video encoder to stop encoding, the method further includes: in response to a third operation on the shooting control, starting the second video encoder; encoding a third video frame through the second video encoder; generating a third dynamic image, wherein the third dynamic image corresponds to the third video frame; after generating the third dynamic image, controlling the second video encoder to stop encoding; in response to a fourth operation on the shooting control, starting the third video encoder; encoding a fourth video frame through the third video encoder; generating a fourth dynamic image, wherein the fourth dynamic image corresponds to the fourth video frame. In this way, the electronic device can also achieve the process of capturing a moving image with a single shot by the user.

[0010] The third video frame and the fourth video frame are also images to be encoded. Each of the third video frame and the fourth video frame may include multiple frames of images to be encoded. No two of the first video frame, the second video frame, the third video frame, or the fourth video frame are identical. The third video encoder and the fourth video encoder are different video encoders activated at different times by the electronic device.

[0011] In one possible embodiment, before the electronic device controls the first video encoder to stop encoding, the method further includes: adding a first shooting task to a task list in response to the first operation; adding a second shooting task to the task list in response to the second operation; when generating the first dynamic image, clearing the first shooting task, and controlling the first encoder to continue encoding if there are tasks in the task list; and when generating the second dynamic image, clearing the second shooting task, and controlling the first video encoder to stop encoding if there are no tasks in the task list. In this way, the encoder's stopping is determined by the task status in the task list, ensuring that encoding does not stop if there is a shooting request, and submitting the encoded image to the mixer, thereby achieving continuous shooting of dynamic images.

[0012] The task list can be referred to in the examples in Figures 9A and 9B, as well as the descriptions of steps S649, S877, and S880, which are not repeated here. Generating the first dynamic image can refer to the process of receiving the notification of completion of the first video mixing in S876. Similarly, generating the second dynamic image can refer to the process of receiving the notification of completion of the second video mixing in S879.

[0013] In one possible implementation, before the electronic device controls the second video encoder to stop encoding, the method further includes: adding a third shooting task to the task list in response to the third operation; clearing the third shooting task when the third dynamic image is generated; and controlling the third video encoder to stop encoding when no tasks are in the task list. In this way, the encoder's stoppage is determined by the task status in the task list. Encoding is not stopped if a shooting request is received, and the encoded image is submitted to the mixer, thereby enabling continuous shooting of dynamic images.

[0014] The above description may refer to S649 , and generating the third dynamic image may refer to the process of receiving the first video mixing completion notification in S648 .

[0015] In one possible implementation, after the electronic device generates the first dynamic image, the method further includes: displaying a second user interface in response to an operation to launch a gallery; the second user interface includes a first thumbnail of a first static image of the first dynamic image; displaying a third user interface in response to an operation on the first thumbnail, the third user interface including the first static image; and playing the first dynamic image on the electronic device in response to an operation on the first static image. In this way, the electronic device completes capturing the dynamic image, and the user can click to play the dynamic image.

[0016] Among them, the operation of launching the gallery can refer to the description of Figure 1H, the second user interface can refer to Figure 1I or Figure 1J, and the third user interface can refer to the description of (a) in Figure 1E, which will not be repeated.

[0017] In a possible implementation, the first dynamic image includes the first static image and a first video, the first video includes M frames of static images, and M is an integer greater than 1.

[0018] There is a corresponding relationship between the M static image frames and the video frames included in the first dynamic image.

[0019] In one possible implementation, the method further includes: in response to the first operation, starting a first audio encoder; encoding a first audio frame via the first audio encoder, wherein the first dynamic image corresponds to the first audio frame; and while the first audio encoder is encoding the first audio frame, in response to the second operation, encoding a second audio frame via the first audio encoder, wherein the second dynamic image corresponds to the second audio frame. In this way, upon receiving a new capture request, the audio encoder continues encoding, ensuring that multiple captures are completed through one encoding step, thereby achieving multiple continuous captures and improving the user experience.

[0020] The first audio frame and the second audio frame are both audio to be encoded. The first audio frame and the second audio frame may each include multiple frames of audio to be encoded. The audio in the first audio frame and the second audio frame are non-repeated. The first dynamic image includes all audio in the first audio frame and part of the audio in the second audio frame. The second dynamic image includes all audio in the second audio frame and part of the audio in the first audio frame.

[0021] In one possible embodiment, generating the first dynamic image includes: mixing a first coded video frame and a first audio-coded video frame by a first mixer to generate the first dynamic image, wherein the first coded video frame is obtained by encoding the first video frame by the first video encoder, and the first coded audio frame is obtained by encoding the first audio frame by the first audio encoder; generating the second dynamic image includes: mixing a second coded video frame and a second audio-coded video frame by a second mixer to generate the second dynamic image, wherein the second coded video frame is obtained by encoding the second video frame by the first video encoder, and the second coded audio frame is obtained by encoding the second audio frame by the second audio encoder. In this way, the electronic device can activate two mixers and consume the encoding results of one encoder, thereby reducing the number of encoder activations while achieving continuous shooting of dynamic images, improving processing efficiency, saving processing resources, and thus preventing lag caused by continuous shooting of dynamic images.

[0022] The first coded video frame and the second coded video frame may both include multiple frames of coded images. There may be partially repeated coded images in the first coded video frame and the second coded video frame.

[0023] In one possible implementation, after the electronic device generates the first and second dynamic images, the method further includes controlling the first video encoder to stop encoding and controlling the first audio encoder to stop encoding. In this way, after obtaining two continuously captured dynamic images, the electronic device can determine the end of encoding, ensuring that the encoded images can be simultaneously provided to both mixers, thereby ensuring the feasibility of continuous shooting.

[0024] In one possible implementation, the method further includes: in response to an operation for starting the camera, obtaining the first video frame through the camera; storing the first video frame in a first video cache queue; after the first video encoder encodes the first video frame to obtain the first encoded video frame, storing the first encoded video frame in a second video cache queue. In this way, the problem of one-time mixed consumption can be solved through the second video cache queue, and input data can also be provided for the mixer that is started later, ensuring the feasibility of one-way encoding and multi-way mixing. Similarly, it also breaks the original problem boundary, improves encoding efficiency, saves processing resources and storage resources, ensures program security, and reduces problems such as freezes.

[0025] In one possible implementation, the method further includes: in response to an operation for starting the camera, obtaining the first audio frame through a microphone; storing the first audio frame in a first audio cache queue; and after the first audio encoder encodes the first audio frame to obtain the first encoded audio frame, storing the first encoded audio frame in a second audio cache queue. In this way, the problem of one-time mixed consumption can be solved through the second video cache queue, and input data can also be provided for the mixer that is started later, ensuring the feasibility of one-way encoding and multi-way mixing. Similarly, it also breaks the original problem boundary, improves encoding efficiency, saves processing resources and storage resources, ensures program security, and reduces problems such as freezes.

[0026] In one possible implementation, the method further includes: in response to an operation on a dynamic mode control, creating the first video cache queue, creating the second video cache queue, creating the first audio cache queue, and creating the second audio cache queue. In this way, during the streaming process, the electronic device can queue the data before and after encoding to ensure that the mixer can obtain the encoded image from a period of time before shooting.

[0027] In one possible implementation, the first video cache queue is stored in the hardware abstraction layer (HAL) of the electronic device, while the second video cache queue, the first audio cache queue, and the second audio cache queue are stored in the application layer of the electronic device. This reduces the pressure on the application layer due to limited application memory, and provides higher YUV image quality, ensuring better encoding and animation quality.

[0028] In one possible embodiment, the method further includes: determining, by a first mixer, a first time based on the shooting time of the first operation and a specific video duration; mixing, by the first mixer, the i-th coded image frame into the first dynamic image when a first time difference between the timestamp of the i-th coded image frame and the timestamp of the first coded image frame is less than or equal to the first time, the i-th coded image frame being a coded video frame in the second video cache queue; and controlling the first mixer to terminate mixing when the first time difference between the timestamp of the i-th coded image frame and the timestamp of the first coded image frame is greater than the first time. In this way, the mixer can determine when mixing is terminated, thereby ensuring the validity of the mixed video.

[0029] The shooting time of the first operation may be Y. For the first time, reference may be made to the relevant descriptions of S625 and S825, which will not be repeated here.

[0030] In one possible implementation, the method further includes: determining, by the first mixer, a second time based on the shooting time and the specific video duration; and selecting, by the first mixer, a coded video frame from a second video cache queue whose timestamp is closest to the second time as the first coded image frame. In this way, the electronic device can produce an animated image of sufficient duration, ensuring a good shooting effect.

[0031] The shooting time of the first operation may be Y, and the second time may refer to the related descriptions of S625 and S825, which will not be repeated here. The first frame of the coded image is the first frame of the coded image mixed by the mixer 1 .

[0032] In one possible implementation, the method further includes: creating the first video encoder and the first audio encoder in response to an operation on the dynamic mode control.

[0033] In a possible implementation, the method further includes: releasing the first video encoder and the first audio encoder in response to an operation for exiting the camera.

[0034] In one possible embodiment, the method further includes: in response to an operation for starting the camera, displaying a first user interface, wherein the first user interface displays an image acquired by the camera, and the first user interface includes a dynamic mode control in a closed state; and in response to an operation for the dynamic mode control, switching the dynamic mode control to an open state. The operation for starting the camera can be referred to in FIG1A , the first user interface, and the user can be referred to in FIG1B , and the dynamic mode control is the dynamic mode switch 111 in FIG1B . Switching the dynamic mode control to an open state can be referred to in FIG1C .

[0035] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a touch screen, a camera, one or more processors, and one or more memories; the one or more processors are coupled to the touch screen, the camera, and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the electronic device performs:

[0036] In response to a first operation applied to a shooting control, a first video encoder is started; a first video frame is encoded by the first video encoder; while the first video encoder is encoding the first video frame, in response to a second operation applied to the shooting control, a second video frame is encoded by the first video encoder; a first dynamic image is generated, the first dynamic image corresponding to the first video frame; a second dynamic image is generated, the second dynamic image corresponding to the second video frame; and the first video encoder is controlled to stop encoding.

[0037] The first video frame and the second video frame are both images to be encoded. The first video frame and the second video frame may include multiple frames of images to be encoded. The images to be encoded in the first video frame and the second video frame are not repeated. The first dynamic image includes all images to be encoded in the first video frame, and the first dynamic image also includes part of the second video frame. The second dynamic image includes all second video frames, and the second dynamic image may also include part of the first video frame. In other words, it can be understood that the images to be encoded used to generate the first dynamic image and the second dynamic image may be partially repeated. The first operation refers to Figure 1C and the related description of S801, and will not be repeated here. The second operation refers to Figure 1F and the related operations of S834, and will not be repeated here. The first video encoder may refer to the video encoders in Figures 6A to 6D, and Figures 8A to 8D.

[0038] In an embodiment of the present application, when multiple requests for shooting dynamic images are received continuously, the electronic device can keep a group of encoders from stopping encoding, provide encoding to multiple mixers, and stop encoding after the videos of all dynamic images are synthesized. This can enable the user to shoot dynamic images continuously and improve the user's shooting experience.

[0039] In one possible embodiment, after controlling the first video encoder to stop encoding, the electronic device further performs the following operations: in response to a third operation on the shooting control, starting the second video encoder; encoding a third video frame through the second video encoder; generating a third dynamic image, wherein the third dynamic image corresponds to the third video frame; after generating the third dynamic image, controlling the second video encoder to stop encoding; in response to a fourth operation on the shooting control, starting the third video encoder; encoding a fourth video frame through the third video encoder; generating a fourth dynamic image, wherein the fourth dynamic image corresponds to the fourth video frame. In this way, the electronic device can also realize the process of capturing a moving image in a single shot by the user.

[0040] The third video frame and the fourth video frame are also images to be encoded. The third video frame and the fourth video frame may each include multiple frames of images to be encoded. No two of the first video frame, the second video frame, the third video frame, and the fourth video frame have the same image frame.

[0041] In one possible embodiment, before the electronic device controls the first video encoder to stop encoding, the electronic device further performs the following operations: adding a first shooting task to a task list in response to the first operation; adding a second shooting task to the task list in response to the second operation; when generating the first dynamic image: clearing the first shooting task, and if there are tasks in the task list, controlling the first encoder to continue encoding; and when generating the second dynamic image: clearing the second shooting task, and if there are no tasks in the task list, controlling the first video encoder to stop encoding. In this way, the encoder's stopping is determined by the task status in the task list, ensuring that encoding does not stop when a shooting request is present, and submitting encoded images to the mixer, thereby enabling continuous shooting of dynamic images.

[0042] The task list can be referred to in the examples in Figures 9A and 9B, as well as the descriptions of steps S649, S877, and S880, which are not repeated here. Generating the first dynamic image can refer to the process of receiving the notification of completion of the first video mixing in S876. Similarly, generating the second dynamic image can refer to the process of receiving the notification of completion of the second video mixing in S879.

[0043] In one possible implementation, before the electronic device controls the second video encoder to stop encoding, the electronic device further performs the following operations: adding a third capture task to the task list in response to the third operation; clearing the third capture task when the third dynamic image is generated; and controlling the third video encoder to stop encoding if no tasks are present in the task list. In this way, the encoder's stoppage is determined by the task status in the task list. Encoding is not stopped if a capture request is present, and encoded images are submitted to the mixer, thereby enabling continuous capture of dynamic images.

[0044] The above description may refer to S649 , and generating the third dynamic image may refer to the process of receiving the first video mixing completion notification in S648 .

[0045] In one possible implementation, after the electronic device generates the first dynamic image, the electronic device further performs the following operations: in response to an operation to launch a gallery, displaying a second user interface; the second user interface includes a first thumbnail of a first static image of the first dynamic image; in response to an operation on the first thumbnail, displaying a third user interface, the third user interface includes the first static image; and in response to an operation on the first static image, the electronic device plays the first dynamic image. In this way, the electronic device completes the capture, and the user can click to play the dynamic image.

[0046] Among them, the operation of launching the gallery can refer to the description of Figure 1H, the second user interface can refer to Figure 1I or Figure 1J, and the third user interface can refer to the description of (a) in Figure 1E, which will not be repeated.

[0047] In a possible implementation, the first dynamic image includes the first static image and a first video, the first video includes M frames of static images, and M is an integer greater than 1.

[0048] There is a corresponding relationship between the M static image frames and the video frames included in the first dynamic image.

[0049] In one possible embodiment, the electronic device further performs the following steps: in response to the first operation, starting a first audio encoder; encoding a first audio frame via the first audio encoder, wherein the first dynamic image corresponds to the first audio frame; and while the first audio encoder is encoding the first audio frame, in response to the second operation, encoding a second audio frame via the first audio encoder, wherein the second dynamic image corresponds to the second audio frame. In this way, upon receiving a new capture request, the audio encoder continues encoding, ensuring that multiple capture controls are encoded all the way through, enabling multiple continuous captures and improving the user experience.

[0050] The first audio frame and the second audio frame are both audio to be encoded. The first audio frame and the second audio frame may each include multiple frames of audio to be encoded. The audio in the first audio frame and the second audio frame are non-repeated. The first dynamic image includes all audio in the first audio frame and part of the audio in the second audio frame. The second dynamic image includes all audio in the second audio frame and part of the audio in the first audio frame.

[0051] In one possible embodiment, the generating of the first dynamic image is specifically performed by: mixing a first coded video frame and a first audio code video frame by a first mixer to generate the first dynamic image, wherein the first coded video frame is obtained by encoding the first video frame by the first video encoder, and the first coded audio frame is obtained by encoding the first audio frame by the first audio encoder; and the generating of the second dynamic image is specifically performed by: mixing a second coded video frame and a second audio code video frame by a second mixer to generate the second dynamic image, wherein the second coded video frame is obtained by encoding the second video frame by the first video encoder, and the second coded audio frame is obtained by encoding the second audio frame by the second audio encoder. In this way, the electronic device can start two mixers and consume the encoding results of one encoder, thereby reducing the number of encoder startups while achieving continuous shooting of dynamic images, improving processing efficiency, saving processing resources, and thus preventing the lag caused by continuous shooting of dynamic images.

[0052] The first coded video frame and the second coded video frame may each include multiple coded images. The first coded video frame and the second coded video frame may each contain partially repeated coded images. The first dynamic image may refer to the first file of this application, and the second dynamic image may refer to the second file of this application.

[0053] In one possible implementation, after the electronic device generates the first and second dynamic images, the electronic device further controls the first video encoder to stop encoding and controls the first audio encoder to stop encoding. In this way, after obtaining two continuously captured dynamic images, the electronic device can determine the end of encoding, ensuring that the encoded images can be simultaneously provided to both mixers, thereby ensuring the feasibility of continuous shooting.

[0054] In one possible implementation, the electronic device further performs the following steps: in response to an operation for starting the camera, obtaining the first video frame through the camera; storing the first video frame in a first video cache queue; and after the first video encoder encodes the first video frame to obtain the first encoded video frame, storing the first encoded video frame in a second video cache queue. In this way, the problem of one-time mixed consumption can be solved through the second video cache queue, and input data can also be provided for the mixer that is started later, ensuring the feasibility of one-way encoding and multi-way mixing. Similarly, it also breaks the original problem boundary, improves encoding efficiency, saves processing resources and storage resources, ensures program security, and reduces problems such as freezes.

[0055] In one possible implementation, the electronic device further performs: in response to an operation for starting the camera, obtaining the first audio frame through the microphone; storing the first audio frame in the first audio cache queue; after the first audio encoder encodes the first audio frame to obtain the first encoded audio frame, storing the first encoded audio frame in the second audio cache queue. In this way, the problem of one-time mixed consumption can be solved through the second video cache queue, and input data can also be provided for the mixer that is started later, ensuring the feasibility of one-way encoding and multi-way mixing. Similarly, it also breaks the original problem boundary, improves encoding efficiency, saves processing resources and storage resources, ensures program security, and reduces problems such as freezes.

[0056] In one possible implementation, the electronic device further executes: in response to an operation on the dynamic mode control, creating the first video cache queue, creating the second video cache queue, creating the first audio cache queue, and creating the second audio cache queue. In this way, during the streaming process, the electronic device can queue the data before and after encoding to ensure that the mixer can obtain the encoded image a period of time before shooting.

[0057] In one possible implementation, the first video cache queue is stored in the hardware abstraction layer (HAL) of the electronic device, while the second video cache queue, the first audio cache queue, and the second audio cache queue are stored in the application layer of the electronic device. This reduces the pressure on the application layer due to limited application memory, and provides higher YUV image quality, ensuring better encoding and animation quality.

[0058] In one possible embodiment, the electronic device further performs the following steps: determining a first time based on the shooting time of the first operation and the specific video duration by a first mixer; mixing the i-th coded image frame with the first dynamic image by the first mixer if a first time difference between the timestamp of the i-th coded image frame and the timestamp of the first coded image frame is less than or equal to the first time, wherein the i-th coded image frame is a coded video frame in the second video cache queue; and controlling the first mixer to end mixing if the first time difference between the timestamp of the i-th coded image frame and the timestamp of the first coded image frame is greater than the first time. In this way, the mixer can determine when mixing ends, thereby ensuring the validity of the mixed video.

[0059] The shooting time of the first operation may be Y. For the first time, reference may be made to the relevant descriptions of S625 and S825, which will not be repeated here.

[0060] In one possible implementation, the electronic device further performs the following steps: controlling the first mixer to determine a second time based on the shooting time and the specific video duration; and selecting, by the first mixer, a coded video frame from a second video cache queue whose timestamp is closest to the second time as the first coded image frame. In this way, the electronic device can produce an animated image of sufficient duration, ensuring a good shooting effect.

[0061] The shooting time of the first operation may be Y, and the second time may refer to the related descriptions of S625 and S825, which will not be repeated here. The first frame of the coded image is the first frame of the coded image mixed by the mixer 1 .

[0062] In a possible implementation, the electronic device further executes: creating the first video encoder and the first audio encoder in response to an operation on the dynamic mode control.

[0063] In a possible implementation, the electronic device further executes: in response to an operation of exiting the camera, releasing the first video encoder and the first audio encoder.

[0064] In one possible embodiment, the electronic device further performs: in response to an operation acting on starting the camera, displaying a first user interface, the first user interface displaying an image acquired through the camera, the first user interface including a dynamic mode control in a closed state; in response to an operation acting on the dynamic mode control, the dynamic mode control switches to an open state.

[0065] The operation of starting the camera can refer to FIG1A , the first user interface, and the user can refer to FIG1B , and the dynamic mode control is the dynamic mode switch 111 in FIG1B . The dynamic mode control is switched to the on state can refer to FIG1C .

[0066] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the shooting method described in the first aspect or any possible implementation method of the first aspect.

[0067] In a fourth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the shooting method as described in the first aspect or any possible implementation of the first aspect.

[0068] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the shooting method as described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] FIG1A to FIG1J are schematic diagrams of a set of shooting scenes in dynamic mode provided by an embodiment of the present application;

[0070] 2A to 2D are schematic diagrams of data structures of a group of photos and videos in a continuous shooting scenario provided by an embodiment of the present application;

[0071] FIG3 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0072] FIG4A is a schematic diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application;

[0073] FIG4B is a schematic diagram of the software and hardware structure of another electronic device provided in an embodiment of the present application;

[0074] FIG5 is a flow chart of a method for processing video data in a dynamic mode provided by an embodiment of the present application;

[0075] 6A to 6C are flowchart diagrams of a method for video processing in a dynamic mode provided by an embodiment of the present application;

[0076] FIG6D is a schematic flow chart of a method for stopping video encoding provided by an embodiment of the present application;

[0077] FIG7A is a schematic diagram of a format structure of a first file provided in an embodiment of the present application;

[0078] FIG7B is a schematic diagram of a timing diagram of audio data and video data encapsulation of video data provided by an embodiment of the present application;

[0079] 8A to 8C are flowchart diagrams of another method for processing video data in a dynamic mode provided by an embodiment of the present application;

[0080] FIG8D is a flowchart of another method for stopping video encoding provided by an embodiment of the present application;

[0081] 9A and 9B are schematic diagrams showing changes in a set of task queues provided in an embodiment of the present application;

[0082] FIG10 is a schematic diagram of hybrid coding of video in a dynamic image provided by an embodiment of the present application;

[0083] 11A to 11E are schematic diagrams of image frame production and consumption of a group of second video cache queues provided in an embodiment of the present application;

[0084] FIG12A is a schematic flow chart of a method for exiting shooting provided in an embodiment of the present application;

[0085] FIG12B is a flowchart of another method for exiting shooting provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0087] In one embodiment provided in the present application, a terminal device such as a mobile phone or tablet computer with shooting and image processing functions (referred to as an electronic device, and hereinafter referred to as electronic device) can be used in a shooting scene, and the user can shoot dynamic images, that is, photos with dynamic effects. The dynamic image (video data in a short period of time) can include videos and images in a short period of time. The electronic device can also save dynamic images, and the user can select an image from a dynamic multi-frame image. Among them, the video included in the dynamic image is a mixture of images captured by the camera and audio captured by the microphone.

[0088] A dynamic image refers to an image with dynamic effects. The dynamic effect can be reflected in the switching display of multiple static images according to preset rules, that is, video playback can produce dynamic effects; a static image is an image with a static picture. In an embodiment of the present application, a dynamic image may include a static image (i.e., a cover image) and a video (such as an MP4 file). For example, when an electronic device displays a dynamic image, it can display the cover of the dynamic image when it does not receive user operations, and display the video corresponding to the dynamic image when it receives user operations (such as long press operations). Among them, the cover image can be called a cover frame. For example, the cover image can have the same content as a frame of image in the video, but a different resolution.

[0089] In the embodiments of the present application, the electronic device may not be limited to a mobile phone or a tablet computer. The electronic device may also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the terminal.

[0090] The following describes the application scenarios involved in the embodiments of this application.

[0091] In daily life, people often use electronic devices such as smartphones and tablets to take photos. Figures 1A to 1J are schematic diagrams of a dynamic image shooting scenario disclosed in this application. The following describes the process of performing dynamic shooting in the preview state of an electronic device in dynamic mode.

[0092] FIG1A exemplarily illustrates a user interface diagram of an electronic device. A user turns on their electronic device, causing the display screen of the electronic device to display the electronic device's desktop, i.e., user interface 100. As shown in FIG1A , user interface 100 may include an icon for at least one application (e.g., weather, calendar, mail, settings, app store, notes, gallery, phone, short message, browser, and camera (camera control 101)). The positions of the application icons and the corresponding application names may be adjusted according to the user's preferences, and this application is not limited thereto.

[0093] It should be noted that the interface diagram of the electronic device shown in FIG1A is an exemplary display of an embodiment of the present application. The interface diagram of the electronic device may also be in other styles, and the present application does not limit this.

[0094] In FIG1A , a user can click on a camera control 101 in a user interface 100. After the electronic device receives the operation on the camera control 101, it can display a user interface 110 shown in FIG1B . FIG1B is an exemplary embodiment of a shooting preview interface shown in FIG1B . As shown in FIG1B , a user uses an electronic device to shoot, that is, the electronic device is currently in a photo shooting mode. In the user interface 110, the electronic device can display the specific controls or switches of FIG1B , which are described in detail below:

[0095] The shooting mode menu 112 may include options for multiple camera modes such as portrait 1121, video 1122, photo 1123, night scene 1124 and more 1125. Different camera modes can achieve different shooting functions. The camera mode pointed to by the "triangle" in the shooting mode menu 112 is used to indicate the initial or user-selected camera mode. As shown in Figure 1B, the "triangle" points to "photo", indicating that the current camera is in photo mode.

[0096] The gallery 113 is used for users to view the pictures and videos they have taken.

[0097] The shooting control 114 is used to enable the electronic device to shoot images or record videos in response to user operations.

[0098] The camera switching control 115 is used to switch the camera for collecting images between the front camera and the rear camera.

[0099] The shooting function switch 116 may include a beauty switch 1161, an AI mode switch 1162, a flash switch 1163, a filter switch 1164, and a setting control 1165.

[0100] The beauty switch 1161 is used to turn on or off the beauty function.

[0101] The AI ​​mode switch 1162 is used to turn on or off the automatic recognition mode function.

[0102] The flash switch 1163 is used to turn on or off the flash.

[0103] The filter switch 1164 is used to turn the filter on or off.

[0104] The setting control 1165 is used to set various parameters when acquiring images.

[0105] In addition, the user interface 110 may further display a dynamic mode switch 111 for turning the dynamic mode on or off. When the dynamic mode is on, the user can click the capture control 114 to capture a moving picture.

[0106] After the user opens the camera and enters the shooting preview interface, the dynamic mode can be turned on or off.

[0107] Optionally, the camera is turned on, the dynamic mode is in the off state, and the user clicks to turn on the dynamic mode. In Figure 1A, in the user interface 100, the user clicks the camera control 101, and in response to the operation of turning on the camera, the electronic device displays Figure 1B. In the user interface 110, the dynamic mode switch 111 is in the off state, and the user can click the dynamic mode switch 111. In response to the operation of the electronic device turning on the dynamic mode, the electronic device can display the user interface 120. Figure 1C is a schematic diagram of a user interface for starting shooting in dynamic mode disclosed in an embodiment of the present application. As shown in Figure 1C, in the user interface 120, the user can click the shooting control and the electronic device can shoot dynamic images. Among them, the dynamic mode switch 121 is in the on state.

[0108] Optionally, the camera is turned on with the dynamic mode enabled. In FIG1A , in user interface 100 , the user clicks camera control 101 . In response to the camera turning on operation, the electronic device displays FIG1C . In user interface 120 , the dynamic mode switch 111 is turned on, and the user can click capture control 114 to enable the electronic device to capture dynamic images.

[0109] Users can click to view dynamic images saved in the gallery. For example, if a user has already captured a dynamic image, as shown in Figure 1D, in user interface 130, the user can click the gallery control. In response to opening the gallery, the electronic device can display the most recently captured dynamic image. As shown in (a) in Figure 1E, the electronic device in user interface 140 can display a recently captured dynamic image. When the user long presses preview area 141, the electronic device can play the dynamic image. The user can process the current dynamic image, and user interface 140 can display dynamic image processing functions 142. Dynamic image processing functions 142 can include sharing, favorites, editing, deleting, and more. User interface 140 can also include an identifier 143A and parameters 143B. Identifier 143A can be used to indicate that the image currently displayed on the interface is a dynamic image; parameters 143B can be used to display information such as the shooting parameters of the image currently displayed on the interface. The above is only one way to view dynamic images. Users can also open the gallery on the desktop to view it, and this application is not limited to this.

[0110] It should be noted that the above-mentioned dynamic mode switch pattern and position display are exemplary illustrations and are not limited in this application. In addition, the display interface of the electronic device for shooting dynamic images in Figure 1E (a) is also an exemplary display and is not limited in this application.

[0111] When the electronic device displays (a) in Figure 1E, the electronic device can click on the parameter 143B control. In response to the above-mentioned click operation, the electronic device can display the detailed parameter information of the picture. As shown in (b) in Figure 1E, the user interface 145 has a new information box 1451 compared to the user interface 140. The information box 1451 is used to display the parameters of the HDR image (preview area 141). Among them, the name of the above image is IMG_20230527_111422, the resolution of the image displayed in the preview area 141 is 3072×4097, the data size of the image displayed in the preview area 141 is 13.53MB, and the storage path of the file of the above image is internal storage / DCIM / Camera / IMG_20230501_111422.jpg. The information in the above information box indicates that the above image is a jpg format image.

[0112] When the electronic device displays (a) of FIG. 1E , the electronic device can click on the edit control 142a. In response to the operation of clicking on the edit control 142a, the electronic device can display the image encoding interface. As shown in (c) of FIG. 1E , in the user interface 146 (image encoding interface), the user can edit the captured dynamic image. The user interface 146 may include a return control 1461, a picture display window 1462, a set as cover control 1463, a save as photo control 1464, a video image frame preview window 1465, and an image editing bar 1466. The video image frame preview window 1465 may include thumbnails of multiple consecutive frames of the dynamic image video. The picture display window 1462 displays the image corresponding to the thumbnail 1465a. The thumbnail 1465a may correspond to the cover frame of the current dynamic image. The image editing bar 1466 may include functions such as dynamic photos, cropping, graffiti, and filters. In the user interface 146 , the dynamic photo is in the open state, and the user can slide the video image frame preview window 1465 left and right to select a frame, and the image displayed in the picture display window 1462 will also change accordingly.

[0113] In the case of a user continuously shooting multiple dynamic images, the user clicks the shooting control 114 once and then clicks the shooting control again within a short period of time. The electronic device cannot respond to the latter shooting operation to obtain the dynamic image. In this case, if the gallery is opened, the electronic device only saves the dynamic image shot previously.

[0114] Exemplarily, as shown in FIG1C , after the user clicks the shooting control 114 for the first time, he immediately clicks to shoot for the second time (assuming that the time difference between the first shooting and the second shooting is 0.7s.). As shown in FIG1F , in the user interface 150, the user clicks the shooting control 114 for the second time. The electronic device has not yet completed processing the dynamic image shot by the first click. Therefore, when the electronic device is processing the dynamic image shot by the first click, it receives the second click to shoot. The electronic device does not have time to process the second dynamic image, and thus cannot capture the second dynamic image. The electronic device can display a preview screen that cannot respond. As shown in FIG1G , in response to the second click to shoot operation, the electronic device cannot process the dynamic image shot by the second click, and the user interface 160 can display a loading buffer icon on the preview screen.

[0115] During the above process, since the electronic device cannot respond to the request for the second click to shoot, the electronic device cannot obtain the two dynamic images. As shown in Figure 1H, after the above two shots (no subsequent shots), in the user interface 170, the user can click on the gallery control 171 (the specific description of Figure 1H can refer to Figure 1A, which is not repeated here). In response to the above operation of opening the gallery, the electronic device can display the user interface 180. As shown in Figure 1I, in the gallery, the electronic device stores 100 photos and 10 videos, including the dynamic image 181 obtained by just clicking the shooting control for the first time. The dynamic image of the second click on the shooting control is not obtained (stored) in the gallery. In this way, the electronic device cannot respond to the user's second shooting operation to generate dynamic images, resulting in a poor experience for users to shoot dynamic images.

[0116] In view of the situation where the above-mentioned implementation cannot continuously shoot dynamic images, in an embodiment of the present application, for the same two shots in Figures 1C and 1F, the electronic device will not display the loading buffer icon for the second click to shoot, but will display a normal preview screen (as shown in Figure 1F). After the above two shots, the electronic device can also open the gallery. As shown in Figure 1J, the user interface 190 includes two dynamic images: dynamic image 191A and dynamic image 191B. Among them, dynamic image 191A is obtained by the first click to shoot, and dynamic image 191B is obtained by the second click to shoot. In this way, the electronic device can respond to the user's second continuous shooting to obtain a dynamic image, thereby improving the user's dynamic image shooting experience.

[0117] In the above two shooting processes, the second shooting is after the first shooting. In the preview screen, the ball slides down the slope over time. Therefore, the ball in the animation 191A is located higher on the slope than the ball in the animation 191B.

[0118] The electronic device can complete the user's single-click shooting of dynamic images. As shown in Figure 2A, after the user clicks to shoot the dynamic image, in response to the user's click-to-shoot 1 operation (as shown in Figure 1C), the electronic device may need to generate the video in the dynamic image. That is, the electronic device starts the video encoder, audio encoder and mixer. Among them, the video encoder, audio encoder and mixer can be started successively in response to the click-to-shoot 1 operation, and may not be started at the same time. The video encoder and audio encoder encode the image and audio respectively. The mixer mixes the encoded image and audio to obtain the video in the dynamic image. When the video is obtained, the video encoder, audio encoder and mixer can be stopped. For the case of shooting a dynamic image with a single click, the video encoder and audio encoder are only used for encoding processing of one video, so the encoding process can be realized. For the case of discontinuous multiple clicks to shoot dynamic images, as shown in Figure 2A, a first group of encoders (audio encoder 1 and video encoder 1) and a mixer 1 are created. The electronic device can start this group of encoders (audio encoder 1 and video encoder 1) and this mixer in response to the user clicking to shoot 1 (Figure 1C). After obtaining a dynamic image, the audio encoder 1, video encoder 1 and mixer 1 are stopped and released. The user can then click again (Figure 1C). In response to clicking to shoot 2, the electronic device creates audio encoder 2, video encoder 2, and mixer 2 and starts them. After obtaining the second dynamic image, the audio encoder 2, video encoder 2 and mixer 2 can be stopped and released.

[0119] Of course, before starting the encoder, the electronic device may create the encoder, and after stopping the encoder, the electronic device may clear (release) the encoder.

[0120] When a user takes a photo, because the image is captured at a specific point in time (the time interval between image captures is extremely short), the user can perform a continuous shooting operation, that is, continuously click the shooting control within a short period of time, and the electronic device can correspondingly capture multiple frames of images. However, according to the above-mentioned user's habitual continuous shooting method, in the above-mentioned dynamic (shooting) mode, the user cannot achieve continuous shooting by continuously clicking the shooting control.

[0121] As shown in Figure 2B, when the user clicks "shoot" for the first time (as shown in Figure 1C), in response to the first click, the electronic device can start the video encoder, audio encoder, and mixer to perform the encoding and mixing process. During the encoding and mixing process, the user clicks "shoot" a second time (as shown in Figure 1F). In response to the second click, the electronic device should start the video encoder, audio encoder, and mixer. For example, after the first click, the video encoder and audio encoder need to perform encoding within 1.5 seconds. 0.7 seconds after the first click, the second click is received. At this time, the electronic device has not yet completed processing the first click and has no time to process the second click. Specifically, the encoding of one video channel corresponds to a group of encoders. The encoding of the second video channel (the video to be encoded by the first click) can only be started when the encoding of the previous video channel (the video to be encoded by the first click) is stopped. In the case of Figure 2B, this group of encoders is already running and cannot respond to the request to start encoding for the second click. As shown in the image in FIG2B , since the second click to shoot fails to respond, the electronic device can shoot the preview interface showing the loading pattern (as shown in FIG1G ), and the electronic device cannot obtain the animated image shot this time.

[0122] In the encoding mixing process shown in Figure 2B, the video encoder encodes the image to be encoded from video cache queue 1, the audio encoder encodes the audio to be encoded from audio cache queue 1, and the mixer directly mixes the encoded image and audio. During the mixing process, the mixer's input image is obtained from the encoder's output. That is, when one frame of data is encoded, the mixer obtains one frame of data and mixes it. This process means that a group of encoders only provides encoding results to one mixer at a time, that is, a group of encoders and a mixer are bound together.

[0123] In order to solve the above problems, in Figure 2C, in the scene of continuous shooting of dynamic images, the electronic device receives a shooting request, and correspondingly starts a group of encoders and a mixer. Compared with the solution of Figure 2B, the electronic device can respond to the second click to shoot, and after the video encoder 1, audio encoder 1 and mixer 1 have been started, it starts the video encoder 2, audio encoder 2 and mixer 2. In the above process, in the same period, the electronic device needs to start multiple sets of encoders and mixers (2 sets, 3 sets or even more), which is a great test for the processing resources of the electronic device and is prone to device jamming. Furthermore, in Figure 2C, the electronic device also has 4 cache queues, namely video cache queue 1, audio cache queue 1, video cache queue 2 and video cache queue 2. This requires occupying a large amount of storage resources of the electronic device at the same time, and the storage resources are tight, which can also cause the electronic device to jam and respond slowly.

[0124] For resource scheduling and program security considerations, the Android system has a limit on the number of encoders that can be used. Generally, electronic devices can only start one set of encoders. Therefore, none of the above solutions can solve the problem of dynamic video continuous shooting.

[0125] In response to the above-mentioned problem, a shooting method is proposed in an embodiment of the present application. After the dynamic mode is started, when the user clicks to shoot a moving picture for the first time, the audio encoder, video encoder and mixer 1 are started. The audio encoder performs audio encoding, the video encoder performs image encoding, and mixer 1 mixes the encoded audio and the encoded image. During the above-mentioned encoding and mixing process, the user clicks to shoot a moving picture for the second time, the electronic device starts mixer 2, and mixes the encoded data of the audio encoder and the video encoder. Mixers 1 and 2 can reuse part of the encoding results for mixing. After mixer 1 completes the mixing, since the encoding mixing of the second click to shoot is not completed, the audio encoder and video encoder continue to encode until mixer 2 completes the mixing, and the audio encoder, video encoder and mixer 2 can be stopped.

[0126] As shown in Figure 2D, when a user clicks the capture control for the first time, in response to the first capture action, the electronic device may activate the video encoder, audio encoder, and mixer 1. The electronic device may then begin encoding and mixing to generate a first video. While the video encoder and audio encoder are encoding, the user clicks the capture control a second time. In response to the second capture action, the electronic device may activate mixer 2 to begin mixing the encoding results of the video encoder and audio encoder. When video mixing for the GIF image generated by the first capture action is complete, mixer 1 stops, but the encoders do not; that is, the video encoder and audio encoder continue encoding. When video mixing for the second capture action is complete, mixer 2 stops. If the electronic device does not receive a third capture action during encoding, it may determine that all capture requests have completed video mixing tasks and stop the video encoder and audio encoder. If a third capture action is received, the video encoder and audio encoder continue encoding, following the same process as for the second capture action and will not be further described.

[0127] In the above embodiment, when multiple requests for shooting dynamic images are received continuously, the electronic device can keep a group of encoders from stopping encoding, provide encoding to multiple mixers, and stop encoding after the videos of all dynamic images are synthesized, thereby enabling the user to continuously shoot dynamic images and improve the user's shooting experience.

[0128] Furthermore, the generation of dynamic images requires the use of multiple frames of images acquired within a continuous period before and after the user clicks to shoot. In Figure 2C, a group of encoders only provides encoded data to one mixer. For example, in Figure 2C, video encoder 1 obtains encoded data from video cache queue 1, and audio encoder 1 cache queue obtains encoded data from audio cache queue 1. The edited results are immediately sent to mixer 1 for consumption and mixing. After mixer 1 consumes the encoded data, the encoded data for this frame is not further stored. This consumption process is a one-time process, making it impossible for multiple mixers to use the encoding results of the same group of encoders simultaneously. For example, when the user clicks to shoot for the first time, mixer 1 consumes the encoded results of the first frame and clears the encoded data of the first frame. When the user clicks to shoot for the second time, mixer 2 also needs to use the encoded results of the first frame. Since the data has been cleared, it cannot be used. Therefore, it is impossible for a group of encoders to provide encoded data to multiple mixers. As shown in Figure 2D, in an embodiment of the present application, the electronic device needs to add two cache queues: a second video cache queue and a second audio cache queue for storage. The encoding results of the video encoder and audio encoder are stored in the second video cache queue and the second audio cache queue, respectively. The second video cache queue and the second audio cache queue store the encoded data within a time period. This allows each mixer to distinguish the data it needs to mix from the second video cache queue and the second audio cache queue, specifically for the time of the current capture request. This allows the mixer to be unbound from a group of encoders. This reduces the number of encoder startups and processing steps, conserving processing resources and ensuring program security.

[0129] It should be noted that the encoder and mixer in Figures 2A to 2D can only be started at the same time, or they can be started one after another after clicking to shoot. The encoder and mixer respond to the same shooting request, and the specific order of starting is not limited in this application. Of course, in the case where the encoder and mixer are stopped at the same time, there may also be a certain time sequence in the actual program execution process, which is also not limited. In the above process, when shooting a continuous animation, if a new shooting request is received during the encoding process, the electronic device can start another mixer. At this time, the two encoders can reuse the encoding results of the same group of encoders (as the input of the mixer). In this way, two videos can be mixed separately. After the mixing of the video of the previous click is completed, mixer 1 can be stopped, and the encoder does not stop. Thereafter, the encoded data of this group of encoders is consumed by mixer 2 until the mixing is completed. When the mixing is completed, if all click-to-shoot tasks are completed (no new click-to-shoot), the encoding is terminated and the encoder is stopped. In this way, it can be ensured that the encoder stops encoding only after the mixing of all the animation videos is completed, ensuring the reliability of the encoder start and stop, providing reliable encoding results for continuous shooting animations, and ensuring the feasibility of continuous shooting of animations. Furthermore, it can also reduce the number of encoders started, reduce the occurrence of electronic device freezes, and ensure the security of resources and programs.

[0130] Furthermore, when shooting continuous motion images, the image only needs to be encoded once, and the video of each motion image is mixed separately to obtain the motion image of each shot, ensuring the feasibility of continuous shooting of multiple motion images. This also reduces the number of encoding times, improves execution efficiency, and saves processing resources.

[0131] The following describes the device involved in the embodiments of the present application.

[0132] FIG3 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0133] The electronic device may include a processor 310, an external memory interface 320, an internal memory 321, a Universal Serial Bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a microphone 370C, a sensor module 380, a camera 393, and a display screen 394. The sensor module 380 may include a touch sensor 380K.

[0134] It is understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0135] The processor 310 may include one or more processing units, for example: the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. Among them, the controller can be the nerve center and command center of the electronic device. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution. A memory can also be set in the processor 310 for storing instructions and data.

[0136] In the embodiment provided in this application, the electronic device can execute the shooting method through the processor 310.

[0137] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes the instructions stored in the internal memory 321 to perform various functional applications and data processing of the electronic device.

[0138] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied in electronic devices. The wireless communication module 360 ​​can provide solutions for wireless communications including wireless local area networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc. applied in electronic devices.

[0139] The electronic device implements display functionality through a GPU, display screen 394, and an application processor. A GPU is a microprocessor for image processing that connects display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or modify display information.

[0140] Display screen 394 is used to display images, videos, etc. Display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (AMOLED), flexible light-emitting diode (FLED), Mini LED, Micro LED, Micro-OLED, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device can include one or N display screens 394, where N is a positive integer greater than 1.

[0141] The electronic device can achieve the acquisition function through an ISP, a camera 393, a video codec, a GPU, a display 394, and an application processor.

[0142] The ISP processes data fed back by camera 393. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image or video. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 393.

[0143] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image or video signal. The ISP outputs the digital image or video signal to the DSP for processing. The DSP converts the digital image or video signal into an image or video signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include multiple cameras 393.

[0144] Video codecs are used to compress or decompress digital video. Electronic devices may support one or more video codecs. This allows them to play or record videos in a variety of encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0145] Touch sensor 380K, also known as a "touch panel," can be disposed on display screen 394. The touch sensor 380K and display screen 394 form a touch screen, also known as a "touch screen." Touch sensor 380K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 394. In other embodiments, touch sensor 380K can also be disposed on the surface of the electronic device, at a location different from that of display screen 394.

[0146] It should be noted that the functions of other modules not mentioned in the electronic device shown in FIG3 can be referred to in relevant technical documents, and this application does not elaborate on them.

[0147] FIG4A is a schematic diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application.

[0148] As shown in FIG4A , the software framework of the electronic device involved in the present application may include an application layer, an application framework layer (framework, FWK), a system library, an Android runtime, a hardware abstraction layer (HAL) and a kernel layer (kernel).

[0149] The application layer may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications (also referred to as applications, some of which are not shown). Among them, the camera application can be used to capture images and videos.

[0150] As shown in FIG4A , the camera application may include a dynamic photographing module, a static photographing module, a video management module, an audio management module, an encoding control module, a data caching module, a mixing module, and a storage module.

[0151] The camera's modes may include, but are not limited to, motion mode, photo mode, video mode, portrait mode, and night mode. These modes may include a motion mode, which allows users to capture dynamic images, including audio and multiple frames. Both the motion camera module and the static camera module can be used to detect user actions and determine the camera's mode. The motion camera module controls encoding and mixing in motion mode, generating video within the dynamic image; the static camera module captures images within the dynamic image, such as the cover frame within the dynamic image, in motion mode.

[0152] The video management module manages video data. For example, it distributes video data configuration information. The video management module can include a video data cache area, such as a video surface. This video surface can be stored in the video acquisition module within the camera HAL and provided to the video encoding module. The video management module can also configure the FWK path for dynamic image encoding, i.e., post-camera.

[0153] The encoding control module may include a video encoding module and an audio encoding module. The video encoding module may control the video encoder to encode the image to be encoded. The audio encoding module may control the audio encoder to encode the audio to be encoded in the audio buffer module.

[0154] The data cache module includes an audio queue module and a video queue module. The data cache module can cache the data encoded by the encoding control module. Among them, the audio queue module can cache the encoded audio output by the audio encoding module, and the video queue module can cache the encoded image output by the video encoding module.

[0155] The mixing module is used to encapsulate the encoded image and encoded audio into video data. This encapsulates the encoded audio data cached by the audio queue module and the encoded video data cached by the video queue module to produce video. When mixing to produce a dynamic image, the mixing module can include N mixers at any given moment, where N is a positive integer. In Figures 6A and 6B, N is 1; in Figures 8A through 8C, N is 2. Each mixer can be used to produce a dynamic image video.

[0156] The storage module is used to store the dynamic image after the mixer mixes, that is, to store the video, image and related description information in the dynamic image. The storage module can control the memory to store.

[0157] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0158] As shown in Figure 4A, the application framework layer may include the camera FWK (framework layer), the media FMK (framework layer), the audio FWK (framework layer), and the post camera. The camera FWK can provide an API interface for applications (e.g., camera applications) to call, and then receive requests from the applications, while maintaining the business logic of the internal flow of requests. Finally, by calling the camera AIDL cross-process interface, the request is sent to the camera service (Camera Service) for processing. Then, it waits for the return of the camera service (Camera Service) result, and then sends the final result to the camera application. Among them, AIDL is the full name of Android Interface Definition Language in English, and its Chinese meaning is Android Interface Definition Language. Similarly, the media FWK can use the API interface to call the corresponding application (e.g., camera application), and then receive requests from the application (e.g., camera application), pass the application's request down, and then return it to the application. The audio FWK can provide an API interface for applications to call, and the application can transmit audio or issue instructions (issue instructions to the audio HAL) through the audio FWK.

[0159] The post camera also provides an interface for applications to call. In dynamic mode, the video management module can configure the post camera. The post camera acts as a new data path for the camera application, receiving encoding requests from the dynamic capture module and the video management module. The post camera can then pass the video buffer data in the camera HAL to the video surface through the post camera and transmit it to the camera application. The video encoding module consumes the encoded data from the video surface, thus forming a data path. It is understood that the application framework layer can also include a window manager, content provider, view system, telephony manager, resource manager, notification manager, etc. The specific meaning of this layer can be found in the relevant technical documentation and will not be explained in detail here.

[0160] The runtime is responsible for system scheduling and management. It consists of a core library and a virtual machine. The core library consists of two parts: one containing the functions that a programming language (e.g., Java) needs to call, and the other the system's core library.

[0161] The application layer and application framework layer run in a virtual machine. The virtual machine executes application layer and application framework layer programming files (for example, Java files) as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0162] The system library can include multiple functional modules, such as the surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), and 2D graphics engine (such as SGL).

[0163] The surface manager is used to manage the display subsystem and provide the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0164] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0165] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing and layer processing.

[0166] A 2D graphics engine is a drawing engine for 2D drawings.

[0167] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and upper-level software, designed to abstract the hardware. The HAL is an abstract interface driven by the device kernel, providing application programming interfaces (APIs) that provide access to underlying devices to higher-level Java API frameworks. The HAL contains multiple library modules, such as the camera HAL, audio HAL, display, and Bluetooth. Each library module implements an interface for a specific type of hardware component. The camera HAL provides interfaces for the camera FWK and post-camera to access hardware components such as the camera. The camera HAL can include a video capture module and a video cache module. The video capture module is used to configure video streams and manage raw video data obtained from the camera. The video cache module can be used to store processed raw image data obtained from the camera by the video capture module. The audio HAL provides interfaces for the audio FWK to access hardware components such as the microphone.

[0168] Assuming the video cache module is located at the application layer, the application cache has memory limitations. The cached YUV image data (the image to be encoded) is large, and the application layer can only compress the image, resulting in poor video quality and pixel density for dynamic images. In the embodiment of the present application, the camera HAL streams and caches image data, which can reduce the storage and processing burden of the application-layer camera, improve the processing efficiency and operating speed of the camera application, and also use sufficient cache space to process the cached image, ensuring the quality of dynamic images.

[0169] The kernel layer is the foundation of the Android operating system. The ultimate functions of the Android operating system are completed through the kernel layer. The kernel layer includes at least the display driver, camera driver, and audio driver.

[0170] The hardware layer can include displays, microphones, cameras, memory, audio encoders, and video encoders. Microphones capture raw audio data, cameras capture raw image data, and memory can store video, image, and dynamic image files. The audio encoder is the hardware that performs audio encoding, and the video encoder is the hardware that performs video encoding.

[0171] It should be noted that the software structure diagram of the electronic device shown in FIG4A provided in this application is only an example.

[0172] The specific module division in different layers of the Android operating system is not limited. For details, please refer to the introduction of the Android operating system software structure in conventional technology. In addition, the shooting method provided in this application can also be implemented based on other operating systems, and this application will not give examples one by one.

[0173] Based on the software and hardware structure of the electronic device shown in Figures 3 and 4A, Figure 4B is a schematic diagram of the software and hardware structure of another electronic device provided in an embodiment of the present application. Combined with Figure 4B, the basic processing logic in a shooting method provided in an embodiment of the present application is introduced from the perspective of software and hardware collaboration.

[0174] In response to the camera opening operation, the dynamic photography module launches the camera application and notifies the camera to begin capturing images. When dynamic mode is enabled, the dynamic photography module requests the creation of a first video cache queue (video cache module) in the camera HAL. The microphone is activated to begin capturing audio. The first audio cache queue (audio cache module) is created in the camera application and initializes audio and video encoding.

[0175] Step (1): Turn on the camera and configure the video stream in dynamic mode.

[0176] The dynamic photography mode can detect user operations (or obtain the on and off status of the current dynamic mode), thereby determining that the current camera mode is dynamic mode and notifying the video management module. Accordingly, the video management module can create a video-surface and send the distribution information of the video data to the post camera. After receiving the distribution information, the post camera can store the address of the video-surface to generate an image reporting channel. Afterwards, the post camera can send the distribution information to the camera HAL, and the camera HAL will receive a successful distribution initialization message. The camera HAL processes and caches the raw image data collected by the camera. The camera HAL notifies the dynamic photography module of the message that the raw image data has been obtained through the camera FWK, which is used to trigger audio acquisition and video data configuration.

[0177] When the user clicks the shooting control to shoot a dynamic image, the collected audio and image are encoded and mixed to generate a video, and the video and image are packaged to obtain a dynamic image.

[0178] Step (2): Data flow of audio acquisition and encoding.

[0179] The audio HAL can obtain the original audio data collected by the microphone and upload it to the audio management module for caching (the first audio cache queue) through the audio FWK. The audio encoding module encodes the audio to be encoded in the first audio cache queue, obtains the encoded audio, and caches it in the audio queue module (the second audio cache queue).

[0180] Step (3): Data flow of image acquisition and encoding.

[0181] In the camera HAL, the video acquisition module processes the raw image data captured by the camera and stores it in the video cache module (first video cache queue) (for example, YUV images). The video cache module can also transmit the image data to the video encoding module via the post camera. The video encoding module encodes the image to be encoded, obtains the encoded image, and caches it in the video queue module (second video cache queue).

[0182] During the dynamic image acquisition process, the original image captured by the camera (or image data that has been processed with effects) needs to be encoded, and the resulting encoded image is packaged and stored. For example, the video encoder can compress the color space (luminance, chrominance, chroma, YUV) data captured by the Camera module into a video stream in MPEG-4 or Advanced Video Coding (AVC) format. The video stream is then sent to the File Muxer module for encapsulation into a video file that can be played by a player, such as in 3GPP, MP4, H.264, and AAC formats.

[0183] Step (4): Data flow of image acquisition in dynamic images.

[0184] The static shooting mode can detect user operations (or obtain the user's operation of clicking the shooting control to shoot a dynamic picture), obtain the image from the camera HAL through the camera FWK, and store this frame of image in the storage module.

[0185] Step (5): Mix the encoded image and audio into a video, and store the video.

[0186] Mixer 1 and Mixer 2 can obtain the encoded data from the video queue module and the audio queue module respectively, and mix them respectively. During the execution of steps (2) and (3), Mixer 1 and Mixer 2 can respectively mix the encoded image and audio into a video and store it in the storage module. After the video and image are stored in the storage module, the video and image can be encapsulated to obtain a dynamic image, and the storage is updated. Two mixers correspond to two dynamic images. The stored dynamic image can be as shown in Figure 1J.

[0187] In addition, when the dynamic camera module and the static camera module start the dynamic mode, the application layer configures the camera HAL. The dynamic camera module can send the first streaming parameter information to the video acquisition module. The first streaming parameter information may include the first width width1 and the first height height1 of the image. For example, width1*height1 is 1440*1080. The video acquisition module stores the first streaming parameter information, and the configuration is completed. After the configuration is completed, the image size stored by the video cache module in the first video cache queue is width1*height1. The static camera module sends the second streaming parameter information to the camera HAL. The second streaming parameter information may include the second width width2 and the second height height2 of the image. For example, width2*height2 is 4000*8000. The camera HAL stores the second streaming parameter information. After the configuration is completed, the camera HAL obtains a large image of the first image and the second image in the dynamic image that meets the above-mentioned second width and second height.

[0188] For the detailed description of each module in FIG4B , please refer to the relevant description of FIG4A , which will not be repeated here.

[0189] The specific implementation of the above embodiment will be described below with reference to FIG. 5 to FIG. 12B .

[0190] FIG5 is a flow chart of a method for processing video data in a dynamic mode provided by an embodiment of the present application.

[0191] It is understood that the user can click the camera application icon (the user opens the camera operation). Accordingly, the dynamic photography module detects the user operation, determines that the camera is activated, and sends a camera activation message to the camera. The camera activation message is used to request the camera to be activated. After receiving the camera activation message, the camera starts to capture images.

[0192] 1. Obtain original image data and original audio data, and configure video data (as shown in steps S500 to S526 in FIG5 ).

[0193] S500: The dynamic photographing module receives a request to start a camera application.

[0194] It is understandable that the user can trigger the electronic device to start the camera application. Accordingly, the dynamic photo taking module in the electronic device can receive the request to start the camera application.

[0195] For example, as shown in FIG1A , a user may click on the camera control 101, and accordingly, the electronic device may receive a request to launch a camera application. In response to the operation of launching the camera application, the electronic device may launch the camera application and display the user interface 110 shown in FIG1B . It is understood that the user may also launch the camera application through voice, gesture, etc., and this application does not limit the specific method by which the user triggers the electronic device to launch the camera application.

[0196] S501: The dynamic photographing module sends a camera start message to the camera.

[0197] Correspondingly, the camera can receive a camera start message sent by the dynamic photography module. The camera start message is used to instruct the camera to start exposure and obtain raw image data.

[0198] S502: The camera starts and collects raw image data.

[0199] It is understood that the camera can be activated after receiving the camera activation message sent by the dynamic photography module. After the camera is activated, it can continuously collect raw image data. Specifically, the camera can collect raw image data at a first time interval. Each frame of raw image data includes a capture timestamp and the image of that frame. For example, the camera may expose once every 33ms to obtain a frame of raw image.

[0200] S503: The camera sends raw image data to the video acquisition module.

[0201] It is understood that the camera can send the collected raw image data to the camera HAL. The camera can send the raw image data to the video acquisition module. Correspondingly, the video acquisition module can receive the raw image data from the camera.

[0202] S504: The video acquisition module processes the original image data and sends the processed image data for display.

[0203] It is understood that when the camera application is in preview mode, the camera HAL (video acquisition module) can process the raw image data and send the processed image to the display screen for display. It is understood that the preview mode mentioned here can include photo mode, dynamic mode, etc.

[0204] S505: The dynamic photography module obtains a request to start the dynamic mode.

[0205] Optionally, when the camera starts the camera application, the dynamic mode is in the off state, and the user can trigger the start of the dynamic mode. Accordingly, the dynamic photo module can receive a request to start the dynamic mode. For example, as shown in Figure 1B, the user turns on the camera and the dynamic mode is in the off state. The user can click the dynamic mode switch 111 (the operation of starting the dynamic mode) to turn on the dynamic mode. It is understandable that the user can also start the dynamic mode through voice, gestures, etc. This application does not limit the specific way in which the user triggers the electronic device to start the dynamic mode.

[0206] Optionally, when the camera starts the camera application, the dynamic mode is turned on, and the dynamic photography module can obtain a request to start the dynamic mode. For example, as shown in Figure 1C, when the user opens the camera, the dynamic mode is already turned on in the shooting preview interface, and the user does not need to trigger the activation of the dynamic mode.

[0207] S506: The dynamic photographing module sends an image data buffering request to the video acquisition module via the camera FWK.

[0208] After the dynamic camera module detects that dynamic mode has been activated, it can send an image data cache request to the video acquisition module. Correspondingly, the video acquisition module can receive the image data cache request from the dynamic camera module. The image data cache request is used to instruct the caching of processed image data in preparation for video encoding.

[0209] There is no restriction on the order in which S505 to S506 and S501 to S504 are executed.

[0210] S507: The video cache module creates a first video cache queue.

[0211] After the video capture module receives an image data cache request from the dynamic camera module, the video cache module may create a first video cache queue. Specifically, after creating the first video cache queue, the video capture module stores the processed image in the first video cache queue. This may be YUV image data. For example, the first video cache queue may include 45 frames of image data. The video cache module stores the first video cache queue. The video capture module may determine the cache address and data size of the first video cache queue.

[0212] S508: The video cache module stores the processed image data in the first video cache queue.

[0213] After the video cache module completes creating the first video cache queue and processes the image data, the processed image data is stored in the first video cache queue.

[0214] S509: The video acquisition module sends an image acquisition notification to the dynamic photography module through the camera FWK.

[0215] When the video acquisition module stores each frame of image data in the first video buffer queue, the video acquisition module sends an image acquisition notification of this frame to the dynamic photography module via the camera FWK. Correspondingly, the dynamic photography module can receive the image acquisition notification from the video acquisition module via the camera FWK.

[0216] S510: The dynamic photographing module determines a first frame image acquisition identifier based on the first image acquisition notification received.

[0217] After executing S505 and S506, the dynamic photography module can determine the first frame image acquisition identifier based on the image acquisition notification when receiving the image acquisition notification for the first time. That is, the dynamic photography module can determine that the video cache module of the current camera HAL has started caching image data.

[0218] S511: The dynamic photographing module sends an audio collection message to the audio collection module.

[0219] The dynamic camera module can send an audio acquisition message to the audio acquisition module. Correspondingly, the audio acquisition module can receive the audio acquisition message from the dynamic camera module. The audio acquisition message is used to instruct the audio acquisition module to start collecting raw audio data, which can be pulse code modulation (PCM) data.

[0220] S512: The audio acquisition module creates a first audio buffer queue.

[0221] After receiving the audio capture message from the dynamic camera module, the audio capture module can create an audio cache queue. Specifically, the audio capture module creates an audio cache mode to cache multiple frames of raw audio data. The audio capture module can determine the cache address and data size of the first audio and video cache queue. The first audio cache queue is used to cache audio data to be encoded. For example, the first audio cache queue may include 50 frames of audio data.

[0222] S513: The audio collection module sends a microphone start message to the microphone.

[0223] After receiving the audio capture message from the dynamic camera module, the audio capture module can send a microphone activation message to the microphone. Correspondingly, the microphone can receive the microphone activation message from the audio capture module. The microphone activation message is used to activate the microphone to collect raw audio data. Information transmission between the audio capture module and the microphone requires the audio FWK and audio HAL to be carried out. For details, please refer to the relevant description of Figure 4A and will not be repeated here.

[0224] S514: The microphone starts and collects raw audio data.

[0225] After the microphone obtains the original audio data, it can start and collect the original audio data. Specifically, the microphone can collect the original audio data at a second time interval. For example, the microphone collects the original audio data every 22ms.

[0226] S515: The microphone sends original audio data to the audio buffer module.

[0227] After the microphone collects raw audio data, it can send the raw audio data to the audio buffer module through the audio HAL and audio FWK. Correspondingly, the audio buffer module can receive the raw audio data from the microphone.

[0228] S516: The audio cache module stores the original audio data in the first audio cache queue.

[0229] After creating the first audio cache queue, the audio cache module can store the received original audio data in the first audio cache queue.

[0230] S517~S520: Encoding initialization.

[0231] S517: The dynamic photographing module sends a video encoding initialization request to the video encoding module.

[0232] Correspondingly, the video encoding module receives a video encoding initialization request from the dynamic photographing module.

[0233] S518: The video encoding module performs video encoding initialization.

[0234] After executing S517 , the video encoding module may initialize video encoding, that is, configure the addresses of the video encoder and the video queue module, and prepare for video encoding.

[0235] S519: The dynamic photographing module sends an audio encoding initialization request to the audio encoding module.

[0236] Correspondingly, the audio encoding module receives the audio encoding initialization request from the dynamic photographing module.

[0237] S520: The audio encoding module performs audio encoding initialization.

[0238] After executing S519, the audio encoding module may initialize audio encoding, ie, configure the addresses of the audio encoder and audio queue module, and prepare for video encoding.

[0239] The execution order of S517 to S518 and S519 to S520 is not limited.

[0240] S521-S526: coded data stream allocation processing.

[0241] S521: The dynamic photography module sends a stream allocation notification to the video management module.

[0242] After receiving a request to activate dynamic mode, the dynamic camera module can send a stream allocation notification to the video management module. Correspondingly, the video management module can receive the stream allocation notification from the dynamic camera module. The stream allocation notification instructs the video management module to configure the data path for image encoding.

[0243] S522: The video management module creates a Video_surface buffer area.

[0244] The Video_surface buffer is the buffer for the video encoder. The video management module can create the Video_surface buffer and determine its storage address and storage size. During the Video_surface creation process, the video management module can determine encoding and streaming parameters. This encoding and streaming parameters can be surface-related information.

[0245] S523: The video management module sends encoding and streaming parameter information to the post camera.

[0246] After executing S522, the video management module may send the encoding and streaming parameter information to the post camera. Correspondingly, the post camera may receive the encoding and streaming parameter information from the video management module.

[0247] Encoding stream allocation parameter information is used to configure the data path during the encoding process. This information can include image width, height, format parameter, usage information, and a first number (number). Image width and height determine the pixel size of the image; format determines whether the surface format is an encoder format, such as 24; usage information indicates the image's purpose for encoding; and the first number (number) indicates the number of buffers requested by the surface.

[0248] S524: post camera stream initialization.

[0249] The post camera initializes streaming based on the encoding streaming parameters. Specifically, based on the encoding streaming parameters, Video_surface creates a buffer queue containing a first number M of buffers, such as 4 or 8. Each empty buffer can be filled with a frame of image to be encoded with width width and height height. Video_surface specifies the encoder format. The buffer queue is used to subsequently transmit the encoded data from the camera HAL to the camera application.

[0250] S525: The post camera sends a message indicating that the stream allocation initialization is successful to the video acquisition module.

[0251] Correspondingly, the video capture module can receive the stream allocation initialization success message from the post camera. The stream allocation initialization success message is used to notify the camera HAL that the stream allocation is successful.

[0252] After the post camera is created, it notifies the video management module that a new channel for dynamic image encoding has been successfully configured. As a new channel, the post camera, unlike the native camera FWK, requires separate transmission of dynamic image encoding data. Upon receiving notification of successful channel configuration, the video management module can prepare to begin video encoding.

[0253] S526: The video acquisition module sends a notification of successful channel configuration to the video management module through the post camera.

[0254] After the video capture module confirms the successful streaming, it can send a notification of successful channel configuration to the post camera. Correspondingly, the video management module can receive this notification from the video capture module via the post camera. At this point, the video management module can confirm that the encoding and streaming is complete.

[0255] In the above process, the execution order of S506, S511, S517, S519 and S521 is not limited.

[0256] At this point, the electronic device has completed the process of acquiring audio and video data, as well as the video data streaming process. In this process, when dynamic mode is enabled, audio and video caching begins, preparing for dynamic image acquisition. The video encoding process is also prepared in advance.

[0257] 2. Video coding and mixing of a single-shot dynamic image (steps S601 to S619 as shown in FIG6A ).

[0258] FIG6A is a flow chart of a method for video encoding in a dynamic mode provided by an embodiment of the present application.

[0259] When the electronic device enters the camera shooting preview screen and the dynamic mode is turned on, the user can click the shooting control on the interface. Correspondingly, the dynamic shooting module detects the user operation and determines that the electronic device needs to record the first video. The following describes the encoding and mixing process of the first video in the dynamic image when the user only shoots a dynamic image once:

[0260] S601: The dynamic photographing module receives a photographing request.

[0261] When the electronic device starts the dynamic mode, the user triggers the shooting, and the dynamic shooting module receives the shooting request. Specifically, the user can click the start shooting control (Figure 1C), or can use voice control to shoot. This application does not limit the action of triggering shooting. Generating dynamic images requires obtaining videos and images. The dynamic shooting mode is used to control the generation of videos, and the static shooting mode is used to control the acquisition of images. After that, the video and image are packaged accordingly to obtain dynamic images.

[0262] S602-S611: Start video encoding. The electronic device encodes the images to be encoded in the first video cache queue to obtain a second video cache queue.

[0263] S602: The dynamic photographing module sends a request to start image encoding to the video encoding module.

[0264] When the dynamic camera module receives a capture request, it can send a start image encoding request to the video encoding module. Correspondingly, the video encoding module can receive a start image encoding request from the dynamic camera module. The start image encoding request is used to instruct the video encoding module to encode the original image data (stored in the video cache module).

[0265] S603: The video encoding module starts the video encoder.

[0266] When the video encoding module receives the image encoding start request from the dynamic camera module, it can start the video encoder. Specifically, the hardware of the electronic device may include a video encoder. At this time, the video encoding module can control the video encoder to start and prepare for encoding.

[0267] S604: The video encoding module sends an image data encoding start request to the video management module.

[0268] After the video encoding module starts the video encoder, it can send an image data start encoding request to the video management module. Correspondingly, the video management module can receive an image data start encoding request from the video encoding module. This image data start encoding request is used to request the video management module to start image encoding. Upon receiving this request, the video encoder begins encoding.

[0269] S605-S611: After the video management module starts encoding, it enters a cyclic video encoding process, encoding one frame of image each time, until the video management module obtains the image data end encoding request and stops encoding.

[0270] S605: The video management module sends an image data encoding request to the video acquisition module through the post camera.

[0271] After the video management module receives the image data encoding request from the video encoding module, it can send an image data encoding request to the video acquisition module via the post camera. Correspondingly, the video acquisition module can receive the image data encoding request from the video management module via the post camera. The image data encoding request is used to request the next frame of image to be encoded stored in the video cache module for encoding.

[0272] Among them, the post camera is used to transmit the data to be encoded in dynamic mode.

[0273] S606: The video buffer module obtains the nth frame of the image to be encoded.

[0274] When the video acquisition module receives an image data encoding request, the video buffer module needs to obtain the nth frame of image to be encoded and pass the one frame of image data stored in the video buffer module to the video encoding module for encoding.

[0275] It should be noted that the video cache module stores an image queue with a first number of frames (the first number can be fixed and unchanging). For example, in this queue (a first-in, first-out image queue), as the camera shoots, image frames are continuously added to the end of the queue. When encoding begins, the first frame to be encoded is selected as the frame at the head of the queue. During the encoding process, the next frame to be encoded is selected as the frame after the last frame to be encoded in the queue. That is, when the video acquisition module (entering the cyclic video encoding) receives an image data encoding request for the first time, the video acquisition module determines the frame at the head of the first video cache queue in the video cache module as the next frame of image data; when the video acquisition module does not receive an image data encoding request for the first time, the video acquisition module determines the frame after the next frame of image data of the last determined image in the first video cache queue as the next frame of image data this time, that is, the images to be encoded are sequentially retrieved. The next frame of image data obtained at this time is all processed original image data, that is, the data cached in S508. In addition, the process of obtaining the n-th frame of the image to be encoded can refer to the relevant exemplary descriptions of Figures 11A to 11E, and will not be repeated here.

[0276] S607: The video buffer module sends the nth frame of the image to be encoded to the video encoding module through the post camera.

[0277] After the video cache module obtains the nth frame to be encoded, it can send the nth frame to be encoded to the video encoding module through the post camera (also known as the post-processing camera). Correspondingly, the video encoding module can receive the nth frame to be encoded from the video cache module through the post camera.

[0278] At this time, the nth frame of image to be encoded includes a frame of image to be encoded and a corresponding timestamp. The timestamp can indicate the time point when the image information of this frame is captured.

[0279] S608: The video encoding module encodes the n-th frame of the image to be encoded to obtain the n-th frame of the encoded image.

[0280] When the video encoding module receives the n-th frame of the image to be encoded from the video buffer module, it can encode the n-th frame of the image to be encoded to obtain the n-th frame of the encoded image.

[0281] S609: The video encoding module sends a request to the video management module to continue encoding the image data.

[0282] After the video encoding module completes encoding of the nth frame of the image to be encoded, it may send a request to the video management module to continue encoding the image data.

[0283] S610: The video encoding module sends the nth frame of encoded image to the video queue module.

[0284] After executing S608, the video encoding module can send the nth frame of encoded image to the video queue module. The nth frame of encoded image includes the encoded image data and a timestamp. The timestamp uses the timestamp of the corresponding frame before encoding, that is, the timestamp of the shooting of this frame of image.

[0285] S611: The video queue module caches the encoded image data.

[0286] After receiving the n-th coded image frame, the video queue module may cache the n-th coded image frame in the second video cache queue of the video queue module.

[0287] In the above process, the video cache module and the video queue module can both be a data cache queue. The video cache module caches the image data before encoding (first video cache queue), and the video queue module caches the image data after encoding (second video cache queue).

[0288] The video management module executes S604 to start encoding and then executes S605. The video management module then executes S609 and can continue to execute S605 in a loop until the video management module receives a request to stop encoding the image data. The video management module can stop encoding and end the loop.

[0289] S612 to S619: Starting audio encoding. The electronic device encodes the audio to be encoded in the first audio buffer queue to obtain a second audio buffer queue.

[0290] S612: The dynamic photographing module sends a request to start audio encoding to the audio encoding module.

[0291] When the dynamic photographing module receives the shooting request, it can send a request to start audio encoding to the audio encoding module. Correspondingly, the audio encoding module can receive the request to start audio encoding from the dynamic photographing module.

[0292] S613: The audio encoding module starts the audio encoder.

[0293] After receiving the audio encoding start request, the audio encoding module may start the audio encoder. Specifically, the hardware of the electronic device may include an audio encoder. At this time, the audio encoding module may control the audio encoder to start and prepare for encoding.

[0294] S614: The audio encoding module sends an audio data encoding start request to the audio buffer module.

[0295] After the audio encoding module starts the audio encoder, it can send a request to the audio management module to start encoding audio and video data. Correspondingly, the audio management module can receive the request from the audio encoding module. The request is used to request the audio management module to start audio and video encoding.

[0296] S615-S619: After audio encoding is started, a loop audio encoding process is entered, encoding one frame of audio each time, until the audio buffer module obtains an audio data end encoding request and stops encoding.

[0297] S615: The audio buffer module sends the mth frame of audio to be encoded to the audio encoding module.

[0298] After receiving the audio and video data start encoding request from the audio encoding module, the audio buffer module can send the mth frame of audio to be encoded to the audio encoding module. Correspondingly, the audio encoding module can receive the mth frame of audio to be encoded from the audio buffer module.

[0299] After executing S614, the audio buffer module executes S615, and then sends the mth frame of audio data to the audio encoding module, which is the first frame of audio data in the queue of the audio buffer module. After executing S617, the audio buffer module executes S615, and then sends the mth frame of audio data to the audio encoding module, which is the next frame of audio data after the last audio data sent in the queue. That is, the audio buffer module extracts the audio data one frame at a time in the queue order.

[0300] The mth frame of audio to be encoded includes an audio frame and an audio timestamp. The audio timestamp is used to indicate the time point when this frame of audio was captured.

[0301] S616: The audio encoding module encodes the mth frame of audio to be encoded to obtain the mth audio to be encoded.

[0302] After receiving the mth frame of audio to be encoded from the audio buffer module, the audio encoding module may encode the mth frame of audio to be encoded to obtain the mth frame of encoded audio.

[0303] S617: The audio encoding module sends a request to the audio buffer module to continue encoding the audio data.

[0304] After obtaining the next frame of encoded audio data, the audio encoding module sends a request to the audio buffer module to continue encoding the audio data. Correspondingly, the audio buffer module can receive the request from the audio encoding module to continue encoding the audio data.

[0305] S618: The audio encoding module sends the mth frame of encoded audio to the audio queue module.

[0306] After the audio encoding module obtains the mth frame of encoded audio, it can send the mth frame of encoded audio to the audio queue module. Correspondingly, the audio queue module can receive the mth frame of encoded audio from the audio encoding module.

[0307] The mth frame of coded audio may include one frame of coded audio and an audio timestamp. The audio timestamp may be the same as the audio timestamp before encoding.

[0308] S619: The audio queue module caches the encoded audio data.

[0309] The audio queue module can cache the mth frame of encoded audio in the second audio cache queue of the audio queue module.

[0310] In the above process, the audio cache module and the audio queue module can both be a data cache queue. The audio cache module caches the audio data before encoding (first audio cache queue), and the audio queue module caches the audio data after encoding (second audio cache queue).

[0311] The audio cache module executes S614, starts encoding, and executes S615. The audio cache module then executes S617 and can continue to execute S615 in a loop until the audio cache module receives a request to stop encoding the audio data. The audio cache module can stop encoding and end the loop.

[0312] FIG6B is a flow chart of a method for video mixing in dynamic mode provided by an embodiment of the present application.

[0313] S620-S633: Mix the encoded audio and image to obtain video data.

[0314] Audio and image are mixed to obtain video data.

[0315] S620: The dynamic photographing module sends a mixer start request to mixer 1.

[0316] When the dynamic photographing module receives the shooting request (after executing S601), it can send a mixer start request to the mixer 1. Correspondingly, the mixer 1 can receive the mixer start request from the dynamic photographing module.

[0317] The mixer start request may include a timestamp Y when the electronic device receives the capture request (in S601 ).

[0318] S621: Mixer 1 starts.

[0319] When the mixer 1 receives the mixer start request, the mixer 1 may start and start mixing.

[0320] S622: The memory creates a first file.

[0321] After the mixer 1 is started, the memory can be controlled to create a first file. The first file is a dynamic image file. The dynamic image file can include video and images.

[0322] S623: Mixer 1 sends a video data start mixing request to the video queue module.

[0323] After creating the first file, the mixer 1 may send a video data start mixing request to the video queue module. Correspondingly, the video queue module may receive the video data start mixing request from the mixer 1.

[0324] S624-S626 image mixing loop: After the video queue module executes S626, it continues to execute S624 to mix the next frame. The loop ends when it is determined in S625 that the next frame image is not to be encapsulated.

[0325] S624: The video queue module sends the i-th frame of encoded image data to mixer 1.

[0326] The video queue module can send the i-th frame of coded image data to the mixer 1. Correspondingly, the mixer 1 can receive the i-th frame of coded image data from the video queue module.

[0327] The i-th frame of coded image data is the earliest coded frame of image data in the cache queue of the current video queue module (the head frame of coded image data in the second video data queue). When the i-th frame of coded image data is consumed (mixed) by the mixer 1, the next frame after the i-th frame becomes the head frame of image data.

[0328] The i-th frame of coded image data may include a frame of coded image and a time stamp.

[0329] S625: The mixer 1 determines whether to encapsulate the next frame of image.

[0330] In S620, mixer 1 obtains timestamp Y. The electronic device can determine the starting frame for mixing (the first coded image frame) from the second video cache queue based on timestamp Y and the specific video duration (or specific number of video frames) and begin encapsulation. Each time a frame is encapsulated in the mixer, it marks the frame as consumed by mixer 1. Before encapsulating a frame, the electronic device determines when to encapsulate the frame, that is, when to end encapsulation, and executes S626. If encapsulation is not required, S626 is not executed. In the second video cache queue, each frame of coded image data includes the coded image frame and the shooting timestamp.

[0331] In a possible implementation, the mixer 1 may determine the starting frame of mixing (determine the first coded image frame of this mixing) from the second video cache queue based on the timestamp Y and the specific video duration, and determine whether to end the encapsulation.

[0332] First, determine the starting frame.

[0333] The mixer 1 may determine the starting frame based on the specific video duration of the timestamp Y and the timestamps of the encoded images in the second video buffer queue. The specific video duration may be preset and is not limited to a specific duration.

[0334] Specifically, the mixer 1 can determine the first time and the second time based on Y and the specific video length, determine which frame to start mixing based on the second time, and determine whether a frame of encoded image is mixed based on the first time, that is, whether to end mixing.

[0335] Exemplarily, the specific video duration is 3s, and the electronic device determines that the images captured within 1.5s before and after Y are mixed. The electronic device can select encoded images with timestamps that meet the time range of Y-1.5s (second time) to Y+1.5s (first time) from the second video cache queue and input them into the mixer 1 for mixing. At this time, the electronic device can determine the image with the closest timestamp to Y1-1.5s in the second video cache queue as the starting frame. Of course, the image with the closest timestamp preset can also be determined from the second video cache queue based on Y-1s as the starting frame. The specific method of selecting the starting frame is not limited.

[0336] Secondly, determine the end of encapsulation. Mixer 1 starts encapsulation and marks each frame of the encapsulated image to indicate that the electronic device has acquired it. When the image frame is acquired next time (acquire a frame of image immediately after the marked image in the second video cache queue), it can be acquired in sequence according to the position of the last mark. When a new (i+1th frame) image frame is acquired, determine whether the current frame to the starting frame meets the specific video duration. If it meets, encapsulation is performed; if not, it is not encapsulated and the encapsulation is ended.

[0337] Optionally, the mixer 1 can determine the timestamp of the first frame of the encapsulated encoded image (the timestamp of the starting frame) and the timestamp of the current frame encapsulated image (the timestamp of the i+1 frame), and whether the first time difference between the two meets the specific video duration. When the first time difference is greater than or equal to (greater than) the specific video duration, it is judged that the specific video duration is met, the next frame of the image is not encapsulated (the mixing of this dynamic image is ended), and S626 is not executed. When the first time difference is less than (less than or equal to) the specific video duration, it is judged that the specific video duration is not met, and the next frame of the image continues to be encapsulated (the above mixing is not completed, and the mixing of this dynamic image is continued), and S626 is continued.

[0338] For example, if the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is greater than 3 seconds, the specific video duration is not met; otherwise, the specific video duration is met. If the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is 2.87 seconds, which is less than 3 seconds, the i+1th frame is encapsulated. If the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is 3.001 seconds, which is greater than 3 seconds, the i+1th frame is not encapsulated and the image mixing ends.

[0339] Optionally, the mixer 1 can determine whether the timestamp of the current frame encapsulated image is less than (less than or equal to) Y and the specific video length to determine the end time. If it is less than (less than or equal to) the end time, the i+1 frame is encapsulated; if it is greater than or equal to (greater than) the end time, the i+1 frame is not encapsulated and the image mixing ends.

[0340] For example, it is determined whether the timestamp of the frame image is less than (less than or equal to) Y+1.5s (the second time) (the start time is Y-1.5s, and the starting frame is selected accordingly). If it is less than or equal to (less than) Y+1.5s, the i+1 frame is packaged; if it is greater than (greater than or equal to) Y+1.5s, the i+1 frame is determined not to be packaged, and the electronic device may determine to end image mixing.

[0341] In another possible implementation, the mixer 1 may determine the starting frame of mixing (determine the first coded image frame of this mixing) from the second video cache queue based on the timestamp Y and the specific video frame number, and determine whether to end the encapsulation.

[0342] The mixer 1 can obtain the encoded image based on the specific video frame number of timestamp Y and the timestamp of each encoded image in the second video cache queue. The electronic device can select a specific number of consecutive video frames in the second video cache queue before and after the timestamp Y and input them into the mixer 1. The specific number of video frames can be preset and not limited. After the mixer 1 starts encapsulation, it marks each encapsulated frame image to indicate that the electronic device has acquired it. When the image frame is acquired next time (acquiring a frame image immediately after the marked image in the second video cache queue), it can be acquired in sequence according to the position of the last mark. When a new (i+1th frame) image frame is acquired, it is determined whether the current frame to the starting frame reaches a specific number of video frames. If it is not reached or is just reached, encapsulation is performed; if it exceeds the specific number of video frames, it is not encapsulated and the encapsulation is ended.

[0343] For example, if the specific video frame number is 40, the electronic device may determine the 20 consecutive frames with timestamps before Y and the 20 consecutive frames after Y1 as the coded images, and input them to the time with the smallest time difference from Y. At this point, the electronic device may determine the 20 frames in the second video cache queue that are before timestamp Y, and determine the frame with the earliest timestamp among these 20 frames as the starting frame. During the sequential encapsulation process, the number of encapsulated frames is determined. If the number of frames is less than or equal to 40, encapsulation is determined to be performed. If the number is greater than 40, encapsulation is not performed, and image mixing ends.

[0344] In S625, the start frame refers to the first frame of the coded image that starts mixing.

[0345] S626: Mixer 1 sends a request to the video queue module to continue mixing images.

[0346] When the mixer 1 determines to encapsulate the next frame of image, it can send a request for continuing image mixing to the video queue module. Correspondingly, the video queue module can receive the request for continuing image mixing from the mixer 1.

[0347] S627: Mixer 1 sends an audio data start mixing request to the audio queue module.

[0348] After creating the first file, the mixer 1 may send an audio data start mixing request to the audio queue module. Correspondingly, the audio queue module may receive the audio data start mixing request from the mixer 1.

[0349] S628-S630 Audio Mixing Loop: After the audio queue module executes S630, it continues to execute S628 to mix the next frame. The loop ends when it is determined in S629 that the next frame of audio is not to be encapsulated.

[0350] S628: The audio queue module sends the kth encoded audio and video data to mixer 1.

[0351] Correspondingly, the mixer 1 can receive the kth encoded audio and video data from the audio queue module.

[0352] S629: Mixer 1 determines whether to encapsulate the next frame of audio.

[0353] In S620, mixer 1 obtains timestamp Y. The electronic device can determine the starting frame of the mix (the first frame of encoded audio) from the second audio cache queue based on timestamp Y and the specific audio duration (or specific number of audio frames) and start encapsulation. Each time a frame of audio is encapsulated in the mixer, it marks that the frame of audio has been consumed by mixer 1. Before encapsulating a frame of audio, the electronic device determines when to encapsulate the frame of audio, that is, when to end encapsulation, and executes S630 for encapsulation; if not encapsulated, S630 is not executed. In the second audio cache queue, each frame of encoded audio data includes the frame of encoded audio and the acquisition timestamp.

[0354] In a possible implementation, the mixer 1 may determine the starting frame of mixing (determine the first frame of encoded audio mixed this time) from the second audio buffer queue based on the timestamp Y and the specific audio duration, and determine whether to end the encapsulation.

[0355] First, determine the starting frame.

[0356] The mixer 1 may determine the start frame based on the specific audio duration of the timestamp Y and the timestamps of the encoded audios in the second audio buffer queue. The specific audio duration may be preset and is not limited to a specific duration.

[0357] Exemplarily, the specific audio duration is 3s, and the electronic device determines that the audio captured within 1.5s before and after Y is mixed. The electronic device can select the encoded audio with a timestamp within the time range of Y-1.5s to Y+1.5s from the second audio cache queue and input it into the mixer 1 for mixing. At this time, the electronic device can determine the audio with the closest timestamp to Y1-1.5s in the second audio cache queue as the starting frame. Of course, the audio with the closest timestamp preset can also be determined from the second audio cache queue based on Y-1s as the starting frame. The selection of the starting frame can refer to the relevant description of Figures 11A to 11E, and the specific method is not limited.

[0358] Secondly, determine the end of encapsulation. Mixer 1 starts encapsulation and marks each frame of encapsulated audio to indicate that the electronic device has acquired it. When the audio frame is acquired next time (acquire a frame of audio immediately after the marked audio in the second audio cache queue), it can be acquired in sequence according to the position of the last mark. When a new (i+1th frame) audio frame is acquired, determine whether the current frame to the starting frame meets the specific audio duration. If it meets, encapsulation is performed; if not, it is not encapsulated and the encapsulation is ended.

[0359] Optionally, the mixer 1 can determine the timestamp of the first frame of the encapsulated encoded audio (the timestamp of the starting frame) and the timestamp of the current frame encapsulated audio (the timestamp of the i+1 frame), and whether the second time difference between the two meets the specific audio duration. When the first time difference is greater than or equal to (greater than) the specific audio duration, it is judged that the specific audio duration is met, the next frame of audio is not encapsulated (the mixing of this dynamic image is ended), and S630 is not executed. When the second time difference is less than (less than or equal to) the specific audio duration, it is judged that the specific audio duration is not met, and the next frame of audio continues to be encapsulated (the above mixing is not completed, and the mixing of this dynamic image is continued), and S630 is continued.

[0360] For example, if the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is greater than 3 seconds, the specific audio duration is not met; otherwise, the specific audio duration is met. If the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is 2.87 seconds, which is less than 3 seconds, the i+1th frame is encapsulated. If the time difference between the timestamp of the i+1th frame and the timestamp of the starting frame is 3.001 seconds, which is greater than 3 seconds, the i+1th frame is not encapsulated and the audio mixing ends.

[0361] Optionally, the mixer 1 can determine whether the timestamp of the current frame encapsulated audio is less than (less than or equal to) Y and the specific audio duration to determine the end time. If it is less than (less than or equal to) the end time, the i+1 frame is encapsulated; if it is greater than or equal to (greater than) the end time, the i+1 frame is not encapsulated and the audio mixing ends.

[0362] For example, it is determined whether the timestamp of the audio frame is less than (less than or equal to) Y+1.5s (the start time is Y-1.5s, and the starting frame is selected accordingly). If it is less than or equal to (less than) Y+1.5s, the i+1 frame is packaged; if it is greater than (greater than or equal to) Y+1.5s, the i+1 frame is determined not to be packaged, and the electronic device may determine to end audio mixing.

[0363] In another possible implementation, the mixer 1 may determine the starting frame of mixing (determine the first frame of encoded audio mixed this time) from the second audio buffer queue based on the timestamp Y and the specific audio frame number, and determine whether to end the encapsulation.

[0364] The mixer 1 can obtain the encoded audio based on the specific audio frame number of timestamp Y and the timestamp of each encoded audio in the second audio cache queue. The electronic device can select a specific number of consecutive audio frames of encoded audio before and after the timestamp Y in the second audio cache queue, and input it into the mixer 1. The specific number of audio frames can be preset and not limited. After the mixer 1 starts encapsulation, it marks each encapsulated frame of audio to indicate that the electronic device has acquired it. When the audio frame is acquired next time (acquiring a frame of audio immediately after the marked audio in the second audio cache queue), it can be acquired in sequence according to the position of the last mark. When a new (i+1 frame) audio frame is acquired, it is determined whether the current frame to the starting frame reaches a specific number of audio frames. If it is not reached or is just reached, encapsulation is performed; if it exceeds the specific number of audio frames, it is not encapsulated and the encapsulation is ended.

[0365] For example, the specific number of audio frames is 60 frames. The electronic device can determine the 30 consecutive frames with timestamps before Y; and the 30 consecutive frames after Y1 as encoded audio, and input them to the time with the smallest time difference with Y. At this time, the electronic device can determine 30 frames of audio before the Y timestamp in the second audio cache queue, and determine the frame with the earliest timestamp of these 30 audio frames as the starting frame. In the process of sequential encapsulation, the number of frames that have been encapsulated is determined. If the number of frames is less than or equal to 30, it is determined to be encapsulated. If it is greater than 30, it is not encapsulated and the audio mixing is terminated.

[0366] The starting frame in S629 refers to the first frame of coded audio that starts mixing.

[0367] S630: Mixer 1 sends an audio mixing continuation request to the audio queue module.

[0368] When it is determined to encapsulate the next frame of audio, an audio continuation mixing request may be sent to the audio queue module. Correspondingly, the audio queue module may receive the audio continuation mixing request from mixer 1.

[0369] The execution order of S624 to S626 and S628 to S630 is not limited and they are executed in parallel.

[0370] S631: The mixer 1 encapsulates the i-th frame of encoded image data and the k-th frame of encoded audio data into the first video.

[0371] When the mixer 1 obtains the i-th frame of coded image data and the k-th frame of coded audio data, the i-th frame of coded image data and the k-th frame of coded audio data may be encapsulated into the first video. Here, i is an integer from 1 to 1, and k is an integer from 1 to K. Both I and K are integers greater than 1.

[0372] S632: Mixer 1 sends a first video mixing completion notification to the dynamic photo module.

[0373] Correspondingly, the dynamic photo module receives a video mixing completion notification from the mixer 1. The first video mixing completion notification may include an identifier of the first video, indicating that the first video has been packaged and can be prepared for subsequent dynamic image splicing.

[0374] S633: The memory stores the first video in a first file.

[0375] After S631 is executed and the mixing is completed, the memory may store the first video in the first file.

[0376] There is no restriction on the execution order between S632 and S633.

[0377] FIG6C is a flow chart of a method for taking photos and packaging dynamic images in a dynamic mode provided by an embodiment of the present application.

[0378] When the electronic device enters the camera shooting preview screen and the dynamic mode is turned on, the user can click the shooting control on the interface. When clicking to shoot (as shown in Figure 1C), both the static shooting module and the dynamic shooting module will receive shooting requests. The action of the dynamic shooting module is performed as S601, and the action of the static shooting module is performed as S634. Thereafter, before the electronic device obtains the first video, it receives other shooting requests, that is, the electronic device processes the user's one shot and stops encoding and mixing after completion. The static shooting module detects the user operation and determines that the electronic device shoots a static image (the first image). When the first video and the first image are obtained, the dynamic image can be encapsulated, as described below:

[0379] S634: The static camera module receives a shooting request.

[0380] In the above S601, when the user clicks to shoot, both the dynamic shooting mode and the static shooting mode will receive a shooting request. The details of S634 can be referred to the description of S601 and will not be repeated here.

[0381] S635: The static camera module sends an image acquisition request to the camera HAL through the camera FWK.

[0382] After executing S634, the static camera module may send an image acquisition request to the camera HAL via the camera FWK. Correspondingly, the camera HAL may receive the image acquisition request from the static camera module via the camera FWK.

[0383] S636: The video cache module obtains thumbnail data of the first image.

[0384] Upon receiving the image acquisition request, the camera HAL may immediately acquire the first image from the camera and obtain thumbnail data for the first image. The thumbnail data may include a thumbnail image of the first image and a timestamp. After the user clicks "shoot," the electronic device acquires a thumbnail image of the first image, which may be displayed in the gallery control shown in FIG1D .

[0385] S637: The camera HAL sends thumbnail data of the first image to the still camera module through the camera FWK.

[0386] After the camera HAL obtains the thumbnail data of the first image, it can send the thumbnail data of the first image to the still camera module via the camera FWK. Correspondingly, the still camera module can receive the thumbnail data of the first image sent by the camera HAL via the camera FWK. The thumbnail data of the first image is of poor quality, with low pixel count and a small data size.

[0387] S638: The static camera module performs dynamic image marking on the first image.

[0388] Dynamic images include both image and video data. Images must be captured and then combined with the video to create a dynamic image. Therefore, dynamic images must be marked in advance. Furthermore, this marking also indicates that the current first image is a thumbnail.

[0389] S639: The memory stores the thumbnail data of the first image.

[0390] After the static photographing module obtains the thumbnail data of the first image, the memory stores it.

[0391] At this time, the gallery can obtain the storage record of the first image and the dynamic image tag, and has already stored a thumbnail of the first image.

[0392] S640: The camera HAL processes the original image of the first image to obtain large image data.

[0393] After the camera HAL processes the image, the large image data is obtained. The pixels of the first image are increased, and the effect is better. The large image data volume of the first image is larger than the thumbnail data.

[0394] S641: The camera HAL sends the large image data of the first image to the static camera module through the camera FWK.

[0395] Correspondingly, the static photographing module may receive the large image data of the first image from the camera HAL through the camera FWK.

[0396] S642: The static camera module updates the dynamic image record.

[0397] After executing S641, the static camera module may update the dynamic image record. The updated dynamic image record may represent the large image data of the currently acquired first image.

[0398] At this time, the image library can obtain the updated storage record and dynamic image tag of the first image, and has already stored the large image data of the first image.

[0399] S643: The memory stores the large image data of the first image.

[0400] After the static camera module obtains the large image data of the first image, the memory stores it.

[0401] S644: The dynamic photographing module sends a first video packaging notification to the storage.

[0402] After the dynamic photographing module obtains the first video mixing completion notification (execution S632), it can send the first video packaging notification to the memory. Correspondingly, the memory can receive the first video packaging notification from the dynamic photographing module.

[0403] S645: The static camera module sends a large image packaging notification of the first image to the storage.

[0404] After executing S642 , the static photographing module may send a large-image packaging notification of the first image to the memory. Correspondingly, the memory may receive the large-image packaging notification of the first image from the static photographing module.

[0405] S646: The memory splices the large image data of the first image and the first video to obtain a first file and stores it.

[0406] In S633, the electronic device has stored the first video in the first file. Under the condition of executing S644 and S645, the memory splices the large image data of the first image and the first video, and encapsulates the first file (dynamic image) based on the description information of the dynamic image. Afterwards, the first file is stored. After the first file is stored in the gallery, click to open the gallery (as shown in Figure 1H), and the gallery preview screen of the electronic device may include the dynamic image of the first file, such as the dynamic image 181 in Figure 1I. If the user clicks to enter this dynamic image, (a) in Figure 1E can be displayed. The user can long press to play this dynamic image. Clicking the edit control 142a, the electronic device can display multiple thumbnails in the video, as well as the large image (cover frame) of the first image. The user long presses the first image, and the electronic device plays the dynamic image. Triggering the playback of the dynamic image can be other operations; you can also long press to not play the dynamic image, which is not limited in this application.

[0407] Optionally, after the stitching is completed, the gallery may be notified to update the record of the first file.

[0408] Figure 7A is a schematic diagram of the format structure of a first file disclosed in an embodiment of the present application. As shown in Figure 7A. The first file is a dynamic image file and may include three fields. The first field is the large image data of the first image (JPEG image, in S643, the large image data of the first image is stored), and the second field is the mixed video data (the first video, the first video stored in S633), for example, an mp4 file. The third field is the description field of the dynamic image (a field describing information), and the third field may include three subfields. Among them, the first subfield is the version number version, for example, currently v2. The second subfield includes the start time start and the duration duration, that is, the second subfield is used to record the start time start and the duration duration of the playback during the gallery preview; the third subfield is TAG, which is used to record the file length of the first video.

[0409] The video data of the first file includes audio data and image data of the encapsulation duration. FIG7B is a schematic diagram of the audio data and video data encapsulation duration of a video data disclosed in an embodiment of the present application. As shown in FIG7B , it is assumed that the first encapsulation duration and the second encapsulation duration are both the duration from T1 to T2. The video data includes a total of K audio frames and I image frames, where K and I are integers greater than 1. The frame numbers of the audio frames may be audio frame 1, audio frame 2, audio frame 3, ..., audio frame K, in sequence. The frame numbers of the image frames may be image frame 1, image frame 2, ..., image frame 1, in sequence. The acquisition duration interval between each two adjacent image frames is the first time interval; the acquisition duration interval between each two adjacent audio frames is the second time interval. For example, the first time interval is 33ms, the second time interval is 22ms, the encapsulation time is 180ms, K is 8, and I is 5.

[0410] 3. After the single shot is completed, the encoding and mixing are stopped (as shown in FIG6D ).

[0411] Figure 6D is a flow chart of a method for stopping video encoding provided by an embodiment of the present application. The method for stopping video encoding is the execution steps after the execution of Figures 6A to 6C, that is, the process of stopping encoding when no continue shooting control is received.

[0412] S647: When the mixer 1 determines that mixing should be stopped, it stops mixing.

[0413] Specifically, during the execution of S625 and S629 , the mixer 1 determines whether to stop mixing. If it is determined that the next frame of image and the next frame of audio will no longer be encapsulated, the mixer 1 determines to stop mixing and stops the mixer from processing.

[0414] S648: The mixer 1 sends a first video mixing completion notification to the dynamic photography module.

[0415] Correspondingly, the dynamic photographing module may receive a first video mixing completion notification from the mixer 1 .

[0416] If it is determined that the first video mixing is completed (S631 is completed), S648 can be executed. S648 and S632 are the same execution steps and will not be repeated.

[0417] S649: The dynamic photographing module determines that the video mixing of the dynamic image is not completed and determines to stop the encoder.

[0418] When the dynamic photography module receives the notification that the first video mixing is completed, the process of determining whether to stop encoding is as follows: when there is currently no video of a dynamic image that has not yet been mixed, the electronic device stops encoding the first video cache queue and the first audio cache queue; when there is currently a video of a dynamic image that has not yet been mixed, the electronic device continues to encode the first video cache queue and the first audio cache queue.

[0419] Specifically, the electronic device receives a shooting request and adds a shooting request task to the task list. When a video mixing completion notification is received, the task corresponding to the shooting request can be determined and cleared. Each time the electronic device receives a mixing completion notification, it first clears the task corresponding to the shooting request and determines whether there is a shooting request task in the current task list. If there is a task being processed, it can be determined that there is an unfinished mixing of the video of the dynamic image. If there is no task (the task list is empty), it can be determined that there is an unfinished mixing of the video of the dynamic image.

[0420] In S648, the electronic device starts the task of the shooting request when and only when it receives one shooting request. After receiving the notification that the first video mixing is completed, the task of the shooting request can be cleared. The electronic device can determine that there is no unfinished mixing of the video of the dynamic image and can determine that the encoder is stopped.

[0421] S650: The dynamic photographing module sends a request to the video encoding module to stop encoding the image data.

[0422] After S648 is completed, the dynamic photographing module executes S650, that is, it can send a request to stop encoding the image data to the video encoding module. Correspondingly, the video encoding module can receive the request from the dynamic photographing module to stop encoding the image data.

[0423] S651: The video encoding module stops the video encoder.

[0424] After the video encoding module receives the request to stop encoding the image data, the video encoding module may stop the video encoder.

[0425] S652: The dynamic photographing module sends a request to the audio encoding module to stop encoding the audio data.

[0426] After S648 is completed, the dynamic photographing module executes S652, that is, it can send a request to stop encoding the audio data to the audio encoding module. Correspondingly, the audio encoding module can receive the request to stop encoding the audio data from the dynamic photographing module.

[0427] S653: The audio encoding module stops the audio encoder.

[0428] After receiving the audio data encoding stop request, the audio encoding module may stop the audio encoder.

[0429] 4. Video encoding and mixing of multiple continuous dynamic image shots (as shown in FIG8A and FIG8B ).

[0430] 8A to 8C are flowchart diagrams of another set of methods for processing video data in a dynamic mode provided by an embodiment of the present application.

[0431] When the electronic device enters the camera shooting preview screen and the dynamic mode is turned on, the user can click the shooting control on the interface multiple times. Accordingly, the dynamic shooting module detects the user's operation of shooting dynamic images multiple times and determines that the electronic device needs to obtain multiple dynamic image videos and photos. The following describes the encoding and mixing process of multiple dynamic images when the user shoots dynamic images multiple times in a row:

[0432] During the two-shot process, the user triggers shooting twice, and both the dynamic shooting module and the static shooting module receive two shooting requests. The dynamic shooting module starts the video encoder to start encoding when it receives the first shooting request. Before stopping the encoder, it receives the second shooting request. The video encoder and audio encoder continue encoding until the video mixing corresponding to the second shooting request is completed, and then the encoding stops. In the above-mentioned continuous dynamic video shooting process, the two dynamic videos and audio encoding only need to perform the encoding process once (one encoding) and mix them separately to obtain the video (multiple mixing). There is no need for the encoder to encode and start separately, which improves execution efficiency and saves encoding energy. It can also solve the problem of continuous shooting of dynamic images and the inability to continuously generate videos.

[0433] Taking the case where the user clicks twice in succession to shoot a dynamic image as an example, the method flow for three or more consecutive shots is similar.

[0434] S801 to S819: When a shooting request is received for the first time, the audio and video encoding process is started.

[0435] S801: The dynamic photographing module receives a first photographing request.

[0436] At this point, with dynamic mode enabled, the user triggers the first capture, and the dynamic capture module receives the first capture request. This triggers encoding and first blending to produce the first video. For details on the user's first capture, refer to S601 and the related description in Figure 1C , and will not be repeated here.

[0437] S802-S811: Start video encoding.

[0438] S802: The dynamic photographing module sends a request to start image encoding to the video encoding module.

[0439] S803: The video encoding module starts the video encoder.

[0440] S804: The video encoding module sends an image data encoding start request to the video management module.

[0441] S805: The video management module sends an image data encoding request to the video acquisition module through the post camera.

[0442] S806: The video buffer module obtains the nth frame of the image to be encoded.

[0443] S807: The video buffer module sends the nth frame of the image to be encoded to the video encoding module through the post camera.

[0444] S808: The video encoding module encodes the n-th frame of the image to be encoded to obtain the n-th frame of the encoded image.

[0445] S809: The video encoding module sends a request to the video management module to continue encoding the image data.

[0446] S810: The video encoding module sends the nth frame of encoded image to the video queue module.

[0447] S811: The video queue module caches the nth frame of the encoded image.

[0448] S812-S819: Start audio encoding.

[0449] S812: The dynamic photographing module sends a request to start audio encoding to the audio encoding module.

[0450] S813: The audio encoding module starts the audio encoder.

[0451] S814: The audio encoding module sends an audio data encoding start request to the audio buffer module.

[0452] S815: The audio buffer module sends the mth frame of audio to be encoded to the audio encoding module.

[0453] S816: The audio encoding module encodes the m-th frame of audio to be encoded to obtain the m-th frame of encoded audio.

[0454] S817: The audio encoding module sends a request to the audio buffer module to continue encoding the audio data.

[0455] S818: The audio encoding module sends the mth frame of encoded audio to the audio queue module.

[0456] S819: The audio queue module caches the mth frame of encoded audio.

[0457] For the detailed description of S801 to S819 , reference may be made to the relevant description of S601 to S619 in FIG. 6A , which will not be repeated here.

[0458] FIG8B is a method flow for calling two mixers to mix the encoded audio and video respectively for two shots to obtain two videos.

[0459] S820 to S833: When the user triggers dynamic shooting for the first time, the mixer 1 mixes the encoded audio and image to obtain a first video.

[0460] S820: The dynamic photographing module sends a mixer start request 1 to mixer 1.

[0461] The mixer start request 1 is used to request the start of the mixer 1. After the dynamic photographing module executes S801, it may execute S820.

[0462] The mixer start request 1 includes the timestamp of the first shooting request, that is, the timestamp Y1 when the electronic device obtains the first shooting operation of the user.

[0463] S821: Mixer 1 starts.

[0464] S822: The memory creates a first file.

[0465] The first file is a file of a dynamic image obtained by triggering dynamic shooting for the first time.

[0466] S823: Mixer 1 sends video data start mixing request 1 to the video queue module.

[0467] S824~S826: Image mixing cycle.

[0468] S824: The video queue module sends the i1th frame of encoded image data to mixer 1.

[0469] Here, i1 is an integer from 1 to I1.

[0470] S825: The mixer 1 determines whether to encapsulate the next frame of image.

[0471] At S820, mixer 1 obtains timestamp Y1. Based on timestamp Y1 and the specific video duration (or specific number of video frames), the electronic device can determine the starting frame for mixing from the second video cache queue and begin encapsulation. During the encapsulation process, the electronic device can determine when to end encapsulation. In the second video cache queue, each frame of coded image data includes the coded image frame and the capture timestamp.

[0472] For the detailed description of S825, please refer to the description of S625 and will not be repeated here.

[0473] S826: Mixer 1 sends image mixing continuation request 1 to the video queue module.

[0474] S827: Mixer 1 sends audio data to the audio queue module to start mixing request 1.

[0475] S828~S830: Audio mixing loop.

[0476] S828: The audio queue module sends the k1th encoded audio and video data to mixer 1.

[0477] Wherein, k1 is an integer from 1 to K1. I1 and K1 are both integers greater than 1.

[0478] S829: Mixer 1 determines whether to encapsulate the next frame of audio.

[0479] For the detailed description of S829, please refer to the description of S629 and will not be repeated here.

[0480] S830: Mixer 1 sends audio continuing mixing request 1 to the audio queue module.

[0481] S831: The mixer 1 encapsulates the i1-th frame coded image data and the k1-th frame coded audio data into the first video.

[0482] The mixer 1 needs to encapsulate all the I1 frame encoded image data and the K1 frame encoded audio data into the first video before the mixer 1 can determine that the encapsulation of the first video is completed.

[0483] S832: Mixer 1 sends a first video mixing completion notification to the dynamic photography module.

[0484] After the first video packaging is completed, the mixer 1 may proceed to execute S832 .

[0485] In an embodiment of the present application, when the dynamic photographing module receives a notification that the mixing of the first video is completed, the electronic device can obtain the first video.

[0486] S833: The memory stores the first video in a first file.

[0487] Among them, the specific description of S820~S833 can refer to the relevant description of S620~S633, and is not repeated here.

[0488] There is no restriction on the execution order between S832 and S833.

[0489] S834-S848: The user touches the dynamic shooting button for the second time, and the mixer 2 mixes the encoded audio and image to obtain a second video.

[0490] S834: The dynamic photographing module receives a second photographing request.

[0491] In this case, with dynamic mode enabled, after the initial capture trigger, the user triggers a second capture before encoding of the first dynamic image is complete. Referring to Figure 8B , the dynamic capture module receives a second capture request before receiving the notification that the first video mixing is complete. The dynamic capture module continues encoding, meaning both the video and audio encoding modules continue encoding. This loop continues, with video and audio encoding continuing until the second mixing is complete.

[0492] In this embodiment of the present application, after encoding is started for a first capture request, a second capture request for capturing a dynamic image is received while the encoder is still in progress. The second capture request is temporally subsequent to the first capture request. In this case, step S834 is executed before step S832, that is, before the first video mixing is completed, the electronic device receives the second capture request.

[0493] Therefore, when the audio encoder and the video encoder have not stopped, the electronic device receives a second shooting request and does not start a new audio encoder and a video encoder for the second shooting request, but keeps the above audio encoder and the video encoder to continue encoding.

[0494] For details of S834, please refer to the relevant contents of S801 and S601, which will not be repeated here.

[0495] S835: The dynamic photographing module sends a mixer start request 2 to mixer 2.

[0496] The mixer start request 2 includes the timestamp of the second shooting request (in S834), that is, the timestamp Y2 when the electronic device obtains the second shooting operation of the user.

[0497] The mixer start request 2 is used to request the mixer 2 to start.

[0498] S836: Mixer 2 starts.

[0499] S837: The memory creates a second file.

[0500] The second file is a file of a dynamic image obtained by triggering dynamic shooting for the second time.

[0501] S838 to S846: The image and audio mixing process of mixer 2 may refer to S623 to S631.

[0502] S838: Mixer 2 sends video data to the video queue module to start mixing request 2.

[0503] S839: The video queue module sends the i2th frame of encoded image data to mixer 2.

[0504] Here, i2 is an integer from 1 to I2.

[0505] S840: The mixer 2 determines whether to encapsulate the next frame of image.

[0506] For details of S840, please refer to the relevant descriptions of S625 and S825, which will not be repeated here.

[0507] S841: Mixer 2 sends image continuing mixing request 2 to the video queue module.

[0508] S842: Mixer 2 sends audio data to the audio queue module to start mixing request 2.

[0509] S843: The audio queue module sends the k2th encoded audio and video data to mixer 2.

[0510] Wherein, k2 is an integer from 1 to K2. I2 and K2 are both integers greater than 1.

[0511] S844: Mixer 2 determines whether to encapsulate the next frame of audio.

[0512] For details of S844, please refer to the relevant descriptions of S629 and S829, which will not be repeated here.

[0513] S845: Mixer 2 sends audio continuing mixing request 2 to the audio queue module.

[0514] S846: The mixer 2 encapsulates the i2-th frame coded image data and the k2-th frame coded audio data into the second video.

[0515] The mixer 2 needs to encapsulate all the I2 frame encoded image data and K2 frame encoded audio data into the second video before the mixer 2 can determine that the second video encapsulation is completed.

[0516] After determinations at S840 and S844, the second video includes I2 frames of coded images and K2 frames of coded audio. The time difference between the first frame (frame 1) and the last frame (frame I2) of the I2-frame coded image is less than or equal to (less than) the first encapsulation duration. The time difference between the first frame (frame 1) and the last frame (frame K2) of the audio in the K2-frame coded audio is less than or equal to (less than) the second encapsulation duration. The time difference between the I2+1th frame of coded image and the 1st frame of coded image is greater than (greater than or equal to) the first encapsulation duration. The time difference between the K2+1th frame of coded audio and the 1st frame of coded audio is greater than (greater than or equal to) the second encapsulation duration. The I2+1th frame of coded image is the frame of image following the I2th frame of coded image in the second video cache queue. The K2+1th frame of coded audio is the frame of audio following the K2th frame of coded audio in the second audio cache queue.

[0517] S847: Mixer 2 sends a second video mixing completion notification to the dynamic photo module.

[0518] After the second video packaging is completed, the mixer 2 may proceed to execute S847 .

[0519] S848: The memory stores the second video in a second file.

[0520] Among them, the specific description of S835~S848 can refer to S620~S633, and the related description of S820~S833, which will not be repeated here.

[0521] There is no restriction on the execution order between S832 and S833.

[0522] FIG8C is a method flow of taking photos in two dynamic shootings, and encapsulating the taken photos and the mixed videos to obtain two dynamic images.

[0523] When the camera is clicked for the first time (as shown in FIG1C ), both the dynamic photo module and the static photo module receive a first capture request. The dynamic photo module performs the following actions as shown in S801, while the static photo module performs the following actions as shown in S849. When the camera is clicked for the second time (as shown in FIG1F ), both the dynamic photo module and the static photo module receive a second capture request. The dynamic photo module performs the following actions as shown in S834, while the static photo module performs the following actions as shown in S862.

[0524] S849-S861: The user triggers the first dynamic shooting, shoots the first image, and packages the first video and the first image to obtain a first file (the dynamic image obtained by the first shooting).

[0525] S849: The static photography module receives a first photography request.

[0526] When the user triggers dynamic shooting for the first time, both the dynamic shooting module and the static shooting module will receive a first shooting request. After receiving the first shooting request, the static shooting module triggers to obtain a first image, which is used to generate the first dynamic image.

[0527] Among them, the specific description of S849 can refer to the description of S801 and S634, and will not be repeated here.

[0528] S850: The static camera module sends an image acquisition request 1 to the camera HAL through the camera FWK.

[0529] S851: The video cache module obtains thumbnail data of the first image.

[0530] S852: The camera HAL sends thumbnail data of the first image to the still camera module through the camera FWK.

[0531] S853: The static camera module performs dynamic image marking on the first image.

[0532] S854: The memory stores the thumbnail data of the first image.

[0533] At this time, the electronic device may display a thumbnail of the first image in the gallery control in the shooting preview screen, as shown in FIG1D .

[0534] S855: The camera HAL processes the original image of the first image to obtain large image data.

[0535] S856: The camera HAL sends the large image data of the first image to the static camera module through the camera FWK.

[0536] S857: The static camera module updates the dynamic image record 1.

[0537] S858: The memory stores the large image data of the first image.

[0538] S859: The dynamic photographing module sends a first video packaging notification to the storage.

[0539] After executing S832, the dynamic photographing module may send a first video packaging notification to the memory.

[0540] S860: The static camera module sends a large-image packaging notification of the first image to the storage.

[0541] S861: The memory splices the large image data of the first image and the first video to obtain a first file and stores it.

[0542] In S833, the electronic device has stored the first video in the first file. After executing S859 and S860, i.e., after receiving the two package notifications of the first video and the first image, the first file can be stitched together. At this point, the electronic device has captured a dynamic image (the first file) and has not yet captured the second dynamic image. The image gallery can display a dynamic image 181 as shown in Figure 1I.

[0543] Among them, the specific description of S849~S861 can refer to the relevant description of S634~S646, which is not repeated here.

[0544] The execution order between S849-S861 and S801-S833 is not restricted. The first video can be acquired first, followed by the first image, or the first image can be acquired first, followed by the first video. When the user first touches the dynamic capture button, both the dynamic capture module and the static capture module receive the first capture request. The static capture module acquires the first image, and the dynamic capture module acquires the first video. These two processes are processed almost in parallel and do not interfere with each other. Therefore, the execution order is not restricted.

[0545] S862-S874: The user triggers a second dynamic shooting, captures a second image, and splices the second video and the second image to obtain a second file (dynamic image obtained by the second shooting).

[0546] S862: The static photography module receives a second photography request.

[0547] When the user triggers dynamic capture for the second time (as shown in Figure 1F), both the dynamic capture module and the static capture module receive a second capture request (execution S834 and S862). After receiving the second capture request, the static capture module triggers the acquisition of a second image, which is used to generate the second dynamic image. The static capture module receives the second capture request before the encoders (video encoder and audio encoder) stop.

[0548] For the detailed description of S862, please refer to S834 and S634 and will not be repeated here.

[0549] S863: The static camera module sends an image acquisition request 2 to the camera HAL through the camera FWK.

[0550] S864: The camera HAL obtains thumbnail data of the second image.

[0551] S865: The camera HAL sends thumbnail data of the second image to the static camera module through the camera FWK.

[0552] S866: The static camera module performs dynamic image marking on the second image.

[0553] S867: The memory stores the thumbnail data of the second image.

[0554] S868: The camera HAL processes the original image of the second image to obtain large image data.

[0555] S869: The camera HAL sends the large image data of the second image to the static camera module through the camera FWK.

[0556] S870: The static camera module updates the dynamic image record 2.

[0557] S871: The memory stores the large image data of the second image.

[0558] In the embodiment of the present application, the large image data of the first image and the second image is the cover image data.

[0559] S872: The dynamic photographing module sends a second video packaging notification to the storage.

[0560] After executing S847, the dynamic photographing module receives the second video mixing completion notification and may send the second video packaging notification to the memory.

[0561] S873: The static camera module sends a large image packaging notification of the second image to the storage.

[0562] S874: The memory splices the large image data of the second image and the second video to obtain a second file and stores it.

[0563] In S848, the electronic device has stored the second video in the second file. After executing S872 and S873, i.e., after receiving the two packaging notifications of the second video and the second image, the second file can be packaged. At this point, the electronic device has captured two dynamic images (first file), and the image gallery can display two dynamic images 191A and 191B as shown in Figure 1J.

[0564] Among them, the specific description of S862~S874 can refer to the relevant description of S634~S646 and S849~S861, which will not be repeated here.

[0565] The execution order between S862-S874 and S834-S848 is not restricted. The second video can be acquired first, followed by the second image, or the second image can be acquired first, followed by the second video. Similarly, when the user touches the dynamic capture button a second time, both the dynamic capture module and the static capture module receive the second capture request. The static capture module acquires the second image, and the dynamic capture module acquires the second video. These two processes are almost parallel and do not interfere with each other. Therefore, the execution order is also not restricted.

[0566] As described in Figures 8A to 8C, when a user continuously shoots dynamic images in a short period of time, the electronic device's dynamic camera mode, if it determines that there are still video mixing tasks to be executed, needs to continue encoding until all videos are mixed. Only when there are no more mixing tasks will the encoding process be stopped. In this way, when shooting dynamic images continuously, the image only needs to be encoded once, and the videos of each dynamic image are mixed separately to obtain the dynamic image captured each time, ensuring the feasibility of the continuous dynamic image solution. This also reduces the number of encoding times, improves execution efficiency, and conserves processing resources.

[0567] 5. After the multiple continuous shooting is completed, the encoding and mixing are stopped (steps S875 to S884 as shown in FIG8D ).

[0568] Figure 8D is a flowchart of another method for stopping video encoding provided by an embodiment of the present application. The method for stopping video encoding is the execution steps after the execution of Figures 8A to 8C, that is, the process of stopping encoding when no continue shooting control is received.

[0569] In an embodiment of the present application, when the electronic device obtains the first video or the second video, it determines whether to stop encoding. Specifically, if there is no video with a dynamic image that has not yet been mixed, the electronic device stops encoding the first video cache queue and the first audio cache queue; if there is a video with a dynamic image that has not yet been mixed, the electronic device continues encoding the first video cache queue and the first audio cache queue.

[0570] S875: When the mixer 1 determines that the mixing should be stopped, the mixer 1 is stopped.

[0571] Among them, S875 can refer to the relevant description of S647 and will not be repeated here.

[0572] S876: Mixer 1 sends a first video mixing completion notification to the dynamic photography module.

[0573] Among them, S876 and S832 are the same execution steps and are not repeated here.

[0574] S877: The dynamic photographing module determines that the video of the dynamic image is not mixed and determines that the encoder does not stop.

[0575] When the dynamic camera module receives a notification of video mixing completion, the process of determining whether to stop encoding is triggered. The process of determining whether to stop encoding is as follows: when there is no video of a dynamic image that has not yet completed mixing, the electronic device stops encoding the first video cache team and the first audio cache queue; when there is a video of a dynamic image that has not yet completed mixing, the electronic device continues to encode the first video cache team and the first audio cache queue. That is, whether there is still a mixing task being processed and no notification of video mixing completion has been received. At this time, the dynamic camera module receives two dynamic shooting messages and only receives the first notification of the completion of the first video mixing (the second notification of the completion of the video mixing has not yet been received). There is still a mixing task being processed, so it is determined not to stop encoding (continue encoding). The logic for determining whether there is a task in the above-mentioned task list can be referred to the relevant description of S649 and will not be repeated here.

[0576] Figure 9A is a schematic diagram of the changes in a task queue disclosed in an embodiment of the present application. Specifically, as shown in Figure 9A, the electronic device receives two shooting requests, and therefore, the task queue includes a task for the first shooting request and a task for the second shooting request. When the electronic device receives a notification that the mixing of the first video is completed, it can clear the task for the first shooting request in the task queue. The cleared task queue includes the task for the second shooting request. The electronic device can determine that the current task queue also includes a task, and can determine that there is an unfinished mixing of the video of the dynamic image. In the embodiment of the present application, the task queue is a task queue established in response to the user clicking to shoot a dynamic image. When a shooting request is received, a corresponding task is created; when the dynamic image and video mixing is completed, the corresponding task is cleared.

[0577] During the above process, a second capture request is received before the dynamic camera module receives the first video mixing completion notification. At this time, the electronic device has not yet received the second video mixing completion notification for the second capture request and can determine that there is an unfinished video mixing of the dynamic image, and determines not to stop the encoder.

[0578] S878: When the mixer 2 determines that mixing should be stopped, the mixer 2 is stopped.

[0579] Among them, S878 can refer to the relevant descriptions of S647 and S875, which will not be repeated here.

[0580] S879: Mixer 2 sends a second video mixing completion notification to the dynamic photography module.

[0581] Among them, S879 and S847 are the same execution steps and are not repeated here.

[0582] S880: The dynamic photographing module determines that the video mixing of the dynamic image is not completed and determines to stop the encoder.

[0583] If the dynamic camera module receives a second video mixing completion notification, the process of determining whether to stop encoding is triggered. At this point, the dynamic camera module has received two dynamic capture messages and two video mixing completion notifications, and no new capture requests were received before receiving the second video mixing completion notification. It is determined that no dynamic image videos have not been mixed (all mixing tasks have been completed), so it is determined to stop encoding and execute S881 and S883.

[0584] FIG9B is a schematic diagram of another task queue change disclosed in an embodiment of the present application. Specifically, as shown in FIG9B , the task queue for the first shooting request has been cleared, and only the task for the second shooting request is included. Upon receiving a notification that the second video mixing is complete, the electronic device can correspondingly clear the task queue for the second shooting request. After the task queue is cleared, there are no tasks. The electronic device can determine that there are no tasks in the current task queue, and can then determine that there is no unfinished mixing of the video of the dynamic image.

[0585] Optionally, before the dynamic photography module receives the second video mixing completion notification, it receives a third capture request, where the third capture request is a user operation subsequent to the second capture request. Upon receiving the second video mixing completion notification, the dynamic photography module determines that the video of the dynamic image of the third capture request has not yet been mixed, and thus may determine that the encoder does not stop and does not execute S881 to S884, i.e., the video encoder and the audio encoder continue encoding.

[0586] S881: The dynamic photographing module sends a request to the video encoding module to stop encoding the image data.

[0587] S882: The video encoding module stops the video encoder.

[0588] S883: The dynamic photographing module sends a request to the audio encoding module to stop encoding the audio data.

[0589] S884: The audio encoding module stops the audio encoder.

[0590] Among them, S881~S883 can refer to the relevant description of S649~S652 above, and will not be repeated here.

[0591] In the above-mentioned continuous shooting process, the dynamic camera module determines whether to stop encoding by determining whether the mixed tasks corresponding to all shooting requests in the current task thread have been processed. Each time a mixed task completion notification is received, the shooting is triggered to make the above-mentioned judgment (S877 and S880). In this way, it can be ensured that encoding can be stopped when the mixed tasks of all shooting requests are completed. When shooting dynamic images continuously, it can avoid the situation where an encoder is started for each shooting request, preventing the problem of excessive occupation of processing resources by encoding. It can also ensure multiple mixing of one encoding, simplify the encoding logic, improve the processing speed, and save processing resources.

[0592] Optionally, when the dynamic mode is activated, after continuous shooting (executing Figures 8A to 8C), no detection is found and the encoding is exited (executing Figure 8D). If the user clicks to shoot a dynamic image again, encoding is restarted and the process of Figures 6A and 6B is executed.

[0593] Optionally, when the dynamic mode is activated, after a single shot (executing Figures 6A and 6B), exit encoding (executing Figure 6D). If the user clicks continuous shooting, re-encoding is performed and the process of Figures 8A to 8C is executed.

[0594] In combination with the video encoding mixing process of Figures 8A to 8D above, Figure 10 is a schematic diagram of mixed encoding of videos in dynamic images exemplarily provided in an embodiment of the present application.

[0595] As shown in Figure 10, when the dynamic mode is turned on, the electronic device starts to cache the first audio cache queue and the first video cache queue. Among them, the number of image frames in the first audio cache queue and the first video cache queue can be determined. For example, the first video cache queue has 45 frames. When the first video cache queue does not reach 45 frames, as new image frames are added to the queue, the number of frames in the first video cache queue continues to increase; when the first video cache queue reaches 45 frames, as new image frames are added to the queue, the old image frames will be dequeued (the last frame at the end of the queue is dequeued), and the 45 frames remain unchanged. In Figure 10, the first video cache queue continues to move forward as the image frames are collected, and the image frames in the first video cache queue will also continue to flow. The first audio cache queue can refer to the description of the first video cache queue and will not be repeated.

[0596] At time T1, the user clicks to shoot once, and the electronic device receives the first shooting request and starts the video encoder, audio encoder and mixer 1. The video encoder obtains the image to be encoded from the first video cache queue, encodes it, and caches the encoded image to the second video cache queue. The audio encoder obtains the audio to be encoded from the first audio cache queue and encodes it, and caches the encoded audio to the second audio cache queue. The second audio cache queue and the second video cache queue can also have a fixed number of frames. As the audio encoder outputs the encoded audio and the video encoder outputs the encoded image, the data frames of the second audio cache queue and the second video cache queue are also constantly flowing. For details, please refer to the description of the first video cache queue above and will not be repeated here. Mixer 1 starts and mixes the encoded data of the second audio cache queue and the second video cache queue.

[0597] At time T2, Mixer 1 has not yet finished mixing the first video, and the video and audio encoders are still encoding. The user clicks "shoot" again, and the electronic device receives a second capture request and starts Mixer 2. Mixer 2 starts and can mix the encoded data in the second audio buffer queue and the second video buffer queue.

[0598] At time T3, mixer 1 completes mixing the first video and stops. Meanwhile, mixer 2 has not yet finished mixing the second video, and the video and audio encoders continue encoding. Mixer 1 can now obtain the first video. The image and audio in the first video are captured from the time before mixer 1 started until it stopped.

[0599] At time T4, mixer 2 completes mixing the second video. No new capture requests were received during the encoding and mixing process, and the mixing tasks for all capture requests are complete. The video encoder, audio encoder, and mixer 2 all stop. Mixer 2 can now obtain the second video. The image and audio in the second video are captured from the time before mixer 2 was started until mixer 2 stopped.

[0600] In the above implementation, the encoder continues encoding while a mixing task is in progress, resolving the issue of delayed encoding response times due to continuous shooting. The multiplexer consumes the encoding results from a group of encoders, responding to multiple shooting requests and mixing the videos separately, ensuring the effect of continuous shooting for animated images and a consistent continuous shooting experience for users. Furthermore, single-channel encoding and multi-channel mixing conserve encoding resources and improve encoding efficiency.

[0601] 11A to 11E are schematic diagrams of image frame production and consumption of a group of second video cache queues according to an embodiment of the present application, and in conjunction with FIG10 , illustrate the specific flow of image frames during the encoding mixing process.

[0602] In an embodiment of the present application, the second video buffer queue maintains a specific queue duration or a specific number of queue frames. As the video encoder continuously encodes, newly encoded image frames are stored in the second video buffer queue, and the earliest image frames are removed from the second video buffer queue, so as to ensure a specific queue duration or a specific number of queue frames.

[0603] Exemplarily, as shown in FIG. 11A, the second video buffer queue includes a total of 45 frames, and each frame of data includes an encoded image frame and a timestamp (t). For example, the encoded image frame 1 corresponds to the timestamp t1, the encoded image frame 2 corresponds to the timestamp t2,..., and the encoded image frame 45 corresponds to the timestamp t45, where t1, t2,..., t45, the time is from far to near (corresponding to the earliest to the latest image frames). The video encoder encodes a new frame of encoded image, frame 46 (t46). The electronic device can perform an enqueue process on frame 46 (t46) and remove the encoded image frame corresponding to the earliest frame t1 from the second video buffer queue. As shown in FIG. 11B, the second video buffer queue becomes frames 2 to 46. As the video encoder encodes a new frame of encoded image frame 47, the second video buffer queue will also dequeue frame 2 and enqueue frame 47, and so on until the encoding ends.

[0604] At time T1, mixer 1 starts consuming the encoded images in the second video buffer queue. As shown in FIG. 11A, mixer 1 is started and obtains the encoded images from the second video buffer queue for mixing. The specific process can refer to the relevant descriptions of S625 and S825, which will not be elaborated here.

[0605] The process of mixer 1 and mixer 2 selecting image frames for mixing (judging whether to mix the frames in the second video buffer queue) can refer to S625. The following will specifically illustrate by examples:

[0606] Exemplarily, the specific video duration is 3s. The timestamp when mixer 1 obtains the first shooting request is T1. Mixer 1 can determine the first time as T1 + 1.5s and the second time as T1 - 1.5s. Mixer 1 can determine the first frame of encoded image (the starting image frame for mixing) to start mixing based on the second time. In the second video buffer queue, if t1 < T1 - 1.5s ≤ t2 (or t1 ≤ T1 - 1.5s < t2), it can be determined that frame 1 corresponding to t1 or frame 2 corresponding to t2 is the first frame of encoded image. Mixer 1 starts from the starting image frame and sequentially obtains image frames in timestamp order.

[0607] Exemplarily, the specific video duration is 3 s. The timestamp of the first shooting request obtained by mixer 1 is T1. It can determine the timestamp of a frame of image based on T1, and determine the first time and the second time based on this timestamp. Among them, for two consecutive frames, frame 44 and frame 45 in the second video cache queue, the timestamps satisfy t44 < T1 ≤ t45, and t45 is the closest to T1 in terms of time. It can be determined that the first time is t45 + 1.5, and the second time is t45 - 1.5 s. Of course, it can also be determined that the first time is t44 + 1.5, and the second time is t44 - 1.5 s. If t1 < t45 - 1.5 s ≤ t2, mixer 1 can determine that frame 1 corresponding to t1 or frame 2 corresponding to t2 is the first encoded image frame.

[0608] At time T2, mixer 2 starts to consume the encoded images in the second video cache queue, and mixer 1 continues to consume the encoded images in the second video cache queue. As shown in FIG. 11C, mixer 1 and mixer 2 respectively obtain encoded images from the second video cache queue for mixing. At this time, the encoded images from frame 36 to frame 80 are included in the second video cache queue. Frame 35 in the second video cache queue will dequeue, and frame 81 will enqueue.

[0609] Exemplarily, the timestamp of the second shooting request obtained by mixer 2 is T2, and the specific video duration is 3 s. Mixer 2 can determine the first time as T2 + 1.5 s and the second time as T2 - 1.5 s. Mixer 2 can determine the starting image frame for starting mixing based on the second time. Exemplarily, in the second video cache queue, t36 < T2 - 1.5 s ≤ t37, it can be determined that frame 36 corresponding to t36 or frame 37 corresponding to t37 is the starting frame. Mixer 2 can start mixing from frame 36 or start mixing from frame 37, and then sequentially obtain image frames according to the timestamp order.

[0610] Exemplarily, after obtaining T2, mixer 2 can determine that the timestamp of the image frame closest to T2 in the second video cache queue is t80. Thus, it can be determined that t80 - 1.5 is the second time, t80 + 1.5 is the first time, and it is determined that t36 < t80 - 1.5 ≤ t37, and frame 36 corresponding to t36 or frame 37 corresponding to t37 is determined as the starting frame.

[0611] In the above process, mixer 1 continues to mix. For example, mixer 1 obtains frame 38. Based on the first time T1 + 1.5 s, it judges that t38 < T1 + 1.5 s and t37 < T1 + 1.5 s, and determines to mix frame 38.

[0612] At time T3, mixer 1 stops consuming the encoded images in the second video buffer queue, and mixer 2 continues to consume the encoded images in the second video buffer queue. As shown in FIG. 11D, both mixer 1 and mixer 2 are consuming the encoded images in the second video buffer queue. The second video buffer queue includes the encoded images from frame 46 to frame 90. The video encoder outputs frame 91, enqueues it, and dequeues frame 45. As shown in FIG. 11E, the second video buffer queue includes the encoded images from frame 47 to frame 91. Mixer 1 finishes mixing and stops consuming, while mixer 2 continues to mix. Among them, mixer 1 and mixer 2 consume some of the same encoded images, for example, frames 36 to frame 90.

[0613] Exemplarily, mixer 1 determines that the first time is T1 + 1.5s, and based on T1 + 1.5s, it determines whether to continue mixing during the mixing process. In the second video buffer queue, t90 < T1 + 1.5s ≤ t91. It can be determined that the mixing ends at frame 90 corresponding to t90 (the mixing includes frame 90); it can also be determined that the mixing ends at frame 91 corresponding to t91 (the mixing includes frame 91). Mixer 1 can then end the mixing.

[0614] Exemplarily, the specific video duration is 3s, and mixer 1 determines that the first time is t45 + 1.5. In the second video buffer queue, if t89 < t45 + 1.5s ≤ t90, it can be determined that the mixing ends at frame 89 corresponding to t89 (the mixing includes frame 89); it can also be determined that the mixing ends at frame 90 corresponding to t90 (the mixing includes frame 90). Mixer 1 can then end the mixing.

[0615] At time T4, mixer 1 stops consuming the encoded images in the second video buffer queue. At this time, both mixer 2 and the video encoder can stop.

[0616] Exemplarily, mixer 2 determines that the first time is T2 + 1.5s, and based on T2 + 1.5s, it determines whether to continue mixing during the mixing process. In the second video buffer queue, t125 < T2 + 1.5s ≤ t126. It can be determined that the mixing ends at frame 125 corresponding to t125 (the mixing includes frame 125); it can also be determined that the mixing ends at frame 126 corresponding to t126 (the mixing includes frame 126). Mixer 2 can then end the mixing (frames 125 and 126 are not shown in the figure).

[0617] Exemplarily, the specific video duration is 3s, and mixer 1 determines that the first time is t80 + 1.5. In the second video buffer queue, if t124 < t80 + 1.5s ≤ t125, it can be determined that the mixing ends at frame 124 corresponding to t124 (the mixing includes frame 124); it can also be determined that the mixing ends at frame 125 corresponding to t125 (the mixing includes frame 125). Mixer 2 can then end the mixing.

[0618] In the embodiment of the present application, if mixer 1 mixes frames 1 to 90 and mixer 2 mixes frames 36 to 125, the first video frame may be frames 1 to 90 and the second video frame may be frames 91 to 125.

[0619] In the above process, the consumption time of mixer 1 is from T1 to T3, and the consumption time of mixer 2 is from T2 to T4. The specific consumption process of mixer 1 and mixer 2 can refer to the processing process of S625, S629, S825, S829, S840 and S844, which will not be repeated here.

[0620] In addition, the processing procedures of the second audio cache queue, the audio encoder, the mixer 1 and the mixer 2 can refer to the above procedures and will not be repeated here.

[0621] 6. When there is no task of shooting dynamic images, exit the camera (steps S1201 to S1225 as shown in FIG12A ).

[0622] Figure 12A is a flow chart of a method for exiting shooting provided by an embodiment of the present application. The method for exiting shooting in Figure 12A is executed by the electronic device after executing Figure 6D or Figure 8D, that is, when the mixing task has been completed and encoding has stopped, and the user triggers the exit of the camera, the electronic device executes the process of exiting the camera, releasing resources, and clearing the audio and video cache queues.

[0623] S1201: The dynamic shooting module receives a request to close the camera application.

[0624] The electronic device receives a user-triggered operation to exit the camera application, and the dynamic capture module may receive a request to close the camera application, and the electronic device may exit the preview capture screen. The operation to exit the camera application may be a user touch operation, a voice command, a finger command, etc.

[0625] S1202: The dynamic photographing module sends a camera closing message to the camera.

[0626] After receiving the request to close the camera application, the dynamic photographing module can send a camera closing message to the camera. Correspondingly, the camera can receive the camera closing message from the dynamic photographing module.

[0627] S1203: The camera stops collecting raw image data and shuts down.

[0628] After receiving the camera shutdown message, the camera may stop collecting raw image data and shut down the camera.

[0629] S1204: The camera sends a camera shutdown notification to the video cache module.

[0630] After the camera is turned off, a camera turn-off notification can be sent to the video cache module. Correspondingly, the video cache module can receive the camera turn-off notification from the camera.

[0631] S1205: The dynamic shooting module sends a first video cache queue clearing request to the video cache module.

[0632] After receiving the request to close the camera application, the dynamic photographing module may send a first video cache queue clearing request to the video cache module. Correspondingly, the video cache module may receive the first video cache queue clearing request from the dynamic photographing module.

[0633] S1206: The video cache module clears the first video cache queue.

[0634] When the video cache module receives the camera shutdown notification and the first video cache queue clearing request, it can clear the first video cache queue.

[0635] S1207: The dynamic shooting module sends a streaming resource release request to the video management module.

[0636] After receiving the request to close the camera application, the dynamic photographing module can send a request to release the allocated stream resources to the video management module. Correspondingly, the video management module can receive the request from the dynamic photographing module to release the allocated stream resources.

[0637] S1208: The video management module releases video stream configuration resources.

[0638] After receiving the request to release the allocation resources from the dynamic camera module, the video management module can release the video stream allocation resources, that is, clear the stored matching encoding path information.

[0639] S1209: The video management module sends a video stream resource release request to the post camera.

[0640] After executing S1208, the video management module may send a video stream resource release request to the post camera. Correspondingly, the post camera may receive the video stream resource release request from the video management module.

[0641] S1210: The post camera releases video stream channel resources.

[0642] After executing S1209 , the post camera may release the video stream channel resources, that is, release the created surface and other resources.

[0643] S1211: The post camera sends a request to the video acquisition module to stop image acquisition.

[0644] After executing S1210, the post camera may send a request to stop image acquisition to the video acquisition module. Correspondingly, the video acquisition module may receive the request from the post camera to stop image acquisition.

[0645] S1212: The video acquisition module releases image cache resources.

[0646] After executing S1211, the video acquisition module may release image cache resources, that is, clear the stream allocation parameter information and the like.

[0647] S1213: The dynamic photographing module sends an audio acquisition stop request to the audio acquisition module.

[0648] After receiving the request to close the camera application, the dynamic photographing module may send an audio stop acquisition request to the audio acquisition module. Correspondingly, the audio acquisition module may receive the audio stop acquisition request from the dynamic photographing module.

[0649] S1214: The audio collection module sends a microphone off message to the microphone.

[0650] After executing S1213, the audio acquisition module may send a microphone off message to the microphone. Correspondingly, the microphone may receive the microphone off message from the audio acquisition module.

[0651] S1215: The microphone stops collecting raw audio data and turns off.

[0652] After executing S1214 , the microphone may stop collecting raw audio data and be turned off.

[0653] S1216: The audio acquisition module sends a first audio cache queue clearing request to the audio cache module.

[0654] After the microphone is turned off, a first audio cache queue clearing request may be sent to the audio cache module. Correspondingly, the audio cache module may receive the first audio cache queue clearing request from the audio acquisition module.

[0655] S1217: The audio cache module clears the first audio cache queue.

[0656] After executing S1216 , the audio cache module may clear the first audio cache queue.

[0657] S1218: The dynamic photographing module sends a video encoder release request to the video encoding module.

[0658] After receiving the request to close the camera application, the dynamic photographing module can send a video encoder release request to the video encoding module. Correspondingly, the video encoding module can receive the video encoder release request from the dynamic photographing module.

[0659] S1219: The video encoding module releases the video encoder.

[0660] After executing S1218 , the video encoding module may release the video encoder.

[0661] S1220: The video encoding module sends a second video cache queue clearing request to the video queue module.

[0662] After executing S1219, the video encoding module may send a second video cache queue clearing request to the video queue module. Correspondingly, the video queue module may receive the second video cache queue clearing request from the video encoding module.

[0663] S1221: The video queue module clears the second video cache queue.

[0664] After executing S1220 , the video queue module may clear the second video cache queue.

[0665] S1222: The dynamic photographing module sends an audio encoder request to the audio encoding module.

[0666] After receiving the request to close the camera application, the dynamic photographing module can send an audio encoder request to the audio encoding module. Correspondingly, the audio encoding module can receive the audio encoder request from the dynamic photographing module.

[0667] S1223: The audio encoding module releases the audio encoder.

[0668] After executing S1222 , the audio encoding module may release the audio encoder.

[0669] S1224: The audio encoding module sends a second audio cache queue clearing request to the audio queue module.

[0670] The audio encoding module may send a second audio cache queue clearing request to the audio queue module. Correspondingly, the audio queue module may receive the second audio cache queue clearing request from the audio encoding module.

[0671] S1225: The audio queue module clears the second audio cache queue.

[0672] After executing S1224 , the audio queue module may clear the second audio buffer queue.

[0673] In the above execution process, the execution order of S1202, S1205, S1213, S1218 and S1222 is not specific, and they are all processed after S1201.

[0674] At this point, all encoders, encoding channels, and pre- and post-encoding data caches in the current dynamic image shooting are cleared, releasing processing and storage resources.

[0675] 7. When there is a task of shooting dynamic images, exit the camera (as shown in steps S1226 to S1230 in FIG12B ).

[0676] Figure 12B is a flow chart of another method for exiting shooting provided by an embodiment of the present application. Figure 12B is an exit process during the execution of Figures 6A to 6B or Figures 8A to 8C.

[0677] When the mixer 1 in the electronic device is mixing the first video, the user triggers the exit camera operation. At this time, the electronic device will also execute steps S1201 to S1225. Furthermore, the electronic device will also execute steps S1226 to S1230. The following is a detailed description:

[0678] S1226: Determine whether a hybrid task is currently being executed.

[0679] Optionally, after receiving the request to close the camera application, the dynamic photography module can determine whether a mixing task is currently being executed. If the mixer 1 is currently processing the mixing task and the mixing has not yet been completed, execute S1227; otherwise, do not execute.

[0680] S1227: The dynamic photographing module sends a mixing stop request to mixer 1.

[0681] When the dynamic photographing module receives a request to close the camera application, it can directly send a mixing stop request to the mixer 1. Correspondingly, the mixer 1 can receive the mixing stop request from the dynamic photographing module.

[0682] S1228: Mixer 1 stops.

[0683] After executing S1227, the mixer 1 stops.

[0684] At this point, the logic for determining whether to stop mixer 1 is not the same as the stop logic triggered in S825 and S829. Instead, upon receiving the mixing stop request, mixer 1 directly stops the mixing cycle. At this point, the first video being mixed by the mixer may not have reached the encapsulation duration. For example, if the encapsulation duration is 3 seconds, mixer 1 receives the mixing stop request when the first video reaches 1.5 seconds and immediately stops. At this point, the duration of the first video is 1.5 seconds.

[0685] S1229: Mixer 1 sends a first video mixing completion notification to the dynamic photography module.

[0686] After executing S1228 , the mixer 1 may send a first video mixing completion notification to the dynamic photographing module. Correspondingly, the dynamic photographing module may receive the first video mixing completion notification from the mixer 1 .

[0687] The execution of S1229 is the same as S832 (S648), except that the triggering conditions are different. S1229 is triggered by the user exiting the camera and mixer 1 ending mixing early.

[0688] S1230: The memory stores the first video as a first file.

[0689] After executing S1228, the memory stores the first file of the first video and the encapsulated first video.

[0690] In the above embodiment, the electronic device determines to obtain the first video in two ways: one is to satisfy the packaging duration and end the mixing to obtain Video 1; the other is that the user exits the camera early, ends the mixing, and obtains Video 2. Compared with the two, the duration of Video 1 is shorter than that of Video 2.

[0691] In the above mixing logic, the priority corresponding to the user operation is the highest, which can ensure that the video mixing is completed while the animated image is obtained and the user operation is coordinated.

[0692] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0693] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0694] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A shooting method, characterized in that: The method is applied to an electronic device, and the method comprises: In response to a first operation on a capture control, starting a first video encoder; encoding the first video frame by the first video encoder; In a process in which the first video encoder encodes the first video frame, in response to a second operation acting on the shooting control, encoding a second video frame by the first video encoder; generating a first dynamic image, wherein the first dynamic image corresponds to the first video frame; generating a second dynamic image, wherein the second dynamic image corresponds to the second video frame; The first video encoder is controlled to stop encoding.

2. The method according to claim 1, characterized in that After controlling the first video encoder to stop encoding, the method further includes: In response to a third operation on the shooting control, starting a second video encoder; encoding the third video frame by the second video encoder; generating a third dynamic image, wherein the third dynamic image corresponds to the third video frame; After generating the third dynamic image, controlling the second video encoder to stop encoding; In response to a fourth operation on the shooting control, starting a third video encoder; encoding the fourth video frame by the third video encoder; A fourth dynamic image is generated, the fourth dynamic image corresponding to the fourth video frame.

3. The method according to claim 1, characterized in that: Before the electronic device controls the first video encoder to stop encoding, the method further includes: In response to the first operation, adding a first shooting task to the task list; In response to the second operation, adding a second shooting task to the task list; When generating the first dynamic image: Clear the first shooting task, When there is a task in the task list, controlling the first encoder to continue encoding; When generating the second dynamic image: Clear the second shooting task, When there is no task in the task list, the first video encoder is controlled to stop encoding.

4. The method according to claim 2, characterized in that: Before the electronic device controls the second video encoder to stop encoding, the method further includes: In response to the third operation, adding a third shooting task to the task list; When the third dynamic image is generated, clearing the third shooting task; When there is no task in the task list, the third video encoder is controlled to stop encoding.

5. The method according to claim 1, characterized in that: After the electronic device generates the first dynamic image, the method further includes: In response to an operation for launching the gallery, displaying a second user interface; the second user interface includes a first thumbnail of a first static image of the first dynamic image; In response to an operation on the first thumbnail, displaying a third user interface, the third user interface including the first static image; In response to an operation performed on the first static image, the electronic device plays the first dynamic image.

6. The method according to claim 5, characterized in that The first dynamic image includes the first static image and a first video, the first video includes M frames of static images, and M is an integer greater than 1.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to the first operation, starting a first audio encoder; encoding a first audio frame by using the first audio encoder, wherein the first dynamic image corresponds to the first audio frame; During the process of the first audio encoder encoding the first audio frame, in response to the second operation, the first audio encoder encodes a second audio frame, and the second dynamic image corresponds to the second audio frame.

8. The method according to claim 7, characterized in that The generating of the first dynamic image comprises: Mixing a first coded video frame and a first audio code video frame by a first mixer to generate the first dynamic image, wherein the first coded video frame is obtained after the first video encoder encodes the first video frame, and the first coded audio frame is obtained after the first audio encoder encodes the first audio frame; The generating of the second dynamic image comprises: The second dynamic image is generated by mixing the second encoded video frame and the second audio code video frame through a second mixer, wherein the second encoded video frame is obtained after the first video encoder encodes the second video frame, and the second encoded audio frame is obtained after the second audio encoder encodes the second audio frame.

9. The method according to claim 8, characterized in that After the electronic device generates the first dynamic image and the second dynamic image, the method further includes: Control the first video encoder to stop encoding, and control the first audio encoder to stop encoding.

10. The method according to claim 1, characterized in that The method further comprises: In response to an operation for starting the camera, acquiring the first video frame through the camera; Storing the first video frame in a first video cache queue; After encoding the first video frame to obtain the first encoded video frame, the first video encoder stores the first encoded video frame in a second video cache queue.

11. The method according to claim 7, characterized in that The method further comprises: In response to an operation for starting the camera, acquiring the first audio frame through a microphone; Storing the first audio frame in a first audio buffer queue; After encoding the first audio frame to obtain the first encoded audio frame, the first audio encoder stores the first encoded audio frame in a second audio cache queue.

12. The method according to claim 10, characterized in that The method further comprises: In response to the operation acting on the dynamic mode control, the first video cache queue is created, the second video cache queue is created, the first audio cache queue is created, and the second audio cache queue is created.

13. The method according to claim 12, characterized in that The first video cache queue is stored in a hardware abstraction layer HAL of the electronic device, and the second video cache queue, the first audio cache queue and the second audio cache queue are stored in an application layer of the electronic device.

14. The method according to claim 10, characterized in that The method further comprises: determining, by a first mixer, a first time based on a shooting time of the first operation and a specific video duration; In a case where a first time difference between a timestamp of an i-th frame of coded image and a timestamp of a first frame of coded image is less than or equal to the first time, mixing the i-th frame of coded image into the first dynamic image through the first mixer, the i-th frame of coded image being a coded video frame in the second video cache queue; When a first time difference between the time stamp of the i-th frame coded image and the time stamp of the first frame coded image is greater than the first time, the first mixer is controlled to end mixing.

15. The method according to claim 14, characterized in that The method further comprises: Determining a second time based on the shooting time and the specific video duration by the first mixer; The first mixer selects a coded video frame with a timestamp closest to the second time from the second video cache queue as the first frame coded image.

16. The method according to claim 1, characterized in that The method further comprises: In response to an operation on a dynamic mode control, the first video encoder and the first audio encoder are created.

17. The method according to claim 16, characterized in that The method further comprises: In response to an operation acting to exit the camera, the first video encoder and the first audio encoder are released.

18. The method according to claim 1, characterized in that The method further comprises: In response to an operation for activating the camera, displaying a first user interface, the first user interface displays an image acquired by the camera, the first user interface including a dynamic mode control in a closed state; In response to an operation acting on the dynamic mode control, the dynamic mode control is switched to an on state.

19. An electronic device, characterized in that: include: One or more processors and one or more memories; the one or more processors are coupled to the one or more memories, the one or more memories are used to store computer program code, the computer program code includes computer instructions, when the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-18.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

Images