Image processing method, electronic device and computer readable storage medium

Abstract

The present application provides an image processing method, an electronic device, and a computer-readable storage medium, and relates to the field of terminal technology. It is applied to electronic devices. In response to a user triggering operation on a first control of a dynamic photo shooting function of a camera application, the electronic device applies for a first encoding buffer for storing multiple video frames; and the size of the first encoding buffer matches the encoding format of the hardware codec of the electronic device, and multiple video frames are obtained based on the preview stream; in response to a user triggering operation on a shooting control, dynamic photos are stored; dynamic photos include videos and pictures, and the video is obtained by calling a hardware codec to encode multiple video frames stored in the first encoding buffer, and the picture is a picture taken in response to the user triggering operation on the shooting control. In the scenario of shooting dynamic photos, it can be ensured that the software interface can access the hardware codec, so that the two can cooperate to implement video encoding and ensure that dynamic photos are obtained.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of mobile smart devices such as smart phones, tablet computers and other electronic devices, most electronic devices support the function of taking photos, which is convenient for users to carry electronic devices with them to record their lives. As users' demand for using electronic devices to record their lives increases, in order to enable the photos taken to record more moments, a technology that can take dynamic photos in the photo mode is proposed. The dynamic photo includes the moment when the user presses the shooting control and the images within a period of time before and after the moment, which can help users record more wonderful moments.

[0003] Dynamic photos are actually composed of pictures and videos. No encoding is required when taking pictures, but encoding is required when taking videos. Therefore, when taking dynamic photos, it is necessary to use the hardware codec of the electronic device to encode the captured video. When using the hardware codec for encoding, it is necessary to call the hardware encoding interface of the hardware codec. However, the hardware codecs provided by different hardware codec manufacturers are different, so it is necessary to use the hardware codec interface provided by the system's underlying multimedia framework to access the hardware codec. The hardware codec interface provided by the system's underlying multimedia framework can unify most of the common hardware encoding interfaces currently available. However, there are still some hardware codecs produced by some manufacturers that cannot be accessed through the hardware codec interface provided by the system's underlying multimedia framework, resulting in the electronic device being unable to take dynamic photos normally when these hardware codecs are used. Summary of the invention

[0004] In view of this, the present application provides an image processing method, an electronic device and a computer-readable storage medium, which can ensure that the software interface can access the hardware codec in the scenario of taking dynamic photos, so that the two can cooperate to achieve video encoding.

[0005] In a first aspect, the present application provides an image processing method, applied to an electronic device, the method comprising:

[0006] Displaying a shooting preview interface of a camera application, the shooting preview interface including a first control and a shooting control; the first control is used to enable a dynamic photo shooting function of the camera application;

[0007] In response to a user triggering operation on a first control, applying for a first encoding buffer; the size of the first encoding buffer matches the encoding format of a hardware codec of the electronic device, and the first encoding buffer is used to store a plurality of video frames, and the plurality of video frames are obtained based on a preview stream;

[0008] In response to a user triggering operation on a shooting control, a dynamic photo is stored; the dynamic photo includes a video and a picture, the video is obtained by calling a hardware codec to encode a plurality of video frames stored in a first encoding buffer, and the picture is a picture taken in response to a user triggering operation on the shooting control.

[0009] In the above implementation process, in response to the user's triggering operation on the first control, the electronic device can turn on the dynamic photo shooting function of the camera application. At this time, the electronic device needs to apply for a first encoding buffer for storing multiple video frames, so that when the user triggers the shooting control, the hardware codec can be called to encode the multiple video frames stored in the first encoding buffer to obtain the video in the dynamic photo, thereby ensuring the normal shooting process of the dynamic photo. Among them, the size of the first encoding buffer is the same as the size of the video frame that can be encoded by the hardware codec, so that the hardware codec can encode normally.

[0010] In the scenario of taking dynamic photos, it can be ensured that the hardware codec is called through the software interface, thereby ensuring the normal operation of video encoding and obtaining dynamic photos.

[0011] In a possible implementation manner of the first aspect, applying for a first encoding buffer includes:

[0012] Get the buffer size based on the size of the preview stream and the encoding format of the hardware codec;

[0013] Apply for the first encoding buffer according to the buffer size.

[0014] In some embodiments, after obtaining the buffer size, the camera application can send a memory allocation request to a software interface for allocating memory space, so that the software interface can allocate a piece of memory space corresponding to the buffer size, that is, the first encoding buffer. Afterwards, the software interface can send the address of the allocated first encoding buffer to the camera application.

[0015] In a possible implementation manner of the first aspect, in response to a triggering operation of the first control by the user, the method further includes:

[0016] Call the software interface to create a layer, and the size of the layer is the preset size;

[0017] Bind the first encoding buffer to the layer.

[0018] In the above implementation process, the size of the layer created by calling the software interface is a preset size. The preset size matches the memory alignment format preset inside the software interface. The memory alignment format preset inside the software interface is provided by the developer or manufacturer who develops the software interface.

[0019] The preset size may be determined based on the actual size of the video frame and a memory alignment format preset within the software interface.

[0020] In a possible implementation of the first aspect, the electronic device includes a camera framework layer and a camera hardware abstraction layer;

[0021] Get the buffer size based on the size of the preview stream and the encoding format of the hardware codec, including:

[0022] The camera framework layer sends a callback function to the camera hardware abstraction layer;

[0023] The camera hardware abstraction layer determines the buffer size based on the size of the preview stream and the encoding format of the hardware codec;

[0024] The camera hardware abstraction layer sends the buffer size to the camera framework layer based on the callback function;

[0025] Apply for the first encoding buffer according to the buffer size, including:

[0026] The camera framework layer applies for the first encoding buffer based on the buffer size.

[0027] In the above implementation process, after the camera framework layer sends a callback function to the camera hardware abstraction layer, the camera hardware abstraction layer can determine the buffer size according to the size of the preview stream and the encoding format of the hardware codec, and then the camera hardware abstraction layer sends the buffer size to the camera framework layer. So that the camera framework layer can apply for the first encoding buffer according to the buffer size. In this way, it can be ensured that the size of the first encoding buffer matches the encoding format of the hardware codec, thereby ensuring that after the video frames in the preview stream are stored in the first encoding buffer, the hardware codec can normally encode the video frames stored in the first encoding buffer, so that dynamic photos can be obtained normally.

[0028] In a possible implementation of the first aspect, the electronic device includes a camera application layer, a camera framework layer and a camera hardware abstraction layer; calling a software interface to create a layer includes: a camera application in the camera application layer calls a software interface to create a layer through the camera framework layer; the method also includes: the camera application layer sends an address of the layer to the camera framework layer; binding a first coding buffer to the layer includes: the camera framework layer binds the address of the first coding buffer to the address of the layer.

[0029] After the camera application calls the software interface to create a layer through the camera framework layer, the address of the first encoding buffer can be bound to the address of the layer, thereby ensuring that the camera application manages the video frames stored in the first encoding buffer through the layer. For example, the camera application can call the hardware codec through the software interface to encode multiple video frames stored in the first encoding buffer.

[0030] In a possible implementation of the first aspect, the method also includes: storing multiple video frames in the preview stream in a first encoding buffer in sequence; the number of the multiple video frames corresponds to the total length of the video; and using a hardware codec to encode all video frames stored in the first encoding buffer in sequence to obtain a video.

[0031] The sizes of the multiple video frames in the preview stream before being stored in the first encoding buffer are different from the sizes of the video frames after being stored in the first encoding buffer. In the process of storing the multiple video frames in the preview stream in the first encoding buffer, it is necessary to pad or crop the data of the video frames according to the size of the first encoding buffer so that the size of the video frames stored in the first encoding buffer is the same as the size of the first encoding buffer, so as to ensure that the hardware codec can encode the video frames stored in the first encoding buffer.

[0032] In a possible implementation manner of the first aspect, the method further includes:

[0033] In response to a user triggering operation on a second control in the shooting preview interface, or in response to a user exiting a camera application, the first encoding buffer is released; the second control is used to turn off a dynamic photo shooting function of the camera application.

[0034] In the above process, the user triggers the second control in the shooting preview interface, indicating that the user wants to disable the dynamic photo function. The user performs an operation to exit the camera application, indicating that the user does not need to shoot dynamic photos. Therefore, the electronic device can release the first encoding buffer to avoid resource waste.

[0035] The operation of exiting the camera application may include ending all processes related to the camera application, or switching the camera application from a foreground running state to a background running state.

[0036] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in the first aspect and any possible design method thereof.

[0037] In a third aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible design thereof.

[0038] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the electronic device to execute the method described in the first aspect and any possible design thereof.

[0039] In a fifth aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the behavior of the electronic device in any of the above aspects and possible implementation methods. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, a sending module or unit, a calling module or unit, an encoding module or unit, and a storage module or unit, etc.

[0040] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor and may also include a memory, for implementing any one of the methods provided in the third to fifth aspects above. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0041] It can be understood that the electronic device described in the second aspect and any possible design thereof provided above, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of viewing dynamic photos provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a camera architecture provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a camera architecture of an electronic device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a static photo shooting scene provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a dynamic photo shooting interface provided in an embodiment of the present application Figure 1 ;

[0048] Figure 7 A schematic diagram of a dynamic photo shooting interface provided in an embodiment of the present application Figure 2 ;

[0049] Figure 8 A schematic diagram of a dynamic photo shooting interface provided in an embodiment of the present application Figure 3 ;

[0050] Fig. 9 A schematic diagram of a dynamic photo shooting scene provided in an embodiment of the present application;

[0051] Fig.10 A schematic diagram of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0053] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0054] Before introducing the embodiments of the present application, the technologies involved in the embodiments of the present application are first introduced in detail.

[0055] 1. A live photo generally includes a picture and a video. For example, a live photo generally includes an image in the Joint Photographic Experts Group (JEPG) format and a video in the Moving Picture Experts Group 4 (MP4) format. A live photo can also be called something else, such as a dynamic picture or a live photo.

[0056] After the user takes a dynamic photo with a mobile phone, the user can view the dynamic photo in the album. Figure 1 As shown, the mobile phone display screen displays the album display interface, which includes thumbnails 101 of dynamic photos taken by the user. Figure 1 It is usually a static picture, which is a picture in a dynamic photo. Figure 1 As shown, in response to the user's operation on the thumbnail 101 of the dynamic photo, the mobile phone can display the display interface of the dynamic photo, and the user can view the dynamic photo in the display interface. Generally, in response to the user's operation on the thumbnail 101 of the dynamic photo, the picture in the dynamic photo is displayed, and the user can trigger the mobile phone to display the video in the dynamic photo by performing a specific operation. For example, the display interface includes a playback control 102. In response to the user's operation on the playback control 102, the mobile phone can play the video included in the dynamic photo in the display interface. In some examples, in response to the user's operation on the thumbnail 101 of the dynamic photo, the mobile phone can display the display interface of the dynamic photo, and the display interface of the dynamic photo may include other forms of controls for triggering the playback of the video included in the dynamic photo.

[0057] 2. Camera architecture

[0058] In the system of electronic devices, the camera architecture is a software architecture that can support the electronic device to realize the shooting function. Figure 2As shown, the camera architecture includes a camera application layer, a camera framework layer, and a camera hardware abstract layer (HAL) layer.

[0059] The camera architecture in electronic devices can adapt to the different hardware configured by various electronic device assembly manufacturers. For example, the same operating system can be configured with different image sensor chips or different hardware codecs. The existing applications in the camera application layer and the existing interfaces in the camera framework layer do not need to be changed, so that the software architecture of the electronic device can be adapted to the hardware. In other words, the hardware of the electronic device can support the normal operation of the software architecture and realize normal photo-taking functions. In addition, the interface provided by the camera HAL layer is also unified, and each hardware manufacturer can implement the hardware configuration according to the interface standard provided by the camera HAL layer.

[0060] 3. Camera architecture buffer

[0061] Buffer is a part of the memory space. In other words, a certain amount of storage space is reserved in the memory space, which is used to buffer input or output data.

[0062] Pixel is the smallest unit in an image. The total number of pixels in an image is the product of the image's pixel width and pixel width height. For example, if an image has a pixel width of 1440 and a pixel height of 1080, then the total number of pixels in the image is 1440*1080, that is, the size of the image is 1440*1080.

[0063] Since an image contains a large number of pixels, for the camera architecture, in order to facilitate image transmission and reduce the number of image reads and writes, a buffer can be used to store all the pixels in the image to be transmitted. If only part of the pixels of the image are read, written and transmitted each time, it will take many times to complete the transmission process of an image, resulting in a waste of transmission time. If an image is stored in a buffer first and then transmitted after the buffer is full, the number of reads and writes will be greatly reduced, which can save time.

[0064] Generally, when starting the shooting function of the camera application, a corresponding buffer can be applied for, such as the buffer corresponding to the camera application, for use by the camera application. For example, in response to the user opening the operation of the dynamic photo shooting function control in the camera application, the electronic device enables the function of shooting dynamic photos. At this time, the camera application of the electronic device will apply for a buffer for storing dynamic photos through the camera framework layer. Since dynamic photos include pictures and videos, the camera framework layer will apply for a picture buffer for storing pictures, as well as a coding buffer for storing video frames that make up the video.

[0065] Since there is only one picture in a dynamic photo, the camera framework layer usually applies for only one picture buffer. However, the video included in the dynamic photo is composed of multiple video frames. In order to increase the transmission speed of the video frames that make up the video, the camera framework layer can apply for multiple encoding buffers.

[0066] The addresses of the requested image buffer and encoding buffer will be sent to the camera HAL layer. In response to the user's operation of the control for taking dynamic photos in the camera application, the camera HAL layer stores the relevant image data in the corresponding image buffer and encoding buffer, and returns the address of the image buffer storing the image and the encoding buffer storing the video frame to the camera framework layer. At this point, since the images do not need to be encoded, the camera framework layer can use the images in the image buffer as the images that make up the dynamic photos. The video frames in the encoding buffer need to be encoded, so the camera application calls the hardware codec through the software interface to encode the video frames in the encoding buffer to encode the video, so as to obtain the video, which can be used as the video that makes up the dynamic photos.

[0067] In the above process, the coding buffer is applied for through the software interface. The size of the coding buffer applied for by the software interface is different from the size of the video frame that can be encoded by the hardware codec. Therefore, there is a situation where the video frames in the coding buffer cannot be sent to the hardware codec for encoding to obtain video.

[0068] Exemplarily, when the actual size of the video frame is known, the size of the to-be-encoded video frame determined according to the memory alignment format required by the hardware codec is different from the size of the encoding buffer actually requested by the software interface.

[0069] For example, the memory alignment format of the encoding buffer is 128 bits for width and 32 bits for height, and the memory alignment format required by the hardware codec is 512 bits for both width and height. Then, after the video frame with an actual size of 1440*1080 is aligned according to the memory alignment format of the encoding buffer, the determined buffer size for buffering the video frame is 1536*1088. Among them, after alignment according to the memory alignment format of the encoding buffer, it is necessary to ensure that the pixel width of the aligned image is an integer multiple of 128. First calculate 1440 / 128=11.25, and round the obtained 11.25 to 12. The 1536 calculated according to 12*128 is the pixel width after the alignment requirement. It is necessary to ensure that the pixel height of the aligned image is an integer multiple of 32. First calculate 1080 / 32=33.75, and round the obtained 33.75 to 34. The 1088 calculated according to 34*32 is the pixel width after the alignment requirement.

[0070] Similarly, after the actual size of the video frame of 1440*1080 is aligned in the memory alignment format required by the hardware codec, with the width and height aligned in 512 units, the size of the video frame to be encoded by the hardware codec is determined to be 1536*1536. Among them, 1440 / 512=2.8125, 2.8125 is rounded to 3, 3*512=1536; 1080 / 512≈2.1, 2.1 is rounded to 3, 3*512=1536.

[0071] The size of the video frame to be encoded that can be encoded by the hardware codec is different from the encoding buffer requested by the camera framework layer. As a result, there may be video frames in the encoding buffer requested by the camera framework layer that cannot be encoded by the hardware codec, and the video cannot be obtained. As a result, dynamic photos including pictures and videos cannot be obtained, and dynamic photos cannot be displayed to users normally.

[0072] To this end, the present application provides an image processing method, which is performed by an electronic device. When the electronic device executes the image processing method, it can determine and apply for a coding buffer for storing video frames based on the size of the video frame that can be encoded by the hardware codec when the dynamic photo shooting function is enabled. In addition, the electronic device can associate the coding buffer with the software interface that calls the hardware codec to ensure that the electronic device can normally use the software interface to call the hardware codec, so that the hardware codec can perform video encoding on the video frames stored in the coding buffer to obtain dynamic photos.

[0073] In order to better understand the embodiments of the present application, the electronic device provided by the embodiments of the present application is first introduced.

[0074] The image processing method provided in the embodiment of the present application can be applied to an electronic device 100. The electronic device 100 can be a mobile phone, a tablet computer, a smart screen, a laptop computer, a vehicle-mounted device, a wearable device (such as a smart watch), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence device, a smart home, and other terminal devices with a shooting function. The embodiment of the present application does not limit the specific type of the electronic device 100 and the installed operating system.

[0075] Combine below Figure 3 The hardware structure of the electronic device 100 is briefly introduced.

[0076] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0077] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0081] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0082] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0083] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0084] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect other electronic devices, such as headphones, AR devices, etc.

[0085] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0086] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142, it can also power the electronic device through the power management module 141. The power management module 141 is used to connect to the battery 142, and the power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the components in the electronic device 100.

[0087] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0088] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0089] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0090] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through the display screen 194.

[0091] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0092] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0093] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0094] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0095] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0096] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0097] In an embodiment of the present application, the camera 193 may include an image sensor chip. In some embodiments, after the function of taking dynamic photos is enabled, the camera 193 may start to acquire images. The images acquired by the camera 193 are stored in real time in the dynamic photo cache queue. For example, when the total duration of the video in the dynamic photo is 3s, the preview stream image 1.5s before the current moment can be copied and retained in the dynamic photo cache queue. In response to the user's operation of the dynamic photo shooting control, the electronic device can obtain the preview stream image 1.5s before the moment of operating the dynamic photo shooting control from the dynamic photo cache queue as a video frame, and since the preview stream image in the dynamic photo buffer queue is updated in real time, after the electronic device obtains the preview stream image 1.5s before the moment of operating the dynamic photo shooting control, the preview stream image in the dynamic photo buffer queue is updated to the preview stream image 1.5s after the moment of operating the dynamic photo shooting control, so that all the preview stream images acquired by the electronic device can form a video with a total duration of 3s.

[0098] The number of images that can be buffered in the dynamic photo cache queue can be determined based on the duration of the preview stream image that can be stored and the frame per second (FPS) of the preview stream image. For example, if the preview stream image 1.5s before the current moment can be copied and retained in the dynamic photo cache queue, if the FPS of the preview stream image is 30, it can be determined that the number of preview stream image frames copied and retained in the dynamic photo cache queue is 45 frames (30*1.5=45).

[0099] The number of images that can be buffered in the dynamic photo cache queue is limited. Therefore, when the dynamic photo cache queue is full of images, the image with the earliest buffering time can be released in the dynamic photo cache queue to ensure the real-time performance of the images in the dynamic photo cache queue.

[0100] In response to the user's operation on the control for taking a dynamic photo, the electronic device can read the preview stream images from the dynamic photo cache queue in chronological order, until the number of preview stream images read is sufficient to obtain a video that meets the requirements of the dynamic photo, and then the electronic device stops reading the preview stream images from the dynamic photo cache queue. After that, the electronic device encodes all the preview stream images read, that is, all video frames into a video. Finally, the electronic device synthesizes the obtained video with the image taken at the time when the user operates the control for taking a dynamic photo into a dynamic photo.

[0101] The electronic device uses the encoding buffer to read the preview stream images from the dynamic photo cache queue in chronological order. Exemplarily, the camera framework layer of the electronic device sends the address of the encoding buffer to the camera HAL layer, so that the camera HAL layer can store the preview stream image with the earliest buffering time in the dynamic photo buffer queue into the encoding buffer, so that the camera framework layer can read the preview stream image stored therein based on the address of the encoding buffer.

[0102] In the embodiment of the present application, buffer is a storage space in the internal memory. The camera framework layer can apply for part of the storage space in the internal memory as a buffer through an interface for applying for buffer.

[0103] In some embodiments, the camera 193 may further include a plurality of image sensor chips. For example, when the camera 193 is a dual-camera, the camera 193 includes two image sensor chips.

[0104] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0105] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc. In some embodiments, the video codec is a hardware codec.

[0106] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0107] The electronic device 100 can implement audio functions through the audio module 170 and the application processor, etc. For example, music playing, recording, etc. The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0108] The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc. The gyroscope sensor may be used to determine the motion posture of the electronic device 100. The ambient light sensor is used to sense the brightness of the ambient light.

[0109] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.

[0110] The following, combined Figure 4 The camera architecture of the electronic device 100 is exemplarily described.

[0111] Figure 4 The camera architecture shown includes a camera application layer, a camera framework layer, a camera HAL layer, and a camera driver layer.

[0112] It should be noted that the embodiments of the present application are based on Figure 4 To illustrate by way of example, in other camera architectures, as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solution of the present application can also be implemented.

[0113] The camera application layer may include a camera application program. The camera application program in the embodiment of the present application may be an application program in an operating system of the electronic device, or may be an application program provided by a third-party software manufacturer.

[0114] The camera framework layer provides an application programming interface (API) and programming framework for the camera applications of the camera application layer. The camera framework layer includes some predefined functions. The camera applications of the camera application layer can obtain the camera services of the lower layer by calling the interface provided by the camera framework layer.

[0115] In some embodiments, the camera application can control the image sensor chip to output images by calling the camera framework layer.

[0116] In some other embodiments, the camera application can apply for a buffer through the camera framework layer. The specific buffer type is related to the photo mode that the camera application is in. For example, in the mode of taking ordinary photos, the buffer applied for by the camera application through the camera framework layer is an ordinary picture buffer. For another example, in the mode of taking videos, the buffer applied for by the camera application through the camera framework layer is a coding buffer. For another example, in the mode of taking dynamic photos, the buffer applied for by the camera application through the camera framework layer is a picture buffer and a coding buffer.

[0117] In some embodiments, the software structure of the electronic device includes a framework layer, and the framework layer includes a camera framework layer and a multimedia hardware codec. The camera framework layer can call the multimedia hardware codec.

[0118] Among them, the multimedia hardware codec is an API for processing audio and video codecs in the software architecture of electronic devices. It is a software module that can provide hardware codec functions. For example, MediaCodec (media codec). When the multimedia hardware codec is used to implement the hardware codec function, the multimedia hardware codec can call the hardware codec. For example, after the camera application calls the camera framework layer to control the output of multiple frames of images by the image sensor chip, it can also call MediaCodec to perform hardware encoding on the multiple frames of images to obtain a video.

[0119] In some other embodiments, the multimedia hardware codec can provide both hardware codec and software codec. For example, MediaCodec can provide both software codec and hardware codec. When using MediaCodec to implement software coding, MediaCodec can use a processor to perform the software coding process.

[0120] The function of the camera HAL layer is to isolate the underlying hardware from other parts of the software architecture. The camera HAL layer can abstract the hardware and provide an abstract and unified interface for the underlying hardware. In this way, the camera framework layer can use a unified API to interact with the hardware through the camera HAL layer without having to pay attention to the differences between different hardware platforms.

[0121] In some embodiments, Figure 4As shown, the camera HAL layer includes a real-time pipeline, a preview offline pipeline, a snapshot offline pipeline, a livephoto manager (livephoto manager), and a camera service. Among them, the realtime pipeline includes an image front end (IFE) and a perception engine. The previewoffline pipeline includes an image processing engine (IPE), an algorithm post-processing node, and a dynamic picture copying node. LivephotoMgr includes a cycle queue and offline dynamic photos (livephoto offline). The camera service can be CfgServer, Post camera. In this application, cyclequeue can refer to a dynamic photo cache queue.

[0122] The camera driver layer is located between the camera HAL layer and the hardware, and can provide information such as hardware capabilities to the layers above the camera driver layer in the form of file nodes. In addition, the upper layer can make the hardware work through the driver of the camera driver layer.

[0123] In some embodiments, when the camera application is in a mode for shooting video, the camera application can access the node in the camera driver layer through the camera HAL layer to obtain capability information of various hardware related to shooting video in the hardware layer, for example, the size, resolution, and zoom ratio of each frame in the video.

[0124] The camera HAL layer can access the node in the camera driver layer through the querycap instruction, and the camera HAL layer sends the obtained capability information to the camera application, so that the camera application stores the queried capability information and provides corresponding functions for the user.

[0125] IFE is used to perform color correction, downsampling, demosaicing and other processing on the images output by the image sensor chip.

[0126] The perception engine is used to detect scene information, for example, scenes including smiling faces, scenes including moving objects, etc. The perception engine is also used to obtain other information such as brightness information and motion state information of electronic devices. The information detected by the perception engine can be stored together with the captured dynamic photos, videos, and static photos to facilitate subsequent processing of the dynamic photos, videos, and static photos. In some examples, the perception engine can detect the motion state information of the electronic device through a gyroscope sensor, detect brightness information through an ambient light sensor, and detect scene information through an NPU.

[0127] The snapshot offline pipeline is used to send the image processed by IFE to the camera application. For example, when the user operates the control for taking a normal photo or the control for taking a dynamic photo, the snapshot offline pipeline can send the image output by the image sensor chip and processed by IFE to the camera application.

[0128] IPE is used to perform secondary processing on the image processed by IFE, including image denoising, sharpening, contrast adjustment, color mapping, etc.

[0129] The algorithm post-processing node is used to perform light processing on the image after IPE processing, for example, electronic image stabilization (EIS).

[0130] The dynamic picture copy node is used for the electronic device to copy each video frame processed by the algorithm post-processing node in real time when the dynamic photo shooting function is turned on, and send the video frame obtained by real-time copying to the circular queue in the dynamic photo cache queue. A certain number of video frames can be stored in the circular queue. In response to the user's operation on the dynamic photo shooting control, the electronic device stores the video frames stored in the circular queue in chronological order through the offline dynamic photos in the empty encoding buffer, and the video frames stored in the encoding buffer are released from the circular queue. Afterwards, the address of the encoding buffer storing the video frame is sent to the multimedia hardware codec.

[0131] When the camera application displays a preview interface, the video frames processed by the algorithm post-processing node are sent to the camera application for preview, providing the user with the image in the preview interface.

[0132] The camera framework layer of the electronic device can send the address of the requested buffer to the corresponding module in the camera HAL layer through the interface of the camera service. For example, when the camera application turns on the dynamic photo function, the camera framework layer calls the multimedia hardware codec to apply for the encoding buffer, and the camera framework layer calls the camera service to apply for the picture buffer. After that, the camera framework layer sends the address of the requested picture buffer to the snapshot offline pipeline through the interface of the camera service, so that the snapshot offline pipeline uses the picture buffer to send the captured picture to the camera framework layer. The camera framework layer sends the address of the requested encoding buffer to the offline dynamic photo through the interface of the camera service, so that the offline dynamic photo uses the encoding buffer to send the captured video frame to the camera framework layer.

[0133] Finally, the multimedia hardware codec combines the pictures received through the picture buffer and the video frames received through the encoding buffer into a dynamic photo.

[0134] The camera driver layer may include a sensor driver and a hardware codec driver. The sensor driver includes information about the image sensor chip, and the sensor driver can drive the image sensor chip to work. The hardware codec driver includes information about the hardware codec, and the hardware codec driver can drive the hardware codec to work.

[0135] In order to explain the relationship between the camera driver and the hardware, Figure 4 The hardware layer is also shown, which includes hardware such as image sensor chips, hardware codecs, etc.

[0136] The drivers in the camera driver layer correspond one-to-one to the hardware. For example, the camera driver may include multiple drivers, each sensor driver corresponds to an image sensor chip, and each sensor driver corresponds to a different image sensor chip.

[0137] The following describes the image processing method provided in the embodiment of the present application in combination with the shooting scene of the mobile phone.

[0138] like Figure 5 As shown, in response to the user's operation on the camera application icon 501, the mobile phone starts running the camera application and displays the shooting interface. In some examples, the mobile phone starts the function of taking ordinary still photos by default, so it can display the following Figure 5 The common static photo shooting interface shown in the figure includes a viewfinder 502 and a shooting control 503. The viewfinder 502 displays a real-time preview image obtained by the image sensor chip.

[0139] In some embodiments, in response to the user's operation on the camera application icon 501, the camera application can perform data stream configuration (referred to as "stream allocation"). For example, in response to the user's operation on the camera application icon 501, the camera application can also access the node in the camera driver layer through the camera framework layer to obtain the capability information of each hardware related to still photo shooting in the hardware layer, such as the image size of still photos, the resolution and zoom supported by the sensor corresponding to still photos, etc. After the camera application obtains the capability information of the hardware related to still photo shooting, the camera application can allocate streams according to the capability information. In the case of shooting ordinary still photos, stream allocation refers to the process of configuring the preview stream and the data stream.

[0140] After the camera application starts running and completes the stream allocation, the images captured by the image sensor chip are divided into a preview stream and a data stream, and the mobile phone can display the preview image in the preview stream. In response to the operation of taking a photo, the mobile phone can obtain the image in the data stream as the captured picture. For example, when the camera application starts running and has the function of taking ordinary still photos, the camera application can control the image sensor chip to start acquiring images in real time. The camera application controls the image sensor chip through the camera framework layer, camera HAL layer, driver layer, etc. The specific process will not be repeated here. Figure 4 As shown in the figure, the IFE in the camera HAL layer can perform preliminary image processing on the image output by the image sensor chip. Afterwards, the image output by the IFE can be sent to the preview offline pipeline as a preview stream image, and can also be sent to the snapshot offline pipeline as a data stream image. The preview stream image sent to the preview offline pipeline is processed by the image processing engine and the algorithm post-processing node, and can be displayed as shown in the figure. Figure 5 In the viewfinder 502 shown in FIG. 5 , if the normal static photo shooting instruction sent by the camera application is not received, the snapshot offline pipeline does not process the data stream image. In response to the user's operation on the shooting control 503, the camera application sends a static photo shooting instruction to the camera HAL layer, and the snapshot offline pipeline processes the data stream image and sends the processed image to the camera application, and finally obtains a static photo stored in the electronic device, that is, the static photo is captured.

[0141] In the above process, the preview stream image is sent to the camera application through the preview stream buffer to be previewed and displayed in the mobile phone's viewfinder 502. In response to the user's operation on the shooting control 503, the data stream image is sent to the camera application through the picture buffer so that the camera application can obtain the captured static photo.

[0142] In some embodiments, during the operation of the camera application, the camera application needs to display image content on the display screen of the mobile phone, so the camera application can create a layer (surface). Afterwards, the camera application can store the content to be displayed in a buffer corresponding to the surface, so as to display the content in the buffer on the display screen of the mobile phone through the surface.

[0143] In the above-mentioned process of taking still photos, the camera application needs to display the preview stream of the still photos and the still photos taken. Therefore, the camera application can create two surfaces, one of which is a preview surface, which corresponds to the preview stream buffer, and the other is a picture surface, which corresponds to the picture buffer. Specifically, after creating the two surfaces, the camera application can apply for a preview stream buffer corresponding to the preview surface and a picture buffer corresponding to the picture surface through the camera framework layer. In addition, the camera application can send a preview request to the camera HAL layer. Afterwards, the camera HAL layer can store the picture output by the image sensor chip in the preview stream buffer, and the preview stream buffer is sent to the camera application. The camera application displays the image in the preview stream buffer in the viewfinder 502 through the preview surface. In response to the user's operation on the shooting control 503, the camera application sends a command to take a still photo through the camera framework layer, so that the camera HAL layer can store the image output by the image sensor chip at the time of the user's operation on the shooting control 503 into the image buffer. After that, the image buffer is sent to the camera application, and the camera application displays the image in the image buffer on the image surface such as Figure 5 The static photo thumbnail 504 is shown. In addition, the camera application can obtain and store the image in the picture buffer to obtain and store the captured static photo.

[0144] The above introduces the scenarios for taking static photos. The following introduces the scenarios for taking dynamic photos.

[0145] In some embodiments, the dynamic photo shooting function can be enabled in a variety of ways. Figure 6-Figure 8 Explains how to enable the dynamic photo shooting function.

[0146] In some examples, the dynamic photo shooting function can be a shooting mode enabled by default after the camera application is launched, such as Figure 6 As shown in FIG. 6 , in response to the user's operation on the camera application icon 601, the mobile phone starts running the camera application and displays the dynamic photo shooting interface. At this time, the mode selection area prompts that the current mode is the dynamic photo shooting mode, that is, the mode in which the dynamic photo shooting function is enabled. The dynamic photo shooting interface includes a viewfinder 602, a shooting control 603, and a mark 604 for indicating that the current interface is the dynamic photo shooting mode. The viewfinder 602 displays a real-time preview image obtained by the image sensor chip.

[0147] In some other examples, when the default shooting mode enabled after the camera application is started is not the dynamic photo shooting mode, the shooting mode can be switched to the dynamic photo shooting mode by means of mode switching. For example, the initial shooting mode is the ordinary static photo shooting mode. Figure 7 As shown in FIG. 1 , the mobile phone interface displays a shooting interface corresponding to the still photo shooting mode, and the shooting interface corresponding to the still photo shooting mode includes a viewfinder 502 and a shooting control 503. In addition, the mode selection area 701 for indicating the shooting mode in the shooting interface indicates that the current mode is the photo shooting mode, that is, the still photo shooting mode. In response to the user's operation in the mode selection area 701, the mobile phone can switch the current shooting mode. For example, Figure 7 As shown in FIG. 1 , the user can switch the shooting mode from the photo shooting mode to the dynamic photo shooting mode by sliding in the mode selection area 701, that is, from the static photo shooting mode to the dynamic photo shooting mode. After the shooting mode is switched, the mobile phone can display the following Figure 7 The dynamic photo shooting mode shown corresponds to the dynamic photo shooting mode. The mode selection area 701 used to indicate the shooting mode in the shooting interface prompts that the current mode is the dynamic mode, that is, the dynamic photo shooting mode. Figure 7 The dynamic photo shooting interface includes a viewfinder 602, a shooting control 603, and a mark 604 for indicating that the current interface is a dynamic photo shooting mode.

[0148] in, Figure 7 The shooting interface corresponding to the static photo shooting mode shown can be the shooting preview interface in the present application, and the mode selection area 701 can be the first control in the present application.

[0149] In some other examples, the mobile phone may include related controls for enabling the dynamic photo shooting function. For example, in the static photo shooting interface displayed by the mobile phone, a control for enabling the dynamic photo shooting function may be included. Figure 8 As shown, the static photo shooting interface displays a control 801 for enabling the dynamic photo shooting function and a shooting control 503. In response to the user clicking the control 801, the mobile phone can turn on the dynamic photo shooting function and display the following Figure 8 The dynamic photo shooting interface shown in the figure. Figure 8 The dynamic photo shooting interface shown includes a control 802 for turning off the dynamic photo shooting function, a viewfinder 803 of the dynamic photo shooting interface, and a shooting control 804. For another example, in the setting interface of the camera application, there may be related controls for enabling the dynamic photo shooting function, such as a switch control.

[0150] in, Figure 8 The static photo shooting interface shown may be the shooting preview interface in the present application, and the control 801 for enabling the dynamic photo shooting function may be the first control in the present application.

[0151] After the mobile phone displays the dynamic photo shooting interface, the user can take dynamic photos in the dynamic photo interface. Figure 8 The dynamic photo shooting interface shown in the figure is used as an example to explain the process of shooting dynamic photos. Fig. 9 As shown, the mobile phone displays a dynamic photo shooting interface. In response to the user's operation on the shooting control 804 in the dynamic photo shooting interface, the mobile phone can obtain dynamic photos and Fig. 9 The dynamic photo shooting interface shown in the figure is used to open the photo display interface control 805 to display the thumbnails of the dynamic photos.

[0152] The following, combined Fig.10 The flowchart shown specifically introduces the image processing process of taking dynamic photos with a mobile phone.

[0153] S1001. In response to a user's operation of enabling a dynamic photo shooting function, a camera application configures a multimedia hardware codec.

[0154] Among them, Figure 6 In the illustrated scenario, the user's operation of enabling the dynamic photo shooting function may be an operation on the icon 601 of the camera application. Figure 7 In the illustrated scenario, the user's operation of enabling the dynamic photo shooting function may be an operation of selecting the dynamic mode in the mode selection area 701. Figure 8In the illustrated scenario, the user's operation of enabling the dynamic photo shooting function may be an operation on the control 801 for enabling the dynamic photo shooting function. The specific scenario can be referred to in the above embodiment, which will not be described in detail here.

[0155] Dynamic photos include pictures and videos. Videos are obtained by encoding multiple video frames output by the image sensor chip. Therefore, in response to the user's operation of enabling the dynamic photo shooting function, the camera application needs to configure the multimedia hardware codec so that the video frames can be encoded using the configured multimedia hardware codec to obtain the video.

[0156] In some embodiments, when the camera application configures the multimedia hardware codec, the actual size of the video frame to be encoded, the video encoding format, and the number of required encoding buffers can be configured. Among them, since the video frames in the video of the dynamic photo are copied from the preview stream, the actual size of the video frame to be encoded is the same as the size of the image in the preview stream. For example, the size of the image in the preview stream is generally 1440*1080, so the size of the video frame to be encoded can be configured to be 1440*1080. The video encoding format can be one of MPEG, MPG, DAT, MP4, etc., and the video encoding format is not limited here. Since the video includes multiple video frames, in order to ensure the reading speed of the video frames, multiple encoding buffers can be applied. The number of encoding buffers can be predetermined according to the actual processing performance. For details, please refer to the relevant technology and will not be repeated here. Of course, you can also apply for a coding buffer.

[0157] In some embodiments, in response to a user's operation of enabling a dynamic photo shooting function, the camera application may select the resolution of the video frame of the dynamic photo to be taken and the resolution of the picture according to the camera preview ratio, and configure the multimedia hardware codec based on the selected resolution of the video frame of the dynamic photo to be taken, so that the size of the encoding buffer requested by the multimedia hardware codec corresponds to the resolution of the video frame.

[0158] For example, the mobile phone may store the corresponding relationship between the preview ratio and the optimal resolution. For example, the preview ratio of 4:3 corresponds to the optimal resolution of 1080P. Then, when the preview ratio corresponding to the current preview interface is 4:3, the camera application may determine that the optimal resolution corresponding to 4:3 is 1080P. In this way, the multimedia hardware codec is configured according to the resolution of 1080P, so that the size of the encoding buffer requested by the multimedia hardware codec is 1440*1080.

[0159] Optionally, if the mobile phone does not obtain a preview image before the user performs an operation to enable the dynamic photo shooting function, such as Figure 6 At this time, after executing S1001, the mobile phone may also execute the following steps S1002-S1010. If the mobile phone has obtained a preview image before the user executes the operation of enabling the dynamic photo shooting function, the mobile phone may not execute S1002-S1010 after executing S1001.

[0160] S1002: The camera application obtains data stream configuration information, and creates a preview surface and a picture surface based on the data stream configuration information.

[0161] The data stream configuration information may include the pixel width and pixel height of the preview image, the pixel width and pixel height of the picture in the dynamic photo, and may also include image format information.

[0162] S1003. The camera application sends the address of the preview surface and the address of the picture surface to the camera framework layer.

[0163] The preview surface is used to manage the preview buffer so that the mobile phone can display the preview image in the preview buffer. The picture surface is used to manage the picture buffer so that the mobile phone can display the picture in the picture buffer.

[0164] S1004: The camera application sends a stream configuration request to the camera framework layer based on the data stream configuration information.

[0165] S1005: The camera application sends a preview request to the camera HAL layer.

[0166] It should be noted that there is no specific restriction on the execution order of S1003, S1004 and S1005, and they may be executed simultaneously.

[0167] S1006: The camera HAL layer obtains the preview stream image and picture from the image sensor chip based on the stream allocation request.

[0168] S1007. The camera framework layer applies for a preview buffer based on the preview surface and applies for a picture buffer based on the picture surface.

[0169] S1008. The camera framework layer sends the addresses of the preview buffer and the image buffer to the camera HAL layer.

[0170] S1009 . In response to receiving the preview request, the camera HAL layer sends the address of the preview buffer storing the image output by the image sensor chip to the camera application in real time.

[0171] Specifically, the camera HAL layer can store the image output by the image sensor chip in the preview buffer, and then send the preview buffer storing the image to the camera application. For details, please refer to the above and Figure 5 Related embodiments.

[0172] S1010. The camera application displays the image in the preview buffer on the mobile phone screen.

[0173] At this point, the phone can display a dynamic photo shooting interface including a preview image.

[0174] In the above scenario, before the user starts the dynamic photo shooting function, the camera application is not in any other shooting mode. At this time, the camera application does not create any surface, such as Figure 6 The scene shown. Then, the camera application needs to create multiple surfaces corresponding to the dynamic photo shooting function. For example, the surfaces corresponding to the dynamic photo shooting function generally include a preview surface, a picture surface, and an encoding surface. The creation of the preview surface and the picture surface can be achieved through the above S1002-S1010.

[0175] S1011. The camera application creates an encoding surface for managing video content in dynamic photos.

[0176] In some embodiments, the camera application can call the configured multimedia hardware codec to create the encoding surface.

[0177] S1012: The camera application sends the address of the encoding surface to the camera framework layer.

[0178] Among them, the multimedia hardware codec is a software interface. After the developer completes writing the multimedia hardware codec, the multimedia hardware codec has set the memory alignment format of the coding buffer it applied for. Therefore, the size of the coding buffer actually corresponding to the coding surface created directly by the multimedia hardware codec may not correspond to the size of the video frame that the hardware codec can encode. Therefore, the mobile phone can execute the following steps S1013-S1016, so that the size of the coding buffer actually corresponding to the coding surface created by calling the multimedia hardware codec can correspond to the size of the video frame that the hardware codec can encode, so that based on the coding surface, a coding buffer with the same size as the size of the video frame that the hardware codec can encode can be applied.

[0179] S1013. The camera framework layer sends a dynamic photo service initialization request to the camera HAL layer, and sends a callback function to the camera HAL layer.

[0180] The dynamic photo service initialization request is used to enable the camera HAL layer to initialize the dynamic photo service. The callback function is used to send the encoded image size calculated by the camera HAL layer based on the encoding capability of the hardware codec back to the camera framework layer.

[0181] The camera HAL layer can implement the initialization of the dynamic photo service through the following S1014:

[0182] S1014: The camera HAL layer creates a dynamic photo cache queue and calculates the encoded image size according to the size of the preview buffer and the encoding capability of the hardware codec.

[0183] In some embodiments, the dynamic photo buffer queue can be used to copy and store the preview image from the dynamic picture copy node in the preview offline pipeline. Therefore, the video frame in the dynamic photo cache queue is the same as the preview image, and therefore, the video frame size is the same as the preview image size. According to the size of the preview buffer and the encoding capability of the hardware codec, the encoded image size can be calculated.

[0184] In some examples, when the size of the preview buffer is 1440*1080, that is, the actual size of the video frame is 1440*1080, and the memory alignment format required by the hardware codec is 512*512 alignment, it can be determined that the size of the video frame that the hardware codec can encode is 1536*1536, that is, the size of the encoded image is 1536*1536. Among them, after alignment according to the encoding that the hardware codec can obtain, it is necessary to ensure that the pixel width of the aligned image is an integer multiple of 512. First calculate 1440 / 512=2.8125, and round the obtained 2.8125 to 3. The 1536 calculated according to 3*512 is the pixel width after the alignment requirement. It is necessary to ensure that the pixel width of the aligned image is an integer multiple of 512. First calculate 1080 / 512≈2.1, and round the obtained 2.1 to 3. The 1536 calculated according to 3*512 is the pixel width after the alignment requirement.

[0185] The encoding capability of the hardware codec may be the encoding format of the hardware codec in the present application, and the encoded image size may be the buffer size in the present application.

[0186] S1015. The camera framework layer obtains the encoded image size based on the callback function, and calls the multimedia codec to apply for the encoding buffer based on the obtained encoded image size.

[0187] The camera framework layer can call the interface for applying for a coding buffer and apply for a coding buffer with the same size as the encoded image.

[0188] At this time, the size of the encoding buffer corresponds to the encoding capability of the hardware codec. Therefore, the hardware codec can subsequently encode the video frames in the encoding buffer, thereby avoiding the situation where the hardware codec cannot encode due to the video frames in the encoding buffer being too large, or avoiding the waste of encoding resources of the hardware codec due to the video frames in the encoding buffer being too small.

[0189] Among them, the encoding buffer in S1015 can be the first encoding buffer in the embodiment of the present application.

[0190] S1016. The camera framework layer establishes a correspondence between the requested encoding buffer and the encoding surface.

[0191] After the camera framework layer establishes a correspondence between the requested encoding buffer and the encoding surface, it can ensure that the size of the video frame in the encoding buffer is the same as the size of the video frame that can be encoded by the hardware codec, so that the hardware codec can normally encode the video frames stored in the encoding buffer.

[0192] After completing S1016, the mobile phone can provide dynamic photo shooting function.

[0193] In some embodiments, Fig. 9 As shown, in response to the user's operation on the shooting control 804, the camera application can send a dynamic photo shooting instruction, and the mobile phone can execute the following steps (S1017-S1021).

[0194] S1017. The camera framework layer sends the address of the encoding buffer to the camera HAL layer.

[0195] S1018. The camera HAL layer stores the video frames in the dynamic photo cache queue into the encoding buffer, and stores the images in the snapshot offline pipeline at the moment when the user operates the shooting control into the picture buffer.

[0196] The size of the encoding buffer is the same as the size of the video frame that can be encoded by the hardware codec.

[0197] The snapshot offline pipeline camera HAL layer is created during the flow distribution process. For details, please refer to the relevant technology and will not be repeated here.

[0198] S1019. The camera HAL layer sends the address of the encoding buffer storing the video frame and the picture buffer storing the image to the camera application.

[0199] In some embodiments, the encoding buffer can rotate between the camera HAL layer and the camera application. For example, after the address of the encoding buffer is sent to the camera HAL layer, the video frames in the dynamic photo cache queue are stored in the encoding buffer, and after the address of the encoding buffer storing the video frames is sent to the camera application, the camera application reads the video frames from the encoding buffer storing the video frames. Afterwards, the camera application again sends the address of the encoding buffer whose content is empty after the reading is completed to the camera HAL layer to ensure that multiple video frames in the dynamic photo cache queue are sent to the camera application. Therefore, the encoding buffer here is also called a rotating buffer.

[0200] In some embodiments, the number of times the camera HAL layer sends the encoding buffer address storing video frames to the camera application is related to the duration of the video in the dynamic photo. For example, when the total duration of the video in the dynamic photo is 3s and the FPS is 30, the number of times the camera HAL layer sends the encoding buffer address storing video frames to the camera application is 90 times, that is, the number of video frames used for video encoding is guaranteed to be 90 frames, so that a video with an FPS of 30 and a total duration of 3s can be obtained.

[0201] After the camera HAL layer sends all video frames required for encoding to the camera application through the encoding buffer, S1020 can be executed:

[0202] S1020: The camera application calls the multimedia hardware codec to perform hardware encoding on the image in the encoding buffer to obtain a video.

[0203] Since the camera framework layer establishes a corresponding relationship between the requested encoding buffer and the encoding surface in S1016, ensuring that the camera application matches the encoding capability of the hardware codec based on the video frame size in the encoding buffer, the camera application can call the multimedia hardware codec to perform normal hardware encoding on the image in the encoding buffer.

[0204] S1021. The camera application reads a picture from the picture buffer, and combines the picture with the video into a dynamic photo.

[0205] At this point, the mobile phone takes dynamic photos.

[0206] In the above embodiments, when the user turns off the dynamic photo function, or when the user switches the dynamic photo shooting mode to other non-dynamic photo shooting modes, or when the user closes the camera application, or when the user switches the camera application to background operation, the mobile phone can release the encoding buffer.

[0207] When the user enables the dynamic photo function again, the above steps S1001-S1021 are re-executed.

[0208] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step in the above-mentioned method embodiment.

[0209] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program runs on an electronic device, the electronic device executes each function or step in the above method embodiment.

[0210] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0215] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the electronic equipment comprises a camera framework layer and a camera hardware abstraction layer; The method comprises: Displaying a shooting preview interface of a camera application, wherein the shooting preview interface includes a first control and a shooting control; the first control is used to enable a dynamic photo shooting function of the camera application; In response to a triggering operation of the first control by a user, sending a callback function to the camera hardware abstraction layer through the camera framework layer; Determine, through the camera hardware abstraction layer, a buffer size according to a size of a preview stream and an encoding format of a hardware codec of the electronic device, and send, through the camera hardware abstraction layer, the buffer size to the camera framework layer based on the callback function; Applying, through the camera framework layer, for a first encoding buffer according to the buffer size; the size of the first encoding buffer matches the encoding format of the hardware codec, the first encoding buffer is used to store a plurality of video frames, and the plurality of video frames are obtained based on the preview stream; In response to a user triggering an operation on the shooting control, a dynamic photo is stored; the dynamic photo includes a video and a picture, the video is obtained by calling the hardware codec to encode multiple video frames stored in the first encoding buffer, and the picture is a picture taken in response to the user triggering an operation on the shooting control.

2. The method according to claim 1, characterized in that In response to a trigger operation of the user on the first control, the method further includes: Calling a software interface to create a layer, the size of the layer being a preset size; Bind the first encoding buffer to the layer.

3. The method according to claim 2, characterized in that The electronic device also includes a camera application layer; The calling of the software interface to create the layer includes: the camera application in the camera application layer calls the software interface through the camera framework layer to create the layer; The method further comprises: the camera application layer sending the address of the layer to the camera framework layer; The binding the first encoding buffer and the layer includes: the camera framework layer binding the address of the first encoding buffer and the address of the layer.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: storing a plurality of video frames in the preview stream in the first encoding buffer in sequence; the number of the plurality of video frames corresponds to the total duration of the video; The hardware codec is used to encode all video frames sequentially stored in the first encoding buffer to obtain a video.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to a user triggering operation on a second control in the shooting preview interface, or in response to a user exiting the camera application, the first encoding buffer is released; the second control is used to turn off a dynamic photo shooting function of the camera application.

6. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the electronic device is caused to perform the method according to any one of claims 1 to 5.

Images