Image processing method and related device

Abstract

The image processing method and related device provided by the embodiments of the present application relate to the field of terminal technology. The method includes: in the preview process of the application framework layer, after the preview frame used for image preview is processed by the image optimization algorithm, the preview frame can be cached as a preview frame for generating dynamic photos. The camera application can generate dynamic photos based on the preview frame for generating dynamic photos cached in the application framework. In this way, the camera application can generate dynamic photos based on the preview frame processed by the image optimization algorithm, thereby improving the quality of the generated dynamic photos and enhancing the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to image processing methods and related devices. Background Art

[0002] Some electronic devices, such as camera apps, offer dynamic photo functionality. This feature captures the image before and after the moment it was taken, creating an image that isn't just static but instead displays some dynamic effects. This allows users to capture moments that would be difficult to capture with a single static photo.

[0003] However, the image quality of dynamic photos generated by some applications is poor, which affects the user experience. Summary of the Invention

[0004] The image processing method and related apparatus provided by the embodiments of the present application utilize, in the preview process at the application framework layer, a preview frame used for image previewing that has been processed by an image optimization algorithm can be cached as a preview frame for generating dynamic photos. The camera application can then generate dynamic photos based on the preview frame cached in the application framework for generating dynamic photos. This allows the camera application to generate dynamic photos based on the preview frame processed by the image optimization algorithm, thereby improving the quality of the generated dynamic photos and enhancing the user experience.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, applied to an electronic device, the method comprising:

[0006] In response to a first operation that triggers the opening of a target application, image processing is performed on the captured image in the first data stream to obtain first image data, which is used for image preview. In response to a second operation that triggers a photo, image processing is performed on the target image in the second data stream to obtain second image data. The target image is the image in the first data stream at the time corresponding to the second operation, and the second image data includes the image obtained when the second operation is triggered. A dynamic photo is generated based on the first image data and the second image data. In this way, in the first data stream, using preview frames processed by an image optimization algorithm to generate dynamic photos can improve the quality of the generated dynamic photos and enhance the user experience.

[0007] In one possible implementation, the first data stream and the second data stream are both set at the application framework layer, and the method further includes: saving the first image data to a preset cache based on the first data stream; generating a dynamic photo based on the first image data and the second image data, including: obtaining the first image data from the preset cache based on the target application, and generating a dynamic photo based on the first image data and the second image data. In this way, the first data stream and the second data stream are both set at the application framework layer, and in the first data stream, after the preview frame used for image preview is processed by the image optimization algorithm, the preview frame can be cached as a preview frame for generating a dynamic photo, and the target application can generate a dynamic photo based on the cached preview frame for generating a dynamic photo. In this way, the target application can generate a dynamic photo based on the preview frame processed by the image optimization algorithm, thereby improving the quality of the generated dynamic photo.

[0008] In one possible implementation, before obtaining the first image data in the preset cache, it also includes: establishing an Android Interface Definition Language AIDL communication connection based on the target application and the application framework layer; obtaining the first image data from the preset cache based on the target application, and generating a dynamic photo based on the first image data and the second image data, including: obtaining the first image data from the preset cache based on the target application, and synthesizing a video file based on the first image data and the collected audio data; generating a dynamic photo based on the video file and the second image data. In this way, by synthesizing a video file based on the first image data and the collected audio data, the audio data and the image data can be time-aligned by the timestamp, so that the audio data stream and the video data stream are kept synchronized during playback or processing. The dynamic photo generated based on the video file and the second image data can capture the picture within a period of time before and after the image is taken, so that the generated image is not only static, but can show some dynamic effects. In this way, some wonderful moments that are difficult to capture with a single static photo can be captured.

[0009] In one possible implementation, before performing image processing on the captured image in the first data stream, the following further comprises: based on the target application and the camera service of the application framework layer, transmitting a first request to the hardware abstraction layer, the first request being used to instruct the hardware abstraction layer to initialize a preview stream; initializing the preview stream based on the hardware abstraction layer and returning the image captured by the camera to the camera service via the preview stream; and based on the camera service, transmitting the captured image to the first data stream. In this way, the first request can carry the camera parameters set by the target application. Based on this first request, the hardware abstraction layer can instruct the camera driver to set the corresponding camera parameters, thereby obtaining an image that meets the user's requirements.

[0010] In one possible implementation, before the second operation, the following is further included: in response to a third operation that triggers turning on the dynamic photo function, based on the target application and the camera service, a second request is transmitted to the hardware abstraction layer, the second request is used to instruct the hardware abstraction layer to reconfigure the preview stream, the second request includes a first identifier, the first identifier is used to indicate whether the dynamic photo function is turned on; based on the first identifier, it is determined that the hardware abstraction layer does not reconfigure the preview stream. In this way, using the value of the scene mode to identify whether the dynamic photo function is turned on facilitates the determination of the status of the dynamic photo. The hardware abstraction layer can determine which mode or function has changed based on the value of the scene mode, simplifying the implementation process of the code in the electronic device, improving the readability and flexibility of the code, and thus being able to concisely indicate the on and off status of the dynamic photo.

[0011] In one possible implementation, determining that the hardware abstraction layer does not reconfigure the preview stream based on the first identifier includes: determining that the hardware abstraction layer does not reconfigure the preview stream if the first identifier indicates that the Live Photo feature is enabled and the hardware abstraction layer records that the first identifier last indicated that the Live Photo feature was disabled. Thus, by not reconfiguring the preview stream when the Live Photo feature is enabled, the hardware abstraction layer can reduce instances of preview frame failure, reduce interface lag, and thus improve the user experience.

[0012] In one possible implementation, based on the first identifier, determining that the hardware abstraction layer does not reconfigure the preview stream includes: when the first identifier indicates that the dynamic photo function is turned on, the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned off, and when the dynamic photo function is turned on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream, determining that the hardware abstraction layer does not reconfigure the preview stream. In this way, the electronic device may include one or more ways to implement dynamic photos, and there will be some differences between different implementation methods. For example, the image processing method of the embodiment of the present application is used to implement the dynamic photo function. When the dynamic photo function is turned on, the hardware abstraction layer may not reconfigure the preview stream, thereby reducing the interface jams. Therefore, by judging whether the electronic device supports the hardware abstraction layer not reconfiguring the preview stream, the preview stream can be flexibly configured according to different implementation processes, thereby improving the flexibility of code execution.

[0013] In one possible implementation, before determining that the hardware abstraction layer does not reconfigure the preview stream, the method further includes: returning a second flag to the camera service based on the hardware abstraction layer, where the second flag is used to instruct the camera service not to trigger the process of reconfiguring the preview stream. Using the second flag to instruct the camera service not to trigger the process of reconfiguring the preview stream simplifies the code implementation process in the electronic device, improving code readability and flexibility, thereby allowing the corresponding execution process to be concisely represented through field flags.

[0014] In one possible implementation, the method further includes: in response to a fourth operation that triggers the disabling of the dynamic photo function, based on the target application and the camera service, transmitting a third request to the hardware abstraction layer, the third request being used to instruct the hardware abstraction layer to reconfigure the preview stream, the third request including a first identifier; and if the first identifier indicates that the dynamic photo function is off, and the hardware abstraction layer records that the last time the first identifier indicated that the dynamic photo function was on, determining that the hardware abstraction layer does not reconfigure the preview stream. In this way, when the dynamic photo function is off, the hardware abstraction layer does not reconfigure the preview stream, which can reduce the situation where no preview frames are output, reduce interface freezes, and thus improve the user experience.

[0015] In one possible implementation, when the first identifier indicates that the dynamic photo function is off and the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is on, determining that the hardware abstraction layer does not reconfigure the preview stream also includes: when the first identifier indicates that the dynamic photo function is off, the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is on, and when the dynamic photo function is on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream, determining that the hardware abstraction layer does not reconfigure the preview stream. In this way, the electronic device may include one or more ways to implement dynamic photos, and there will be some differences between different implementation methods. For example, the image processing method of the embodiment of the present application is used to implement the dynamic photo function. When the dynamic photo function is off, the hardware abstraction layer may not reconfigure the preview stream, thereby reducing the interface jamming. Therefore, by judging whether the electronic device supports the hardware abstraction layer not to reconfigure the preview stream, the preview stream can be flexibly configured according to different implementation processes, thereby improving the flexibility of code execution.

[0016] In a second aspect, embodiments of the present application provide an image processing device, which may be an electronic device or a chip or system-on-chip within an electronic device. The device may include a processing unit. The processing unit is configured to implement any processing-related method performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is configured to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or system-on-chip within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit within the electronic device located external to the chip (e.g., a read-only memory, random access memory, etc.).

[0017] Exemplarily, the processing unit is used to perform image processing on the captured image in the first data stream to obtain first image data; is also used to perform image processing on the target image in the second data stream to obtain second image data; and is also used to generate dynamic photos based on the first image data and the second image data.

[0018] In one possible implementation, the processing unit is further used to save the first image data to a preset cache based on the first data stream; specifically, it is further used to obtain the first image data from the preset cache based on the target application, and generate a dynamic photo based on the first image data and the second image data.

[0019] In one possible implementation, the processing unit is also used to establish an Android interface definition language AIDL communication connection based on the target application and the application framework layer; specifically, it is also used to obtain first image data from a preset cache based on the target application, and synthesize a video file based on the first image data and the collected audio data; and generate a dynamic photo based on the video file and the second image data.

[0020] In one possible implementation, the processing unit is further used to transmit a first request to the hardware abstraction layer based on the target application and the camera service of the application framework layer, where the first request is used to instruct the hardware abstraction layer to initialize the preview stream; it is also used to initialize the preview stream based on the hardware abstraction layer and return the image captured by the camera to the camera service through the preview stream; it is also used to transmit the captured image to the first data stream based on the camera service.

[0021] In a possible implementation, the processing unit is further configured to transmit a second request to the hardware abstraction layer based on the target application and the camera service; and is further configured to determine, based on the first identifier, that the hardware abstraction layer does not reconfigure the preview stream.

[0022] In one possible implementation, the processing unit is further configured to determine that the hardware abstraction layer does not reconfigure the preview stream when the first identifier indicates that the dynamic photo function is turned on and the hardware abstraction layer records that the last time the first identifier indicated that the dynamic photo function was turned off.

[0023] In one possible implementation, the processing unit is further used to determine that the hardware abstraction layer does not reconfigure the preview stream when the first identifier indicates that the dynamic photo function is turned on, the hardware abstraction layer records the last time that the first identifier indicates that the dynamic photo function is turned off, and when the dynamic photo function is turned on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream.

[0024] In a possible implementation, the processing unit is further configured to return a second identifier to the camera service based on the hardware abstraction layer.

[0025] In one possible implementation, the processing unit is further used to pass a third request to the hardware abstraction layer based on the target application and the camera service, and is further used to determine that the hardware abstraction layer does not reconfigure the preview stream when the first identifier indicates that the dynamic photo function is turned off and the last time the first identifier recorded by the hardware abstraction layer indicated that the dynamic photo function is turned on.

[0026] In one possible implementation, the processing unit is further used to determine that the hardware abstraction layer does not reconfigure the preview stream when the first identifier indicates that the dynamic photo function is turned off, the hardware abstraction layer records the last time that the first identifier indicates that the dynamic photo function is turned on, and when the dynamic photo function is turned on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream.

[0027] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors and a memory, the memory being coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, and the one or more processors being used to call computer instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when the computer program runs on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is configured to execute a computer program or instruction to perform the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0031] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, wherein instructions are stored in the at least one memory. The memory may be a storage unit within the chip, such as a register or cache, or a storage unit of the chip (such as a read-only memory or random access memory).

[0032] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a camera preview interface provided in an embodiment of the present application;

[0034] Figure 2 More schematic diagrams of interfaces of a camera provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a method for implementing a dynamic photo function provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of module interaction for a dynamic photo function provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the execution flow of a dynamic photo function provided in an embodiment of the present application;

[0040] Figure 8A timing diagram of module calls when the dynamic photo function is turned on, provided in an embodiment of the present application;

[0041] Figure 9 A timing diagram of module calls when the dynamic photo function is turned off, provided in an embodiment of the present application;

[0042] Figure 10 A timing diagram of module calls when another dynamic photo function provided in an embodiment of the present application is turned on;

[0043] Figure 11 A timing diagram of module calls when another dynamic photo function is turned off provided in an embodiment of the present application;

[0044] Figure 12 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0045] Figure 13 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0047] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0048] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0049] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or plural.

[0050] Some applications in electronic devices can provide functions such as preview, photo taking, and video recording. Taking the camera application as an example, users can take photos or videos through the camera application.

[0051] For example, Figure 1 The camera application's photo preview interface 100 is shown. Users can preview images and take photos or videos on this interface 100. In some scenarios, this photo preview interface 100 may also be referred to as a preview interface or a capture interface. The photo preview interface 100 may include a toolbar 101, a preview window 102, camera mode options 103, an album shortcut control 104, a capture control 105, and a camera flip control 106.

[0052] The toolbar 101 may be used to display one or more functional controls, which may include a flash control, a dynamic photo control 107 , a short video control, and a setting control, etc. It is understandable that the toolbar 101 may also include more or fewer functional controls.

[0053] The preview window 102 can be used to obtain the image to be previewed and display the preview image in real time. In some scenarios, the preview window 102 can also be called a viewfinder.

[0054] The camera mode option 103 may be used to display one or more shooting mode options. The one or more shooting mode options may include: a night scene mode option, a portrait mode option, a photo mode option, a video mode option, and more options 108. It is understood that the camera mode option 103 may also include more or fewer shooting mode options.

[0055] The album quick control 104 can display thumbnails of captured images or videos. In addition, in response to a user triggering the album quick control 104, the electronic device can launch an album application and display the captured images and videos. It is understood that the user operations mentioned in the embodiments of the present application may include, but are not limited to, touch (e.g., click), voice control, gestures, and other operations, and the embodiments of the present application are not limited thereto.

[0056] The shooting control 105 can be used to control the electronic device to shoot. The camera flip control 106 can be used to switch the camera, thereby controlling the electronic device to use the front camera or the rear camera to shoot images.

[0057] It is understandable that the photo preview interface 100 is only a schematic interface, and the actual interface may include more or less content and controls; the icon of the dynamic photo control 107 can also be set to other styles, which is not limited in the embodiment of the present application.

[0058] When the user is previewing or taking a photo, he or she can choose to turn on the dynamic photo function. The dynamic photo function can capture the picture within a period of time before and after the image is taken, so that the generated image is not only static, but can show some dynamic effects. In this way, some wonderful moments that are difficult to capture with a single static photo can be captured. For example, the dynamic photo function can save the captured image and 1.5 seconds of video before and after the shooting moment, a total of 3 seconds of video, into a file to generate a dynamic photo. Among them, 1.5 seconds is only an exemplary value, and the specific length of the video before and after the shooting moment is not limited in the embodiments of the present application. The images described in the embodiments of the present application can also be understood as pictures or photos.

[0059] For example, the user may trigger the operation of the dynamic photo control 107 in the photo preview interface 100 , and in response to the triggering operation, the camera application may turn on the dynamic photo function.

[0060] Alternatively, the user may trigger the operation of more options 108 in the photo preview interface 100, and in response to the trigger operation, such as Figure 2 As shown, the camera application can display a more interface 200, which can include a dynamic photo control 201. The user can trigger the operation of the dynamic photo control 201, and in response to the triggering operation, the camera application can turn on the dynamic photo function.

[0061] It is understood that the above are two ways to enable the dynamic photo function, and the camera application may also include other ways to enable the dynamic photo function. For example, in the photo preview interface 100, the dynamic photo control 107 can be located in other locations of the interface 100, such as in the camera mode option 103, or in any other possible location in the interface 100, and this embodiment of the application is not limited thereto.

[0062] In some implementations, such as Figure 3 As shown, electronic devices can use Figure 3 The architecture shown is used to implement the dynamic photo function.

[0063] When the dynamic photo feature is enabled, the hardware abstraction layer of the electronic device can start the dynamic photo path during the configuration of the preview stream. The preview stream can be used to process and cache preview frames for image preview, and the dynamic photo path can be used to process and cache preview frames for generating dynamic photos.

[0064] It is understood that the preview frames cached in the preview stream and the preview frames cached in the dynamic photo path can be image frames in different caches. The resolution of the preview frames cached in the preview stream and the resolution of the preview frames cached in the dynamic photo path can be the same or different. After the camera application obtains the preview frames cached in the preview stream, it can display them in the preview window, for example, Figure 1 The preview window 102 is displayed. After the camera application obtains the preview frame cached by the dynamic photo path, it can be used to generate a dynamic photo.

[0065] For example, after acquiring image frames captured by the camera, the hardware abstraction layer can pass the captured image frames to the dynamic photo path as preview frames corresponding to the dynamic photo. The hardware abstraction layer can also pass the image at the time of capture to the photo algorithm path. In some implementations, this photo algorithm path can also be referred to as a photo algorithm link.

[0066] In the dynamic photo pipeline, the image front end (IFE) preprocesses preview frames, including color correction, downsampling, and 3A statistics (auto exposure, auto focus, and auto white balance). It then passes the preprocessed preview frames to the image processing engine (IPE). The IPE can perform image processing on the preview frames, including cropping, noise reduction, and detail enhancement.

[0067] After the image processing engine processes the image, it can cache the preview frame of the dynamic photo and return it to the camera application, or the camera application can retrieve it from the cache. Furthermore, the image processing engine can pass the preview frame of the dynamic photo to the scene semantic detection module for scene semantic detection, which then returns the detection results to the camera application. Based on this detection result, the camera application can add attributes such as tags to the dynamic photo to facilitate user viewing.

[0068] The photo algorithm pipeline performs image processing such as cropping, noise reduction, and detail enhancement to generate an image at the moment of capture that can be displayed and saved, and returns this image to the camera application. In some scenarios, the image generated by the photo algorithm pipeline at the moment of capture can be called the large image.

[0069] Furthermore, the camera application can generate a dynamic photo based on the preview frame and the large image of the dynamic photo.

[0070] However, in some scenarios, it is necessary to perform image optimization algorithm processing on the preview frame and large image of dynamic photos. For example, image optimization algorithm processing includes image beauty algorithm, filter algorithm and other processing, and electronic devices usually perform these image optimization algorithm processing in the application framework layer. Figure 3 The corresponding architecture caches preview frames of dynamic photos in the hardware abstraction layer and passes them to the application layer. However, the preview frames obtained by the application layer are not processed by the image optimization algorithm, resulting in relatively poor quality of the generated dynamic photos compared to the preview images. Adding image optimization algorithm processing to the preview frames in the dynamic photo path of the hardware abstraction layer alone would increase the computing power of the electronic device and reduce its performance.

[0071] In view of this, the image processing method provided in the embodiments of the present application, in the preview process at the application framework layer, after the preview frame used for image preview is processed by the image optimization algorithm, the preview frame can be cached as a preview frame for generating dynamic photos. The camera application can generate dynamic photos based on the preview frame for generating dynamic photos cached in the application framework. In this way, the camera application can generate dynamic photos based on the preview frame processed by the image optimization algorithm, thereby improving the quality of dynamic photos and enhancing the user experience.

[0072] The electronic device of the embodiment of the present application may also be a terminal device in any form. For example, the electronic device may include: a mobile phone, a tablet computer, a PDA, a laptop computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a wearable device, an electronic device in a 5G network or a future evolved public land mobile communication network. The electronic devices in the mobile network (PLMN) are not limited to this in the embodiments of the present application.

[0073] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0074] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in the Internet of Things (IoT) system. The IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0075] The electronic device in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0076] In the embodiments of the present application, electronic devices or network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as the Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0077] For example, Figure 4 A schematic structural diagram of an electronic device is shown.

[0078] The electronic device may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, an antenna 2, a mobile communication module 450, a wireless communication module 460, an audio module 470, a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, a sensor module 480, a button 490, a motor 491, an indicator 492, a camera 493, a display screen 494, and a subscriber identification module (SIM) card interface 495, etc. The sensor module 480 may include a pressure sensor 480A, a gyroscope sensor 480B, an air pressure sensor 480C, a magnetic sensor 480D, an acceleration sensor 480E, a distance sensor 480F, a proximity light sensor 480G, a fingerprint sensor 480H, a temperature sensor 480J, a touch sensor 480K, an ambient light sensor 480L, a bone conduction sensor 480M, etc.

[0079] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] Processor 410 may include one or more processing units. For example, processor 410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0081] A memory may also be provided in the processor 410 for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that have just been used or are recycled by the processor 410. If the processor 410 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. This avoids repeated accesses and reduces the waiting time of the processor 410, thereby improving the efficiency of the system. For example, in an embodiment of the present application, the processor 410 may be used to process a camera application's photo request, and may also be used to process an image using an optimization algorithm to generate dynamic photos, etc.

[0082] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0083] The internal memory 421 can be used to store computer executable program code, and the executable program code includes instructions. The internal memory 421 may include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 410 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 421 and / or instructions stored in a memory provided in the processor. For example, in an embodiment of the present application, the internal memory 421 can be used to store preview frames of dynamic photos and captured images, and can also be used to store relevant codes for implementing dynamic photo functions, etc.

[0084] The electronic device implements display functionality through a GPU, display screen 494, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs that execute program instructions to generate or modify display information. The electronic device can implement camera functionality through an ISP, camera 493, video codec, GPU, display screen 494, and application processor.

[0085] In some embodiments, the electronic device may include 1 or N cameras 493 , where N is a positive integer greater than 1. The camera 493 may be used to capture still images or videos.

[0086] Display screen 494 is used to display images, videos, etc. Display screen 494 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0087] Figure 5 This is a schematic diagram of the software structure of an electronic device according to an embodiment of the present application. A layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into five layers: the application layer, the application framework layer, the Android runtime and system libraries, the hardware adaptation layer (HAL), and the kernel layer.

[0088] The application layer can also be called the application layer, which can include a series of application packages. Figure 5 As shown, the application package may include applications such as camera application, gallery application, phone application, etc. Applications may include system applications and third-party applications.

[0089] In an embodiment of the present application, the camera application may include a camera capture session module (CameraCaptureSession), a camera device module (CameraDevice), and a dynamic photo mode implementation module, etc.

[0090] The camera capture session module manages and controls camera capture sessions, such as creating sessions, setting single or repeated capture requests, and managing the allocation and release of camera resources. Single capture requests are used for photo taking, while repeated capture requests are used for previewing or recording videos.

[0091] The camera device module in the application layer provides applications with interfaces for starting and stopping camera preview, photo taking, and video recording, thereby enabling the parameters passed by the application to be passed to the camera service (CameraService) in the application framework layer. The camera device module also returns status changes of the camera hardware device, such as opening, closing, and disconnecting, to the application via a callback interface. For example, in embodiments of the present application, the camera capture session module and the camera device module can be used to manage and control different capture sessions, including configuring capture requests and processing capture results.

[0092] The dynamic photo mode implementation module can be used to generate dynamic photo files. The dynamic photo mode implementation module may include an advanced audio coding (AAC) queue, an advanced video coding (AVC) queue, a mixing module (muxer), a session service host module (ServiceHostGlobalSession), and a dynamic photo service host module (ServiceHostLivePhoto).

[0093] The AAC queue can be used to cache audio data collected by the electronic device and the corresponding timestamps, while the AVC queue can be used to cache image frames and the corresponding timestamps.

[0094] The mixing module can be used to manage and synchronize the audio data stream and the video data stream, so that the audio data stream and the video data stream can be kept synchronized during playback or processing.

[0095] The session service host module can be used to receive images delivered by the write module (write) in the post-processing camera algorithm module (ServiceHost) and generate dynamic photo files based on these images. In this embodiment of the application, the image delivered by the write module can also be referred to as a large image, which can be understood as the image generated by the ServiceHost module after the user triggers the photo operation.

[0096] The Live Photo service host module manages the service logic for Live Photos, such as storage, processing, and transmission. Based on the Android Interface Definition Language (AIDL) interface, the Live Photo service host module establishes an inter-process communication connection with the Live Photo Manager Proxy module of the ServiceHost module, enabling data exchange between the camera application and the hardware algorithm (HwAlgo).

[0097] The application framework layer, also known as the Framework layer, provides an application programming interface (API) and programming framework for applications in the application layer. The Framework layer can include some predefined functions.

[0098] like Figure 5As shown, the Framework layer may include an activity manager, a window manager, a resource manager, a notification manager, a content provider, a view system, etc. For details, please refer to the description of the relevant technology and will not be repeated here.

[0099] In an embodiment of the present application, the Framework layer may further include a camera service (CameraService) and a post-processing camera algorithm module (ServiceHost).

[0100] A camera service is responsible for communicating with the device's camera hardware. For example, it can interact with the camera through the hardware abstraction layer to implement functions such as opening and closing the camera, capturing images, and so on. A camera service also provides APIs for accessing camera functions. By calling these APIs, applications can control various camera functions, such as starting a preview, capturing photos, recording videos, adjusting focus, and setting exposure.

[0101] The camera service can include the camera device client module (CameraDeviceClient), the camera device module (Camera3Device), and the camera output stream module (Camera3OutputStream).

[0102] The camera device client module can be used to provide an interface for the camera application to interact with the camera hardware. For example, the camera application can access the camera function through the interface of the camera device client module to capture images.

[0103] The camera device module in the application framework layer can be used to manage and control the specific implementation of camera hardware. For example, the camera device module can provide an interface for the camera application to call, thereby enabling communication between the upper-layer application and the hardware abstraction layer. The camera device module can also handle image capture requests, such as configuring capture parameters, initiating the capture process, and processing capture results.

[0104] The camera output stream module manages and processes the output of image frames captured by the camera. For example, the camera output stream can configure and control image frame parameters such as resolution, format, and frame rate. The camera output stream can also pass image frames captured by the camera to the data source module (FilterCameraSource) in the ServiceHost module.

[0105] The ServiceHost module can include a preview session (normalSession), a photo session (ppsession), a dynamic photo management proxy module (LivePhotoManagerProxy), and a dynamic photo management module (LivePhotoManager).

[0106] A preview session handles preview-related processes. For example, it processes each frame of the preview scene in real time and returns the results to the application for display. A preview session can include a data source module (FilterCameraSource), a preview algorithm module, a forking module (Forklpp), and a preview module (preview).

[0107] The data source module in the preview session receives image frames from the camera output stream module and transmits them to the photo data source module (FilterPPSource) in the photo session. The preview algorithm module applies image optimization algorithms to the image frames, such as applying beauty enhancement and filters. The stream diversion module copies the image frames and transmits them to the dynamic photo management module. The preview module transmits the image processed by the image optimization algorithm to the camera application, which then displays the preview image in the camera application's preview window.

[0108] A photo session is used to handle photo-related processes. It includes a photo data source module (Filter PPSource), an image optimization algorithm processing module, an encoding module (JPEG Encoder), and a writing module (write).

[0109] The photo data source module can receive image frames from the preview session data source module and pass them to the image optimization algorithm processing module, which can perform image optimization algorithm processing on the image frames, such as applying beauty and filters to the image frames.

[0110] The encoding module can be used to convert image frame formats, for example, converting YUV images to JPEG. The writing module can be used to write images to a file path and pass the generated images to the camera application's session service host module through a callback function for use in generating dynamic photos.

[0111] It is understood that in some scenarios, image frames processed in a preview session may also be referred to as preview frames, and images generated by the preview session may include preview images for display on the camera interface. Image frames processed in a photo session may also be referred to as photo frames, and images generated by the photo session may include images generated by the photo session, which may also be referred to as large images.

[0112] The dynamic photo management proxy module can be used to interact with the dynamic photo service host module of the camera application, and can also be used to interact with the dynamic photo management module to obtain cached image frames.

[0113] The live photo management module may include a LivePhotoSession module, which may include a CycleQueue. The live photo management module is responsible for creating and destroying live photo session modules. The live photo management module may also cache preview frames delivered by the offloading module in the CycleQueue through the LivePhotoSession module.

[0114] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for controlling and managing the Android system.

[0115] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0116] The application layer and framework layer run in a virtual machine. The virtual machine executes the Java files in the application and framework layers as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiments of the present application, the virtual machine can be used to configure the photo stream or preview stream, as well as to create and destroy channels for dynamic photos.

[0117] The system library can also be called the Native layer. The Native layer can include multiple functional modules, such as the media library, function library, and graphics processing library.

[0118] The Hardware Abstraction Layer (HAL) is an abstracted layer between the kernel and the Android runtime. It encapsulates hardware drivers and provides a unified interface for upper-layer applications. For example, in this embodiment, the HAL can retrieve image frames captured by the camera and pass them to the camera device module of the camera service.

[0119] The kernel layer is the layer between hardware and software. The kernel layer may include camera driver, display driver, audio driver, battery driver, Bluetooth driver, CPU driver, USB driver, etc.

[0120] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0121] The following describes the method of the embodiment of the present application in detail through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.

[0122] For example, taking the scenario of taking a photo with the dynamic photo function turned on, Figure 6 A schematic diagram of module interaction for implementing the dynamic photo function in an embodiment of the present application is shown.

[0123] When the dynamic photo function is turned on, the user can trigger the photo-taking operation, such as Figure 1 In the embodiment of the present invention, the user may trigger the operation of the shooting control 105. In response to the triggering operation, the camera application may call a pre-processing function (preCapture) to perform a pre-processing process.

[0124] During the preprocessing process, the camera application can establish a photo session, initialize the image name, create a file path for storing the image, and determine the photo parameters. The camera application can also create a task process for dynamic photos. These parameters may include exposure value (EV), zoom, and parameters corresponding to dynamic photos. The preprocessing process can also send a photo request to the camera capture session module. This request can carry some or all of the photo parameters, which can then be passed to the camera capture session module.

[0125] The camera capture session module can pass a photo request to the camera device module. The camera device module can send a request list (submitRequestList) to the camera device client module of the camera service. This request list can include parameters for turning dynamic photos on or off, as well as photo parameters such as exposure time, white balance, and focal length.

[0126] The camera device client module can pass a capture request (process_capture_request) to the hardware abstraction layer based on the camera device module. After obtaining the image frames captured by the camera, the hardware abstraction layer can return the result (process_capture_result) to the camera device module of the camera service. The returned result may include the captured image frames, and the camera device module can return the result to the camera output stream module.

[0127] It's understandable that during the camera capture process, the preview image still needs to be displayed in the preview window of the camera application, and the preview process continues. That is, the capture stream and the preview stream are executed in parallel. The preview stream is established when the preview starts, and its lifecycle spans the entire preview process. The capture stream is created once per capture and is destroyed after the capture is complete. Therefore, the camera output stream module can obtain both the preview frame for image preview and the image at the time of capture, and pass the preview frame and the image at the time of capture to the ServiceHost module.

[0128] On the one hand, the ServiceHost module can process the preview frame through the preview stream and return it to the camera application for preview interface display. On the other hand, the ServiceHost module can process the image at the time of shooting through the photo stream and return it to the camera application for storage and display.

[0129] Corresponding to the first aspect, the data source module of the preview session can receive the preview frames transmitted by the camera output stream module and pass the preview frames to the preview algorithm module. The preview algorithm module can perform image optimization algorithm processing on the preview frames, such as applying beauty and filters to the preview frames, and pass the processed preview frames to the diversion module. The diversion module can pass the preview frames to the preview module, and the preview module can pass the image obtained after the image optimization algorithm processing to the camera application, thereby displaying the preview image in the preview window of the camera application.

[0130] It is understood that when the dynamic photo function is enabled, the diversion module can also pass the preview frame to the circular queue in the dynamic photo management module for caching, thereby being used for dynamic photo generation. Optionally, the diversion module can also pass the preview frame to other storage controls in the dynamic photo management module for caching, which is not limited in this embodiment of the application.

[0131] Corresponding to the second aspect, the data source module of the preview session can also receive the image at the moment of capture delivered by the camera output stream module and pass it to the photo data source module of the photo session. The photo data source module can pass the image to the image optimization algorithm processing module, which can perform image optimization algorithm processing, such as applying beauty effects or filters, and then pass the processed image to the encoding module.

[0132] Optionally, the image optimization algorithm processing module and the preview algorithm module in the preview session can execute the same image processing process, so that the display effect of the image seen by the user in the preview window and the image generated by taking a photo can be consistent.

[0133] The encoding module converts images, for example, from YUV to JPEG, and passes the converted images to the writing module. The writing module writes the images to a file path, allowing users to access them through the gallery, album, or the camera's album shortcut controls. The writing module also passes the generated images to the camera app's session service host module via a callback function for use in generating dynamic photos.

[0134] When the dynamic photo function is turned on, the dynamic photo service host-side module of the camera application can establish an AIDL interface communication connection with the dynamic photo management proxy module of the ServiceHost module, and request to obtain the image frames used to generate dynamic photos. The dynamic photo management proxy module can initialize the relevant parameters of the image cache surface, which can be used to cache image frames before and after the photo is taken, so as to generate dynamic photos. The dynamic photo management proxy module can pass the request to obtain image frames to the dynamic photo management module, and the dynamic photo management module can obtain image frames from the circular queue based on the dynamic photo session module and return them to the dynamic photo management proxy module. The dynamic photo management proxy module can cache the image frames in the image cache surface, and the dynamic photo management proxy module can also return the image frames in the image cache surface to the dynamic photo service host-side module of the camera application.

[0135] In a possible implementation, since dynamic photos require 1.5 seconds of video before and after the capture time, the image cache surface can cache image frames 1.5 seconds or longer before the current moment. This can also be understood as image frames 1.5 seconds or longer before the capture time to generate the 1.5 seconds of video before the capture time. Over time, the image cache surface can cache image frames 1.5 seconds or longer after the capture time to generate the 1.5 seconds of video after the capture time.

[0136] After the dynamic photo service host module acquires the image frame, it can pass it to the AVC queue. This allows the mixing module to generate a video file based on the audio data cached in the AAC queue, the image data cached in the AVC queue, and information such as timestamps. This video file can include an MP4 format video file. It is understood that the mixing module can time-align the audio and image data using timestamps, so that the generated video file has a matching image and audio.

[0137] Furthermore, the dynamic photo mode implementation module can obtain the video file generated by the mixing module and the captured image obtained in the session service host module, and synthesize the captured image and the video file to generate a dynamic photo file.

[0138] Figure 7 A schematic diagram of the execution flow of implementing the dynamic photo function in an embodiment of the present application is shown.

[0139] The execution process involves the application process, the service host, and the hardware algorithm. The application process includes the camera application process and the process corresponding to the live photo service host module (ServiceHostLivePhoto.java); the service host includes the process corresponding to the live photo management proxy module (LivephotoManagerProxy.cpp); and the hardware algorithm includes the process corresponding to the live photo management module (LivephotoManager.cpp).

[0140] For ease of description, the following uses the module names as examples to introduce the interaction process between modules, omitting the description of processes. However, in the actual execution process, each module can be executed in a process. It is understandable that the processes mentioned in the embodiments of this application can also be threads in some implementations, without limitation.

[0141] The specific execution process may include the following processes: (1) the process of opening the dynamic photo, (2) the process of applying the request for each frame, (3) the process of applying the acquisition of semantic detection tags, and (4) the process of closing the dynamic photo.

[0142] (1) The process of opening dynamic photos.

[0143] In response to a user-triggered photo-taking operation, the camera application can call the dynamic photo service host-side module's dynamic photo callback function setProcLivePhotoCallback(surface, framenumber, mProcLivePhotoCallback). The parameters passed by this callback function include: image cache surface, frame number framenumber, and dynamic photo callback object mProcLivePhotoCallback. The dynamic photo service host-side module can call the dynamic photo management agent module's image cache function setSurfaceForCycle(surface, bufferNum). The parameters passed by this function include: image cache surface and frame number bufferNum.

[0144] The image buffer surface can be used to store image frames, and the frame number framenumber can be used to set the surface size. It can be understood that both framenumber and bufferNum can represent frame numbers, and framenumber and bufferNum can be understood as parameters used in different functions. That is, through inter-module calls, the value of the framenumber parameter in the function setProcLivePhotoCallback(surface, framenumber, mProcLivePhotoCallback) can be passed to the bufferNum parameter in the function setSurfaceForCycle(surface, bufferNum).

[0145] The dynamic photo management proxy module can initialize the relevant parameters of the image cache surface, and can also call the dynamic photo management module's initialization dynamic photo parameter function initLivePhotoParameter(cameraId). The input parameter cameraId passed by this function can represent the camera ID. Different cameraId values ​​can represent different cameras. For example, the camera can include the front camera and the rear camera.

[0146] The dynamic photo management module can further initialize the dynamic photo session module (LivephotoSession.cpp) and the circular queue.

[0147] After live photo initialization is complete, the live photo management module calls the onLivePhotoInit(livePhotoImageBufferInfo) function to return the live photo initialization result to the live photo management proxy module. The livePhotoImageBufferInfo parameter passed to this function may include information such as the format usage of the image buffer surface. The live photo management proxy module can set the format of the image buffer surface and call the postLivePhotoInitResult() function to set the live photo initialization result, returning the live photo initialization result to the live photo service host module.

[0148] The dynamic photo service host-side module can call the dynamic photo initialization completion callback function onLivePhotoInitComplete(res==0) ​​of the dynamic photo callback object mProcLivePhotoCallback, and then return the dynamic photo initialization result to the camera application. Among them, res==0 can indicate that the returned result is true, that is, the dynamic photo initialization is successful. It is understandable that in some abnormal circumstances, the result of the dynamic photo initialization completion function returned by the dynamic photo service host-side module can be false, that is, onLivePhotoInitComplete(false).

[0149] The camera app can also call the setLivePhotoTagCallback() function of the Live Photo service host module to set the live photo tag callback function. This callback function can be used to execute the relevant process for determining the category of the live photo. The categories of live photos can include people, animals, scenery, etc. In this callback function, a tag callback object mLivePhotoTagCallback can be created.

[0150] It is understandable that there is no distinction in the order of execution between setting the dynamic photo tag callback function and returning the dynamic photo initialization result. The camera application can also call the dynamic photo tag callback function before returning the dynamic photo initialization result. This is not limited to the embodiments of the present application.

[0151] (2) The process of applying requests for each frame.

[0152] The camera application can call the getNextFrameBuffer() function of the dynamic photo service host module. The dynamic photo service host module can further call the getNextFrameBuffer() function of the dynamic photo management agent module.

[0153] The live photo management proxy module can obtain a cache buffer for storing live photos from the image cache surface. For example, the live photo management proxy module can execute anw->dequeueBuffer to obtain the cache buffer from the image cache surface. The live photo management proxy module can also call the live photo request function processLivePhotoRequest() of the live photo management module and pass the buffer to the live photo management module, instructing the live photo management module to place the obtained image frame into the cache buffer.

[0154] The live photo management module can retrieve image frames from the circular queue and store them in a buffer. The live photo management module can also call the live photo retrieval completion function onLivePhotoCompleted(livePhotoResult) to pass the retrieved image frames to the live photo management proxy module, where the parameter livePhotoResult can include the buffer.

[0155] The dynamic photo management proxy module can return the image frame to the image cache surface through the cache buffer. For example, the dynamic photo management proxy module can execute anw->queueBuffer to put the image frame into the image cache surface. The dynamic photo management proxy module can also call the function postLivePhotoProcData() to set the dynamic photo data. The parameters passed in this function may include the timestamp of the image frame, the frame number of the image frame, and other information, and pass these parameter information to the dynamic photo service host module. Among them, the frame number of the image frame can be used to mark the position of the cover in the video, so that it can be used for later video editing.

[0156] The live photo service host module can encapsulate information such as the image frame timestamp and frame count into a Bundle container and call the onLivePhotoPictureCallback(bundle) function of the live photo callback object mProcLivePhotoCallback, returning the data in the Bundle container to the camera application via the parameter bundle. A Bundle container can be understood as a container containing key-value pairs for storing data information.

[0157] (3) Apply the process of obtaining semantic detection labels.

[0158] The camera app can call the getLivePhotoTag(Bundle tagBundle) function of the Live Photo Service host module to obtain live photo tags. The Live Photo Service host module can parse the tagBundle parameter to obtain the live photo's name ID, start time, end time, and capture time, and call the getLivePhotoTag() function to pass these parameters to the Live Photo Management Agent module. The Live Photo Management Agent module can then call the processLivePhotoGetTag() function to pass these parameters to the Live Photo Management module.

[0159] After obtaining the live photo tag, the live photo management module can call the live photo tag result function onLivePhotoAlgoTagResult() to return the result to the live photo management agent module. The live photo management agent module can call the live photo tag setting function postLivePhotoTagData(). The parameters passed in this function may include tag status state, live photo name id, live photo tag, and other information, and pass these parameters to the live photo service host module.

[0160] It is understood that when performing tag detection, there may be cases where the detection fails and the tag detection is invalid. The tag state can indicate whether the detected tag is valid. For example, state 1 can indicate that the detected tag is valid, and state 0 can indicate that the detected tag is invalid. The specific value of the tag state is not limited in the embodiments of this application.

[0161] The dynamic photo service host module can encapsulate the dynamic photo's state, name id, tag, and other information into a Bundle container, and call the dynamic photo tag callback function onTagResult (bundle) of the tag callback object mLivePhotoTagCallback to return the data in the Bundle container to the camera application through the parameter bundle.

[0162] (4) Turn off the dynamic photo process.

[0163] The camera application can call the dynamic photo service host-side module's dynamic photo closing function flushLivePhoto(), the dynamic photo service host-side module can further call the dynamic photo management agent module's dynamic photo closing function flushLivePhoto(), and the dynamic photo management agent module can call the dynamic photo management module's dynamic photo session closing function flushLivePhotoSession().

[0164] After the dynamic photo management module completes the process of closing the dynamic photo session, it can call the dynamic photo closing function onLivePhotoFlush(res) to return the execution result to the dynamic photo management proxy module.

[0165] In some scenarios, when the user triggers the operation of turning on or off the dynamic photo function, the interface may freeze.

[0166] This is because after the camera app enters the preview interface, it instructs the electronic device to create a preview stream at the hardware abstraction layer. Similarly, after the user triggers a photo operation, the camera app instructs the electronic device to create a photo stream at the hardware abstraction layer. When the user triggers the dynamic photo function to turn on or off, the camera app determines that turning on or off the dynamic photo function causes the parameters of the image captured by the camera to change. Therefore, the camera app instructs the electronic device to reconfigure the preview stream or photo stream at the hardware abstraction layer. Reconfiguring the preview stream may take hundreds of milliseconds, during which time no frames will be previewed, causing the interface to freeze.

[0167] Figure 8 It shows the calling sequence diagram of related modules when the dynamic photo function is turned on.

[0168] When the user turns on the dynamic photo function, the camera application can update the status of the dynamic photo switch and send a repeating request function setRepeatingRequest() to the camera service.

[0169] The camera service's camera device module (Camera3Device) uses a thread loop to process requests. When an application sends a new, repeating request, the thread loop executes the updateSessionParams() function to update the session parameters carried in the request. In some scenarios, these session parameters are also called session keys.

[0170] In a possible implementation, in the loop thread, the session key (sessionkey) in the request will be compared with the previous session key (sessionkey). If the key-value pair in (sessionkey) changes, it means that the scene mode has changed and the flow needs to be reconfigured. The loop thread will call the hardware abstraction layer's reconfiguration request function isReconfigurationRequired(), and the hardware abstraction layer can determine whether the flow needs to be reconfigured.

[0171] It is understandable that in different modes or different functions of the camera application, the electronic device can set different scene mode values, and use the scene mode value to identify whether the dynamic photo function is turned on, so as to facilitate the judgment of the status of the dynamic photo. In this way, the hardware abstraction layer can determine which mode or function has changed based on the value of the scene mode, simplifying the implementation process of the code in the electronic device, improving the readability and flexibility of the code, and thus being able to concisely represent the on and off status of the dynamic photo. Among them, the scene mode can be represented by the sceneMode field or the cameraSceneMode field, and can also be defined as other fields, which is not limited in the embodiments of the present application.

[0172] For example, taking the value of the scene mode corresponding to the dynamic photo function as 73, when the dynamic photo function is turned on, the value of the scene mode is 73; when the dynamic photo function is turned off, the value of the scene mode is 0. In this way, when the value of the scene mode changes from 0 to 73, the hardware abstraction layer can determine that the dynamic photo function is turned on; when the value of the scene mode changes from 73 to 0, the hardware abstraction layer can determine that the dynamic photo function is turned off. The value of the scene mode corresponding to the specific dynamic photo function can be pre-set by the electronic device, and the embodiments of the present application are not limited thereto. For the sake of ease of description, the subsequent description will be based on the example of the scene mode value of 73 corresponding to the dynamic photo function.

[0173] When the hardware abstraction layer determines that the dynamic photo function is enabled, it returns a result indicating that stream reconfiguration is required to the camera service. For example, the result indicating that stream reconfiguration is required may be true, while the result indicating that stream reconfiguration is not required may be false. Electronic devices may use other fields or values ​​to indicate whether stream reconfiguration is required, and this embodiment of the application is not limited thereto.

[0174] After the camera service receives the result, it calls the reconfigureCamera() function to reconfigure camera parameters. Within this function, it executes the ConfigureStream() function to instruct the Hardware Abstraction Layer to reconfigure the stream and issue a capture request (process_capture_request). The Hardware Abstraction Layer then sends the reconfigured stream parameters to the camera driver, which then controls the camera to capture images based on the new parameters.

[0175] The camera application can also call the setProcLivePhotoCallback(surface, framenumber, mProcLivePhotoCallback) function to set the dynamic photo callback function to the post-processing camera algorithm module (ServiceHost), and pass the image cache, etc. The post-processing camera algorithm module can initialize the dynamic photo session, etc. For the specific process, please refer to Figure 7 The relevant description of the process of opening dynamic photos in (1) of the corresponding embodiment will not be repeated here.

[0176] Figure 9 The following diagram shows the call sequence diagram of related modules when the dynamic photo function is turned off. The call sequence of specific modules can be referred to Figure 8 The relevant description of the corresponding embodiment will not be repeated here.

[0177] and Figure 8 Unlike the scene in which the Live Photo feature is turned on, the hardware abstraction layer detects that the scene mode value changes from 73 to 0, determines that the Live Photo feature is turned off, and returns a result indicating that stream reconfiguration is required to the camera service. For example, the result indicating that stream reconfiguration is required can be true.

[0178] After the camera application obtains the result, it can call the flushLivePhoto() function to close the dynamic photo function in the post-processing camera algorithm module (ServiceHost). The post-processing camera algorithm module can clear the callback and destroy the dynamic photo session. For the specific process, please refer to Figure 7 The relevant description of the process of closing dynamic photos in (4) of the corresponding embodiment is not repeated here.

[0179] It is understandable that in Figure 8 and Figure 9 In the corresponding process, when the dynamic photo function is turned on or off, the hardware abstraction layer determines that the scene mode has changed and returns true to the camera service. The camera service then triggers the process of reconfiguring the flow. The reconfiguration process takes several hundred milliseconds to execute, causing the interface to freeze.

[0180] In the embodiment of the present application, since the dynamic photo caching process is executed in the preview process of the application framework layer, turning the dynamic photo function on or off does not require reconfiguring the preview stream or photo stream at the hardware abstraction layer. Therefore, when executing the method of the embodiment of the present application, when the hardware abstraction layer determines whether the dynamic photo function is turned on or off, it can return a result to the camera service that does not require stream reconfiguration, thereby avoiding stream reconfiguration and reducing interface lag.

[0181] Figure 10 The timing diagram of calling relevant modules when the dynamic photo function is turned on in the method of the embodiment of the present application is shown. The timing diagram of calling specific modules can be referred to Figure 8 The relevant description of the corresponding embodiment will not be repeated here.

[0182] and Figure 8 The difference between opening a scene with the dynamic photo function in is that the hardware abstraction layer can determine the value of the current scene and scene mode to determine whether the stream needs to be reconfigured.

[0183] For example, if the current scene is a preset field and the value of the scene mode changes from 0 to 73, the hardware abstraction layer can return a result indicating that stream reconfiguration is not required to the camera service, such as false. If the current scene is not a preset field, or the value of the scene mode does not change from 0 to 73 or from 73 to 0, the hardware abstraction layer can return a result indicating that stream reconfiguration is required to the camera service, such as true.

[0184] Among them, the preset field is used to identify that the current electronic device is using the method of the embodiment of the present application, that is, the image frames of the dynamic photo cached by the ServiceHost module and the generated large image. For the sake of convenience, the architecture corresponding to the image processing method of the embodiment of the present application can also be referred to as the post-processing camera algorithm architecture or the ServiceHost architecture. The definition of the specific preset field is not limited by the embodiment of the present application.

[0185] Figure 11 The timing diagram of calling relevant modules when the dynamic photo function is turned off in the method of the embodiment of the present application is shown. The timing diagram of calling specific modules can be referred to Figure 9 The relevant description of the corresponding embodiment will not be repeated here.

[0186] and Figure 9 The difference between opening a scene with the dynamic photo function in is that the hardware abstraction layer can determine the value of the current scene and scene mode to determine whether the stream needs to be reconfigured.

[0187] For example, if the current scene is a preset field and the value of the scene mode changes from 73 to 0, the hardware abstraction layer can return a result indicating that stream reconfiguration is not required to the camera service, such as false. If the current scene is not a preset field, or the value of the scene mode does not change from 73 to 0 or from 0 to 73, the hardware abstraction layer can return a result indicating that stream reconfiguration is required to the camera service, such as true.

[0188] Figure 12 The image processing method of an embodiment of the present application is shown, which is applied to an electronic device. The method includes:

[0189] S1201: In response to a first operation of triggering an opening of a target application, perform image processing on a captured image in a first data stream to obtain first image data, where the first image data is used for image preview.

[0190] In the embodiment of the present application, the target application can be understood as an application that can generate dynamic photos. For example, the target application can include a camera application.

[0191] The first operation can be understood as an operation of opening a target application or displaying an image preview. The first operation may include but is not limited to operations such as clicking an icon of a target application, opening a target application by voice control, and opening a target application by gesture, and the embodiment of the present application does not limit this.

[0192] The first data stream can be understood as Figure 6 In the corresponding embodiment, the preview session (normalSession) in the post-processing camera algorithm module can be used to process preview-related processes, such as image processing of the captured image. Figure 5 and Figure 6 The relevant descriptions in the corresponding embodiments are omitted here.

[0193] The first image data can be understood as Figure 6 In the corresponding embodiment, the image data output by the preview algorithm module in the post-processing camera algorithm module can be used to display on the preview interface of the target application.

[0194] S1202. In response to a second operation that triggers a photo, perform image processing on a target image in a second data stream to obtain second image data; wherein the target image is an image at a time corresponding to the second operation in the first data stream, and the second image data includes an image obtained by triggering the second operation.

[0195] In the embodiment of the present application, the second operation can be understood as a photo-taking operation, which may include but is not limited to operations such as clicking a photo button, voice control photo-taking, and gesture control photo-taking, which are not limited in the embodiment of the present application. For example, clicking a photo button includes clicking Figure 1The shooting controls 105 in .

[0196] The second data stream can be understood as Figure 6 In the corresponding embodiment, the photo session (ppsession) in the post-processing camera algorithm module can be used to process the process related to photo taking, such as performing image processing on the image obtained by taking a photo. Figure 5 and Figure 6 The relevant descriptions in the corresponding embodiments are omitted here.

[0197] The second image data may include one or more images, and the electronic device may generate a captured image with better quality based on the one or more images, thereby presenting a better image effect to the user.

[0198] The target image can be understood as Figure 6 In the corresponding embodiment, the image is transmitted from the data source module of the preview session to the photo data source module of the photo session.

[0199] The second image data can be understood as Figure 6 In the corresponding embodiment, the image data output by the writing module in the post-processing camera algorithm module can be used as the image at the time of shooting, and can also be used to generate dynamic photos.

[0200] S1203: Generate a dynamic photo based on the first image data and the second image data.

[0201] In the embodiment of the present application, the implementation of generating a dynamic photo based on the first image data and the second image data can refer to Figure 6 The relevant descriptions in the corresponding embodiments are omitted here.

[0202] It is understandable that, in the first data stream, using the preview frames processed by the image optimization algorithm to generate dynamic photos can improve the quality of the generated dynamic photos and enhance the user experience.

[0203] Optional, in Figure 12 On the basis of the corresponding embodiment, the first data stream and the second data stream are both set at the application framework layer, and the method may further include: saving the first image data to a preset cache based on the first data stream; generating a dynamic photo based on the first image data and the second image data, which may include: obtaining the first image data from the preset cache based on the target application, and generating a dynamic photo based on the first image data and the second image data.

[0204] In the embodiment of the present application, the preset cache can be understood as Figure 6 The specific process of storing data in the preset cache and obtaining data can be referred to the CycleQueue in the corresponding embodiment. Figure 6The relevant descriptions in the corresponding embodiments are omitted here.

[0205] It is understood that both the first and second data streams are set up at the application framework layer. In the first data stream, after the preview frame used for image preview is processed by the image optimization algorithm, it can be cached as a preview frame for generating a dynamic photo. The target application can then generate a dynamic photo based on the cached preview frame used for generating the dynamic photo. In this way, the target application can generate a dynamic photo based on the preview frame processed by the image optimization algorithm, thereby improving the quality of the generated dynamic photo.

[0206] Optional, in Figure 12 On the basis of the corresponding embodiment, before obtaining the first image data in the preset cache, it can also include: establishing an Android interface definition language AIDL communication connection based on the target application and the application framework layer; obtaining the first image data from the preset cache based on the target application, and generating a dynamic photo based on the first image data and the second image data, which can include: based on the target application, obtaining the first image data from the preset cache, and synthesizing a video file based on the first image data and the collected audio data; generating a dynamic photo based on the video file and the second image data.

[0207] In the embodiment of the present application, the implementation of synthesizing a video file based on the first image data and the collected audio data, and generating a dynamic photo based on the video file and the second image data can refer to Figure 6 The relevant descriptions in the corresponding embodiments are omitted here.

[0208] It is understood that when synthesizing a video file based on the first image data and the collected audio data, the audio data and image data can be time-aligned using timestamps, maintaining synchronization between the audio and video data streams during playback or processing. A dynamic photo generated based on the video file and the second image data can capture the image within a period of time before and after the image was captured, making the generated image not only static but also capable of displaying some dynamic effects. This allows the capture of wonderful moments that are difficult to capture with a single static photo.

[0209] Optional, in Figure 12 On the basis of the corresponding embodiment, before performing image processing on the captured image in the first data stream, it can also include: based on the camera service of the target application and the application framework layer, passing a first request to the hardware abstraction layer, the first request is used to instruct the hardware abstraction layer to initialize the preview stream; initializing the preview stream based on the hardware abstraction layer, and returning the image captured by the camera to the camera service through the preview stream; passing the captured image to the first data stream based on the camera service.

[0210] In the embodiment of the present application, the first request can be understood as Figure 6 The corresponding capture request (process_capture_request) in the embodiment. For the specific interaction between the hardware abstraction layer and the application framework layer, please refer to Figure 6 The relevant descriptions in the corresponding embodiments are omitted here.

[0211] It is understandable that the first request may carry the photographing parameters set by the target application. Based on the first request, the hardware abstraction layer may instruct the camera driver to set the corresponding photographing parameters, thereby obtaining an image that meets the user's requirements.

[0212] Optional, in Figure 12 On the basis of the corresponding embodiment, before the second operation, it may also include: in response to the third operation of triggering the opening of the dynamic photo function, based on the target application and the camera service, passing a second request to the hardware abstraction layer, the second request is used to instruct the hardware abstraction layer to reconfigure the preview stream, the second request includes a first identifier, the first identifier is used to indicate whether the dynamic photo function is turned on; based on the first identifier, it is determined that the hardware abstraction layer does not reconfigure the preview stream.

[0213] In the embodiment of the present application, the third operation can be understood as the operation of opening the dynamic photo function. The third operation may include but is not limited to operations such as clicking the dynamic photo button, opening the dynamic photo by voice control, and opening the dynamic photo by gesture control. The embodiment of the present application does not limit this. For example, clicking the dynamic photo button includes clicking Figure 1 Dynamic Photo controls in 107.

[0214] The second request can be understood as Figure 6 The first identifier can be understood as the capture request (process_capture_request) in the corresponding embodiment. Figure 8 The identifier corresponding to the scene mode in the corresponding embodiment is, for example, the sceneMode field or the cameraSceneMode field. The specific value of the first identifier can be referred to Figure 8 The relevant description of the scene mode in the corresponding embodiment will not be repeated.

[0215] The scene mode value is used to identify whether the dynamic photo function is turned on, making it easier to determine the status of the dynamic photo. The hardware abstraction layer can determine which mode or function has changed based on the scene mode value, simplifying the code implementation process in electronic devices and improving code readability and flexibility, so that the on and off status of the dynamic photo can be concisely represented.

[0216] Optional, in Figure 12On the basis of the corresponding embodiment, determining that the hardware abstraction layer does not reconfigure the preview stream based on the first identifier may include: when the first identifier indicates that the dynamic photo function is turned on, and the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned off, determining that the hardware abstraction layer does not reconfigure the preview stream.

[0217] In the embodiment of this application, Figure 8 Taking the description in the corresponding embodiment as an example, the first identifier indicating that the dynamic photo function is on can be understood as the scene mode value being 73, and the first identifier indicating that the dynamic photo function is off can be understood as the scene mode value being 0. If the first identifier indicates that the dynamic photo function is on, and the hardware abstraction layer records the last time that the first identifier indicated that the dynamic photo function was off, indicating that the scene mode value has changed from 0 to 73, and the dynamic photo function is on, it can be determined that the hardware abstraction layer does not need to reconfigure the preview stream.

[0218] It is understandable that when the dynamic photo function is turned on, the hardware abstraction layer does not reconfigure the preview stream, which can reduce the situation of no preview frames and reduce interface freezes, thereby improving the user experience.

[0219] Optional, in Figure 12 On the basis of the corresponding embodiment, determining that the hardware abstraction layer does not reconfigure the preview stream based on the first identifier may include: when the first identifier indicates that the dynamic photo function is turned on, the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned off, and when the dynamic photo function is turned on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream, determining that the hardware abstraction layer does not reconfigure the preview stream.

[0220] In an embodiment of the present application, when the dynamic photo function is turned on, the electronic device supports the situation where the hardware abstraction layer does not reconfigure the preview stream. This can be understood as the electronic device using the image processing method of an embodiment of the present application to implement the dynamic photo function, and can also be understood as the dynamic photo corresponding to the image frame cached in the post-processing camera algorithm module (ServiceHost) of the application framework layer.

[0221] It is understandable that electronic devices may include one or more ways to implement dynamic photos, and there may be some differences between different implementation methods. For example, the image processing method of the embodiment of the present application is used to implement the dynamic photo function. When the dynamic photo function is turned on, the hardware abstraction layer may not reconfigure the preview stream, thereby reducing the interface jam. Therefore, by determining whether the electronic device supports the hardware abstraction layer not to reconfigure the preview stream, it is possible to flexibly configure the preview stream according to different implementation processes, thereby improving the flexibility of code execution.

[0222] Optional, in Figure 12On the basis of the corresponding embodiment, before determining that the hardware abstraction layer does not reconfigure the preview stream, the method may further include: returning a second identifier to the camera service based on the hardware abstraction layer, where the second identifier is used to instruct the camera service not to trigger the process of reconfiguring the preview stream.

[0223] In the embodiment of the present application, the second identifier can be understood as Figure 10 The preset field in the corresponding embodiment is used to indicate the result returned by the hardware abstraction layer to the camera service that no stream reconfiguration is required. For example, the second flag may include false. Based on the second flag, the camera service will not call the stream reconfiguration function.

[0224] Using the second identifier to instruct the camera service not to trigger the process of reconfiguring the preview stream simplifies the code implementation process in the electronic device and improves the code readability and flexibility, so that the corresponding execution process can be concisely represented by the field identifier.

[0225] Optional, in Figure 12 On the basis of the corresponding embodiment, the method may further include: in response to the fourth operation that triggers the turning off of the dynamic photo function, based on the target application and the camera service, passing a third request to the hardware abstraction layer, the third request is used to instruct the hardware abstraction layer to reconfigure the preview stream, and the third request includes a first identifier; when the first identifier indicates that the dynamic photo function is turned off, and the last time the first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned on, determining that the hardware abstraction layer does not reconfigure the preview stream.

[0226] In the embodiment of the present application, the fourth operation can be understood as the operation of closing the dynamic photo function. The fourth operation may include but is not limited to operations such as clicking the dynamic photo button, voice control to close the dynamic photo, gesture control to close the dynamic photo, etc., which are not limited in the embodiment of the present application. For example, clicking the dynamic photo button includes clicking Figure 1 Dynamic Photo controls in 107.

[0227] The third request can be understood as Figure 6 Corresponding to the capture request (process_capture_request) in the embodiment. Figure 8 Taking the description in the corresponding embodiment as an example, the first identifier indicates that the dynamic photo function is turned off, and the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned on, indicating that the value of the scene mode changes from 73 to 0, and the dynamic photo function is turned off. It can be determined that the hardware abstraction layer does not need to reconfigure the preview stream.

[0228] It is understandable that when the dynamic photo function is turned off, the hardware abstraction layer does not reconfigure the preview stream, which can reduce the situation of no preview frames and reduce interface freezes, thereby improving the user experience.

[0229] Optional, in Figure 12 On the basis of the corresponding embodiment, when the first identifier indicates that the dynamic photo function is turned off and the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned on, determining that the hardware abstraction layer does not reconfigure the preview stream may also include: when the first identifier indicates that the dynamic photo function is turned off, the last first identifier recorded by the hardware abstraction layer indicates that the dynamic photo function is turned on, and when the dynamic photo function is turned on or off, the electronic device supports the hardware abstraction layer not reconfiguring the preview stream, determining that the hardware abstraction layer does not reconfigure the preview stream.

[0230] In an embodiment of the present application, when the dynamic photo function is turned off, the electronic device supports the hardware abstraction layer without reconfiguring the preview stream. This can be understood as the electronic device using the image processing method of an embodiment of the present application to implement the dynamic photo function, and can also be understood as the dynamic photo corresponding to the image frame cached in the post-processing camera algorithm module (ServiceHost) of the application framework layer.

[0231] It is understandable that electronic devices may include one or more ways to implement dynamic photos, and there may be some differences between different implementation methods. For example, the image processing method of the embodiment of the present application is used to implement the dynamic photo function. When the dynamic photo function is turned off, the hardware abstraction layer may not reconfigure the preview stream, thereby reducing the interface jam. Therefore, by determining whether the electronic device supports the hardware abstraction layer not to reconfigure the preview stream, it is possible to flexibly configure the preview stream according to different implementation processes, thereby improving the flexibility of code execution.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0233] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0234] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0235] like Figure 13 FIG1 is a schematic diagram of the structure of a chip provided by an embodiment of the present application. The chip 1300 includes one or more (including two) processors 1301, a communication line 1302, a communication interface 1303, and a memory 1304.

[0236] In some embodiments, the memory 1304 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0237] The methods described in the above embodiments of the present application can be applied to the processor 1301 or implemented by the processor 1301. The processor 1301 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by a hardware integrated logic circuit in the processor 1301 or by instructions in the form of software. The above processor 1301 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. The processor 1301 can implement or execute the methods, steps, and logic block diagrams related to each processing disclosed in the embodiments of the present application.

[0238] The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-established in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). The storage medium is located in memory 1304, and processor 1301 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the method described above.

[0239] The processor 1301 , the memory 1304 , and the communication interface 1303 may communicate with each other via a communication line 1302 .

[0240] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0241] Embodiments of the present application also provide a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are fully or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium capable of computer storage or a data storage device such as a server or data center that includes one or more available media. For example, available media may include magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0242] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0243] As one possible designation, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Furthermore, any connection may also be appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers.

[0244] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An image processing method, applied to electronic equipment, characterized in that: The application framework layer of the electronic device is provided with a camera service and a post-processing camera algorithm module, and the method includes: In response to a first operation triggering the opening of a camera application, the application framework layer performs image processing on an image captured in a first data stream to obtain first image data, and saves the first image data to a preset cache, wherein the first image data is used for image preview. In response to a second operation triggering the taking of a photo, the application framework layer performs image processing on a target image in a second data stream to obtain second image data; wherein the target image is an image in the first data stream at a time corresponding to the second operation, and the second image data includes an image obtained when the second operation is triggered. Obtaining the first image data from the preset cache based on the camera application, and generating a dynamic photo based on the first image data and the second image data; Before responding to the second operation of triggering a photo-taking operation, the method further includes: In response to the third operation, the camera application updates the state of the dynamic photo switch and sends a repeat request to the camera service, so that the camera service sends a second request to the hardware abstraction layer, where the second request includes values ​​of the current scene and the scene mode, and is used to instruct the hardware abstraction layer to reconfigure the preview stream; When the third operation is to enable the dynamic photo function, and the current scene is a preset field, and the value of the scene mode is a second value that is different from the first value last obtained by the hardware abstraction layer, the hardware abstraction layer returns a result of not reconfiguring the preview stream to the camera service; When the third operation is to turn off the dynamic photo function, and the current scene is the preset field, and the value of the scene mode is the first value, which is different from the second value last obtained by the hardware abstraction layer, the hardware abstraction layer returns a result of not reconfiguring the preview stream to the camera service; Among them, the preset field indicates that the electronic device uses the post-processing camera algorithm module to cache the preview frame and the image at the shooting moment for generating dynamic photos, the first value indicates that the dynamic photo function is turned off, and the second value indicates that the dynamic photo function is turned on.

2. The method according to claim 1, characterized in that Before obtaining the first image data from the preset cache, the method further includes: An Android Interface Definition Language (AIDL) communication connection is established based on the camera application and the application framework layer.

3. The method according to claim 2, characterized in that The acquiring the first image data from the preset cache based on the camera application, and generating a dynamic photo based on the first image data and the second image data, includes: Based on the camera application, obtaining the first image data from the preset cache, and synthesizing a video file according to the first image data and the collected audio data; The dynamic photo is generated based on the video file and the second image data.

4. The method according to any one of claims 1 to 3, characterized in that Before performing image processing on the image collected in the first data stream, the method further includes: Delivering a first request to the hardware abstraction layer based on the camera application and the camera service in the application framework layer, where the first request is used to instruct the hardware abstraction layer to initialize a preview stream; Initialize the preview stream based on the hardware abstraction layer, and return the image captured by the camera to the camera service through the preview stream; The captured image is delivered to the first data stream based on the camera service.

5. The method according to claim 1, wherein The method further comprises: When the third operation is to turn on the dynamic photo function, when the current scene is not the preset field, or the value of the scene mode is not switched from the first value to the second value, the hardware abstraction layer returns the result of reconfiguring the preview stream to the camera service.

6. The method according to claim 1, characterized in that The method further comprises: When the third operation is to turn off the dynamic photo function, when the current scene is not the preset field, or the value of the scene mode is not switched from the second value to the first value, the hardware abstraction layer returns the result of reconfiguring the preview stream to the camera service.

7. The method according to claim 1, characterized in that The post-processing camera algorithm module is set in the application framework layer, and the post-processing camera algorithm module includes a preview session and a photo session, wherein the preview session includes a preview algorithm module for performing image optimization algorithm processing on the preview frame; the photo session includes an image optimization algorithm processing module for performing image optimization algorithm processing on the image at the shooting moment.

8. The method according to claim 1, characterized in that The method further comprises: The camera application obtains a semantic detection result corresponding to the dynamic photo, where the semantic detection result includes a status of the dynamic photo, a name of the dynamic photo, and a tag of the dynamic photo; The camera application adds tags to the dynamic photo according to the semantic detection result.

9. An electronic device, characterized in that: The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 8.

Images