A frame rate adjustment method for preview algorithm and electronic device

Abstract

The present application provides a frame rate adjustment method for a preview algorithm and an electronic device, which are applied to the field of electronic technology, comprising: in response to triggering a photographing event, executing a photographing algorithm to process preview frame image data, determining a target execution frame rate of the preview algorithm per unit time according to a predicted execution time reserved for the preview algorithm in a resource consumption bottleneck unit during this photographing process and a time consumed by executing the preview algorithm once in the resource consumption bottleneck unit during this photographing process, wherein the resource consumption bottleneck unit refers to a computing unit that consumes the longest time to execute the photographing algorithm during this photographing process, and finally executing the preview algorithm according to the target execution frame rate, which can reduce the number of executions of the preview algorithm, thereby reducing the frequency of the photographing algorithm and the preview algorithm competing for device resources with each other, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a frame rate adjustment method of a preview algorithm and an electronic device. Background Art

[0002] Currently, electronic devices such as mobile phones are equipped with cameras, and electronic devices can use cameras to capture images, such as photographing people, photographing scenery, etc.

[0003] After the electronic device opens the camera application, it can call the preview algorithm to process the image data collected by the camera, thereby converting it into an image displayed on the screen of the electronic device. When the user triggers a photo event, the electronic device can call the photo algorithm to capture the image and perform a series of processing on the captured image to obtain the taken photo.

[0004] In some cases, when the electronic device responds to the user's trigger to take a photo and runs the photo algorithm, the preview algorithm is also running, that is, there will be a scenario where the photo algorithm and the preview algorithm are running at the same time, which may cause the photo algorithm and the preview algorithm to frequently compete with each other for device resources (device resources such as CPU resources, CPU resources and NPU resources, etc.), which may cause the preview image displayed in the shooting preview interface to become stuck or the time to take a single photo to increase, resulting in a poor user experience. Summary of the invention

[0005] The frame rate adjustment method of the preview algorithm and the electronic device provided in the present application reduce the frequency of the camera algorithm and the preview algorithm competing for device resources, thereby improving the user experience.

[0006] In order to achieve the above purpose, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a frame rate adjustment method of a preview algorithm, which can be applied to an electronic device, including:

[0008] In response to the user opening the camera operation, the camera application is started, and the preview algorithm is called to process the captured image data in real time. When it is determined that the photo event is triggered, the photo algorithm is executed to process the preview frame image data. Then, the resource consumption bottleneck unit of the photo algorithm in this photo process, the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, and the execution time reserved for the preview algorithm in the resource consumption bottleneck unit in this photo process are predicted, wherein the resource consumption bottleneck unit refers to the longest computing unit for executing the photo algorithm in this photo process. Then, according to the execution time and the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, the target execution frame rate of the preview algorithm per unit time is determined, and finally the preview algorithm is executed according to the target execution frame rate, which can reduce the frequency of the photo algorithm and the preview algorithm competing for device resources, save the execution time of the photo algorithm, and thus improve the user experience.

[0009] In some possible implementations, the number of preview algorithms that need to be executed may be multiple. In this case, the present application can predict the time consumed by each preview algorithm in the resource consumption bottleneck unit during the current photo shooting process and the total time consumed by the preview algorithms in the resource consumption bottleneck unit during the current photo shooting process, and then determine the difference between the total time consumed by the preview algorithms in the resource consumption bottleneck unit during the current photo shooting process and the execution time. Finally, based on the difference value, determine the target difference value corresponding to each preview algorithm, and determine the target execution frame rate corresponding to preview algorithm i based on the target difference value, the time consumed by preview algorithm i in the resource consumption bottleneck unit during the current photo shooting process, and the time consumed by executing preview algorithm i once in the resource consumption bottleneck unit, where i is a positive integer.

[0010] The specific formula can be shown as follows:

[0011]

[0012] in, is the target gap value corresponding to the preview algorithm i, G is the gap value, is the time consumed by the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process, It can be the total time consumed by each preview algorithm in the resource consumption bottleneck unit during the photo shooting process.

[0013] In some possible implementations, the target execution frame rate corresponding to the preview algorithm i is determined, and the specific formula may be as follows:

[0014]

[0015] in, is the target execution frame rate corresponding to the preview algorithm i, The time consumed by executing the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process , is the target gap value corresponding to the preview algorithm i.

[0016] In some possible implementations, the average time consumed by executing the photo algorithm in each computing unit in the N photos closest to the current photo moment can be determined, where N is a positive integer, and then the computing unit with the longest average is determined as the resource consumption bottleneck unit.

[0017] In some possible implementations, taking the computing unit including GPU, NPU and CPU as an example, when the average time consumed by executing the photo algorithm in the CPU in N photos is equal to the average time consumed by executing the photo algorithm in the NPU in N photos and is greater than the average time consumed by executing the photo algorithm in the GPU in N photos, since the CPU is generally a multi-core processor, the resource competition in the CPU is smaller than the resource competition in the GPU, and the GPU is determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined subsequently according to the resource consumption in the GPU.

[0018] When the average time consumed by executing the photo algorithm in the CPU during N photo shooting is equal to the average time consumed by executing the photo algorithm in the GPU during N photo shooting and is greater than the average time consumed by executing the photo algorithm in the NPU during N photo shooting, since the CPU is generally a multi-core processor, the resource competition in the CPU is relatively smaller than the resource competition in the GPU, and the GPU is determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined subsequently according to the resource consumption in the GPU.

[0019] When the average time consumed by executing the photo algorithm in the NPU during N photo shooting is equal to the average time consumed by executing the photo algorithm in the GPU during N photo shooting and is greater than the average time consumed by executing the photo algorithm in the CPU during N photo shooting, the unit with the larger gap value G between the NPU and the GPU is determined as the resource consumption bottleneck unit. The gap value G here can be explained in detail in step S504. The larger the gap value G, the more intense the resource competition between the photo algorithm and the preview algorithm in the current computing unit. The unit with the stronger resource competition can be determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined based on the resource consumption in the resource consumption bottleneck unit later.

[0020] In some possible implementations, the weight coefficient K can be determined to be proportional to the time consumed by the camera algorithm in the resource consumption bottleneck unit during the current camera process. The weight coefficient is used to balance the resource usage of the photo algorithm and the preview algorithm. The execution time R reserved for the preview algorithm in the resource consumption bottleneck unit during this photo shooting process is determined according to the difference between the time T consumed by executing the photo algorithm during this photo shooting process and the product.

[0021] The specific formula can be as follows:

[0022]

[0023] In some possible implementations, the average time consumed by executing the photo algorithm in the resource consumption bottleneck unit in the N most recent photos taken at the current photo moment can be calculated. Determined as the time consumed by executing the photo algorithm in the resource consumption bottleneck unit during this photo shooting process .

[0024] The average time consumed by the preset frame rate internal resource consumption bottleneck unit to execute the preview algorithm Determine the time consumed by executing the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process .

[0025] The average time of executing the photo algorithm for the N most recent photos taken at the time of this photo. The time T consumed by executing the photographing algorithm during this photographing process is determined.

[0026] In a second aspect, the present application provides an electronic device, including: a processor and a memory;

[0027] One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device performs the method described in any one of the first aspects.

[0028] In a third aspect, the present application provides a computer storage medium, comprising computer instructions, which, when executed on a mobile terminal, enables the electronic device to execute the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of opening a camera provided in an embodiment of the present application;

[0030] Figure 2 An interface diagram of a shooting preview interface provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of another photo taking algorithm and preview algorithm provided in an embodiment of the present application being executed in a computing unit;

[0032] Figure 4 A schematic diagram of a photo taking algorithm and a preview algorithm competing for device resources provided in an embodiment of the present application;

[0033] Figure 5 A diagram of a frame rate adjustment method of a preview algorithm provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 7 A software architecture diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0037] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0038] Currently, electronic devices such as mobile phones are equipped with cameras, and electronic devices can use cameras to capture images, such as photographing people, photographing scenery, etc.

[0039] For example, Figure 1 As shown, when a user needs to take a photo, the user can perform an operation to open the camera on the display screen of the electronic device, that is, the user touches the camera icon, and the touch sensor in the display screen can receive the user's touch operation on the camera icon and report the touch operation on the camera icon to the processor. After receiving the touch operation on the camera icon, the processor can respond to the above touch operation and start the application corresponding to the camera icon (which can be referred to as the camera application). In some possible implementation methods, the user can also enable the electronic device to start the camera application in other ways. For example, through voice commands or other preset gestures, etc., which are not limited in the embodiments of the present application.

[0040] When the electronic device opens the camera application, the lens receives the light reflected from the target scene (the scene to be photographed) and projects the light onto the sensor to generate an optical image. The sensor then converts the optical image into an electrical signal and sends it to the ISP. After the ISP receives the electrical signal sent by the sensor, it converts the electrical signal into a digital image signal through analog-to-digital conversion. The electronic device can then call the preview algorithm to process the digital image signal processed by the ISP, thereby converting the digital image signal into an image signal in standard RGB, YUV and other formats. The preview algorithm can also perform processing such as scaling the image, and finally convert the processed image signal into an image displayed on the screen of the electronic device, which can then be displayed as follows: Figure 2 The shooting preview interface shown in the figure allows users to display the preview image from the shooting preview interface of the camera application, where the preview algorithm refers to a series of computing methods and technologies used to process and display preview frames in real time during video shooting or image capture, which may include image compression algorithms, scene recognition algorithms, face recognition and face tracking algorithms, etc.

[0041] In some possible implementations, after the user opens the camera application, the camera application can automatically enter the photo taking mode, such as Figure 2 As shown, the user can trigger the photo shooting operation by touching the shooting key 201 in the shooting preview interface of the camera application. When the user triggers the photo shooting operation, the touch sensor can obtain the photo shooting operation triggered by the user and report the photo shooting operation to the processor, and the processor can control the camera application to shoot the photo. Specifically, when the electronic device responds to the user triggering the photo shooting operation, it can call the photo shooting algorithm to capture the image and perform a series of processing on the captured image to obtain the captured photo, wherein the photo shooting algorithm is a series of calculation methods and processing technologies for improving image quality, enhancing shooting effects, and optimizing device performance. For example, the photo shooting algorithm may include an automatic exposure algorithm, an automatic focus algorithm, and an automatic white balance algorithm.

[0042] In some cases, when the electronic device responds to a user triggering a photo-taking operation and runs a photo-taking algorithm, the preview algorithm is also running, that is, there is a scenario in which the photo-taking algorithm and the preview algorithm are running at the same time.

[0043] For example, the link of the photo-taking algorithm is relatively long. For example, taking a series of algorithms including automatic exposure algorithm, automatic focus algorithm and white balance algorithm, each algorithm can be deployed to execute at a different stage, and each algorithm can be deployed to execute in a different computing unit (such as CPU, GPU, NPU and DSP, etc.).

[0044] like Figure 3As shown, a photo-taking algorithm includes three stages (here they can be called stage a, stage b and stage c) as an example. For example, stage a takes the execution of an automatic exposure algorithm and the automatic exposure algorithm is deployed in three computing units: CPU, DSP and GPU. Stage b takes the execution of a white balance algorithm and the white balance algorithm is deployed in three computing units: CPU, GPU and NPU. Stage c takes the execution of an autofocus algorithm and the autofocus algorithm is deployed in two computing units: CPU and GPU.

[0045] In this way, when an electronic device calls a photo-taking algorithm, it needs to call the corresponding algorithm of each stage in turn to process the image. For example, when the automatic exposure algorithm is called, since it is deployed in three computing units, namely CPU, DSP and GPU, the automatic exposure algorithm needs to be processed in CPU, DSP and GPU in turn.

[0046] In some possible implementations, the preview algorithm executed may include multiple different algorithms, such as a compression algorithm and a downsampling algorithm. Figure 3 In the example, the preview algorithm includes 5 algorithms. Each algorithm may also be deployed in different computing units. For example, preview algorithm 1 is deployed in three computing units: CPU, GPU, and NPU, preview algorithm 2 is deployed in two computing units: CPU and NPU, preview algorithm 3 is deployed in two computing units: CPU and GPU, preview algorithm 4 is also deployed in two computing units: CPU and NPU, and preview algorithm 5 is also deployed in two computing units: CPU and GPU. It should be noted that since the GPU is a multi-core processor, the multiple different algorithms included in the preview algorithm can be executed simultaneously in the GPU. Since the NPU and GPU are single-core processors, the multiple different algorithms included in the preview algorithm need to be executed sequentially in the NPU or GPU and cannot be executed simultaneously.

[0047] Since the camera application needs to capture image frames in real time and needs to update the preview image displayed in the shooting preview interface in real time, in order to display the preview image smoothly and without lag in the shooting preview interface, the execution frame rate of the preview algorithm generally set at present (referring to the number of executions of the preview algorithm every preset number of times and within a unit time of 1s) is relatively large. For example, if the electronic device captures 30 image frames in 1s and executes the preview algorithm once every 2 captured frames, the frame rate of the preview algorithm is 15fps, that is, it is executed 15 times within the unit time.

[0048] like Figure 3As shown in the figure, the preview algorithm is executed multiple times during the execution of the photo algorithm. This may cause the photo algorithm and the preview algorithm to frequently occupy device resources (device resources such as CPU resources, CPU resources and NPU resources, etc.). Figure 3 As shown in the example, for example, when the a phase of the photo algorithm is executed in the CPU, preview algorithm 1, preview algorithm 2, preview algorithm 3, preview algorithm 4 and preview algorithm 5 all compete with the photo algorithm for CPU resources. When the a phase of the photo algorithm is executed in the GPU, preview algorithm 1 and preview algorithm 5 may compete with the photo algorithm for GPU resources. When the b phase of the photo algorithm is executed in the CPU, preview algorithm 1, preview algorithm 2, preview algorithm 3, preview algorithm 4 and preview algorithm 5 may also compete with the photo algorithm for CPU resources. When the a phase of the photo algorithm is executed in the GPU, preview algorithm 1, preview algorithm 3 and preview algorithm 5 may compete with the photo algorithm for GPU resources. When the b phase of the photo algorithm is executed in the NPU, preview algorithm 2 may compete with the photo algorithm for NPU resources. When the c phase of the photo algorithm is executed in the CPU, preview algorithm 1, preview algorithm 2, preview algorithm 3, preview algorithm 4 and preview algorithm 5 may also compete with the photo algorithm for CPU resources. When the c phase of the photo algorithm is executed in the GPU, preview algorithm 1, preview algorithm 3 and preview algorithm 5 may compete with the photo algorithm for GPU resources. For example, Figure 4 As shown, taking the execution of the photo algorithm and the preview algorithm in the NPU as an example, the execution time of the photo algorithm alone in the NPU is, for example, 3ms, and the execution time of the photo algorithm alone in the NPU is, for example, 1ms. When the photo algorithm and the preview algorithm are executed simultaneously in the NPU, the photo algorithm and the preview algorithm frequently grab device resources from each other, which may cause the processing time of the photo algorithm or the preview algorithm to increase (that is, the processing delay of the algorithm increases). For example, the processing time of the photo algorithm in the NPU becomes 5ms, which may cause the preview image displayed in the shooting preview interface to become stuck or the time to take a single photo to increase, resulting in a poor user experience.

[0049] In view of this, the technical solution of the present application is proposed after research. The present application provides a frame rate adjustment method of a preview algorithm, including: in response to a user opening a camera operation, starting a camera application, calling a preview algorithm to process captured image data in real time, in response to triggering a photo event, executing a photo algorithm to process preview frame image data, predicting the resource consumption bottleneck unit of the photo algorithm in this photo process, the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, and the execution time reserved for the preview algorithm in the resource consumption bottleneck unit in this photo process, wherein the resource consumption bottleneck unit refers to the longest calculation unit for executing the photo algorithm in this photo process, according to the predicted execution time reserved for the preview algorithm in the resource consumption bottleneck unit in this photo process and the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, determining the target execution frame rate of the preview algorithm per unit time, and finally executing the preview algorithm according to the target execution frame rate. Since the calculated target execution frame rate is less than the original frame rate preview algorithm of the preview algorithm, executing the preview algorithm according to the target execution frame rate can reduce the number of executions of the preview algorithm, thereby reducing the frequency of mutual occupation of device resources between the photo algorithm and the preview algorithm, thereby improving the user experience.

[0050] In order to make the technical solution of the present application clearer and easier to understand, the following describes the frame rate adjustment method of the preview algorithm provided by the present application embodiment in combination with the above embodiments and corresponding drawings, taking the electronic device as the execution subject. Figure 5 As shown, the frame rate adjustment method of the preview algorithm provided in the embodiment of the present application may include:

[0051] S501: In response to a user opening a camera operation, starting a camera application.

[0052] The electronic device launches a camera application in response to a user opening a camera operation.

[0053] Specifically, the electronic device receives a user's touch operation on the camera icon and reports the touch operation on the camera icon to the processor. After receiving the touch operation on the camera icon, the processor can respond to the touch operation and start the application corresponding to the camera icon (which may be referred to as the camera application). In some possible implementations, the user can also enable the electronic device to start the camera application in other ways. For example, through voice commands or other preset gestures, etc., which are not limited in the embodiments of the present application.

[0054] When the electronic device starts the camera application, it needs to call some functions to activate the camera function to ensure that the camera application can correctly access and control the camera hardware. At the same time, the camera device can be powered on, where powering on the camera device means providing power to the camera hardware to switch it from the off state, sleep state or low power state to the working state, so that it can perform photo taking, video recording or other image capture functions.

[0055] After the camera device is powered on, the electronic device can initialize the camera application, that is, configure some initialization parameters such as exposure time, sensitivity, and white balance required to start capturing images for the camera application. After the camera application is initialized, the output stream of the camera application can be configured.

[0056] Electronic devices can use the kernel layer to set parameters such as resolution, frame rate, and data format for each data stream, and then create a pipeline for capturing and processing images or videos based on the configured data stream. Algorithm nodes can be configured in the pipeline, such as automatic exposure, automatic white balance, and automatic focus. In this way, the configuration of the camera output stream can be realized. The channel configuration of the camera application is completed here. After the channel configuration of the camera application is completed, the electronic device can request to start the preview image frame function, and then call the preview algorithm to process the captured image data in real time, so that the electronic device can display Figure 2 The shooting preview interface shown displays the image frame data processed by the preview algorithm, wherein starting the preview image frame function refers to requesting to activate the real-time preview function of the camera. There can be multiple preview algorithms called, and each preview algorithm can set a different execution frame rate according to demand (that is, it is executed once every N image frames captured, and N can be a positive integer). For example, the execution frame rate of preview algorithm 1 is f1, and the execution frame rate of preview algorithm 2 is f2.

[0057] S502: In response to the user clicking on the photo control, executing a photo algorithm to process the image.

[0058] When the user needs to take a photo, he can click the photo control in the shooting preview interface. In response to the user clicking the photo control in the shooting preview interface, the electronic device can execute a photo algorithm to process the image frame data, thereby outputting and displaying the taken photo, wherein the photo algorithm may include an automatic exposure algorithm and an automatic focus algorithm, etc.

[0059] S503: Determine a resource consumption bottleneck unit of the photographing algorithm and performance overheads of the photographing algorithm and the preview algorithm in the resource consumption bottleneck unit.

[0060] The electronic device can determine the resource consumption bottleneck unit of the photographing algorithm and the performance overhead of the photographing algorithm and the preview algorithm in the resource consumption bottleneck unit, wherein the resource consumption bottleneck unit refers to the computing unit that consumes the longest time to execute the photographing algorithm in the current photographing process, and the performance overhead of the photographing algorithm and the preview algorithm in the resource consumption bottleneck unit may include the average of the time consumed to execute the photographing algorithm in the resource consumption bottleneck unit in the case of N photographs. And the average time consumed by the preset frame rate internal resource consumption bottleneck unit to execute preview algorithm i For example, if the resource consumption bottleneck unit of the preset frame rate executes the preview algorithm 5 times in total, then the average time consumed by executing the preview algorithm 5 times is determined as It should be noted that N here can be set according to actual needs, such as 5 or 3, and the preset frame rate can also be set according to actual needs, such as 30 frames. This is just an example and is not limited here.

[0061] Exemplarily, the electronic device may determine the average time consumed by executing the photographing algorithm in each computing unit in the N photographs closest to the current photographing moment, and then may use the computing unit with the longest average as the resource consumption bottleneck unit.

[0062] For example, the electronic device can obtain the execution time of the photographing algorithm in each of the N photographs closest to the current photographing moment, and the time consumed by the photographing algorithm in each computing unit (including CPU, NPU and GPU as an example here) during each photographing process. For example, taking N as 5 as an example, the current photograph is the Kth photograph, then the time consumed by the photographing algorithm in each computing unit during each photographing process of the k-1, k-2, k-3, k-4 and k-5 photographs is obtained. For example, during the k-1 photographing process, the time consumed by the photographing algorithm in the computing unit CPU is 200ms, the time consumed by the photographing algorithm in the computing unit NPU is 300ms, and the time consumed by the photographing algorithm in the computing unit GPU is 200ms. During the k-2 photographing process, the time consumed by the photographing algorithm in the computing unit CPU is 240ms, the time consumed by the photographing algorithm in the computing unit NPU is 350ms, and the time consumed by the photographing algorithm in the computing unit GPU is 280ms. Based on a similar principle, the time consumed by the photographing algorithm in each computing unit during each photographing process is obtained.

[0063] After obtaining the time consumed by executing the photographing algorithm in each computing unit during each photographing process, the electronic device can determine the average time consumed by each computing unit to execute the photographing algorithm in the case of N photographing times. Taking the CPU as an example, the electronic device can determine the sum of the time consumed by the CPU to execute the photographing algorithm during each photographing process, and determine the average time consumed by the CPU to execute the photographing algorithm during each photographing process based on the quotient of the sum and the number of photographing times N.

[0064] For example, if the average time consumed by the CPU to execute the photo algorithm in the case of N photos is 200ms, the average time consumed by the NPU to execute the photo algorithm in the case of N photos is 300ms, and the average time consumed by the GPU to execute the photo algorithm in the case of N photos is 200ms, then the resource consumption bottleneck unit (i.e., the computing unit with the longest average time consumed to execute the photo algorithm in the case of N photos) can be determined to be the NPU unit.

[0065] In this way, the average time consumed by executing the photographing algorithm in the resource consumption bottleneck unit in the case of N photographing times and the average time consumed by executing the photographing algorithm in the case of N photographing times can be determined.

[0066] The electronic device also needs to determine the average time consumed by the resource consumption bottleneck unit to execute preview algorithm i within the preset frame rate. For example, the preset frame rate is 30 frames, and the resource consumption bottleneck unit is NPU. For example, the electronic device executes preview algorithm i once every time it captures 3 image frames. If the preview algorithm i is executed 10 times in total when 30 image frames are captured, the electronic device can determine the average time consumed by the 10 preview algorithms i executed in the NPU within the 30 frames closest to the current photo moment.

[0067] In some possible implementations, there may be two computing units that consume equal time to execute the photography algorithm during the current photography process, that is, the electronic device determines the average time consumed by each computing unit to execute the photography algorithm in the N photos most recent to the current photography moment, and there are two computing units with equal means.

[0068] Exemplarily, the computing unit includes a GPU, an NPU, and a CPU.

[0069] When the average time consumed by executing the photo algorithm in the CPU during N photo shooting is equal to the average time consumed by executing the photo algorithm in the NPU during N photo shooting and is greater than the average time consumed by executing the photo algorithm in the GPU during N photo shooting, since the CPU is generally a multi-core processor, the resource competition in the CPU is relatively small compared to the resource competition in the NPU, and the NPU is determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined subsequently based on the resource consumption in the NPU.

[0070] When the average time consumed by executing the photo algorithm in the CPU during N photo shooting is equal to the average time consumed by executing the photo algorithm in the GPU during N photo shooting and is greater than the average time consumed by executing the photo algorithm in the NPU during N photo shooting, since the CPU is generally a multi-core processor, the resource competition in the CPU is relatively smaller than the resource competition in the GPU, and the GPU is determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined subsequently according to the resource consumption in the GPU.

[0071] When the average time consumed by executing the photo algorithm in the NPU during N photo shooting is equal to the average time consumed by executing the photo algorithm in the GPU during N photo shooting and is greater than the average time consumed by executing the photo algorithm in the CPU during N photo shooting, the unit with the larger gap value G between the NPU and the GPU is determined as the resource consumption bottleneck unit. The gap value G here can be explained in detail in step S504. The larger the gap value G, the more intense the resource competition between the photo algorithm and the preview algorithm in the current computing unit. The unit with the stronger resource competition can be determined as the resource consumption bottleneck unit, so that the target execution frame rate of the preview algorithm can be determined based on the resource consumption in the resource consumption bottleneck unit later.

[0072] S504: Calculate the target execution frame rate of the preview algorithm.

[0073] Electronic devices can Determined as the time consumed by executing the photo algorithm in the resource consumption bottleneck unit during this photo shooting process Will Determine the time consumed by executing the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process .Will The time T consumed by executing the photographing algorithm during this photographing process is determined.

[0074] The electronic device can calculate the time consumed by executing the photo algorithm in the current photo process according to the time consumed by T and the time consumed by executing the photo algorithm in the resource consumption bottleneck unit in the current photo process. The difference between determines the execution time R reserved for the preview algorithm in the resource consumption bottleneck unit for this photo shooting. It should be noted that here, only the determination of R based on T is used as an example for explanation. The principle implemented here is to balance the resource consumption of the photo shooting algorithm and the preview algorithm within a fixed time. The present application may also have other implementation methods, such as setting a threshold time and determining R based on the threshold time. The specific method is similar to formula (1) and will not be described here.

[0075] In some cases, T and The values ​​of may be equal, therefore, a weight coefficient K can also be set to balance the resource usage of the photo taking algorithm and the preview algorithm. The value of K can be set to 0.7, 0.8, and 0.9, etc. Of course, this is just an example, and those skilled in the art can set the value of K according to the actual situation without limitation. The difference between the two can be used to determine R as shown in formula (1):

[0076] (1)

[0077] After determining R, the electronic device can calculate the difference G between R and the execution time of the photo preview algorithm in the resource consumption bottleneck unit:

[0078] In some possible implementations, the preview algorithm executed may be one, and the formula for calculating G may be:

[0079] (2)

[0080] in, is the frame rate of the preview algorithm within a unit time of 1s, which refers to the number of executions of preview algorithm i every preset number of executions and within a unit time of 1s. For example, if the electronic device captures 30 image frames in 1s and executes preview algorithm i every 3 frames, then It is 10fps. Here, since the preview algorithm executed is one as an example, i is equal to 1.

[0081] According to G, the time consumed by the preview algorithm i in the resource consumption bottleneck unit at the target execution frame rate during the current photo shooting process is determined. The specific formula is shown in (3):

[0082] (3)

[0083] The formula for determining the target execution frame rate can be shown as (4):

[0084] (4)

[0085] In some possible implementations, multiple preview algorithms may be executed, and the formula for calculating G may be:

[0086] (5)

[0087] in, is the frame rate of preview algorithm i within a unit time of 1s, which refers to the number of executions of preview algorithm i every preset number of executions and within a unit time of 1s. For example, if an electronic device captures 30 image frames in 1s and executes preview algorithm i once every 3 captured frames, then For example, the electronic device captures 60 frames per second and executes the preview algorithm i every 3 frames captured. is 20fps, T and The unit is ms, It can represent the time consumed by the preview algorithm i in the resource consumption bottleneck unit within a unit time of 1s during this photo shooting process. It can represent the time consumed by the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process. It can represent the total time consumed by each preview algorithm in the resource consumption bottleneck unit during the photo shooting process.

[0088] After determining G, we can calculate the value of G allocated to each preview algorithm. The specific formula can be shown as (6):

[0089] (6)

[0090] in, is the value of G allocated to each preview algorithm i, that is, the difference between the target execution time of preview algorithm i in the resource consumption bottleneck unit during this photo shooting process and the time consumed by preview algorithm i in the resource consumption bottleneck unit during this photo shooting process.

[0091] according to Determine the time consumed by preview algorithm i in the resource consumption bottleneck unit at the target execution frame rate during this photo shooting process. The specific formula is shown in (7):

[0092] (7)

[0093] Combining formulas (5), (6) and (7), we can derive the target execution frame rate of preview algorithm i within a unit time of 1s: Formula (11) can be derived as shown in formulas (8), (9) and (10). The target execution frame rate here can be understood as the target execution times of preview algorithm i when it is executed once every preset number of times and within a unit time of 1s.

[0094] (8)

[0095] (9)

[0096] (10)

[0097] (11)

[0098] S505: Replace the original frame rate of the preview algorithm with the target execution frame rate.

[0099] After calculating the target execution frame rate, the electronic device may use the target execution frame rate to replace the original frame rate of the preview algorithm.

[0100] Exemplarily, as shown in Table 1, the preview algorithms include preview algorithm 1, preview algorithm 2, preview algorithm 3, preview algorithm 4 and preview algorithm 5 as examples for introduction:

[0101] Table 1

[0102]

[0103] As can be seen in Table 1, preview algorithm 1, preview algorithm 3 and preview algorithm 4 all calculate target execution frame rates that are smaller than the original frame rate. The target execution frame rate can be used to replace the original frame rate of the preview algorithm so that the preview algorithm is executed according to the corresponding target execution frame rate. This can reduce the number of executions of the preview algorithm, thereby reducing the frequency of the camera algorithm and the preview algorithm competing for device resources, thereby improving the user experience.

[0104] S506: When the execution of the photo-taking algorithm is completed, the original frame rate of the preview algorithm is restored.

[0105] When the execution of the photo-taking algorithm is completed, the resources of the electronic device are sufficient, and there is no problem of the photo-taking algorithm and the preview algorithm competing for device resources. The electronic device can restore the original frame rate of the preview algorithm so that the preview algorithm is executed according to the corresponding original frame rate.

[0106] In this application, in response to the user opening the camera operation, the camera application is started, the preview algorithm is called to process the captured image data in real time, and in response to the user clicking the photo control, the photo algorithm is executed to process the image, and then the resource consumption bottleneck unit of the photo algorithm and the performance overhead of the photo algorithm and the preview algorithm in the resource consumption bottleneck unit are determined, wherein the resource consumption bottleneck unit refers to the computing unit that consumes the longest time to execute the photo algorithm in this photo process, and the performance overhead of the photo algorithm and the preview algorithm in the resource consumption bottleneck unit may include the average of the time consumed by executing the photo algorithm in the resource consumption bottleneck unit in the case of N photos and the average of the time consumed by executing the preview algorithm i in the resource consumption bottleneck unit within the preset frame rate. The electronic device can calculate the target execution frame rate of the preview algorithm according to the performance overhead of the photo algorithm and the preview algorithm in the resource consumption bottleneck unit, and replace the original frame rate of the preview algorithm with the target execution frame rate, so that the preview algorithm is executed according to the target execution frame rate. Since the target execution frame rate calculated based on the performance overhead of the camera algorithm and the preview algorithm in the resource consumption bottleneck unit is lower than the original frame rate of the preview algorithm, executing the preview algorithm according to the target execution frame rate can reduce the number of executions of the preview algorithm, thereby reducing the frequency of the camera algorithm and the preview algorithm competing for device resources and improving the user experience.

[0107] The following is a brief introduction to the hardware structure of the electronic device provided by the present application in conjunction with the accompanying drawings. In some embodiments, the electronic device may be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 6 As shown, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0108] The method provided in the present application can be executed on an electronic device. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device can be as follows: Figure 6 As shown, Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0109] like Figure 6 As shown, the electronic device may include a processor 610, a sensor module 620, a camera 630, a display screen 640, etc. The sensor module 620 may include a touch sensor 650A, etc.

[0110] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0111] The processor 610 may include one or more processing units, for example, the processor 610 may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors, and the photo-taking algorithm and the preview algorithm may be run in the CPU, GPU, and NPU.

[0112] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0113] In some embodiments, the processor 610 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, and a mobile industry processor interface (MIPI).

[0114] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 610 may include multiple groups of I2C buses. The processor 610 may be coupled to the touch sensor 650A, the charger, the flash, the camera 630, etc. through different I2C bus interfaces. For example, the processor 610 may be coupled to the touch sensor 650A through the I2C interface, so that the processor 610 communicates with the touch sensor 650A through the I2C bus interface to realize the touch function of the electronic device.

[0115] The MIPI interface can be used to connect the ISP to the processor 610, and to connect the processor 610 to peripheral devices such as the display screen 640. The MIPI interface can include a camera serial interface (CSI), a display serial interface (DSI), and the like.

[0116] In some embodiments, the ISP can transmit the processed digital image signal to the processor 610 through the CSI interface in the MIPI standard interface to realize the shooting function of the electronic device. The processor 610 and the display screen 640 communicate through the DSI interface, and finally convert it into an image displayed on the screen of the electronic device to realize the display function of the electronic device.

[0117] The electronic device implements the display function through a GPU, a display screen 640, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 640 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 610 may include one or more GPUs that execute program instructions to generate or change display information.

[0118] The display screen 640 is used to display images, videos, etc. The display screen 640 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 640, where N is a positive integer greater than 1.

[0119] A series of graphical user interfaces (GUIs), camera application interfaces, etc. can be displayed on the display screen 640 of the electronic device. These GUIs are the main screen of the electronic device. Generally speaking, the size of the display screen 640 of the electronic device is fixed, and only limited controls can be displayed on the display screen 640 of the electronic device. A control is a GUI element, which is a software component included in an application. It controls all data processed by the application and the interactive operations on these data. Users can interact with the control through direct manipulation to read or edit the relevant information of the application. Generally speaking, controls can include visual interface elements such as icons, controls, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0120] The electronic device can realize the shooting function through the ISP, the camera 630, the video codec, the GPU, the display screen 640 and the application processor.

[0121] The ISP is used to process the data fed back by the camera 630. For example, when taking a photo, the shutter is opened, and the ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 630.

[0122] The camera 630 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto a photosensitive element (also called an image sensor). The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 630, where N is a positive integer greater than 1.

[0123] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0124] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0125] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition.

[0126] The touch sensor 650A is also called a "touch control device". The touch sensor 650A can be set on the display screen 640, and the touch sensor 650A and the display screen 640 form a touch screen, also called a "touch control screen". The touch sensor 650A is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. For example, the user can perform an operation to open the camera on the display screen, that is, the user touches the camera icon, the touch sensor can receive the user's touch operation on the camera icon, and report the touch operation on the camera icon to the application processor. The application processor can respond to the above touch operation, determine the triggering of the photo taking event, start the application corresponding to the camera icon, and provide visual output related to the touch operation through the display screen 640. In other embodiments, the touch sensor 650A can also be set on the surface of the electronic device, which is different from the position of the display screen 640.

[0127] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0128] The software architecture of the present application is introduced below. The software architecture of the electronic device of the present application can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example for exemplary description. Figure 7 As shown:

[0129] like Figure 7 As shown, the software architecture of the electronic device of the present application can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example for exemplary description.

[0130] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely the application layer, the application framework layer, the Android runtime, the system library, and the kernel layer.

[0131] The application layer may include a series of application packages, for example, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0132] The application framework layer provides an application programming interface (API) and a programming framework for the application. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework may include a view system and a display system Android interface definition language (AIDL) interface and an activity recognition management service ARMS.

[0133] The application framework layer may include a photo-taking algorithm calling module, a preview algorithm calling module, and a preview algorithm frame rate adjustment module.

[0134] Among them, the camera algorithm calling module is used to call the camera algorithm to process the image.

[0135] The preview algorithm calling module is used to call the preview algorithm to process the captured image data in real time.

[0136] The preview algorithm frame rate adjustment is used to predict the resource consumption bottleneck unit of the photo algorithm in this photo process, the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, and the execution time reserved for the preview algorithm in the resource consumption bottleneck unit in this photo process. Among them, the resource consumption bottleneck unit refers to the computing unit with the longest time consumed in executing the photo algorithm in this photo process. The target execution frame rate of the preview algorithm per unit time is determined according to the execution time and the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit in this photo process, and the preview algorithm is executed according to the target execution frame rate.

[0137] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display interface of a camera application can include a view for displaying text and a view for displaying images.

[0138] The application layer and framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0139] The kernel layer is a layer between hardware and software. The kernel layer provided in the embodiment of the present application includes a display driver, an audio driver, and a sensor driver.

[0140] The technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program codes.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for adjusting the frame rate of a preview algorithm, characterized in that: include: In response to triggering a photographing event, executing a photographing algorithm to process the preview frame image data; Predict the resource consumption bottleneck unit of the photographing algorithm in this photographing process, the time consumed by executing a preview algorithm in the resource consumption bottleneck unit in this photographing process, and the execution time reserved for the preview algorithm in the resource consumption bottleneck unit in this photographing process. The resource consumption bottleneck unit refers to the computing unit that consumes the longest time to execute the photographing algorithm in this photographing process; Determine a target execution frame rate of the preview algorithm per unit time according to the execution time and the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit during the current photographing process; The preview algorithm is executed according to the target execution frame rate.

2. The method according to claim 1, characterized in that When there are multiple preview algorithms, the method further includes: Predicting the time consumed by each preview algorithm in the resource consumption bottleneck unit during the current photographing process and the total time consumed by the multiple preview algorithms in the resource consumption bottleneck unit during the current photographing process; The determining the target execution frame rate according to the execution time and the time consumed by executing a preview algorithm in the resource consumption bottleneck unit during the current photographing process includes: Determine the difference between the total execution time of the multiple preview algorithms in the resource consumption bottleneck unit and the execution time during the current photo shooting process; According to the gap value, determining a target gap value corresponding to each preview algorithm; The target execution frame rate corresponding to the preview algorithm i is determined according to the target gap value, the time consumed by the preview algorithm i in the resource consumption bottleneck unit during the current photo shooting process, and the time consumed by executing the preview algorithm i once in the resource consumption bottleneck unit, where i is a positive integer.

3. The method according to claim 2, characterized in that According to the gap value, the target gap value corresponding to each preview algorithm is determined, and the specific formula is as follows: ； in, is the target gap value corresponding to the preview algorithm i, G is the gap value, is the time consumed by the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process, is the total time consumed by the execution of the multiple preview algorithms in the resource consumption bottleneck unit during this photographing process, T is the time consumed by the execution of the photographing algorithm during this photographing process, The time consumed by executing the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process, is the frame rate of the preview algorithm within a unit time of 1s, and n is the number of preview algorithms executed during this photo-taking process.

4. The method according to claim 2, characterized in that: The specific formula for determining the target execution frame rate corresponding to the preview algorithm i is as follows: ； in, is the target execution frame rate corresponding to the preview algorithm i, The time consumed by executing the preview algorithm i in the resource consumption bottleneck unit during this photo shooting process, is the target gap value corresponding to the preview algorithm i, T is the time consumed by executing the photo algorithm during this photo shooting process, It is the frame rate of the preview algorithm within 1s.

5. The method according to claim 1, characterized in that The resource consumption bottleneck unit of the photographing algorithm predicted during the photographing process includes: Determine the average time consumed by executing the photo algorithm in each computing unit in the N photos taken most recently to the current photo moment, where N is a positive integer; The computing unit with the longest mean is determined as the resource consumption bottleneck unit.

6. The method according to claim 5, characterized in that The computing unit includes a GPU, an NPU, and a CPU. When the average time consumed by executing the photographing algorithm in the CPU during the N photographing times is equal to the average time consumed by executing the photographing algorithm in the NPU during the N photographing times and is greater than the average time consumed by executing the photographing algorithm in the GPU during the N photographing times, the NPU is determined as the resource consumption bottleneck unit; When the average time consumed by executing the photographing algorithm in the CPU during the N photographing times is equal to the average time consumed by executing the photographing algorithm in the GPU during the N photographing times and is greater than the average time consumed by executing the photographing algorithm in the NPU during the N photographing times, the GPU is determined as the resource consumption bottleneck unit; When the average value of the time consumed by executing the photographing algorithm in the NPU in the N photographing times is equal to the average value of the time consumed by executing the photographing algorithm in the GPU in the N photographing times and is greater than the average value of the time consumed by executing the photographing algorithm in the CPU in the N photographing times, the unit with the larger gap value between the NPU and the GPU is determined as the resource consumption bottleneck unit, and the gap value is the difference between the execution time reserved for the preview algorithm in the computing unit for this photographing and the execution time of the preview algorithm in the computing unit for this photographing.

7. The method according to claim 1, characterized in that Predict the execution time reserved for the preview algorithm in the resource consumption bottleneck unit during this photo shooting process, including: Determine the product of the weight coefficient and the time consumed by executing the photo algorithm in the resource consumption bottleneck unit during the current photo shooting process, wherein the weight coefficient is used to balance the resource usage of the photo shooting algorithm and the preview algorithm; The execution time reserved for the preview algorithm in the resource consumption bottleneck unit during the current photographing process is determined according to the difference between the time consumed by executing the photographing algorithm during the current photographing process and the product.

8. The method according to claim 7, characterized in that The average value of the time consumed by executing the preview algorithm in the resource consumption bottleneck unit within the preset frame rate is determined as the time consumed by executing the preview algorithm once in the resource consumption bottleneck unit during this photo shooting process; The average of the time consumed by executing the photographing algorithm in the resource consumption bottleneck unit in the N most recent photographing times before the current photographing moment is determined as the time consumed by executing the photographing algorithm in the resource consumption bottleneck unit during the current photographing process; The average of the time taken to execute the photographing algorithm in the N most recent photographing times is determined as the time consumed by executing the photographing algorithm in the current photographing process.

9. An electronic device, characterized in that: include: Processor and memory; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that: The method comprises computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

Images