Image processing method, electronic device, storage medium and chip system

Abstract

The image processing method, electronic device, storage medium and chip system provided by the embodiments of the present application relate to the field of terminal technology. The method includes: statistics on the execution status of each detection algorithm, the priority configured by each detection algorithm, the dynamically variable frame rate configured by each detection algorithm, and the operating status of the electronic device, etc., to comprehensively decide the execution of the detection algorithm. In this way, when the electronic device executes each detection algorithm, it can reduce the conflict generated when calling hardware resources, improve the smoothness of image preview, so that in the image preview scene, the interface of the electronic device can display the image evenly and continuously, and the picture is continuous and without jamming, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, an electronic device, a storage medium and a chip system. Background Art

[0002] Some electronic devices may support functions such as image preview, photo taking, and video recording. The image preview scene may include a scene of image preview displayed on the interface before taking a photo, and may also include a scene of image preview displayed on the interface during video recording.

[0003] However, in some image preview scenarios, electronic devices may experience interface freezes, which reduces the user experience. Summary of the invention

[0004] The image processing method, electronic device, storage medium and chip system provided in the embodiments of the present application can count the execution status of each detection algorithm, the priority configured by each detection algorithm, the dynamically variable frame rate configured by each detection algorithm, and the operating status of the electronic device, etc., and comprehensively decide the execution of the detection algorithm. In this way, when the electronic device executes each detection algorithm, it can reduce the conflict generated when calling hardware resources, improve the smoothness of image preview, and thus improve the user experience.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes:

[0006] At the first moment, the system performance level of the electronic device is the first level, and M of the N image processing algorithms are executed; wherein the execution priority of the M image processing algorithms is higher than the execution priority of the remaining image processing algorithms in the N image processing algorithms, N is an integer greater than 1, and M is a positive integer less than or equal to N; at the second moment, the system performance level of the electronic device is the second level, and L of the N image processing algorithms are executed; wherein the execution priority of the L image processing algorithms is higher than the execution priority of the remaining image processing algorithms in the N image processing algorithms, L is an integer greater than or equal to 0 and less than M, and the system performance corresponding to the first level is higher than the system performance corresponding to the second level. In this way, when the electronic device executes each detection algorithm, it can reduce the conflict generated when calling hardware resources, improve the smoothness of image preview, and thus improve the user experience.

[0007] In one possible implementation, before executing M of the N image processing algorithms, the method further includes: adjusting the execution priority of the N image processing algorithms; and determining to execute the M image processing algorithms based on the adjusted execution priority of the N image processing algorithms and the system performance level of the electronic device. The electronic device determines the image processing algorithm to be executed based on the priorities of multiple detection algorithms and the system performance of the electronic device. In this way, conflicts when calling hardware resources can be reduced, so that in the image preview scene, the interface of the electronic device can display the image evenly and continuously, and the picture is continuous and without lag.

[0008] In a possible implementation, the electronic device is configured with preset priorities, maximum execution frequencies, and minimum execution frequencies of N image processing algorithms, and the execution priorities of the N image processing algorithms are adjusted, including: if there are k consecutive frames in the electronic device that have not executed the target image processing algorithm, then on the basis of the preset priority of the target image processing algorithm, the execution priority of the target image processing algorithm is increased; wherein the N image processing algorithms include the target image processing algorithm, k is less than the maximum execution frequency of the target image processing algorithm, and k is greater than the minimum execution frequency of the target image processing algorithm. The electronic device can determine the priority of the algorithm according to the actual execution status of each algorithm, and for the target image processing algorithm that has not been executed for k consecutive frames, the electronic device can increase the execution priority of the algorithm. In this way, each algorithm can be executed within a reasonable time, the detection results of the algorithm can be updated in time, and the accuracy of image processing can be improved.

[0009] In a possible implementation, the closer k is to the maximum value of the execution frequency of the target image processing algorithm, the higher the execution priority of the target image processing algorithm. The closer k is to the maximum value of the execution frequency of the target image processing algorithm, the more frames the target image processing algorithm has not been executed in succession, and the longer the electronic device has not executed the target image processing algorithm. For image processing algorithms that have not been executed for multiple consecutive frames, the priority of the algorithm can be increased. In this way, the algorithm can be executed as quickly as possible, thereby reasonably scheduling multiple image processing algorithms in the electronic device.

[0010] In a possible implementation, adjusting the execution priorities of the N image processing algorithms includes: if the execution of the target image processing algorithm depends on the execution results of one or more other image processing algorithms among the N image processing algorithms, adjusting the target image processing algorithm and the one or more image processing algorithms to the same priority. In this way, the algorithms with dependencies are adjusted to the same priority, which does not affect or block the execution of the algorithms, thereby improving the execution efficiency of the algorithms.

[0011] In a possible implementation, a correspondence between a system performance level and a maximum number of executable image processing algorithms is configured in the electronic device. Based on the adjusted execution priorities of the N image processing algorithms and the system performance level of the electronic device, determining to execute M image processing algorithms includes: accumulating the first preset durations of the first i image processing algorithms among the N image processing algorithms according to the execution priorities of the N image processing algorithms to obtain a total execution duration, wherein i is a positive integer less than N, and the first preset duration of the i-th image processing algorithm is determined based on multiple previous execution durations of the i-th image processing algorithm; when the total execution duration is greater than a second preset duration and i-1 is greater than the maximum number of executable image processing algorithms corresponding to the first level, determining to execute M image processing algorithms, wherein M is equal to the maximum number of executable image processing algorithms corresponding to the first level; when the total execution duration is greater than the second preset duration and i-1 is less than the maximum number of executable image processing algorithms corresponding to the first level, determining to execute M image processing algorithms, wherein M is equal to i-1. The second preset duration is set as the display duration of each frame of image. In this way, the electronic device can execute a detection algorithm with a higher priority first within the time of displaying a frame of image, so that the electronic device can process the image based on the detection result of the high-priority detection algorithm to obtain a better image display effect.

[0012] In a possible implementation, if the execution time of the first image processing algorithm among the M image processing algorithms exceeds the first preset time of the first image processing algorithm, one or more image processing algorithms among the M image processing algorithms will not be executed, and the execution priority of the one or more image processing algorithms is lower than the execution priority of the first image processing algorithm. When the execution time of the first image processing algorithm exceeds the first preset time of the first image processing algorithm, it means that the execution time of the first image processing algorithm exceeds the predicted time corresponding to the algorithm, and the algorithm with a lower priority does not have time to execute, so the detection algorithm with a lower priority may not be executed. In this way, the impact on the operating performance of the electronic device can be reduced, the jamming of the electronic device can be reduced, and by not executing the detection algorithm with a lower priority, the impact on the image display can be reduced as much as possible, thereby not affecting the user experience.

[0013] In a possible implementation, the system performance corresponding to the first level is higher than the system performance corresponding to the second level, including: the frequency utilization rate of the device in the electronic device corresponding to the first level is lower than the frequency utilization rate of the device corresponding to the second level, and / or the temperature of the device corresponding to the first level is lower than the temperature of the device corresponding to the second level; wherein the device includes a device related to the execution of the image processing algorithm. The system performance is determined by considering the worst system environment in which the electronic device is located. In this way, when executing the image processing algorithm, the number of algorithms executed can be effectively controlled so that the performance of the electronic device will not become worse, thereby effectively maintaining the operating efficiency of the electronic device.

[0014] In one possible implementation, the device includes a central processing unit CPU, a graphics processing unit GPU, and / or a neural network processor NPU. In this way, the central processing unit CPU can be used to provide high-performance computing power, quickly process images, and can also work with the graphics processing unit GPU and the neural network processor NPU to further improve the speed and performance of image processing. The graphics processing unit GPU can be used for image enhancement, rendering, noise reduction and other processing, and can also increase the image processing speed and shorten the image processing time, thereby improving the operating efficiency of electronic devices. The neural network processor NPU can be used for image recognition, speech recognition and other processing, and can have higher execution efficiency when processing neural network calculations, which can improve the operating efficiency of electronic devices and reduce power consumption.

[0015] In a possible implementation, the N image processing algorithms include one or more of the following algorithms: a face detection algorithm, a face tracking algorithm, a subject detection algorithm, a motion detection algorithm, a scene detection algorithm, and an image quality assessment algorithm. The face detection algorithm can be used to detect faces and output face detection results. The face tracking algorithm can correct the current frame based on the face detection results of the previous frames, thereby reducing the jitter of the image. The subject detection algorithm can be used to identify and locate objects or subjects in the image, and after identifying the subject category, the image can be classified. The motion detection algorithm can detect whether there is a moving target in the current shooting scene based on multiple images, so that the electronic device can provide a more suitable shooting mode to the user. The scene detection algorithm can be used to detect the current shooting scene. The image quality assessment algorithm can evaluate the image quality of the current image. In this way, the image processing algorithm can identify and classify objects and features in the image, enhance the image, so that the image has a better display effect, thereby improving the user experience.

[0016] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The device may include a processing unit. The processing unit is used to implement any method related to processing performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements a method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements a method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0017] Exemplarily, the processing unit is used to execute M image processing algorithms out of N image processing algorithms; and is also used to execute L image processing algorithms out of N image processing algorithms.

[0018] In one possible implementation, the processing unit is used to adjust the execution priorities of N image processing algorithms, and is also used to determine to execute M image processing algorithms based on the adjusted execution priorities of the N image processing algorithms and the system performance level of the electronic device.

[0019] In a possible implementation, the processing unit is configured to increase the execution priority of the target image processing algorithm based on a preset priority of the target image processing algorithm.

[0020] In a possible implementation, the closer k is to the maximum value of the execution frequency of the target image processing algorithm, the higher the execution priority of the target image processing algorithm.

[0021] In a possible implementation, the processing unit is configured to adjust the target image processing algorithm and one or more image processing algorithms to the same priority.

[0022] In one possible implementation, the processing unit is used to accumulate the first preset durations of the first i image processing algorithms among the N image processing algorithms according to the execution priorities of the N image processing algorithms to obtain the total execution duration, and is also used to determine the execution of M image processing algorithms.

[0023] In a possible implementation, if the execution time of a first image processing algorithm among the M image processing algorithms exceeds a first preset time of the first image processing algorithm, one or more image processing algorithms among the M image processing algorithms are not executed.

[0024] In one possible implementation, the system performance corresponding to the first level is higher than the system performance corresponding to the second level, including: the frequency utilization rate of the devices in the electronic device corresponding to the first level is lower than the frequency utilization rate of the devices corresponding to the second level, and / or the temperature of the devices corresponding to the first level is lower than the temperature of the devices corresponding to the second level; wherein the devices include devices related to executing image processing algorithms.

[0025] In one possible implementation, the device includes a central processing unit CPU, a graphics processing unit GPU, and / or a neural network processor NPU.

[0026] In one possible implementation, the processing unit is used for N image processing algorithms including one or more of the following algorithms: face detection algorithm, face tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, and image quality assessment algorithm.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory, wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors are used to call computer instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0030] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0031] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0032] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a flow chart of an electronic device executing multiple algorithms provided in an embodiment of the present application;

[0036] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the execution flow of a perception decision module provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0041] 1. Terminology

[0042] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0043] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0044] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0045] 2. Electronic equipment

[0046] The electronic device of the embodiment of the present application may also be a terminal device in any form. For example, the electronic device may include: a mobile phone, a tablet computer, a PDA, a laptop computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, an electronic device in a 5G network or a public land mobile communication network (public land mobile communication network) to be evolved in the future. The electronic devices in the mobile network (PLMN) are not limited to this in the embodiments of the present application.

[0047] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0048] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in an Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0049] The electronic device in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0050] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0051] For example, Figure 1 A schematic structural diagram of an electronic device is shown.

[0052] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0053] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may include hardware, software, or a combination of software and hardware.

[0054] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0055] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system. For example, in an embodiment of the present application, the processor 110 may be used to obtain basic information of the CPU, GPU, and NPU of the electronic device, and may also be used to obtain the processing delay of each detection algorithm, the frame skipping situation of each detection algorithm, and the total delay of the detection algorithm processing, etc.

[0056] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0057] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor. For example, in an embodiment of the present application, the internal memory 121 may be used to store relevant codes of a face detection algorithm, a face tracking focus algorithm, a subject detection algorithm, a motion detection algorithm, a scene detection algorithm, an image quality assessment algorithm, etc.

[0058] In the embodiment of the present application, the electronic device realizes the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information. The electronic device can realize the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0059] In some embodiments, the electronic device may include 1 or N cameras 193 , where N is a positive integer greater than 1. The camera 193 may be used to capture still images or videos.

[0060] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1. For example, in an embodiment of the present application, the display screen may be used to display a preview image processed by a plurality of detection algorithms.

[0061] Figure 2 It is a software structure block diagram of the electronic device of the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system and hardware are layered, including the application layer, application framework layer, hardware adaptation layer (HAL), driver layer, and hardware layer from top to bottom.

[0062] The application layer can also be called the application layer, which can include a series of application packages. Figure 2 As shown, the application package may include applications such as camera, gallery, etc. Applications may include system applications and third-party applications.

[0063] The application framework layer can also be called the Framework layer. The Framework layer can provide application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The Framework layer can include some predefined functions.

[0064] For example, in an embodiment of the present application, the Framework layer may include a camera management module and a camera device module, etc. Among them, camera management can be used to obtain a list of available cameras on the device, and can also open a specified camera, etc. The camera device module can be used to manage and control the specific implementation of camera hardware. Exemplarily, the camera device module can provide an interface for the camera application to call, so as to realize the communication between the upper application and the hardware abstraction layer. In addition, the Framework layer may include an activity manager, a window manager, a resource manager, a notification manager, a content provider, and a view system, etc. For details, please refer to the relevant technology, which will not be repeated here.

[0065] It is understandable that the software architecture may also include Android runtime and system libraries, etc. For details, please refer to the relevant technology and will not be described here. HAL is an abstract layer structure between the kernel layer and Android runtime.

[0066] The HAL layer can be a package of the hardware driver, providing a unified interface for the call of the upper application. For example, in an embodiment of the present application, the HAL layer may include a camera hardware abstraction layer and a camera algorithm software architecture. Among them, the camera hardware abstraction layer may include multiple camera devices. The camera algorithm software architecture may include a pre-processing module, a preview processing module, a perception decision module, a perception detection module, and a post-processing module.

[0067] The pre-processing module can be used to perform preliminary processing on the raw image data after the image sensor has collected the raw image data. The preliminary processing may include but is not limited to one or more of the following: denoising, white balance adjustment, color correction, sharpening, and lens distortion correction.

[0068] The preview processing module can be used to process the image data transmitted by the pre-processing module, such as adjusting the tuning parameters of the image. The tuning parameters may include but are not limited to one or more of the following parameters: exposure parameters, white balance parameters, color parameters, contrast and brightness parameters, sharpening parameters, noise suppression parameters, lens correction parameters, special effects and filter parameters, autofocus parameters, etc.

[0069] The perception decision module can make a comprehensive decision on the execution of each detection algorithm based on the execution status of each detection algorithm and the operating status of the electronic device.

[0070] The perception detection module can be used to execute detection algorithms related to image semantic understanding, such as the face detection algorithm, face tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, image quality assessment algorithm, etc. in the embodiments of the present application.

[0071] The post-processing module can be used to further process and optimize the image data transmitted by the preview processing module, so as to improve the quality of the image, for example, including but not limited to one or more of the following processes: image enhancement processing, high dynamic range processing, image synthesis processing, special effects and filter processing, image compression and encoding processing, metadata embedding processing, and multi-frame processing.

[0072] Among them, image enhancement processing can include enhancing image details and edges, reducing noise in images, adjusting image color performance, optimizing image brightness and contrast, etc. High dynamic range processing can generate images with a higher dynamic range, thereby retaining more details in bright and dark areas. Special effects and filter processing can include beauty effects, color filters, blur effects, etc. The post-processing module can embed various metadata (Metadata) in the image file, such as shooting time, geographic location, camera settings (such as exposure, ISO, white balance, etc.), etc., for subsequent image management and search. Multi-frame processing can be used to merge multiple frames of images to reduce noise and increase brightness, etc.

[0073] The driver layer is used to drive hardware resources. The driver layer may include multiple driver modules. The driver layer may include camera device drivers, CPU drivers, GPU drivers, and NPU drivers. For example, in an embodiment of the present application, the camera algorithm software architecture may be used to obtain information such as the frequency of the CPU driver, the frequency of the GPU driver, and the frequency of the NPU driver.

[0074] The hardware layer includes image sensors, image signal processors ISP, central processing units CPU, graphics processing units GPU and neural network processors NPU, etc.

[0075] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0076] Some electronic devices may support functions such as image preview, photo taking, and video recording. The image preview scene may include the scene of image preview displayed on the interface before taking a photo, and may also include the scene of image preview displayed on the interface during video recording. In order to improve the user experience, some electronic devices may also execute a variety of algorithms related to image semantic understanding, such as face detection algorithms, scene detection algorithms, etc.

[0077] In possible implementations, Figure 3 A schematic diagram of a flow chart showing an electronic device executing multiple algorithms.

[0078] After the image sensor captures the image, it can be passed to the pre-processing module. After the pre-processing module performs preliminary processing, the processed image can be passed to the preview processing module and the perception detection module respectively.

[0079] The preview processing module can be used to perform image processing on the preview stream, such as adjusting the tuning parameters of the image, and passing the processed image to the post-processing module. After further processing the image, the post-processing module can pass the image to the application for displaying the preview image, and the application can then display the preview image in the interface.

[0080] It is understandable that the preview processing module and the post-processing module may rely on the detection results output by the perception detection module when performing image processing. The detection results include but are not limited to the detection results of one or more of the following detection algorithms: face detection algorithm, face tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, image quality assessment algorithm, etc.

[0081] Among them, the face detection algorithm can be used to detect the face and output the face detection result. The face detection result can include information such as the age and gender of the person. In a possible implementation, the perception detection module can detect the face once every 2 frames, that is, every other frame detection.

[0082] After detecting a face, on the one hand, the perception detection module can pass the face detection result to the preview processing module, and the preview processing module can adjust the image based on the face detection result, such as adjusting the brightness of the area near the face in the image, so that the image can present a better display effect.

[0083] On the other hand, the perception detection module can pass the face detection results to the face tracking algorithm, which can correct the current frame based on the face detection results of the previous frames, thereby reducing image jitter. In a possible implementation, the perception detection module can execute the face tracking algorithm every 4 frames, correct the current frame based on the images of the previous 4 frames, and output the detection results of the 5th frame.

[0084] The subject detection algorithm can be used to identify and locate objects or subjects in an image. For example, objects or subjects can include people, animals, various objects, etc. After identifying the subject category, the image can be classified. In a possible implementation, the perception detection module can perform subject detection for each frame. After detecting the subject, the perception detection module can pass the information to the motion detection algorithm.

[0085] The motion detection algorithm can detect whether there is a moving target in the current shooting scene based on multiple images. If there is a moving target, the perception detection module can pass the target type and motion information of the moving target to the post-processing module. The target type can include people, animals or other objects, and the motion information can include information such as the coordinates of the running target. The post-processing module can pass the preview image to the application, and can also instruct the application to display corresponding prompts in the interface. For example, the application interface can display a motion mode icon to prompt the user that the current operating mode can be selected; or, the application interface can display a toast prompt to prompt the user that the current operating mode can be selected to obtain a better image. In a possible implementation, the perception detection module can perform motion detection for each frame.

[0086] The scene detection algorithm can be used to detect the current shooting scene. For example, the shooting scene includes but is not limited to one or more of the following scenes: natural scenes, urban and architectural scenes, human activity scenes, animal and plant scenes, etc. Among them, natural scenes can include sunrise scenes, sunset scenes, etc.; urban and architectural scenes can include urban street scenes, indoor scenes, etc.; human activity scenes can include party scenes, work scenes, etc.; animal and plant scenes can include zoo scenes, garden scenes, etc. After detecting the scene, the perception detection module can pass the relevant information of the scene to the image quality assessment algorithm. In a possible implementation, the perception detection module can perform scene detection for each frame.

[0087] The image quality assessment algorithm can assess the image quality of the current image. In a possible implementation, the perception detection module can detect the image quality assessment algorithm every 3 frames and pass the algorithm result to the post-processing module. The post-processing module can process the image based on the result of the image quality assessment algorithm, for example, it can adjust the brightness and color of the image. The detection interval frame rate of the above-mentioned different algorithms of the specific perception detection module can be set according to actual conditions.

[0088] However, in some image preview scenarios, electronic devices may have interface freeze problems. When the perception detection module executes the above-mentioned detection algorithm, it may use hardware resources such as NPU and / or GPU, and the preview processing module and the post-processing module may also use hardware resources such as NPU and / or GPU. In addition, when taking pictures, hardware resources such as NPU and / or GPU may also be used in the photo stream. In this way, the electronic device may use hardware resources such as NPU and / or GPU at the same time, which may cause conflicts when calling hardware resources, causing the perception detection module to execute each detection algorithm. The execution speed slows down, resulting in freezes when previewing the image, and the smoothness of the preview interface is poor.

[0089] In view of this, the image processing method provided in the embodiment of the present application can count the execution status of each detection algorithm, the priority configured by each detection algorithm, the dynamically variable frame rate configured by each detection algorithm, and the operating status of the electronic device, etc., and comprehensively decide the execution of the detection algorithm. In this way, when the electronic device executes each detection algorithm, it can reduce the conflict generated when calling hardware resources, improve the smoothness of image preview, so that in the image preview scene, the interface of the electronic device can display the image evenly and continuously, and the picture is continuous and without lag, thereby improving the user experience.

[0090] The following specific embodiments are used to describe the method of the present application in detail. The following embodiments may be combined with each other or implemented independently, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0091] Figure 4 A schematic flow chart of an image processing method proposed in an embodiment of the present application is shown.

[0092] Relative to Figure 3 The corresponding implementation method is: Figure 4 The corresponding image processing method also includes a perception decision module. The perception decision module can comprehensively control the execution of the detection algorithm based on the statistical data of the frames before the current moment, the priority configured by each detection algorithm, the dynamically variable frame rate configured by each detection algorithm, and the current operating state of the electronic device, thereby improving the image processing speed, reducing the interface freeze in the image preview scene, and improving the user experience.

[0093] The statistical data of the frames before the current moment include the execution time of each detection algorithm, the execution interval frame rate of each detection algorithm, and the total execution time of all detection algorithms before the current moment. The current operating state of the electronic device may include the system performance of the current electronic device, the usage of hardware resources such as the current NPU and / or GPU, and the temperature of the current electronic device.

[0094] It is understandable that after each detection algorithm is executed, the processing result can be returned to the perception decision module, and the processing result can be used to indicate that the algorithm execution is completed. In this way, the perception decision module can count the execution status of each detection algorithm in the current frame, so as to be used for subsequent more accurate judgment on whether frame reduction processing or control of the execution of the detection algorithm is required. If a detection algorithm times out, the perception decision module can calculate the timeout time of the detection algorithm, which is convenient for subsequent statistics of the execution time of the detection algorithm, and for judging whether a low-priority detection algorithm needs to be executed.

[0095] Optionally, if a detection algorithm performs abnormally, the abnormal processing result can be returned to the perception decision module. In this way, when the perception decision module performs statistics and predictions on the detection algorithm, the data corresponding to the abnormal execution situation cannot be used as valid data, thereby performing statistics and predictions on the detection algorithm more accurately.

[0096] In a possible implementation, the perception decision module can set enable or disenable for each detection algorithm in the perception detection module. Exemplarily, if a detection algorithm needs to be executed, the perception decision module can set the enable flag corresponding to the detection algorithm to an enabled state; if a detection algorithm is not executed, the perception decision module can set the enable flag corresponding to the detection algorithm to a disabled state. In this way, the thread scheduling module that executes the detection algorithm can determine whether to execute the corresponding detection algorithm based on the enable flag. Optionally, if the performance of the electronic device is poor and some detection algorithms do not need to be executed, the perception decision module can also instruct the electronic device to perform frame reduction processing.

[0097] from Figure 4 It can be seen that in the embodiment of the present application, the detection results output by the perception detection module are mostly passed to the post-processing module. Since it takes a certain amount of time for the perception detection module to output the detection results, and it also takes a certain amount of time for the preview image to be processed by the preview processing module and then passed to the post-processing module, that is, when the post-processing module is performing image processing, it can more easily obtain the detection results output by the perception detection module, without spending too much time waiting, which is conducive to the synchronous execution of the image processing method flow and improves the execution efficiency of the algorithm.

[0098] Understandably, Figure 4 The corresponding embodiment only takes the face detection algorithm, face tracking algorithm, subject detection algorithm, and motion detection algorithm as examples to illustrate the process of the image processing method of the embodiment of the present application. The perception detection module can also execute the scene detection algorithm, image quality assessment algorithm, and other algorithms, which are not limited in the embodiment of the present application.

[0099] Optionally, the detection results of each detection algorithm output by the perception detection module can be passed to the preview processing module or the post-processing module respectively. For example, the detection result of the face detection algorithm can be passed to the preview processing module or the post-processing module; the detection result of the motion detection algorithm can be passed to the preview processing module or the post-processing module. The detection results of each specific detection algorithm can be passed to the preview processing module or the post-processing module according to the actual situation, which is not limited in the embodiments of the present application.

[0100] Figure 5 A schematic diagram of the execution flow of the perception decision module is shown.

[0101] S501. Obtain basic information of the current electronic device.

[0102] In an embodiment of the present application, the basic information of the electronic device may include but is not limited to one or more of the following information: CPU frequency, GPU frequency, NPU frequency, CPU temperature, GPU temperature, and NPU temperature.

[0103] Among them, taking the frequency of the CPU as an example, the embodiment of the present application can calculate the frequency utilization of the current CPU based on the frequency of the CPU. The frequency utilization can represent the ratio of the execution frequency of the current CPU to the full load frequency of the CPU, and can also be understood as the ratio between the workload of the CPU and the available processing power of the CPU. The frequency utilization of the CPU can be divided into the following intervals: [0%, 80%), [80%, 90%), [90%, 100%), and 100%, corresponding to the frequency levels 0, 1, 2, and 4 of the GPU, respectively. For example, when the frequency utilization of the CPU is within the interval of [0%, 80%), the frequency level of the GPU is 0; when the frequency utilization of the CPU is within the interval of [80%, 90%), the frequency level of the GPU is 1; when the frequency utilization of the CPU is within the interval of [90%, 100%), the frequency level of the GPU is 2; when the frequency utilization of the CPU is 100%, the frequency level of the GPU is 4. In this way, overall, when the CPU frequency utilization is low, the smaller the GPU frequency level value is, the higher the available frequency utilization of the GPU is, and the better the performance of the electronic device is; when the CPU frequency utilization is high, the larger the GPU frequency level value is, the lower the available frequency utilization of the GPU is, and the worse the performance of the electronic device is.

[0104] It is understandable that the CPU frequency utilization interval and the CPU frequency level can be divided according to actual conditions. For example, the frequency level of the GPU can also be set to 0, 1, 2, 3, or 1, 2, 3, 4, which is not limited in the embodiments of the present application. Similarly, the division of the frequency utilization level of the GPU and the division of the frequency utilization level of the NPU are similar to the division of the frequency utilization level of the CPU, but the GPU and the NPU can have their own corresponding frequency utilization intervals and corresponding frequency levels, which will not be repeated.

[0105] Taking the temperature of the CPU as an example, the temperature of the CPU can be divided into the following intervals: [0, 40), [40, 43), [43,45), and above 45, which correspond to the temperature levels 0, 1, 2, and 4 of the GPU respectively. For example, when the temperature of the CPU is in the interval [0, 40), the temperature level of the GPU is 0; when the temperature of the CPU is in the interval [40, 43), the temperature level of the GPU is 1; when the temperature of the CPU is in the interval [43, 45), the temperature level of the GPU is 2; when the temperature of the CPU is above 45 degrees, the temperature level of the GPU is 4. In this way, in general, when the temperature of the CPU is low, the smaller the value of the temperature level of the GPU is, the better the performance of the electronic device is; when the temperature of the CPU is high, the larger the value of the temperature level of the GPU is, the worse the performance of the electronic device is.

[0106] It is understandable that the temperature range of the CPU and the temperature level of the CPU can be divided according to the actual situation. For example, the temperature level of the GPU can also be set to 0, 1, 2, 3, or 1, 2, 3, 4, which is not limited in the embodiment of the present application. The division of the temperature level of the GPU and the division of the temperature level of the NPU are similar to the division of the temperature level of the CPU, but the GPU and the NPU can have their own corresponding temperature ranges and corresponding temperature levels, which will not be repeated.

[0107] S502: The perception decision module determines the current system environment level.

[0108] After calculating the CPU frequency utilization level and the CPU temperature level, the higher level value can be taken. For example, if the CPU frequency utilization level is 3 and the CPU temperature level is 4, the CPU level is 4. It can be understood that the GPU and NPU can also get their corresponding levels respectively.

[0109] In a possible implementation, the current system environment level can take the highest value among the CPU, GPU and NPU. For example, if the CPU level is 4, the GPU level is 2, and the NPU level is 3, the perception decision module can determine that the current system environment level is 4.

[0110] It can also be understood that the current system environment level setting is related to the frequency utilization of the current device, and / or the temperature of the current device, and the device may include a CPU, a GPU, and an NPU, etc. The lower the frequency utilization of the current device, and / or the lower the temperature of the current device, the higher the system performance of the electronic device. For example, when the frequency utilization level of the device is 4 and the temperature level of the device is 3, the system environment level is 4, that is, the current system environment level setting is related to the frequency utilization of the current device. When the frequency utilization level of the device is 3 and the temperature level of the device is 4, the system environment level is 4, that is, the current system environment level setting is related to the temperature of the current device. When the frequency utilization level of the device is 4 and the temperature level of the device is 4, the system environment level is 4, that is, the current system environment level setting is related to the frequency utilization of the current device and the temperature of the current device.

[0111] It can be understood that taking the highest level value can estimate and take into account the worst system environment of the electronic device. In this way, when executing the detection algorithm, the number of detection algorithms can be effectively controlled so that the performance of the electronic device will not become worse, thereby effectively maintaining the operating efficiency of the electronic device.

[0112] S503, the perception decision module obtains the execution time and frame skipping of each detection algorithm, as well as the total execution time of all detection algorithms.

[0113] In the embodiment of the present application, the execution time of each detection algorithm can also be understood as the average execution time of each detection algorithm. In some scenarios, the execution time of an algorithm can also be called the processing delay of the algorithm, and the average execution time of an algorithm can also be called the average processing delay of the algorithm.

[0114] The frame skipping of a detection algorithm can be understood as the detection algorithm not being executed for several consecutive frames.

[0115] The total execution time of all detection algorithms can be understood as the sum of the average execution times of each detection algorithm.

[0116] In a possible implementation, the perception decision module may record the execution status of each detection algorithm through a Map data structure. Optionally, the perception decision module may also record the execution status of each detection algorithm through other data structures or storage methods, which is not limited in the embodiment of the present application.

[0117] Taking the face detection algorithm as an example, the Map data structure can record the execution of the face detection algorithm for multiple consecutive frames, for example, the execution of the face detection algorithm for about 20 consecutive frames can be recorded. Taking 20 consecutive frames as an example, the Map can record whether the face detection algorithm is executed in each frame.

[0118] If the face detection algorithm is executed in a certain frame, the Map will record information such as the execution duration of the face detection algorithm, the start time and end time of the face detection algorithm execution, and the execution result of the face detection algorithm.

[0119] If the face detection algorithm is not executed in a certain frame, it means that there is a frame skipping situation, and the frame skipping situation of the face detection algorithm will be recorded in the Map. For example, among 20 consecutive frames, if only the nth frame is not executed, it means that the face detection algorithm has skipped 1 frame; if the nth frame and the n+1th frame are not executed, it means that the face detection algorithm has skipped 2 frames; if the nth frame to the n+mth frame are not executed, it means that the face detection algorithm has skipped m+1 frames. Among them, n and m are both integers greater than or equal to 0, and n+m is less than or equal to 20.

[0120] The perception decision module can calculate the average execution time of the face detection algorithm based on the execution of the face detection algorithm for about 20 consecutive frames. For example, if 16 of the 20 frames record the execution time of the face detection algorithm, the perception decision module can add up the execution time of the 16 face detection algorithms and divide it by 16 to get the average execution time. In this way, when predicting the execution time of the algorithm, the perception decision module can use the average execution time as the predicted time, which is convenient for subsequent judgment of whether the execution time of the detection algorithm exceeds the predicted time, so as to perform corresponding processing.

[0121] It is understandable that by calculating the average execution time of the detection algorithm through multiple sets of data, more accurate and objective data can be obtained. If the number of consecutive frames recorded is too small, the data obtained may be accidental and insufficient to represent the average execution time of the detection algorithm; if the number of consecutive frames recorded is too large, the memory occupied is large and the amount of calculation also increases. Therefore, the perception decision module can record the execution of the face detection algorithm for about 20 consecutive frames. The specific number of consecutive frames recorded is not limited in the embodiments of the present application, as long as the average execution time of the detection algorithm can be calculated more accurately without occupying more memory.

[0122] S504: Obtain the priority of each detection algorithm configuration and the dynamically variable frame rate.

[0123] The electronic device may be pre-configured with priorities and dynamically variable frame rates corresponding to various detection algorithms.

[0124] In a possible implementation, the smaller the value of the priority, the higher the corresponding priority. For example, the priority of the face detection algorithm can be set to 1, the priority of the face tracking algorithm can be set to 10, the priority of the subject detection algorithm can be set to 20, and the priority of the motion detection algorithm can be set to 30. Optionally, the electronic device can also set the larger the value of the priority, the higher the corresponding priority. The execution priority of each specific detection algorithm can be configured according to actual business needs, and is not limited in the embodiments of the present application.

[0125] Among them, the dynamically variable frame rate can be understood as the interval frame rate at which the perception detection module executes the detection algorithm is variable, and can also be understood as the period at which the perception detection module executes the detection algorithm is variable.

[0126] In a possible implementation, the electronic device can configure the maximum value Max and the minimum value Min of the execution frame rate for each detection algorithm. For example, if the minimum value Min of the execution frame rate configured by a detection algorithm is 2, and the maximum value Max of the execution frame rate is 4, it means that the dynamic variable frame rate of the detection algorithm can be 2 frames to 4 frames, and the perception detection module can execute the detection algorithm once every 2 frames, once every 3 frames, or once every 4 frames. It can be understood that if the minimum value and the maximum value of the execution frame rate configured by a detection algorithm are equal, it means that the execution frame rate of the detection algorithm is fixed.

[0127] The perception detection module can determine the interval frame rate for executing the detection algorithm according to the actual operation of the electronic device. For example, if the performance of the electronic device is good, the perception detection module can execute the detection algorithm every 2 frames; if the performance of the electronic device deteriorates, the perception detection module can execute the detection algorithm every 3 frames; if the performance of the electronic device continues to deteriorate, the perception detection module can execute the detection algorithm every 4 frames.

[0128] In this way, when the performance of the electronic device is better, the interval frame rate of executing the detection algorithm can be reduced, the number of executions of the detection algorithm can be increased, and the accuracy of the detection results can be improved; when the performance of the electronic device is poor, the interval frame rate of executing the detection algorithm can be increased, the number of executions of the detection algorithm can be reduced, and the computing power of the electronic device can be reduced, thereby reducing the consumption of the performance of the electronic device.

[0129] S505. For the current frame, the perception decision module determines the priority of each detection algorithm to be executed.

[0130] The perception decision module can decide the priority of each detection algorithm to be executed for the current frame based on the information in step S503 and step S504.

[0131] The perception decision module can adjust the priority of these detection algorithms according to the execution status of each detection algorithm. For example, taking the dynamic variable frame rate of a detection algorithm configured by the electronic device as 1 frame to 10 frames, if the detection algorithm has not been executed for i consecutive frames, i is an integer greater than 1 and less than 10, the perception decision module can increase the priority of the detection algorithm. If the detection algorithm has not been executed for j consecutive frames, j is an integer greater than i and less than 10, the perception decision module can continue to increase the priority of the detection algorithm.

[0132] In this way, the electronic device determines the priority of each detection algorithm according to its actual execution status, and can increase the execution priority of a detection algorithm that has not been executed for multiple consecutive frames. This allows the perception decision module to reasonably schedule multiple image processing algorithms in the electronic device, so that each algorithm can be executed within a reasonable time, and the detection results of the algorithm can be updated in time, thereby improving the accuracy of image processing.

[0133] In a possible implementation, the perception decision module can traverse the detection algorithms from high to low according to their priorities, that is, traverse the detection algorithms with high priorities first. Assuming that the detection algorithm with the highest priority is the face detection algorithm, the perception decision module can traverse the face detection algorithm first, and the perception decision module can determine whether the face detection algorithm needs to be executed in the current frame based on the dynamically variable frame rate of the face detection algorithm. For example, if the dynamically variable frame rate of the face detection algorithm is 2 to 3 frames, and there have been 2 frames in which the face detection algorithm has not been executed, the perception decision module can determine that the face detection algorithm needs to be executed in the current frame.

[0134] It is understandable that among the detection algorithms executed by the perception detection module, there are some dependencies between the algorithms, for example, the execution of detection algorithm A depends on the detection result of detection algorithm B. In this case, the electronic device can pre-configure the detection algorithms with the same priority with the dependencies, so as not to affect or block the execution of the detection algorithms and improve the execution efficiency of the algorithms.

[0135] Exemplarily, when the perception decision module increases the priority of detection algorithm A, it can be determined whether detection algorithm A depends on the detection results of other detection algorithms. If the perception decision module determines that detection algorithm A depends on the detection results of detection algorithm B, the perception decision module can simultaneously increase the priority of detection algorithm A and detection algorithm B. Optionally, if the performance of the electronic device is poor, the perception decision module may not increase the priority of detection algorithm A and detection algorithm B because increasing the priority of multiple detection algorithms may affect the performance of the electronic device.

[0136] There may be multiple dependencies between detection algorithms, which can also be understood as indirect dependencies between detection algorithms. For example, the execution of detection algorithm A depends on the detection result of detection algorithm B, the execution of detection algorithm B depends on the detection result of detection algorithm C, the execution of detection algorithm C depends on the detection result of detection algorithm D, etc. This situation is similar to the above implementation. The perception decision module can simultaneously increase the priority of detection algorithm A, detection algorithm B, detection algorithm C, and detection algorithm D. Alternatively, the perception decision module may not increase the priority of detection algorithm A, detection algorithm B, detection algorithm C, and detection algorithm D according to the performance of the electronic device.

[0137] S506: Select an algorithm with a higher priority to execute according to the system environment level.

[0138] In a possible implementation, the perception decision module can successively accumulate the average execution time of each detection algorithm in order from high to low priority to obtain the total execution time of the algorithms involved in the accumulation. When the total execution time is less than or equal to the preset time, the perception decision module can execute the detection algorithms involved in the accumulation in order from high to low priority. Among them, the preset time can be set to the display time of a frame of image. It is understandable that in order not to affect the display effect, the electronic device can display 30 frames and more than 30 frames of images per second. Taking the display of 30 frames of images per second as an example, the display time of each frame of the image is about 30 milliseconds (ms), so the preset time can be set to be less than or equal to 30ms. The preset time can be set according to the actual situation, and the specific value of the preset time is not limited in the embodiment of the present application.

[0139] Exemplarily, assuming that there are N detection algorithms sorted from high to low in priority, when it is calculated that the total execution time of the first M algorithms is less than or equal to the preset time, and the total execution time of the first M+1 algorithms is greater than the preset time, the perception decision module can instruct the perception detection module to execute the first M algorithms. Wherein N is an integer greater than 1, and M is an integer greater than or equal to 0 and less than or equal to N. In this way, within the time of displaying a frame of image, the detection algorithm with higher priority can be executed first, so that the electronic device can process the image based on the detection result of the detection algorithm with higher priority to obtain a better image display effect.

[0140] Optionally, the perception decision module may also consider the performance of the electronic device and set an upper limit on the number of detection algorithms to be executed. If the performance of the current electronic device is relatively good, a larger number of detection algorithms may be allowed to be executed, for example, all or most of the detection algorithms may be allowed to be executed. If the performance of the current electronic device is relatively poor, a smaller number of detection algorithms may be allowed to be executed.

[0141] Exemplarily, taking the perception detection module including 20 detection algorithms as an example, for the current frame, if the system environment level of the current electronic device is 0, indicating that the performance of the electronic device is good, then the upper limit of the number of detection algorithms executed can be set in the electronic device to 20, that is, all algorithms are allowed to be executed. If the system environment level of the current electronic device is 1, indicating that the performance of the electronic device is good, then the upper limit of the number of detection algorithms executed can be set in the electronic device to 15; if the system environment level of the current electronic device is 2, indicating that the performance of the electronic device is poor, then the upper limit of the number of detection algorithms executed can be set in the electronic device to 5; if the system environment level of the current electronic device is 4, indicating that the performance of the electronic device is very poor, in order not to affect the operating performance of the electronic device, the upper limit of the number of detection algorithms executed can be set in the electronic device to 0, that is, no detection algorithm is allowed to be executed. The upper limit of the number of detection algorithms executed corresponding to the specific different system environment levels can be set according to the actual situation, and the value of the specific upper limit is not limited in the embodiment of the present application.

[0142] It can be understood that, taking the system environment level of the electronic device as 1 as an example, the upper limit of the number of detection algorithms to be executed in the electronic device can be set to 15. If the above-mentioned perception decision module calculates and determines that the value of M is greater than 15, the perception decision module can instruct the perception detection module to execute the 15 algorithms with the highest priority among the M detection algorithms. If the above-mentioned perception decision module calculates and determines that the value of M is less than or equal to 15, the perception decision module can instruct the perception detection module to execute M detection algorithms. The judgment method for the system environment level of the electronic device being 2 is similar to that for the system environment level of the electronic device being 1, and will not be repeated.

[0143] It can be understood that if the perception detection module executes a detection algorithm with a higher priority, and the execution time of the detection algorithm exceeds the predicted time corresponding to the detection algorithm, the perception decision module can determine that the detection algorithm with a lower priority has no time to execute, then the perception decision module can instruct the perception detection module not to execute the detection algorithm with a lower priority.

[0144] In an embodiment of the present application, if a detection algorithm with a higher priority times out, the perception decision module can instruct not to execute a detection algorithm with a lower priority, thereby reducing the impact on the operating performance of the electronic device and reducing the lag of the electronic device. Not executing a detection algorithm with a lower priority can minimize the impact on image display, thereby not affecting the user experience.

[0145] Figure 6 The image processing method of the embodiment of the present application is shown, which is applied to an electronic device. The method includes:

[0146] S601. At a first moment, the system performance level of the electronic device is the first level, and M image processing algorithms out of N image processing algorithms are executed; wherein the execution priority of the M image processing algorithms is higher than the execution priority of the remaining image processing algorithms out of the N image processing algorithms, N is an integer greater than 1, and M is a positive integer less than or equal to N.

[0147] In the embodiment of the present application, the first moment can be understood as the moment when the system performance level of the electronic device is the first level and M image processing algorithms among N image processing algorithms are executed.

[0148] The system performance level of electronic equipment can be understood as Figure 5 Corresponding to the system environment level in the embodiment, the system performance level can be determined based on the basic information of the electronic device. The basic information of the specific electronic device can be referred to Figure 5 The system environment level can refer to the relevant description in step S501 of the corresponding embodiment. Figure 5 The relevant description in step S502 of the corresponding embodiment is not repeated. The N image processing algorithms may include the face detection algorithm, face tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, image quality assessment algorithm, etc. mentioned in the embodiment of the present application.

[0149] S602. At the second moment, the system performance level of the electronic device is the second level, and L of the N image processing algorithms are executed; wherein the execution priority of the L image processing algorithms is higher than the execution priority of the remaining image processing algorithms in the N image processing algorithms, L is an integer greater than or equal to 0 and less than M, and the system performance corresponding to the first level is higher than the system performance corresponding to the second level.

[0150] In the embodiment of the present application, the second moment can be understood as the moment when the system performance level of the electronic device is the second level and L of the N image processing algorithms are executed.

[0151] The first level and the second level can be any level in the system performance level, but it is required that the system performance corresponding to the first level is higher than the system performance corresponding to the second level.

[0152] The image processing method provided in the embodiment of the present application can make a comprehensive decision on the execution of the detection algorithm based on the priority of multiple detection algorithms and the system performance of the electronic device. In this way, when the electronic device executes each detection algorithm, it can reduce the conflict generated when calling hardware resources, improve the smoothness of image preview, and thus improve the user experience.

[0153] Optional, in Figure 6On the basis of the corresponding embodiment, before executing M image processing algorithms among N image processing algorithms, it can also include: adjusting the execution priority of the N image processing algorithms; and determining to execute the M image processing algorithms based on the adjusted execution priority of the N image processing algorithms and the system performance level of the electronic device.

[0154] In the embodiment of the present application, the specific process of adjusting the execution priority of N image processing algorithms can be referred to Figure 5 The relevant description of step S505 of the corresponding embodiment; based on the adjusted execution priority of the N image processing algorithms and the system performance level of the electronic device, the process of determining the execution of the M image processing algorithms can refer to Figure 5 The relevant description in step S506 of the corresponding embodiment is not repeated here.

[0155] It is understandable that the electronic device determines the image processing algorithm to be executed based on the priority of multiple detection algorithms and the system performance of the electronic device. In this way, conflicts when calling hardware resources can be reduced, so that in the image preview scene, the interface of the electronic device can display the image evenly and continuously, and the picture is continuous and without lag.

[0156] Optional, in Figure 6 On the basis of the corresponding embodiments, the electronic device is configured with preset priorities, maximum execution frequencies and minimum execution frequencies of N image processing algorithms. Adjusting the execution priorities of the N image processing algorithms may include: if there are k consecutive frames in the electronic device where the target image processing algorithm has not been executed, then based on the preset priority of the target image processing algorithm, increasing the execution priority of the target image processing algorithm; wherein the N image processing algorithms include the target image processing algorithm, k is less than the maximum execution frequency of the target image processing algorithm, and k is greater than the minimum execution frequency of the target image processing algorithm.

[0157] In the embodiment of the present application, the preset priority can be understood as Figure 5 In step S504 of the corresponding embodiment, the electronic device is pre-configured with priorities corresponding to various detection algorithms.

[0158] The maximum and minimum execution frequencies can be understood as Figure 5 The dynamic variable frame rate of the algorithm in step S504 of the corresponding embodiment. The maximum value of the execution frequency can be understood as the maximum value of the interval frame rate of the algorithm execution, or as the maximum value Max of the algorithm execution frame rate. Similarly, the minimum value of the execution frequency can be understood as the minimum value of the interval frame rate of the algorithm execution, or as the minimum value Min of the algorithm execution frame rate.

[0159] For details on improving the execution priority of the target image processing algorithm, please refer to Figure 5The relevant descriptions in step S504 and step S505 of the corresponding embodiment are not repeated here.

[0160] It is understandable that the electronic device can determine the priority of the algorithm according to the actual execution status of each algorithm. If the target image processing algorithm has not been executed for k consecutive frames, the electronic device can increase the execution priority of the algorithm. In this way, each algorithm can be executed within a reasonable time, the detection results of the algorithm can be updated in time, and the accuracy of image processing can be improved.

[0161] Optional, in Figure 6 On the basis of the corresponding embodiment, the closer k is to the maximum value of the execution frequency of the target image processing algorithm, the higher the execution priority of the target image processing algorithm.

[0162] In the embodiment of the present application, the closer k is to the maximum value of the execution frequency of the target image processing algorithm, the more frames the target image processing algorithm has not been executed in a row, and the longer the electronic device has not executed the target image processing algorithm. For an image processing algorithm that has not been executed for multiple consecutive frames, the priority of the algorithm can be increased. For details, please refer to Figure 5 The relevant description in step S505 of the corresponding embodiment will not be repeated. In this way, the algorithm can be executed as quickly as possible, thereby reasonably scheduling multiple image processing algorithms in the electronic device.

[0163] Optional, in Figure 6 Based on the corresponding embodiments, adjusting the execution priorities of N image processing algorithms may include: if the execution of the target image processing algorithm depends on the execution results of one or more other image processing algorithms among the N image processing algorithms, then adjusting the target image processing algorithm and one or more image processing algorithms to the same priority.

[0164] In the embodiment of the present application, the execution logic of the dependencies between algorithms can refer to Figure 5 The relevant description in step S505 of the corresponding embodiment is not repeated here. It can be understood that adjusting the algorithms with dependencies to the same priority level can avoid affecting or blocking the execution of the algorithms, thereby improving the execution efficiency of the algorithms.

[0165] Optional, in Figure 6On the basis of the corresponding embodiment, the electronic device is configured with a correspondence between the system performance level and the maximum number of executable image processing algorithms. Based on the adjusted execution priorities of the N image processing algorithms and the system performance level of the electronic device, determining to execute M image processing algorithms may include: according to the execution priorities of the N image processing algorithms, accumulating the first preset durations of the first i image processing algorithms in the N image processing algorithms to obtain a total execution duration, wherein i is a positive integer less than N, and the first preset duration of the i-th image processing algorithm is determined based on multiple execution durations of the i-th image processing algorithm before; when the total execution duration is greater than the second preset duration, and i-1 is greater than the maximum number of executable image processing algorithms corresponding to the first level, determining to execute M image processing algorithms, wherein M is equal to the maximum number of executable image processing algorithms corresponding to the first level; when the total execution duration is greater than the second preset duration, and i-1 is less than the maximum number of executable image processing algorithms corresponding to the first level, determining to execute M image processing algorithms, wherein M is equal to i-1.

[0166] In the embodiment of the present application, the maximum number of executable image processing algorithms can be understood as the upper limit of the number of executable image processing algorithms. The specific correspondence between the system performance level and the maximum number of executable image processing algorithms can be referred to. Figure 5 The relevant description in step S506 of the corresponding embodiment is not repeated here.

[0167] The first preset duration of the i-th image processing algorithm can be understood as the duration of the i-th image processing algorithm predicted by the electronic device, or can be understood as Figure 5 The predicted duration in step S503 of the corresponding embodiment. Determined based on the duration of multiple executions of the i-th image processing algorithm before, it may include being determined based on the average duration of multiple executions of the i-th image processing algorithm before, it may also include being determined based on the maximum duration of multiple executions of the i-th image processing algorithm before, it may also be determined based on the shortest duration of multiple executions of the i-th image processing algorithm before, or it may adopt other calculation methods, which are not limited in the embodiments of the present application.

[0168] The second preset duration can be understood as Figure 5 The preset duration in step S506 of the corresponding embodiment, for example, the second preset duration may include the display duration of each frame of the image, and may also be set according to actual conditions, which is not limited in the embodiment of the present application.

[0169] The second preset duration is set as the display duration of each frame of image. In this way, the electronic device can execute a detection algorithm with a higher priority first within the time of displaying a frame of image, so that the electronic device can process the image based on the detection result of the high-priority detection algorithm to obtain a better image display effect.

[0170] Optional, in Figure 6 Based on the corresponding embodiment, if the execution time of the first image processing algorithm among the M image processing algorithms exceeds the first preset time of the first image processing algorithm, one or more image processing algorithms among the M image processing algorithms will not be executed, and the execution priority of one or more image processing algorithms is lower than the execution priority of the first image processing algorithm.

[0171] In the embodiment of the present application, the first image processing algorithm may be any one of the M image processing algorithms. The first preset duration of the first image processing algorithm may be understood as being determined based on the duration of multiple previous executions of the first image processing algorithm. The specific determination method may refer to the related description of the first preset duration of the i-th image processing algorithm mentioned above, and will not be described in detail.

[0172] It can be understood that not executing one or more of the M image processing algorithms can be understood as a state of the electronic device, or it can be understood as the electronic device setting the enable state of the one or more image processing algorithms to disabled, or it can be understood as the electronic device instructing the algorithm scheduling thread not to call the one or more image processing algorithms, etc., so as to achieve the purpose of not executing the one or more image processing algorithms.

[0173] When the execution time of the first image processing algorithm exceeds the first preset time of the first image processing algorithm, it means that the execution time of the first image processing algorithm exceeds the predicted time corresponding to the algorithm, and the algorithm with lower priority has no time to execute, so the detection algorithm with lower priority may not be executed. In this way, the impact on the operating performance of the electronic device can be reduced, the jamming of the electronic device can be reduced, and by not executing the detection algorithm with lower priority, the impact on the image display can be reduced as much as possible, thereby not affecting the user experience.

[0174] Optional, in Figure 6 On the basis of the corresponding embodiments, the system performance corresponding to the first level is higher than the system performance corresponding to the second level including: the frequency utilization rate of the devices in the electronic device corresponding to the first level is lower than the frequency utilization rate of the devices corresponding to the second level, and / or the temperature of the devices corresponding to the first level is lower than the temperature of the devices corresponding to the second level; wherein the devices include devices related to executing image processing algorithms.

[0175] In the embodiment of the present application, the frequency utilization rate of the device can be understood as the ratio of the current execution frequency of the device to the full load frequency of the device, or it can be understood as the ratio between the workload of the device and the available processing capacity of the device.

[0176] From the relationship described above, we can see that when the frequency utilization rate of a device is relatively low, it means that the device still has available frequency space, and accordingly, the system performance will be relatively good; when the temperature of a device is relatively low, it means that the device is in good operating condition and the system performance is relatively good. For details, please refer to Figure 5 The relevant description in step S502 of the corresponding embodiment is not repeated here.

[0177] It is understandable that the system performance is determined by considering the worst system environment in which the electronic device is located. In this way, when executing the image processing algorithm, the number of executed algorithms can be effectively controlled so that the performance of the electronic device will not become worse, thereby effectively maintaining the operating efficiency of the electronic device.

[0178] Optional, in Figure 6 Based on the corresponding embodiments, the device includes a central processing unit CPU, a graphics processing unit GPU, and / or a neural network processor NPU.

[0179] In the embodiment of the present application, the central processing unit CPU can be used to provide high-performance computing power, quickly process images, and can also work together with the image processor GPU and the neural network processor NPU to further improve the speed and performance of image processing.

[0180] The image processor GPU can be used for image enhancement, rendering, noise reduction and other processing. It can also increase the image processing speed and shorten the image processing time, thereby improving the operating efficiency of electronic devices.

[0181] The neural network processor NPU can be used for image recognition, voice recognition and other processing. It can have higher execution efficiency when processing neural network calculations, and can improve the operating efficiency of electronic devices and reduce power consumption.

[0182] Optional, in Figure 6 On the basis of the corresponding embodiments, the N image processing algorithms include one or more of the following algorithms: face detection algorithm, face tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, and image quality assessment algorithm.

[0183] In an embodiment of the present application, a face detection algorithm can be used to detect a face and output a face detection result. A face tracking algorithm can correct the current frame based on the face detection results of the previous frames, thereby reducing image jitter. A subject detection algorithm can be used to identify and locate objects or subjects in an image, and after identifying the subject category, the image can be classified. A motion detection algorithm can detect whether there is a moving target in the current shooting scene based on multiple images, so that the electronic device can provide a more suitable shooting mode to the user. A scene detection algorithm can be used to detect the current shooting scene. An image quality assessment algorithm can assess the image quality of the current image.

[0184] From this we can see that image processing algorithms can identify and classify objects and features in images, enhance images, make images have better display effects, and thus improve user experience.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0186] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0187] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0188] like Figure 7FIG. 7 is a schematic diagram of a chip structure provided by an embodiment of the present application. The chip 700 includes one or more (including two) processors 701 , a communication line 702 , a communication interface 703 and a memory 704 .

[0189] In some implementations, the memory 704 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0190] The method described in the above embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in the form of software. The above processor 701 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 701 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0191] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 704, and the processor 701 reads the information in the memory 704 and completes the steps of the above method in combination with its hardware.

[0192] The processor 701 , the memory 704 , and the communication interface 703 may communicate with each other via the communication line 702 .

[0193] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0194] The embodiment of the present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0195] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0196] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers.

[0197] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: At a first moment, the system performance level of the electronic device is the first level, and M image processing algorithms among N image processing algorithms are executed; wherein the execution priority of the M image processing algorithms is higher than the execution priority of the remaining image processing algorithms among the N image processing algorithms, N is an integer greater than 1, and M is a positive integer less than or equal to N; At the second moment, the system performance level of the electronic device is the second level, and L image processing algorithms among the N image processing algorithms are executed; wherein the execution priority of the L image processing algorithms is higher than the execution priority of the remaining image processing algorithms among the N image processing algorithms, L is an integer greater than or equal to 0 and less than M, and the system performance corresponding to the first level is higher than the system performance corresponding to the second level; Before executing M of the N image processing algorithms, the method further includes: adjusting the execution priorities of the N image processing algorithms; and determining to execute the M image processing algorithms based on the adjusted execution priorities of the N image processing algorithms and the system performance level of the electronic device; The electronic device is configured with preset priorities, maximum execution frequencies, and minimum execution frequencies of the N image processing algorithms, and the adjusting of the execution priorities of the N image processing algorithms includes: If there are k consecutive frames in the electronic device in which the target image processing algorithm has not been executed, the execution priority of the target image processing algorithm is increased based on the preset priority of the target image processing algorithm; wherein the N image processing algorithms include the target image processing algorithm, the k is less than the maximum value of the execution frequency of the target image processing algorithm, and the k is greater than the minimum value of the execution frequency of the target image processing algorithm.

2. The method according to claim 1, characterized in that: The closer k is to the maximum value of the execution frequency of the target image processing algorithm, the higher the execution priority of the target image processing algorithm.

3. The method according to claim 2, characterized in that The adjusting the execution priorities of the N image processing algorithms further includes: If the execution of the target image processing algorithm depends on the execution results of one or more other image processing algorithms among the N image processing algorithms, the target image processing algorithm and the one or more image processing algorithms are adjusted to the same priority.

4. The method according to any one of claims 1 to 3, characterized in that: The electronic device is configured with a correspondence between a system performance level and a maximum number of executable image processing algorithms, and determining to execute the M image processing algorithms based on the adjusted execution priorities of the N image processing algorithms and the system performance level of the electronic device includes: According to the execution priorities of the N image processing algorithms, first preset durations of first i image processing algorithms among the N image processing algorithms are accumulated to obtain a total execution duration, wherein i is a positive integer less than N, and the first preset duration of the i-th image processing algorithm is determined based on multiple previous execution durations of the i-th image processing algorithm; When the total execution time is greater than a second preset time, and i-1 is greater than the maximum number of executable image processing algorithms corresponding to the first level, determine to execute the M image processing algorithms, wherein M is equal to the maximum number of executable image processing algorithms corresponding to the first level; When the total execution time is greater than the second preset time and i-1 is less than the maximum number of executable image processing algorithms corresponding to the first level, determine to execute the M image processing algorithms, wherein M is equal to i-1.

5. The method according to claim 4, characterized in that If the execution time of a first image processing algorithm among the M image processing algorithms exceeds a first preset time of the first image processing algorithm, one or more image processing algorithms among the M image processing algorithms will not be executed, and the execution priority of the one or more image processing algorithms is lower than the execution priority of the first image processing algorithm.

6. The method according to any one of claims 1 to 3, characterized in that: The system performance corresponding to the first level is higher than the system performance corresponding to the second level, including: the frequency utilization rate of the devices in the electronic device corresponding to the first level is lower than the frequency utilization rate of the devices corresponding to the second level, and / or the temperature of the devices corresponding to the first level is lower than the temperature of the devices corresponding to the second level; wherein the devices include devices related to executing image processing algorithms.

7. The method according to claim 6, characterized in that The device includes a central processing unit (CPU), a graphics processing unit (GPU), and / or a neural network processor (NPU).

8. The method according to any one of claims 1 to 3, characterized in that: The N image processing algorithms include one or more of the following algorithms: face detection algorithm, face focus tracking algorithm, subject detection algorithm, motion detection algorithm, scene detection algorithm, and image quality assessment algorithm.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

Images