Camera control method and device, chip system, storage medium and program product
Through dynamic loading of perception algorithms that match the device status, the lag and crash problems of camera applications in electronic devices are solved, and the user experience and application stability are improved.
Patent Information
- Application Number
- CN202411496663.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-22
AI Technical Summary
During the startup or operation of the camera application, electronic devices may stutter or crash, affecting the user experience, and the existing technology is difficult to effectively solve this problem.
By obtaining the operating status parameters of electronic devices, such as temperature, CPU resources and GPU resources, corresponding perception algorithms are dynamically loaded to match the device status, reduce unnecessary algorithm operation, improve device support capabilities, and avoid lag and crashes.
It effectively reduces the lag and crash of camera applications, improves user experience, and ensures the smoothness of preview screen and functional stability of camera applications.
Smart Images

Figure CN119012002A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a camera control method, device, chip system, storage medium and program product. Background Art
[0002] With the development of the field of terminal technology, camera applications in electronic devices can support more and more functions. For example, after the electronic device starts the camera application and before receiving an input operation indicating shooting, the electronic device can display a camera application preview interface so that the user can preview the picture to be shot. In addition, during this process, the camera application can perform object recognition, such as identifying the shooting object as a person, food or beautiful scenery, etc., to display the identification of the recognized object in the preview interface; or, the camera application can also perform target tracking, for example, during the movement of the person to be photographed, the preview interface can always focus on the person, etc.
[0003] However, during the startup or operation of the camera application, the camera application may freeze or crash, affecting the user experience. Summary of the invention
[0004] The embodiments of the present application provide a camera control method, device, chip system, storage medium and program product, which are applied to the field of terminal technology. Based on the running state of the electronic device, the perception algorithm of the camera application matching the running state of the electronic device can be run, so that the running state of the electronic device can support the running of the perception algorithm with a high probability, thereby reducing the problems of camera application freeze or flash back, and improving the user experience.
[0005] In a first aspect, an embodiment of the present application proposes a camera control method, which is applied to an electronic device, wherein the electronic device is installed with a camera application, the method comprising: obtaining a first parameter, the first parameter being used to indicate a first operating state of the electronic device, and the first parameter comprising parameters related to one or more of the following: the temperature of the electronic device, central processing unit CPU resources, graphics processing unit GPU resources or memory; when the camera application is running in the foreground, based on the first parameter, loading a first algorithm corresponding to the first parameter, the first algorithm being used to implement the function of the camera application.
[0006] The first parameter may be some indicators related to the operation of the electronic device described below. The parameters related to the CPU resource may be the CPU performance parameters described below. The process of obtaining the first parameter may refer to the following Figure 8 Related description of the embodiment. The first operating state can be understood as the state of the electronic device. The first algorithm can be part or all of the perception algorithms described below. The electronic device loads the first algorithm corresponding to the first parameter, that is, the electronic device can load the corresponding first algorithm based on the first operating state of the electronic device.
[0007] Loading the first algorithm can also be understood as loading a configuration file of the first algorithm, and the electronic device can configure the first algorithm based on the configuration file of the first algorithm. The configuration file may include, for example, various parameters required for the operation of the first algorithm, such as parameters required to be set for the operation of the first algorithm (thresholds related to the operation of the first algorithm, preprocessing steps for processing input data, etc.), and environmental parameters, that is, parameters for setting hardware such as a GPU and / or a DSP in order to run the first algorithm.
[0008] In one possible case, the first algorithm may be loaded based on an input operation indicating to start the camera application and a first parameter. That is, when the camera application is started, the electronic device loads the first algorithm based on the first parameter.
[0009] In another possible case, the first algorithm may be loaded during the operation of the camera application based on the improvement of the operating state of the electronic device. For example, before loading the first algorithm, the electronic device has already loaded algorithm 1 (which may be one or more algorithms), then based on the improvement of the operating state of the electronic device, the electronic device loads the first algorithm again, and the first parameter may correspond to the first algorithm and algorithm 1.
[0010] In another possible case, the first algorithm may be the remaining algorithm after discarding or deleting the remaining algorithms based on the deterioration of the operating state of the electronic device during the operation of the camera application. Loading the first algorithm can also be understood as the electronic device discarding algorithm 2 (which may be one or more algorithms). That is, before loading the first algorithm, the electronic device has already loaded the first algorithm and algorithm 2, then based on the deterioration of the operating state of the electronic device, the electronic device discards algorithm 2, and the algorithms loaded in the electronic device still have the first algorithm.
[0011] For example, the status of the electronic device in the following text is divided into very poor, relatively poor, and relatively good. When the first operating state is very poor, the first algorithm can be a high-priority perception algorithm in the following text. Similarly, when the first operating state is relatively poor, the first algorithm can be a high-priority and medium-priority perception algorithm in the following text; when the first operating state is relatively good, the first algorithm can be all the perception algorithms in the following text.
[0012] In the camera control method of the present application, during the startup or operation of the camera application, the electronic device can obtain a first parameter that can be used to indicate the operation status of the electronic device, and can load the corresponding first algorithm based on the first parameter, so that the electronic device can realize the function of the camera application by running the first algorithm during the operation of the camera application. For example, when the first parameter indicates that the operation status of the current electronic device is poor, some perception algorithms can be loaded (for example, perception algorithms that have a greater impact on the camera preview can be loaded) to realize the function of the camera application. In this way, the electronic device can load matching perception algorithms based on the operation status of the electronic device, so that the operation status of the electronic device can support the probability that the electronic device runs these algorithms. For example, when the operation status of the electronic device is poor, the electronic device loads fewer perception algorithms, and the number of perception algorithms run by the electronic device during the operation of the camera application is relatively small, which helps to reduce the problems of the camera application being stuck or flashing back due to the inability of the electronic device to support the operation of multiple perception algorithms, and helps to make the preview screen of the camera application displayed by the electronic device more fluent, thereby improving the user experience.
[0013] In one possible implementation, after loading the first algorithm corresponding to the first parameter, the method also includes: obtaining a second parameter, the second parameter is used to indicate a second operating state of the electronic device, the second operating state is better than the first operating state; based on the second parameter, loading a second algorithm, the second algorithm is used to implement the function of the camera application, and the second parameter corresponds to the first algorithm and the second algorithm.
[0014] It is understandable that during the operation of the camera application, the state of the electronic device may improve (for example, Fig.12 Corresponding to process 2 in the embodiment, the electronic device switches from the first operating state to the second operating state. Exemplarily, the electronic device switches from the first operating state to the second operating state may be the electronic device switches from very poor to relatively poor or relatively good; or the state of the electronic device switches from relatively poor to relatively good. After the state of the electronic device improves, the electronic device may load more perception algorithms, namely the second algorithm, so that the camera application can achieve more functions.
[0015] It should be noted that after the state of the electronic device is switched from the first operating state to the second operating state, since the electronic device has already loaded the first algorithm, the electronic device may not repeatedly load the first algorithm at this time.
[0016] Exemplarily, when the electronic device switches from a very bad state to a worse state, the first algorithm may be a high-priority perception class algorithm described below, and the second algorithm may be a medium-priority perception class algorithm described below; when the electronic device switches from a very bad state to a better state, the first algorithm may be a high-priority perception class algorithm described below, and the second algorithm may be a medium-priority perception class algorithm and a low-priority perception class algorithm described below; when the electronic device switches from a worse state to a better state, the first algorithm may be a high-priority perception class algorithm and a medium-priority perception class algorithm described below, and the second algorithm may be a low-priority perception class algorithm described below.
[0017] In this way, when the electronic device is in a better state, the electronic device can load more perception algorithms for the camera application to realize more camera functions. It can provide users with more camera functions and enhance user experience while reducing the freeze of the camera application preview interface.
[0018] In one possible implementation, after loading the first algorithm corresponding to the first parameter, the method also includes: obtaining a third parameter, the third parameter is used to indicate a third operating state of the electronic device, the third operating state is worse than the first operating state; based on the third parameter, deleting the third algorithm included in the first algorithm, the first parameter corresponds to the third algorithm, and the third parameter corresponds to an algorithm in the first algorithm other than the third algorithm.
[0019] It is understandable that during the operation of the camera application, the state of the electronic device may deteriorate (for example, Fig.12 Corresponding to process 3 in the embodiment, the electronic device switches from the first operating state to the third operating state. Exemplarily, the electronic device switches from the first operating state to the third operating state may be the electronic device switches from a better state to a worse or very worse state; or the electronic device switches from a worse state to a very worse state. After the state of the electronic device deteriorates, the third algorithm in the first algorithm may be deleted.
[0020] It should be understood that deleting the third algorithm in the first algorithm can be understood as the electronic device discarding the third algorithm (that is, not running the third algorithm, or the electronic device not configuring the third algorithm, or the electronic device deleting the relevant parameters obtained to run the third algorithm, etc.).
[0021] Exemplarily, when the electronic device switches from a worse state to a very worse state, the first algorithm may be a high-priority perception class algorithm and a medium-priority perception class algorithm described below, and the third algorithm may be a medium-priority algorithm described below; when the electronic device switches from a better state to a worse state, the first algorithm may be all the perception class algorithms described below (a high-priority perception class algorithm, a medium-priority perception class algorithm, and a low-priority perception class algorithm), and the third algorithm may be a low-priority algorithm described below; when the electronic device switches from a better state to a very worse state, the first algorithm may be all the perception class algorithms described below, and the second algorithm may be a high-priority perception class algorithm and a medium-priority perception class algorithm described below.
[0022] In this way, the electronic device can dynamically adjust the loaded perception algorithms based on the current operating status. When the operating status of the electronic device deteriorates, the perception algorithms loaded for the camera application can be reduced, so that the electronic device can support fewer perception algorithms running by the camera application, which helps to reduce the lag during the operation of the camera and improve the user experience.
[0023] In a possible implementation, loading a first algorithm corresponding to a first parameter includes: determining, based on the first parameter, that the first operating state is a first level; and loading the first algorithm based on a first corresponding relationship, wherein the first corresponding relationship includes a corresponding relationship between the first level and the first algorithm.
[0024] The first level may be level 1, level 2 or level 3 as described below, and the process of determining that the first operating state is the first level may be understood as the process of the perception module confirming the level of the electronic device state as described below. Figure 8 The first corresponding relationship can refer to the relevant description of the corresponding relationship 2 below, which will not be repeated here.
[0025] In this way, the corresponding perception algorithm can be loaded based on the level of the state of the electronic device, and based on the first correspondence, the electronic device can load the perception algorithm suitable for the current state of the electronic device in different states, so that the electronic device can support the probability of successfully calling the perception algorithm of the camera application with a higher probability, which helps to reduce freezes and other phenomena and improve user experience.
[0026] In one possible implementation, before loading the first algorithm, the method also includes: based on the frequency and / or algorithm type of calling the algorithm for implementing the preview function of the camera application, dividing the algorithms in the first algorithm set pre-set in the electronic device into different priorities, the algorithms in the first algorithm set are used to implement the preview function of the camera application, the first algorithm includes the algorithm with the highest priority in the first algorithm set, and the first corresponding relationship includes the corresponding relationship between the first level and the priority of the first algorithm.
[0027] Among them, the frequency of calling the algorithm can be used to indicate the importance of the algorithm, that is, the higher the frequency of calling the algorithm, the more important the algorithm is, the greater the possibility that the algorithm affects the preview function, and the higher the priority of the algorithm can be set. The algorithm category can be understood as the type of algorithm, for example, the algorithm can be related to implementing the preview function of the camera application, etc. The algorithm can, for example, include the face recognition-related algorithms or tracking-related algorithms described below. For example, the electronic device can set the face recognition-related algorithm to have a higher priority. Different priorities can, for example, include the high priority, medium priority, and low priority described below.
[0028] It should be understood that the algorithm for implementing the preview function call can be understood as all the perception algorithms mentioned below, for example, it can be an algorithm mounted on a tiny stream. The tiny stream can be understood as an image stream that processes object detection (such as recognition, positioning, and tracking, etc.) and related special effects (such as beauty, stickers, and adding filters, etc.), where the image stream can be understood as a data stream that transmits image data. And the tiny stream can also be understood as a low-resolution image stream that can be used for real-time preview, reducing the burden of processing and transmission, thereby improving the response speed of the camera application. Alternatively, the algorithm for implementing the preview function call can also be understood as the perception algorithm run by the camera application when implementing the preview function.
[0029] Further, the electronic device may load a first algorithm of a corresponding priority based on the first level. Exemplarily, when the first level is level 1 described below, the first algorithm may include a high priority algorithm; when the first level is level 2 described below, the first algorithm may include a high priority and a medium priority algorithm; when the first level is level 1 described below, the first algorithm may include all algorithms in the first algorithm set.
[0030] In this way, the electronic device can load the first algorithm of the corresponding priority based on the first level, and can load a higher priority algorithm when the system resources are tight, or load more priority algorithms when the system resources are sufficient, so as to reduce the phenomenon of camera application freeze when the system resources of the electronic device are tight. And when the system resources of the electronic device are sufficient, more camera functions can be provided to improve the user experience.
[0031] In one possible implementation, after loading the first algorithm corresponding to the first parameter, the method further includes: processing tasks in a first task queue through a first thread in a first thread pool, where the first task queue is a task queue bound to the first thread pool, and the tasks in the first task queue are related to the operation of the first algorithm.
[0032] Among them, the first task queue may include (record or cache) tasks that call (or run) the first algorithm. Exemplarily, when the first algorithm includes a high-priority algorithm, the first task queue may be used to cache tasks that call high-priority perception algorithms. Accordingly, the first thread pool may be used to process tasks that call the first algorithm. For example, when the first algorithm is a high-priority perception algorithm, the first thread pool may include thread pool 1 described below, and the first thread may include threads in thread pool 1 described below; or, when the first algorithm is a high-priority and medium-priority algorithm, the first thread pool may include thread pool 1 and thread pool 2 described below, and the first thread may include threads in thread pool 1 and thread pool 2 described below.
[0033] In this way, by reusing the threads in the thread pool to run the first algorithm, the electronic device can reduce the overhead of creating or destroying threads, improve task processing efficiency, save system resources, and help improve the response speed of the camera application and enhance the user experience.
[0034] In one possible implementation, the first thread includes a first stable thread, which is a stable thread in a first thread pool; the method also includes: receiving a first operation instructing to exit the camera application; based on the first operation, setting the state of the first stable thread to a sleep state, destroying the remaining threads used to process tasks corresponding to the camera application, and exiting the camera application.
[0035] The first stable thread can refer to the following description of the stable threads in thread pool 1, which will not be repeated here. It should be understood that the first stable thread can be used to run a higher priority perception algorithm.
[0036] It can be understood that the task corresponding to the camera application can be understood as a task that calls a perception algorithm. After receiving the first operation indicating to exit the camera application, the electronic device can set the first stable thread from an active state to a dormant state. It can also be understood that the electronic device may not destroy the first stable thread after exiting the camera application. The remaining threads are, for example, threads in thread pool 1 to thread pool n except the stable thread in thread pool 1.
[0037] In this way, when the camera application exits, the electronic device can destroy the remaining threads to release system resources. The first stable thread can be left undestroyed, so that after the camera application is restarted, it can quickly restore the ability to call higher priority perception algorithms, reduce the overhead of the electronic device to recreate threads, improve the response speed of the camera application, and improve the smoothness of the camera operation.
[0038] In one possible implementation, after exiting the camera application, the method further includes: receiving a second operation instructing to start the camera application, wherein the time interval between the moment when the second operation is received and the moment when the first operation is received is less than a first preset duration; based on the second operation, starting the camera application; in the case where there are tasks in the first task queue, setting the state of the first stable thread from the sleep state to the active state, and processing the tasks in the first task queue through the first stable thread.
[0039] The first preset time length may be the preset time length mentioned below. That is, the first preset time length may be understood as the time length for not destroying the first stable thread after exiting the camera application.
[0040] In this way, after the electronic device exits the camera application and restarts the camera application within the first preset time period, it can continue to use the first stable thread to process tasks in the first task queue (i.e., call a higher priority perception algorithm), which can reduce the overhead of the electronic device to recreate threads, improve the response speed of the camera application, and improve the smoothness of the camera operation.
[0041] In one possible implementation, after exiting the camera application, the method also includes: at a first moment, destroying a first stable thread, and the time interval between the first moment and the moment when the first operation is received is greater than or equal to a first preset duration; receiving a third operation instructing to start the camera application, and the moment when the third operation is received is later than the first moment; based on the third operation, starting the camera application and creating a first stable thread.
[0042] That is, after the first preset time after exiting the camera application, if the electronic device does not restart the camera application, the electronic device can destroy the first stable thread (for example, refer to the relevant description of S1124 below). And when receiving the third operation of starting the camera application after the first preset time after exiting the camera application, the electronic device can re-create the first stable thread to process the tasks in the first task queue, so that the first stable thread can reduce unnecessary system resources occupied and improve system performance.
[0043] In one possible implementation, the first thread includes a first stable thread and a variable thread; before processing the tasks in the first task queue through the first thread in the first thread pool, the method also includes: receiving a fourth operation indicating to start the camera application; based on the fourth operation, creating a first stable thread; when the number of tasks in the first task queue exceeds a first threshold, creating a variable thread; processing the tasks in the first task queue includes: processing the first task in the first task queue through the first stable thread, and processing the second task in the first task queue through the variable thread.
[0044] The first threshold may be the threshold a mentioned below, the first task may be the task 1 in the task queue 1 mentioned below, and the second task may be the task 2 in the task queue 1 mentioned below. That is, when the number of tasks in the first task queue exceeds the first threshold, the electronic device may create a variable thread, and use the first stable thread to process the first task in the first task queue, and use the variable thread to process the second task in the first task queue.
[0045] In this way, when tasks are piled up in the task queue, the electronic device can create more variable threads to quickly process the piled up tasks, improve the system's response speed and the smoothness of the camera application's operation, and improve the user experience.
[0046] In a possible implementation, the method further includes: after the variable thread completes processing the second task, destroying the variable thread.
[0047] Among them, the destruction of the variable thread can refer to the relevant description of destroying the variable thread in the thread pool 1 in S1123, which will not be repeated here.
[0048] In this way, after processing the second task, the electronic device can destroy the variable thread and release the corresponding system resources, so that the variable thread does not occupy too many system resources, which helps to improve the system performance.
[0049] In a second aspect, an embodiment of the present application provides a camera control device, which may be an electronic device, or a chip or chip system in an electronic device. The camera control device may include a camera and a processing unit. The processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements a camera control method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements a camera control method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).
[0050] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.
[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.
[0052] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect or any possible implementation manner of the first aspect.
[0053] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.
[0054] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).
[0055] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 A schematic diagram of an interface during the operation of a camera application;
[0057] Figure 2 This is a schematic diagram of the interface of another camera application during operation;
[0058] Figure 3 is a flowchart of a camera control method;
[0059] Figure 4 It is a schematic diagram of a task queue and thread pool;
[0060] Figure 5 A schematic diagram of the process of starting a camera application provided in an embodiment of the present application;
[0061] Figure 6 This is a schematic diagram of the process of running the camera application in an embodiment of the present application;
[0062] Figure 7 A schematic block diagram of a hardware architecture of an electronic device provided in an embodiment of the present application;
[0063] Figure 8 A schematic block diagram of a software architecture of an electronic device provided in an embodiment of the present application;
[0064] Fig. 9 A schematic diagram of a process for setting the priority of a perception algorithm for a task module provided in an embodiment of the present application;
[0065] Fig.10 A schematic diagram of a process for a sensing module according to an embodiment of the present application to confirm the status of an electronic device;
[0066] Fig.11 A schematic diagram of a flow chart of a camera control method provided in an embodiment of the present application;
[0067] Fig.12 A schematic flow chart of a second camera control method provided in an embodiment of the present application;
[0068] Fig.13 A schematic flow chart of a third camera control method provided in an embodiment of the present application;
[0069] Fig.14 A schematic diagram of the process of processing tasks by the thread pool module provided in an embodiment of the present application;
[0070] Fig.15 A schematic block diagram of a camera control device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:
[0072] 1. Thread
[0073] It can refer to the basic unit in the operating system of an electronic device that can be independently scheduled and executed. An electronic device can perform tasks through threads.
[0074] Exemplarily, in response to an input operation of starting the camera, such as clicking a corresponding icon, the electronic device may create one or more threads, wherein the one or more threads may be used to execute one or more tasks during the operation of the camera.
[0075] It should be understood that when an electronic device creates a thread, it needs to allocate system resources to the thread so that the thread can run independently and process tasks. System resources may also be, for example, CPU resources, etc. And system resources may include, but are not limited to: memory resources and processor time (CPU time), etc. Among them, memory resources can be understood as the memory space that can be occupied by the thread running, and processor time can be understood as the time allocated to the thread to execute tasks on the processor.
[0076] 2. Thread Pool
[0077] It is a thread management mechanism that maintains a certain number of threads that can be reused to perform multiple tasks instead of creating and destroying threads for each task.
[0078] 3. Other terms
[0079] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0080] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0081] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.
[0082] 4. Electronic equipment
[0083] The electronic devices of the embodiments of the present application may include handheld devices, vehicle-mounted devices, etc. with face recognition functions. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (PLANs), and wireless terminals in smart cities. The embodiments of the present application do not limit this.
[0084] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.
[0085] In addition, in the embodiments of the present application, the electronic device may also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.
[0086] The electronic device in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.
[0087] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.
[0088] With the development of terminal technology, the functions of camera applications in electronic devices have become more abundant. For example, in the process of shooting images or videos through the camera application, the camera application can perform target tracking or object recognition to improve the shooting effect of the camera application and improve the user experience.
[0089] Among them, target tracking can be used to track the subject (i.e., tracking target) in the camera application preview interface, and can keep the camera focus on the moving subject. The subject tracked by target tracking can be a person, an animal, or other objects.
[0090] Combine the following Figure 1 Taking a person as an example, the process of implementing person tracking during the operation of a camera application is described.
[0091] Figure 1 This is a schematic diagram of the interface of a camera application during operation. Figure 1 As shown, Figure 1 The interface shown in the figure can be understood as a preview interface. The preview interface is an interface in which an electronic device displays a real-time image when shooting an image or video through a camera application. Through the preview interface, the user can see the image captured by the camera before pressing the shutter.
[0092] Exemplarily, in response to a user's operation of starting a camera application, the electronic device may display a preview interface of the camera application. The preview interface may include a preview window, and an image may be previewed in the preview window.
[0093] Figure 1 (a) includes a first preview window 101, wherein the first preview window 101 can display the picture to be photographed by the user, that is, the preview picture.
[0094] It can be understood that the camera application can perform face recognition on the preview screen in the first preview window 101, so that the camera application can recognize the face in the preview screen and display the recognition box 102 on the portrait in the first preview window 101, for example, the recognition box 102 can be displayed at the face position, etc.
[0095] The recognition frame 102 can mark the detected face position in real time, so that the user can confirm that the face has been recognized and focused. As the recognized face position moves, the recognition frame 102 will also move with the movement of the face position.
[0096] For example, Figure 1 As shown in (b), the position of the face changes from Figure 1 The position shown in (a) is moved to Figure 1 The recognition frame 102 will also move along with the position shown in (b) so that the recognition frame 102 remains on the face of the person in the first preview window 101.
[0097] Similarly, if Figure 1 As shown in (c), the position of the face changes from Figure 1 The position shown in (b) is moved to Figure 1 The identification frame 102 will also move to the position shown in (c) Figure 1 The position shown in (c).
[0098] Understandably, Figure 1 In the face tracking process shown, the camera application needs to perform face recognition and face tracking. In the face recognition process, the camera application can call the face recognition algorithm to implement it, and in the face tracking process, the camera application can call the target tracking algorithm to implement it.
[0099] In addition, object recognition can be used to identify a shooting object in a shooting interface of a camera, where the shooting object can be a person, a landscape, or a delicacy, etc.
[0100] Combine the following Figure 2 Taking the photographing subjects of food and scenery as an example, the process of implementing object recognition during the operation of the camera application is explained.
[0101] Figure 2 This is a schematic diagram of the interface of another camera application during operation. Figure 2 As shown, Figure 2 The interface displayed in can be understood as a preview window.
[0102] Figure 2 (a) includes a second preview window 201. The camera application can perform object recognition on the photographed object displayed in the second preview window 201, so that the camera application can recognize that the photographed object in the second preview window 201 is food, and can display a first mark 202 in the second preview window 201. The first mark 202 is used to indicate that the second preview window 201 includes food.
[0103] akin, Figure 2 (b) includes a third preview window 203 , the camera application can recognize that the subject in the third preview window 203 is sunrise or sunset, and can display a second mark 204 in the third preview window 203 , the second mark 204 is used to indicate that the third preview window 203 includes sunrise or sunset.
[0104] Understandably, Figure 2 In the object recognition process shown, the camera application can call the object detection algorithm or the scene recognition algorithm to implement it.
[0105] Above Figure 1 The face recognition algorithm or target tracking algorithm involved in the process shown, and Figure 2 The object detection algorithm or scene recognition algorithm involved in the process shown can be understood as a perception algorithm. During the operation of the camera application, the camera application can call various perception algorithms to implement various functions.
[0106] It can be understood that the perception algorithm can be an algorithm for processing and analyzing the image or video data captured by the camera. The perception algorithm can improve the user experience when the user uses the camera application. In addition to the various algorithms mentioned above, the perception algorithm can also include expression recognition algorithms, scene segmentation algorithms, etc., which are not specifically limited here.
[0107] It can be understood that during the operation of the camera application, the camera application can implement various functions of the camera by calling multiple perception algorithms. That is, the camera application implementing various functions can be understood as the process of the camera application calling the perception algorithm. Therefore, in the embodiment of the present application, the process of the camera application processing a task can also be understood as the process of the camera application calling the perception algorithm. For the sake of brevity, this will not be repeated below.
[0108] As users pursue a higher shooting experience, the camera application can realize more and more functions, which means that the camera application may call multiple algorithms during operation. Therefore, the camera application may need to process multiple tasks at the same time during operation.
[0109] When the camera application needs to process multiple tasks at the same time, in order to enable the camera application to efficiently implement various functions, the camera application can create multiple threads to process multiple tasks in parallel, so as to shorten the time it takes for the camera application to process multiple tasks. For example, the camera application can create a corresponding thread for each task and destroy the corresponding thread after the task is completed.
[0110] It is understandable that the camera application can allocate certain system resources (such as memory resources) to the thread when creating the thread, and recycle the system resources when destroying the thread. However, as the number of tasks that the camera application needs to handle continues to increase, the electronic device needs to frequently create and destroy threads to handle tasks. For example, if the camera application needs to call 20 algorithms, the electronic device may need to create 20 threads to handle the tasks, etc., then the camera application operation process may involve the creation and destruction of 20 threads. Moreover, during the operation of the camera, as the shooting mode is constantly switched, the threads created by the electronic device for the camera application also need to be constantly destroyed and created. For example, in portrait mode, the electronic device creates 20 threads for the camera application to run. As the shooting mode switches from portrait mode to photo mode, the electronic device needs to destroy the 20 threads and needs to create the threads again after the portrait mode is switched to photo mode.
[0111] Therefore, during the operation of the camera application, such as in scenarios such as mode switching, the electronic device will frequently create and destroy threads, which may cause a large amount of system resources to be frequently allocated and recycled, resulting in a large amount of system resources being occupied. The camera application may not be able to allocate sufficient resources for the tasks that need to be processed in time, resulting in task response delays and even application freezes.
[0112] At present, electronic devices can use thread pool and task queue mechanism to realize reasonable call of system resources, so as to solve the above problems. This is described in detail below.
[0113] Figure 33 is a flow chart of a camera control method 300. The method 300 comprises the following steps:
[0114] S301: In response to an input operation instructing to start a camera application, such as clicking an icon of a camera application, the electronic device starts a camera application.
[0115] It should be understood that after the electronic device receives the input operation indicating the start of the camera application, there may be a period of time during which the camera application will not display the preview interface. That is, after the camera application receives the input operation indicating the start of the camera application and starts to start the camera application, the first few frames of images acquired by the camera application may not be displayed, so that no images are displayed in the preview window of the electronic device. During this period of time, the electronic device can, for example, execute S302 and S303.
[0116] S302: Starting a camera application based on the electronic device, and loading a configuration file of a perception algorithm on the electronic device.
[0117] Among them, the configuration file of the perception algorithm may include various parameters of the perception algorithm. And the configuration file of the perception algorithm may be, for example, but not limited to, in a format such as JSON or XML. The configuration file of the perception algorithm may be pre-set in the electronic device, for example, 20 kinds of perception algorithms may be pre-set in the electronic device. Moreover, after the electronic device starts the camera application, the electronic device may load the configuration files of multiple perception algorithms that may be called by the camera application, for example, the configuration files of all perception algorithms are pre-set in the electronic device.
[0118] In addition, for perception algorithms loaded by electronic devices, the electronic devices can divide all loaded perception algorithms into different groups according to algorithm types, etc. For example, perception algorithms related to faces (such as face recognition algorithms, etc.) are the first group of perception algorithms; artificial intelligence (AI) scene detection algorithms, etc. can be the second group of perception algorithms; the remaining perception algorithms can be the third group of perception algorithms, etc.
[0119] S303: Based on starting a camera application on the electronic device, the electronic device initializes multiple thread pools, configures a task queue for each thread pool, and creates a default number of threads in each thread pool.
[0120] Initializing multiple thread pools can also be understood as creating and configuring multiple thread pools. Configuring multiple thread pools can be understood as setting parameters for each thread pool in the multiple thread pools, such as thread idle time, thread survival time, and other parameters. Configuring a task queue for each thread pool can also be understood as part of initializing multiple thread pools. Each thread pool in the multiple thread pools can be associated with a task queue. The task queue corresponding to a thread pool can be used to record the tasks that the thread pool needs to process.
[0121] A default number of threads may be created in each of the multiple thread pools, and the default number may be a preset positive integer. And the threads in each thread pool may be used to process tasks in the task queue corresponding to each thread pool.
[0122] In addition, the tasks that can be executed by each thread pool may be different, so the tasks that may be added to the task queues corresponding to each thread pool may also be different.
[0123] For example, Figure 4 A schematic diagram of a task queue and a thread pool is shown. Figure 4 As shown, the multiple thread pools include thread pool 1, thread pool 2 and thread pool 3. The electronic device can configure (or bind) task queue 1 for thread pool 1, configure (or bind) task queue 2 for thread pool 2, and configure (or bind) task queue 3 for thread pool 3.
[0124] When the camera's pending tasks include Task 1, Task 2, Task 3, and Task 4, the operating system of the electronic device can store each task in a different task queue based on a preset rule when processing the camera's pending tasks. The preset rule can be, for example, different tasks (called perception algorithms) that can be executed by each thread pool, and task allocation can be performed based on the tasks that can be executed by each thread pool.
[0125] Exemplarily, assuming that task 1 can be implemented by calling perception algorithm 1, and perception algorithm 1 can be called in thread pool 1; task 2 can be implemented by calling perception algorithm 2, and thread pool 2 can call perception algorithm 2; task 3 can be implemented by calling perception algorithm 3, and task 4 can be implemented by calling perception algorithm 4; and perception algorithm 3 and perception algorithm 4 can be called in thread pool 3, then the electronic device can add task 1 to the task queue 1 corresponding to thread pool 1, and add task 2 to the task queue 2 corresponding to thread pool 2, and add task 3 and task 4 to the task queue 3 corresponding to thread pool 3.
[0126] In addition, it can also be understood that different thread pools can call perception algorithms in different groups. Exemplarily, the perception algorithms that can be called by thread pool 1 can be set as the first group of perception algorithms; the perception algorithms that can be called by thread pool 2 can be set as the second group of perception algorithms, and the perception algorithms that can be called by thread pool 3 can be set as the third group of perception algorithms. It is assumed that perception algorithm 1 belongs to the first group of perception algorithms, perception algorithm 2 belongs to the second group of perception algorithms, and perception algorithms 3 and perception algorithms 4 belong to the third group of perception algorithms. When the operating system of the electronic device processes the tasks to be executed of the camera application, it can assign tasks to the tasks to be executed based on the group to which the tasks belong. For example, the tasks that call the first group of perception algorithms can be assigned to task queue 1, the tasks that call the second group of perception algorithms can be assigned to task queue 2, and the tasks that call the third group of perception algorithms can be assigned to task queue 3.
[0127] Furthermore, the electronic device can use the threads in each thread pool to execute the tasks in the corresponding task queue. Figure 4 As shown, assuming that thread pool 1 creates thread 1, thread 1 can be used to execute task 1 in task queue 1; assuming that thread pool 2 creates thread 2, thread 2 can be used to execute task 2 in task queue 2; assuming that thread pool 3 creates thread 3, thread 4 and thread 5, the threads in thread 3, thread 4 and thread 5 can be used to execute tasks 3 and task 4 in task queue 3.
[0128] In this way, during the operation of the camera application, when the camera application needs to implement various functions, the electronic device can add tasks that can be processed by each thread pool to the task queue corresponding to each thread pool.
[0129] S304: During the running of the camera application, various functions are implemented through multiple thread pools.
[0130] It should be understood that the implementation of S304 can refer to the description of S303, which will not be repeated here.
[0131] Optionally, during the execution of the camera application, the electronic device may not destroy and recreate the threads in the multiple thread pools that have been created.
[0132] In this way, the phenomenon of frequent creation and destruction of threads by electronic devices can be reduced.
[0133] S305: In response to the input operation instructing to exit the camera application, the electronic device exits the camera application.
[0134] It is understandable that when the user requests to exit the camera application, the camera application may release the resources allocated to each thread pool. Optionally, the resources allocated to the thread pool may include recycling thread resources (such as memory resources, etc.), destroying threads in the thread pool, etc.
[0135] S306: In response to the input operation instructing to restart the camera application, the electronic device restarts the camera application.
[0136] It should be understood that the implementation method of restarting the camera application of the electronic device can refer to S302-S304, that is, the camera application can reload the configuration file of the perception algorithm and reinitialize multiple thread pools, etc., which will not be repeated here.
[0137] It can be seen from the above method 300 that in scenarios such as camera startup, switching shooting modes or camera exiting, the electronic device creates and destroys threads less frequently, and creates and destroys a smaller number of threads. For example, assuming that the camera application needs to call 20 algorithms, the electronic device can process tasks based on 5 threads in the 3 created thread pools. Even if the electronic device frequently exits and starts the camera application, the threads frequently created and destroyed by the electronic device are these 5 threads. In addition, during the operation of the camera application, as the shooting mode is switched, the electronic device may not destroy the 5 threads. Therefore, this can reduce the large amount of system resources occupied due to the frequent creation and destruction of threads.
[0138] However, the following problems still exist during the actual operation of the camera application.
[0139] Problem 1: When launching the camera app, the app freezes, for example, the preview interface cannot be displayed normally.
[0140] Each time an electronic device starts a camera application, it will load configuration files of all perception algorithms pre-set in all electronic devices. When an electronic device starts a camera application, if the electronic device is under high load, the state of the electronic device may not support the call of various perception algorithms to implement various functions of the camera application, resulting in the preview interface of the camera application not being displayed normally, affecting the user experience.
[0141] For example, Figure 5 This is a schematic diagram of the process of starting the camera application provided in an embodiment of the present application. Figure 5 As shown in (a), in response to the user clicking the icon of the camera application, the electronic device can load the configuration file of the perception algorithm and perform processes such as initializing the thread pool. In this process, the interface of the camera application displayed by the electronic device can be as follows: Figure 5 As shown in (b) of FIG. 1 . At this time, the camera application cannot display the preview window interface, that is, the displayed interface does not include the preview screen. Further, after the camera application is initialized and the configuration file is loaded, the electronic device can display the following Figure 5 The preview screen shown in (c) is shown.
[0142] exist Figure 5 In the process shown, if the electronic device is in very poor operating condition, such as excessive load, etc., when the electronic device loads the configuration files of all perception algorithms, the electronic device may not be able to call each perception algorithm normally, so that the electronic device cannot support the normal operation of the camera application, and the camera application may crash or fail to display normally. Figure 5 (c) etc.
[0143] Question 2: During the startup of the camera application, although the electronic device loads the configuration files of all perception algorithms, during the actual operation of the camera application, if the operating state of the electronic device is very poor, such as when the load is too heavy, the electronic device may not be able to normally call some perception algorithms to implement various functions of the camera application, resulting in freezes and other phenomena during the operation of the camera application.
[0144] For example, Figure 6 This is a schematic diagram of the process of running the camera application in the embodiment of the present application. Figure 6 As shown in the preview interface, the camera application uses the target tracking algorithm and face recognition algorithm to track people. Figure 6 As shown in (a), at this moment, the person tracked by the electronic device is in Figure 6 As the tracked person moves from Figure 6 (a) Move to Figure 6 In the case of (b), when the electronic device is under high load, the camera application will still execute all preset perception algorithms (such as the face recognition algorithm and target tracking algorithm mentioned above) to track people. However, the current operating state of the electronic device may not support the call of more perception algorithms. As a result, the camera application may not be able to display the preview interface normally, causing the camera application to freeze. For example, the electronic device displays Figure 6 (b), the abnormal picture 602 displayed in the fourth preview window 601, etc., affects the user experience.
[0145] In order to solve the above technical problems, the present application provides a camera control method, which can confirm the operation of appropriate perception algorithms based on the current state of the electronic device (such as the current temperature of the electronic device, CPU performance parameters, GPU resource usage or memory occupancy, etc.) when starting the camera or switching the camera mode. For example, when the current state of the electronic device is poor, a perception algorithm that affects the camera preview can be run to improve the smoothness of the preview screen when the electronic device is running the camera, thereby improving the user experience.
[0146] It is understandable that when an electronic device starts a camera application, the camera application needs to load configuration files of multiple perception algorithms preset in the electronic device and initialize multiple thread pools. When exiting the camera application, the camera application needs to release the resources allocated to each thread pool and recycle thread resources, etc. Therefore, when the electronic device restarts the camera application after exiting the camera, it may need to re-initialize the operation (for example, load configuration files of multiple perception algorithms and initialize multiple thread pools), reallocate system resources, and affect the smoothness of the camera operation.
[0147] Combine the following Figures 7 to 13 The camera control method of the present application is described in detail. The camera control method of the embodiment of the present application can be executed by an electronic device with a camera, or by a chip, chip system or processor that supports the electronic device to implement the camera control method, or by a logic module or software that can implement all or part of the functions of the electronic device. The present application does not impose specific restrictions on this. The camera control method of the embodiment of the present application is described in detail below with the electronic device as the execution subject.
[0148] In order to better understand the electronic device in the embodiment of the present application, Figure 7 and Figure 8 The hardware structure and software structure of the electronic device in the embodiment of the present application are described in detail.
[0149] Figure 7 A structural schematic diagram of an electronic device 700 suitable for the present application is shown.
[0150] The electronic device 700 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0151] It should be noted that Figure 7 The structure shown does not constitute a specific limitation on the electronic device 700. In other embodiments of the present application, the electronic device 700 may include Figure 7 The electronic device 700 may include more or fewer components than those shown. Figure 7 Combinations of some of the components shown, or electronic device 700 may include Figure 7 Subcomponents of some of the components shown. For example, Figure 7 The illustrated proximity light sensor 180G may be optional. Figure 7 The components shown may be implemented in hardware, software, or a combination of software and hardware.
[0152] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices.
[0153] In some embodiments, the electronic device 700 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
[0154] In a possible implementation, based on an input operation indicating to start a camera application, the processor 110 may call the camera 193 to collect image data.
[0155] The display screen 194 may be used to display an interface of a camera application. For example, after the camera 193 captures image data, the processor 110 may process the image data and may display the processed image data on the display screen 194 .
[0156] In one possible implementation, the processor 110 may process the image data collected by the camera 193 using different perception algorithms to obtain processed image data.
[0157] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0158] Figure 8 This is a schematic diagram of the architecture (including software system and some hardware) of the embodiment of the present application. Figure 8 As shown, the application architecture is divided into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the application architecture can be divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer, the driver layer, and the hardware layer.
[0159] like Figure 8 As shown, the application layer may include applications such as camera, gallery, etc. The application framework layer may include a camera access interface and a perception module API, wherein the camera access interface may include camera management and camera devices.
[0160] Optionally, the perception module API can be used to provide an interface for monitoring various indicators of the electronic device, wherein the various indicators of the electronic device may include some operation-related indicators, such as the temperature of the electronic device, CPU performance parameters, GPU resource usage, memory occupancy, and other parameters that can represent the operating status of the electronic device. CPU performance parameters may include, but are not limited to, one or more of the following: CPU usage, CPU temperature, or CPU power consumption. GPU resources may include, but are not limited to, one or more of the following: GPU computing core, video memory capacity, and memory bandwidth.
[0161] The hardware abstraction layer (HAL) can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware. Figure 8 As shown, the hardware abstraction layer may include a camera hardware abstraction module, a thread pool module, a perception module, and a task module.
[0162] The task module may set the priority of the perception algorithm. Optionally, the task module may set the priority of the perception algorithm based on the category of the perception algorithm, for example, the perception algorithm related to face recognition may be set to a higher priority.
[0163] For example, Fig. 9A schematic diagram of the process of setting the priority of the perception algorithm for the task module provided in the embodiment of the present application. Fig. 9 As shown, the task module can set the algorithm priority for the perception algorithm based on the category of the perception algorithm. For example, the priority of the perception algorithm can include high priority, medium priority or low priority.
[0164] It should be noted that the above classification of the priority levels into three levels of high priority, medium priority and low priority is only an exemplary description, and does not specifically limit the number of priorities. In the actual application process of the embodiments of the present application, the number of priorities can be set to define more or fewer priority levels according to specific needs, system resource conditions or application scenarios to better meet user needs.
[0165] It should be noted that the naming of the priority levels as high priority, medium priority and low priority is only an exemplary description and does not limit the priority names. In the actual application of the embodiments of the present application, these priorities can be represented by other names, such as the first level, the second level and the third level, etc., which can be replaced and adjusted as needed.
[0166] The perception module can monitor some indicators related to the operation of the electronic device in real time, and can determine the level of the state of the electronic device based on some indicators related to the operation of the electronic device. The state of the electronic device can be understood as the operating state of the electronic device. For example, the perception module can monitor some indicators related to the operation of the electronic device in real time after the electronic device is turned on.
[0167] That is, the state of the electronic device can be divided into different levels. For example, the state of the electronic device can be divided into level 1, level 2 and level 3, and different levels can indicate different operating states of the current electronic device. Level 1, level 2 and level 3 can respectively represent different operating states of the electronic device. For example, level 1 can indicate that the operating state of the electronic device is very poor, level 2 can indicate that the operating state of the electronic device is poor, and level 3 can indicate that the operating state of the electronic device is good, etc.
[0168] Optionally, the electronic device is pre-set with a correspondence 1, which includes correspondences between some indicators related to the operation of the electronic device and multiple levels. The perception module can determine the level of the state of the electronic device based on the correspondence 1 and some indicators related to the operation of the electronic device acquired in real time.
[0169] Exemplarily, the electronic device may be preset with multiple thresholds corresponding to indicators such as the temperature of the electronic device, CPU performance parameters, GPU resource usage, and memory occupancy. For example, the temperature of the electronic device corresponds to normal threshold 1 (e.g., 70°C), warning threshold 1 (e.g., 80°C), and critical threshold 1 (e.g., 90°C). Similarly, the CPU performance parameters and GPU resource usage correspond to normal threshold 2 (e.g., 60%), warning threshold 2 (e.g., 80%), and critical threshold 2 (e.g., 90%); the memory occupancy corresponds to normal threshold 3 (e.g., 70%), warning threshold 3 (e.g., 85%), and critical threshold 3 (e.g., 95%).
[0170] During the operation of the camera application, the perception module can monitor in real time indicators such as the temperature, CPU performance parameters, GPU resource usage, and memory occupancy of the electronic device, and compare the real-time status of these indicators with the corresponding normal thresholds, warning thresholds, and crisis thresholds, and confirm the level of the status of the electronic device based on the comparison results.
[0171] For example, when indicators such as the temperature, CPU performance parameters, GPU resource utilization, and memory occupancy rate of the electronic device are all less than or equal to their corresponding normal thresholds, it indicates that the electronic device is in good operating condition, and the level of the status of the electronic device may be, for example, level 3; when indicators such as the temperature, CPU performance parameters, GPU resource utilization, and memory occupancy rate of the electronic device are all less than their corresponding critical thresholds and one or more of them exceed the warning threshold, it may indicate that the electronic device is in poor operating condition, and the perception module may confirm that the status of the electronic device is level 2; when one or more of the indicators such as the temperature, CPU performance parameters, GPU resource utilization, and memory occupancy rate of the electronic device exceed their corresponding critical thresholds, it may indicate that the electronic device is in very poor operating condition, and the perception module may confirm that the status of the electronic device is level 1.
[0172] It should be understood that the above-described method for the perception module to confirm the level of the status of the electronic device is only an example and is not restrictive. In actual application, other suitable methods can be selected according to specific needs and environments, and are not limited here.
[0173] In addition, the perception module can also determine the perception algorithm corresponding to the level based on correspondence 2 and the level of the state of the electronic device.
[0174] Among them, the correspondence 2 may include a correspondence between multiple levels of the state of the electronic device and multiple perception algorithms of algorithm priority. Exemplarily, level 1 corresponds to a high-priority perception algorithm; level 2 corresponds to a high-priority perception algorithm and a medium-priority perception algorithm; level 3 corresponds to a high-priority perception algorithm, a medium-priority perception algorithm, and a low-priority perception algorithm.
[0175] In this way, the perception module can load the corresponding perception algorithm based on the level of the electronic device's status, so that the electronic device is more likely to support the perception algorithm successfully called by the camera application, which helps to reduce phenomena such as lag.
[0176] The thread pool module can be used to create at least one thread pool, and can create threads in the thread pool to process tasks. In addition, the thread pool module can adjust the status of threads in each thread pool based on the perception class algorithm loaded by the perception module.
[0177] In addition, the thread pool module can also create a corresponding task queue for each thread pool in at least one thread pool. Different task queues can be used to cache (or record) tasks of perception algorithms of different priorities. Exemplarily, when thread pool 1 and thread pool 2 are created in the thread pool module, the thread pool module can create a corresponding task queue 1 for thread pool 1, wherein task queue 1 can be used to store tasks that execute high-priority perception algorithms.
[0178] Similarly, the thread pool module can also create a corresponding task queue 2 for thread pool 2, wherein task queue 2 can be used to store tasks that execute medium priority and / or low priority perception algorithms.
[0179] For example, Fig.10 A schematic diagram of a process for a sensing module to confirm the status of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the perception module can determine the level of the state of the electronic device based on some indicators related to the operation of the electronic device, such as level 1, level 2 or level 3.
[0180] When the status of the electronic device is level 1, it means that the status of the electronic device is very poor, and the perception module can load high-priority perception algorithms; when the status of the electronic device is level 2, it means that the status of the electronic device is poor, and the perception module can load high-priority and medium-priority algorithms; when the status of the electronic device is level 3, the status of the electronic device is good, and the perception module can load all perception algorithms, that is, the camera application can load high-priority, medium-priority and low-priority perception algorithms.
[0181] Furthermore, HAL can also add tasks corresponding to high-priority algorithms to task queue 1 corresponding to thread pool 1, and thread pool 1 processes the tasks in task queue 1; add tasks corresponding to medium-priority and / or low-priority perception algorithms to task queue 2 corresponding to thread pool 2, and thread pool 2 processes the tasks in task queue 2.
[0182] It is understandable that in the embodiment of the present application, when the electronic device is in a very poor state, the system resources may be limited, and the thread pool module can only run the threads in thread pool 1, that is, only use the threads in thread pool 1 to run high-priority perception algorithms, so that the key perception algorithm tasks can be prioritized during the camera operation process. In this way, when the electronic device is in a very poor state or the electronic device is in a poor state, the limited system resources can be applied to the high-priority algorithm, which improves the smoothness of the camera application during operation, reduces the occurrence of freezes in the camera preview interface, and improves the user experience.
[0183] It should be noted that the above description of the thread pool module creating thread pool 1 and thread pool 2 is only an exemplary description and does not specifically limit the number and naming of thread pools. In the actual application of the camera control method provided in the embodiment of the present application, the number of thread pools can be flexibly configured according to the specific needs of the system, available resources, and different application scenarios, which may involve creating multiple or fewer thread pools.
[0184] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. Figure 8 As shown, the driver layer may include a camera device driver, a CPU driver, a GPU driver, a temperature sensor driver, and a memory management driver, etc.
[0185] Among them, the CPU driver can be used to monitor CPU performance parameters (such as CPU usage, CPU temperature or CPU power consumption, etc.); the GPU driver can be used to monitor GPU resource usage; the memory management driver can be used to monitor memory usage; the temperature sensor driver can be used to monitor the temperature of electronic equipment.
[0186] In one implementation, the perception module can obtain the status of the CPU of the current electronic device through the CPU driver; can obtain the status of the GPU of the current electronic device through the GPU driver; can obtain the usage of the memory of the current electronic device through the memory management driver; and can obtain the temperature of the current electronic device through the temperature sensor driver.
[0187] The hardware layer includes cameras, CPU, GPU, temperature sensors, and memory, etc.
[0188] The above describes in detail the software system used in the embodiments of the present application.
[0189] Understandably, Figure 8 ] is an example of a software module included in an electronic device. Each layer of the software structure of the electronic device may also include more or fewer software modules. Exemplarily, the application layer of the electronic device may also include other application programs, such as information, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, Alipay and other application programs. This application does not specifically limit this.
[0190] Combine the following Figure 11~Figure 13 The camera control method 1100 of the embodiment of the present application is described in detail through a specific embodiment. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.
[0191] There are five processes described in method 1100:
[0192] Process 1 ( S1101 to S1111 ): The camera application is started, and the status of the electronic device is level 1 .
[0193] Process 2 (S1112 to S1118): During the operation of the camera application, the state of the electronic device switches from level 1 to level 2, that is, the state of the electronic device becomes better.
[0194] Process 3 (S1119 to S1121): During the operation of the camera application, the state of the electronic device switches from level 2 to level 1, that is, the state of the electronic device deteriorates.
[0195] Process 4 (S1122 to S1124): The camera application exits.
[0196] Process 5 (S1125 to S1130): Restarting the camera application within a preset time period after the camera application is exited.
[0197] It should be understood that the method 1100 may be executed by an electronic device, and the hardware structure of the electronic device may be as follows: Figure 7 As shown, the software structure can be as follows Figure 8 As shown, Figure 11~Figure 13 The functions of the various software modules shown in Figure 8 .
[0198] like Figure 11~Figure 13 As shown, method 1100 includes the following steps:
[0199] S1101. The task module sets the priority of the perception algorithm.
[0200] For example, refer to Fig. 9As shown, the task module can set the corresponding algorithm priority for each perception algorithm based on the category of the perception algorithm. For example, the perception algorithm can include high priority, medium priority and low priority perception algorithms. Optionally, the task module can set the algorithm priority of each perception algorithm when the electronic device is turned on or the electronic device operating system is started.
[0201] S1102: Based on an input operation instructing to start a camera application, the camera application instructs a thread pool module to initialize a thread pool.
[0202] It can be understood that after the user triggers the operation of starting the camera application, the camera application can instruct the thread pool module to initialize the thread pool, based on which the thread pool module can respond to the start of the camera application in a timely manner.
[0203] S1103. The thread pool module initializes a preset number of thread pools and creates a task queue for each thread pool.
[0204] It should be understood that based on the instruction of the camera application, the thread pool module can initialize the thread pool. The initialization operation may include creating and configuring a preset number of thread pools, and creating and configuring a task queue for each thread pool, and each task in the task queue can be executed by a thread in the corresponding thread pool.
[0205] It should be understood that the understanding of initializing the thread pool can refer to the description at S303, which will not be repeated here.
[0206] Optionally, the thread pool module may create one or more thread pools based on predefined configuration parameters, and the predefined configuration parameters may include parameters such as a preset number and thread idle time of each thread pool in the preset number of thread pools. For example, when the preset number is 2, the thread pool module may create two thread pools, such as thread pool 1 and thread pool 2. Correspondingly, the thread pool module may also create and configure task queue 1 for thread pool 1, and create and configure task queue 2 for thread pool 2.
[0207] Among them, the threads in thread pool 1 can be used to execute high-priority perception algorithms, and the threads in thread pool 2 can be used to execute medium-priority and / or low-priority perception algorithms, and there is no restriction here.
[0208] In this way, after the camera application is started, the thread pool module can pre-create one or more thread pools and their corresponding task queues, which can improve the response speed of the camera application running the perception algorithm.
[0209] S1104: Based on the input operation indicating starting the camera application, the camera application indication sensing module confirms the status of the electronic device.
[0210] Among them, the input operation based on the instruction to start the camera application can be, for example but not limited to: the user clicks the icon of the camera application, the user initiates a voice command to instruct the electronic device to turn on the camera, or the user clicks the shortcut key to start the camera application to start the camera application, etc.
[0211] Confirming the status of an electronic device can also be understood as confirming some indicators related to the operation of the electronic device. For example, indicators such as the temperature of the electronic device, CPU performance parameters, GPU resource usage, and memory occupancy. The method of obtaining these indicators can refer to Figure 8 The description in will not be repeated here.
[0212] S1105: Based on some indicators related to the operation of the electronic device, the sensing module confirms that the state of the electronic device is level 1. That is, the operation state of the electronic device is very poor.
[0213] Among them, some indicators related to the operation of the electronic device can reflect the quality of the operation status of the electronic device. Correspondingly, the level of the status of the electronic device can also indicate the quality of the operation status of the electronic device.
[0214] It should be understood that the way in which the sensing module confirms the level of the status of the electronic device can be referred to in Figure 8 The description of the above will not be repeated here.
[0215] It should be understood that S1102, S1103 and S1104 can be executed in parallel or sequentially, and the embodiments of the present application do not specifically limit this.
[0216] S1106. The perception module loads the high-priority perception class algorithm corresponding to level 1.
[0217] Among them, the high-priority perception algorithm corresponding to loading level 1 can also be understood as the configuration file of the high-priority perception algorithm corresponding to loading level 1, etc.
[0218] It should be understood that since the electronic device is in a very poor operating state, even if the electronic device loads all the perception algorithms, it may be difficult for the electronic device to run more perception algorithms at the same time, and it may cause the camera application to freeze or crash.
[0219] Therefore, after the perception module confirms that the state of the electronic device is very poor, that is, after the perception algorithm confirms that the state of the electronic device is level 1, in order to make the camera application run smoothly, the perception module can load part of the perception algorithm. Exemplarily, in order to meet the operating requirements of the camera application with a high probability, the perception module can optimize the perception algorithm during the operation of the camera by loading the high-priority perception algorithm corresponding to level 1. The high-priority perception algorithm may include, for example, an algorithm with a higher probability of being called.
[0220] It should be understood that the perception module can determine the high-priority perception algorithm corresponding to level 1 based on the correspondence 2. Figure 8 The description of the above will not be repeated here.
[0221] S1107. The perception module indicates the high-priority perception class algorithm to the task module.
[0222] The task module can set the task of adding a high-priority perception algorithm to task queue 1 based on the indication of the high-priority perception algorithm, so that medium-priority perception algorithms and low-priority perception algorithms are not added to task queue 1.
[0223] Alternatively, the task module may also instruct the thread pool module to set the task queue 1 to add a task of the high-priority perception algorithm based on the indication of the high-priority perception algorithm.
[0224] S1108. The perception module instructs the thread pool module to run a high-priority perception algorithm.
[0225] It can be understood that after the perception class algorithm loads a high-priority perception class algorithm, the perception class algorithm can instruct the thread pool module to run the high-priority perception class algorithm so that the thread pool module can set the status of each thread pool accordingly.
[0226] S1109. The thread pool module creates a stable thread in thread pool 1, sets the thread priority, and performs a core binding operation.
[0227] It should be understood that Fig.10 ,For high-priority perception algorithms, the thread pool module can use the threads in thread pool 1 to process the tasks corresponding to the high-priority perception algorithms.
[0228] It can be understood that, under the premise of caching high-priority perception algorithm tasks in the task queue corresponding to thread pool 1, the thread priority of the threads in thread pool 1 can be set to the highest, so that high-priority perception algorithms can be processed preferentially when the resources of the electronic device are limited, thereby improving the smoothness of the camera operation process.
[0229] Correspondingly, on the premise that high-priority perception algorithm tasks are cached in the task queue corresponding to thread pool 1, and medium-priority perception algorithm tasks are stored in the task queue corresponding to thread pool 2, the thread priority of threads in thread pool 1 can be set higher than the thread priority of threads in thread pool 2, so that when high-priority perception algorithms and medium-priority perception algorithms compete for CPU resources, high-priority perception algorithms can be processed first, thereby improving the stability and reliability of electronic devices under high load conditions and improving user experience.
[0230] In the embodiment of the present application, after setting the thread priority of the stable thread, a core binding operation can be performed for the stable thread, that is, the stable thread can be bound to a specific CPU core to run, so that the stable thread can always be executed on the same CPU core without switching between multiple cores. In this way, the interference of other tasks on the stable thread in thread pool 1 when executing high-priority perception algorithm tasks can be reduced, thereby improving the stability of the camera application during operation and improving the user experience.
[0231] S1110 , the thread pool module sets the state of the stable thread to an active (running) state based on the task in task queue 1 , and can use the stable thread in thread pool 1 to process task 1 in task queue 1 .
[0232] In the embodiment of the present application, the state of the stable thread can be divided into an active (run) state and a dormant (sleep) state. Among them, the active state can refer to the state in which the stable thread is executing a task or preparing to execute a task. In the active state, the stable thread may occupy CPU resources or other system resources (such as memory resources, etc.). Correspondingly, the dormant state can refer to the situation in which the stable thread is temporarily not executing a task and is in a waiting state.
[0233] It can be understood that the initial state of the stable thread after being created can be a dormant state. When thread pool 1 needs to run a perception algorithm, the thread pool module can set the state of the stable thread to an active state based on the tasks in task queue 1. That is, when there are tasks in task queue 1, the thread pool module can set the state of the stable thread in thread pool 1 to an active state; otherwise, the thread pool module can keep the state of the stable thread in a dormant state.
[0234] Further, after the thread pool module sets the state of the stable thread to the active state, the thread pool module can use the stable thread in thread pool 1 to process task 1 in task queue 1.
[0235] In the specific implementation process, the thread pool module can monitor the tasks in task queue 1. When there is a task (such as task 1) in task queue 1 that needs to be processed, the thread pool module can set the state of the stable thread in thread pool 1 to an active state. It can be understood that the stable thread set to an active state can read task 1 from task queue 1 and start processing task 1. Optionally, after task 1 is completed, the thread pool module can also reset the stable thread in the active state to a dormant state.
[0236] S1111 . When the length of task queue 1 is greater than or equal to threshold a, the thread pool module creates a variable thread in thread pool 1 to process tasks in task queue 1 .
[0237] The threshold a may be a preset positive integer, such as 2 or 3. The length of task queue 1 may be understood as the number of tasks included in task queue 1. During the operation of the camera application, as the camera application needs to implement different functions, the electronic device (HAL) may add more tasks to task queue 1, and the tasks added to task queue 1 are high-priority perception algorithm tasks.
[0238] In order to reduce the backlog of tasks and response delays, when the length of task queue 1 is greater than or equal to threshold a, the thread pool module can create a variable thread, so that when the stable thread in thread pool 1 is processing part of the tasks in task queue 1, the variable thread can also process other tasks in task queue 1. Different from stable threads, variable threads in the thread pool can be dynamically destroyed and created. That is, when the length of task queue 1 is greater than or equal to threshold a, the thread pool module can create a variable thread in thread pool 1 and use the variable thread to execute other tasks in task queue 1. In addition, after the variable thread has processed other tasks, the thread pool module can destroy the variable thread to save system resources.
[0239] It should be noted that, when the length of task queue 1 is less than threshold a, all tasks in task queue 1 can be processed by stable threads in thread pool 1, and the thread pool module may not create a variable thread.
[0240] It should also be noted that the thread pool module can monitor the threads in the thread pool 1 and the task queue 1 in real time. As the number of tasks added to the task queue 1 gradually increases, the thread pool module can continuously create new variable threads for the newly added tasks. That is, each thread in the thread pool 1 is in the state of processing tasks, and when a new task appears in the task queue 1, the thread pool module can create a new variable thread in the thread pool 1 and use the new variable thread to process the new task.
[0241] In the embodiment of the present application, without affecting the stable thread, the thread pool module can quickly expand the processing capacity of thread pool 1 by creating and destroying variable threads when there is a backlog of tasks in task queue 1. This flexibility enables tasks to be processed in a timely manner, reduces task backlogs and system response delays, helps improve the smoothness of the camera application during operation, and improves the user experience.
[0242] During the operation of the camera, the sensing module can monitor the state of the electronic device in real time, that is, obtain some indicators related to the operation of the electronic device in real time. The above describes the process of starting the camera application when the electronic device is in a very poor operating state. As the operating state of the electronic device improves, for example, the sensing module detects that the state of the electronic device switches from level 1 to level 2, the electronic device can perform the following steps.
[0243] It should be noted that in the implementation of the present application, the thread pool module creates a variable thread in the thread pool corresponding to the target task queue when it is determined that the length of the target task queue is greater than or equal to the threshold a. The thread pool module can also create a variable thread in the thread pool corresponding to the target task queue when it is determined that the stable thread in the thread pool corresponding to the target task queue (or the thread pool corresponding to the target task queue) is in a running (busy) state; or, when it is determined that the number of tasks accumulated (unprocessed) in the target task queue exceeds the threshold b, a variable thread can be created in the thread pool corresponding to the target task queue, and the number of variable threads created can be determined based on the number of accumulated tasks.
[0244] The target task queue may be, for example, task queue 1 and / or task queue 2.
[0245] S1112: The sensing module detects that the state of the electronic device is switched from level 1 to level 2. That is, the operating state of the electronic device is improved.
[0246] It should be understood that the way in which the perception module determines that the status of the electronic device is level 2 is similar to the way in which the perception module determines that the status of the electronic device is level 1. Please refer to the implementation of S1105 and will not be repeated here.
[0247] S1113, the perception module loads the perception class algorithm with medium priority.
[0248] It should be understood that as the operating state of the electronic device improves, that is, after the perception module detects that the state of the electronic device switches from level 1 to level 2, the perception module can load more perception algorithms to enable the camera application to achieve more functions. For example, the perception module can load a perception algorithm with a medium priority.
[0249] It should be understood that the implementation method of loading a medium-priority perception class algorithm by the perception module is similar to that of loading a high-priority perception class algorithm. Please refer to the above description and will not be repeated here.
[0250] It should be noted that after the perception module detects that the state of the electronic device has switched to level 2, the operating state of the electronic device can satisfy the camera application to call more perception algorithms. In addition to high-priority perception algorithms, medium-priority perception algorithms can include algorithms with relatively high call probabilities. Therefore, the perception module can also load medium-priority perception algorithms. Since the perception module in S1106 above has already loaded a high-priority perception algorithm, there is no need to load a high-priority perception algorithm here.
[0251] S1114. The perception module indicates a high-priority perception algorithm and a medium-priority perception algorithm to the task module.
[0252] The task module may set a task of adding a high-priority perception algorithm to task queue 1 and a task of adding a medium-priority perception algorithm to task queue 2 based on the indication.
[0253] S1115. The perception module instructs the thread pool module to run the perception class algorithm with the priority.
[0254] S1116. The thread pool module creates a stable thread in thread pool 2, sets the thread priority, and performs a core binding operation.
[0255] S1117. The thread pool module sets the state of the stable thread to an active state based on the tasks in task queue 2, and uses the stable threads in thread pool 2 to run the tasks in task queue 2.
[0256] S1118. When the length of task queue 2 is greater than or equal to threshold a, the thread pool module creates a variable thread in thread pool 2 for executing tasks in task queue 2.
[0257] It can be understood that the implementation of S1115-S1118 is similar to the implementation of S1108-S1111, and the relevant description of S1108-S1111 can be referred to, which will not be repeated here.
[0258] It should be understood that the above process 2 is only an example. In some possible situations, the perception module may detect that the state of the electronic device switches from level 1 to level 3, etc., then correspondingly, the medium-priority perception class algorithm in S1113 to S1118 can be replaced by a medium-priority perception class algorithm and a low-priority perception class algorithm. The specific implementation method is similar to the implementation method of S1113 and S1118, and can refer to the above description, which will not be repeated here.
[0259] It should be noted that during the execution of S1113-S1118 by the electronic device, the threads in thread pool 1 are still processing tasks corresponding to high-priority perception algorithms in task queue 1. For the sake of brevity, only the process of the thread pool module creating and using threads in thread pool 2 to process tasks corresponding to medium-priority algorithms is described here.
[0260] As the camera application runs, the operating state of the electronic device may further deteriorate. For example, the state of the electronic device switches from level 3 to level 2 or level 1; or the state of the electronic device switches from level 2 to level 1, etc. Fig.12 , this process is described by taking the state of an electronic device switching from level 2 to level 1 as an example.
[0261] S1119. The sensing module detects that the status of the electronic device switches from level 2 to level 1.
[0262] Optionally, after S1119, method 1100 further includes: the perception module indicating a high-priority perception class algorithm to the task module.
[0263] The task module can set the task queue 1 to add the task of the high-priority perception algorithm based on the indication, and delete the task of the medium-priority perception algorithm in the task queue 2.
[0264] S1120. The perception module instructs the thread pool module to stop running the perception class algorithm with medium priority.
[0265] It should be understood that after the perception module detects that the state of the electronic device switches from level 2 to level 1, that is, the state of the electronic device switches from a relatively poor state to a very poor state, it indicates that the current state of the electronic device does not support running more perception algorithms, and the thread pool module can reduce the number of perception algorithms being run. Therefore, the perception module can instruct the thread pool module to stop running the perception algorithm of medium priority.
[0266] Correspondingly, based on the instructions of the perception module, the thread pool module can stop running the medium-priority perception algorithm and continue running the high-priority perception algorithm to improve the stability of the camera application during operation and reduce the occurrence of lag when the electronic device is in poor condition.
[0267] S1121, the thread pool module sets the state of the stable thread in thread pool 2 to a dormant state. The threads in thread pool 1 can process the tasks in task queue 1 normally.
[0268] It should be understood that after the thread pool module stops running the perception class algorithm of medium priority, for thread pool 2 that executes the perception class algorithm of medium priority, the state of the stable thread in thread pool 2 can be set to a dormant state to save system resources. Setting the state of the stable thread in thread pool 2 to a dormant state does not destroy the stable thread in thread pool 2. In this way, when the state of the electronic device is frequently switched, the execution capability of the stable thread pool 2 can be quickly restored, thereby reducing the overhead and delay of recreating the thread, improving the response speed of the system and the smoothness of the camera application operation, and improving the user experience.
[0269] As the camera application exits, some threads may be destroyed in order to release system resources. The following describes the destruction of threads in thread pool 2 and variable threads in thread pool 1 as an example.
[0270] S1122: Based on the input operation instructing to exit the camera application, the camera application instructs the thread pool module to exit the camera application.
[0271] S1123, the thread pool module clears the task queue, destroys the threads in thread pool 2, and the variable threads in thread pool 1.
[0272] It can be understood that in response to the user's operation of exiting the camera application, the camera application indicates to the thread pool module that the electronic device has exited the camera application. Correspondingly, based on the instruction of the camera application, the thread pool module can stop running the perception algorithm.
[0273] It should be noted that, when there are tasks in task queue 1, the thread pool module can clear the tasks in task queue 1; when there are tasks in task queue 1 and task queue 2, the thread pool module can clear the tasks in task queue 1 and task queue 2.
[0274] It can be understood that when the electronic device exits the camera application, the thread pool module may have the following situations:
[0275] Case 1: Thread pool 1 includes stable threads, and thread pool 2 has no threads. In this case, the thread pool module can clear the task queue.
[0276] Case 2: Thread pool 1 includes stable threads and variable threads, and there are no threads in thread pool 2. At this time, the thread pool can clear the task queue and destroy the variable threads in thread pool 1.
[0277] Case 3: Thread pool 1 includes stable threads, or thread pool 1 includes stable threads and variable threads, and thread pool 2 includes stable threads, or thread pool 2 includes stable threads and variable threads, then the thread pool module clears the task queue, and destroys the threads in thread pool 2, and / or destroys the variable threads in thread pool 1.
[0278] That is, for thread pool 1, if there are variable threads in thread pool 1, the thread pool module can destroy the variable threads in thread pool 1, and if there are no variable threads in thread pool 1, the thread pool module may not perform this step. For thread pool 2, the possible situations at this time are: there are stable threads in thread pool 2, or there are stable threads and variable threads in thread pool 2, then the thread pool module can destroy each thread in thread pool 2.
[0279] In order to reduce frequent thread destruction and creation, when the camera application exits, some threads can be kept alive within a preset time, that is, they are not destroyed during the preset time. The following description is taken as an example of including stable threads in thread pool 1 within the preset time.
[0280] That is, destroying the stable thread in thread pool 1 can be regarded as a thread with a keep-alive mechanism, that is, after the electronic device exits the camera application, the electronic device may not immediately destroy the stable thread, but allow the stable thread to continue to exist for a preset period of time after the camera application exits the camera. When the user restarts the camera application, the electronic device can immediately use the existing stable thread to perform tasks without having to wait for the thread to be re-created and initialized, thereby significantly shortening the startup time of the camera application and improving the user experience.
[0281] In addition, it should be noted that, during the above camera application exit process, the thread pool module may not destroy the stable threads in thread pool 1, and may set the stable threads in thread pool 1 to a dormant state.
[0282] S1124: If the camera application is not restarted within the preset time, the thread pool module destroys the stable thread in thread pool 1.
[0283] It can be understood that if the electronic device does not re-enter the camera application within a preset time, the thread pool module can destroy the stable thread in thread pool 1 to release system resources.
[0284] In addition, if the electronic device re-enters the camera application within the preset time period, the electronic device may execute S1125 to S1130 as follows.
[0285] S1125. Within a preset time period, based on the input operation indicating starting the camera application, the camera application indicates that the sensing module confirms the status of the electronic device.
[0286] It can be understood that the implementation of S1125 is similar to the implementation of S1104, and reference can be made to the above description, which will not be repeated here.
[0287] Similar to process 1, the perception module may confirm that the status of the electronic device is at various levels. The following takes the perception module reconfirming that the status of the electronic device is level 1 as an example to introduce the process of the electronic device re-entering the camera application within a preset time period.
[0288] S1126. The sensing module confirms that the status of the electronic device is level 1.
[0289] S1127, the perception module loads the high-priority perception class algorithm corresponding to level 1.
[0290] S1128. The perception module indicates the high-priority perception class algorithm to the task module.
[0291] The task module can set the task of adding a high-priority perception algorithm to the task queue 1 based on the indication.
[0292] S1129. The perception module instructs the thread pool module to run a high-priority perception algorithm.
[0293] It can be understood that the implementation of S1126 to S1129 is similar to the implementation of S1105 to S1108, and the relevant description of S1105 to S1108 can be referred to, which will not be repeated here.
[0294] S1130 , the thread pool module re-enables the stable threads in thread pool 1 , and uses the stable threads to process the tasks in task queue 1 .
[0295] The thread pool module re-enabling the stable thread in thread pool 1 can also be understood as: when there are tasks recorded in task queue 1, the thread pool module sets the stable thread in thread pool 1 from the dormant state to the active state, so that the stable thread in thread pool 1 can process the tasks in task queue 1.
[0296] Through the above implementation, the thread pool module does not destroy the stable thread in the thread pool 1 within the preset time after the electronic device exits the camera application. After the camera application is started again within the preset time, the thread pool module can quickly re-enable the stable thread to process the tasks in the task queue 1, thereby reducing the frequent destruction and creation of stable threads in the thread pool 1, and helping to reduce the occurrence of freezes when the camera application is restarted, thereby improving the response speed of the camera application and enhancing the user experience.
[0297] It can be understood that method 1100 takes the thread pool module including thread pool 1 and thread pool 2 as an example to introduce the camera control method provided by the embodiment of the present application, and in the actual use of the camera application, the thread pool module may be configured with more or fewer thread pools depending on the performance of the electronic device. Fig.14 Introduce the thread pool module in the embodiment of the present application.
[0298] Fig.14 A schematic diagram of the process of processing tasks by the thread pool module provided in the embodiment of the present application. Fig.14 As shown, the preset number of thread pools in the electronic device may be n thread pools. n is a preset positive integer, such as 3. After the camera application is started, the thread pool module may initialize thread pool 1, thread pool 2, ..., thread pool n. And each thread pool in thread pool 1, thread pool 2, ..., thread pool n corresponds to a task queue, for example, thread pool 1 corresponds to task queue 1, thread pool 2 corresponds to task queue 2, ..., thread pool n corresponds to task queue n.
[0299] In addition, the thread pool module can also create a stable thread for each of the n thread pools, and can set the state of the stable thread to an active state or a dormant state, and the thread pool module can set thread priorities and core binding operations for threads in each of the n thread pools.
[0300] It should be understood that the thread pool module initializes n thread pools, sets stable threads to active or dormant states, sets thread priorities, and implements core binding operations in a manner similar to the implementation methods of S1103 to S1109, and can refer to the above description, which will not be repeated here.
[0301] It can be understood that thread pool 1, thread pool 2, ..., thread pool n can process perception algorithms of different algorithm priorities. Therefore, the electronic device can add tasks corresponding to perception algorithms of different algorithm priorities in task queues 1 to task queues n respectively. And the stable threads in each thread pool in thread pool 1 to thread pool n can process tasks in task queues 1 to task queues n respectively.
[0302] For example, if n is 3, thread pool 1 can process high-priority perception algorithms, and electronic devices can add tasks corresponding to high-priority perception algorithms in task queue 1; thread pool 2 can process medium-priority perception algorithms, and electronic devices can add tasks corresponding to medium-priority perception algorithms in task queue 2; thread pool 3 can process low-priority perception algorithms, and electronic devices can add tasks corresponding to low-priority perception algorithms in task queue 3, and so on.
[0303] In addition, when there is a thread pool in the thread pool 1 to the thread pool n that is in a busy state, the thread pool module may create a variable thread for the thread pool in the busy state.
[0304] Exemplarily, for thread pool 1, when the state of thread pool 1 is busy, the thread pool module can create a variable thread for thread pool 1 to process tasks accumulated in the task queue corresponding to thread pool 1. Moreover, when the variable thread is idle, that is, after the variable thread has processed the task, the thread pool module can recycle the resources of the variable thread.
[0305] For other thread pools other than thread pool 1 (such as thread pool 2 and thread pool 3), after the electronic device exits the camera application, or when the thread pool state is idle (which can be understood as a stable thread in the thread pool being idle), the thread pool module can recycle the resources of stable threads in other thread pools and destroy stable threads in other thread pools. In addition, similar to thread pool 1, other thread pools can also determine the state of the thread pool when processing tasks in the task queue. When the thread pool state is busy, if a new task appears in the task queue, other thread pools can also create a variable thread to process the new task, and the resources of the variable thread can be recycled after the new task is processed.
[0306] It should be noted that when n is greater than or equal to 3, the implementation of the camera control method is similar to method 1100, and reference may be made to the above description, which will not be repeated here.
[0307] It should be understood that in the above embodiment, the relative size of the step numbers does not indicate the order in which the steps are executed, but is determined by the internal logic between the steps.
[0308] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.
[0309] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0310] The camera control method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above list sorting method.
[0311] Fig.15A schematic block diagram of a camera control device 1500 provided in an embodiment of the present application. The device 1500 includes a processor 1501, a communication interface 1502, and a memory 1503. The processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through an internal connection path, the memory 1503 is used to store instructions, and the processor 1501 is used to execute the instructions stored in the memory 1503. The communication interface 1502 can be used to send signals to other devices (such as the processor 1501 or the touch screen of the terminal device), and can also be used to receive signals from other devices (such as the memory 1503). Exemplarily, the communication interface 1502 reads the instructions stored in the memory 1503 and sends the instructions to the processor 1501.
[0312] It should be understood that the device 1500 can be specifically a terminal device in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the terminal device in the above method embodiment. Optionally, the memory 1503 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 1501 may be used to execute instructions stored in the memory, and when the processor 1501 executes instructions stored in the memory, the processor 1501 is used to execute the various steps and / or processes of the above method embodiment.
[0313] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0314] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0315] The camera control method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above related descriptions, which will not be repeated here.
[0316] An embodiment of the present application provides an electronic device, which terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.
[0317] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.
[0318] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.
[0319] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.
[0320] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.
[0321] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0322] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.
Claims
1. A camera control method, characterized in that: Applied to an electronic device, the electronic device is equipped with a camera application, and the method includes: Acquire a first parameter, where the first parameter is used to indicate a first operating state of the electronic device, and the first parameter includes parameters related to one or more of the following: temperature, central processing unit (CPU) resources, graphics processing unit (GPU) resources, or memory of the electronic device; When the camera application is running in the foreground, a first algorithm corresponding to the first parameter is loaded based on the first parameter, where the first algorithm is used to implement the function of the camera application.
2. The method according to claim 1, characterized in that After loading the first algorithm corresponding to the first parameter, the method further includes: Acquire a second parameter, where the second parameter is used to indicate a second operating state of the electronic device, where the second operating state is superior to the first operating state; Based on the second parameter, a second algorithm is loaded, where the second algorithm is used to implement the function of the camera application, and the second parameter corresponds to the first algorithm and the second algorithm.
3. The method according to claim 1, characterized in that After loading the first algorithm corresponding to the first parameter, the method further includes: Acquire a third parameter, where the third parameter is used to indicate a third operating state of the electronic device, where the third operating state is worse than the first operating state; Based on the third parameter, a third algorithm included in the first algorithm is deleted, the first parameter corresponds to the third algorithm, and the third parameter corresponds to algorithms in the first algorithm except the third algorithm.
4. The method according to any one of claims 1 to 3, characterized in that The loading of the first algorithm corresponding to the first parameter includes: Based on the first parameter, determining that the first operating state is a first level; The first algorithm is loaded based on a first corresponding relationship, where the first corresponding relationship includes a corresponding relationship between the first level and the first algorithm.
5. The method according to claim 4, characterized in that Before loading the first algorithm, the method further includes: Based on the frequency and / or algorithm type of calling the algorithm for implementing the preview function of the camera application, the algorithms in the first algorithm set pre-set in the electronic device are divided into different priorities, the algorithms in the first algorithm set are used to implement the preview function of the camera application, the first algorithm includes the algorithm with the highest priority in the first algorithm set, and the first corresponding relationship includes the corresponding relationship between the first level and the priority of the first algorithm.
6. The method according to claim 1, characterized in that After loading the first algorithm corresponding to the first parameter, the method further includes: The tasks in the first task queue are processed by the first thread in the first thread pool, where the first task queue is a task queue bound to the first thread pool, and the tasks in the first task queue are related to the operation of the first algorithm.
7. The method according to claim 6, characterized in that The first thread includes a first stable thread, and the first stable thread is a stable thread in the first thread pool; The method further comprises: receiving a first operation instructing to exit the camera application; Based on the first operation, the state of the first stable thread is set to a dormant state, the remaining threads for processing the tasks corresponding to the camera application are destroyed, and the camera application is exited.
8. The method according to claim 7, characterized in that After exiting the camera application, the method further includes: A second operation instructing to start the camera application is received, and a time interval between a time when the second operation is received and a time when the first operation is received is less than a first preset time length; Based on the second operation, starting the camera application; In a case where there are tasks in the first task queue, the state of the first stable thread is set from the dormant state to an active state, and the tasks in the first task queue are processed by the first stable thread.
9. The method according to claim 7, characterized in that: After exiting the camera application, the method further includes: At a first moment, the first stable thread is destroyed, and a time interval between the first moment and a moment when the first operation is received is greater than or equal to a first preset duration; receiving a third operation instructing to start the camera application, wherein a time at which the third operation is received is later than the first time; Based on the third operation, the camera application is started and the first stable thread is created.
10. The method according to any one of claims 6 to 9, characterized in that The first thread includes a first stable thread and a variable thread; Before processing the tasks in the first task queue by the first thread in the first thread pool, the method further includes: receiving a fourth operation instructing to start the camera application; Based on the fourth operation, creating the first stable thread; When the number of tasks in the first task queue exceeds a first threshold, creating the variable thread; The processing of tasks in the first task queue includes: The first task in the first task queue is processed by the first stable thread, and the second task in the first task queue is processed by the variable thread.
11. The method according to claim 10, characterized in that The method further comprises: After the variable thread finishes processing the second task, the variable thread is destroyed.
12. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1-11.
13. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.
15. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Camera application configuration method and device and storage medium
CN117149294A
Analysis method and device for application system architecture
CN117270960A
ARM (Advanced RISC Machines) platform-oriented edge reasoning framework optimization method and system
CN117608847A
Task processing method and electronic equipment
CN118467149A
Image processing method, electronic equipment, storage medium and chip system
CN118741298A
