Camera resource management method and related equipment

Abstract

The embodiment of the present application provides a camera resource management method and related equipment, the method comprising: obtaining node information of multiple nodes corresponding to at least two shooting modes respectively; determining the common resources and dynamic resources of at least two shooting modes; establishing an association relationship between the dynamic resources of at least two shooting modes; in response to setting the current shooting mode to one of the at least two shooting modes, loading the common resources and dynamic resources of at least two shooting modes; performing shooting operations according to the common resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode; in response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, performing shooting operations according to the common resources, dynamic resources and the association relationship between the dynamic resources corresponding to the other shooting mode. The embodiment of the present application can avoid the time-consuming reloading of node resources required for shooting mode switching, which causes freezes in the preview interface.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, which belongs to shooting technology, and in particular to a camera resource management method and related equipment. Background Art

[0002] With the development of terminal technology, smart terminal devices such as smartphones and tablets are equipped with cameras, have shooting functions, and support different shooting modes, such as portrait mode, night scene mode, macro mode, etc. Each shooting mode corresponds to a camera pipeline, which provides the camera resource collection required for the corresponding shooting mode. In actual use, the terminal device needs to switch between different shooting modes. Each time the shooting mode is switched, the camera pipeline needs to be switched accordingly, and the camera resource collection corresponding to the switched camera pipeline needs to be reloaded. The reloading of the camera resource collection can easily cause the camera preview interface to be blurred or stuck, thereby affecting the user experience. Summary of the invention

[0003] In view of the above, it is necessary to provide a camera resource management method and related equipment to solve the problem that when the above-mentioned terminal device switches the shooting mode, reloading the machine resource collection easily causes the camera preview interface to be blurred or stuck.

[0004] In a first aspect, the present application provides a camera resource management method, which is applied to an electronic device, and the method includes: obtaining node information of multiple nodes corresponding to at least two shooting modes respectively; determining the common resources and dynamic resources of the at least two shooting modes based on the node information of the multiple nodes; establishing an association relationship between the dynamic resources of the at least two shooting modes; in response to setting the current shooting mode to one of the at least two shooting modes, loading the common resources and dynamic resources of the at least two shooting modes; performing a shooting operation based on the common resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode; in response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, performing the shooting operation based on the common resources, dynamic resources and the association relationship between the dynamic resources corresponding to the other shooting mode.

[0005] Through the above technical solution, when controlling the camera application to enter the shooting mode, the node resources corresponding to multiple shooting modes can be loaded, and the common resources, dynamic resources in the node resources and the dependencies between the node resources can be determined. When the shooting mode is switched to another shooting mode, only the dependencies between the node resources can be switched without reloading the node resources, thereby avoiding the time-consuming reloading of the node resources causing the preview interface of the camera application to become stuck.

[0006] In a possible implementation, each of the at least two shooting modes corresponds to a camera pipeline, and the camera pipeline includes multiple nodes and dependencies between the multiple nodes, which are used to implement the shooting function corresponding to each shooting mode.

[0007] The above technical solution manages multiple node resources through the camera pipeline, which can improve the image processing efficiency during the shooting process.

[0008] In one possible implementation, the node information includes necessary information, node attributes and basic connection relationships. The necessary information includes a necessary identifier of each node in the current shooting mode. The node attributes include node name, node parameters, node status, and processing functions. The basic connection relationship includes input and output ports of each node.

[0009] The above technical solution pre-sets the node information to facilitate the flow distribution of nodes in various shooting modes and generate corresponding camera pipelines.

[0010] In a possible implementation, each node includes a configuration file, and obtaining node information of multiple nodes corresponding to at least two shooting modes includes: obtaining the node information of each node from the configuration file of each node through a dynamic resource manager.

[0011] The above technical solution stores the node information in a configuration file, making it easier to obtain the node information through a dynamic resource manager.

[0012] In one possible implementation, the determining of the common resources and dynamic resources of the at least two shooting modes based on the node information of the multiple nodes includes: comparing the node information of the multiple nodes corresponding to the at least two shooting modes, taking the nodes with the same node names and the same basic connection relationships as the common resources, and taking the nodes with different node names and the nodes with the same node names but different basic connection relationships as the dynamic resources.

[0013] The above technical solution can improve the resource loading efficiency in different shooting modes by predetermining the common resources and dynamic resources of different shooting modes, thereby improving the data processing efficiency during the shooting process.

[0014] In a possible implementation manner, the association relationship between the dynamic resources includes a dependency relationship between nodes in the dynamic resources in the at least two shooting modes.

[0015] The above technical solution pre-establishes the dependency relationship between nodes in dynamic resources under different shooting modes, so as to facilitate the reallocation of camera pipelines corresponding to the shooting mode when the shooting mode is switched, thereby improving the data processing efficiency during the shooting process.

[0016] In a possible implementation, establishing the association relationship between the dynamic resources of the at least two shooting modes includes: setting, by a dynamic resource manager, an input port and an output port of each node in the dynamic resource in the at least two shooting modes.

[0017] The above technical solution can accurately establish the association relationship between the dynamic resources of at least two shooting modes by setting the input port and output port of each node in the dynamic resources under the at least two shooting modes, so as to facilitate the reallocation of the camera pipeline corresponding to the shooting mode when the shooting mode is switched, thereby improving the data processing efficiency during the shooting process.

[0018] In a possible implementation, the establishing the association relationship between the dynamic resources of the at least two shooting modes further includes: writing the association relationship between the dynamic resources into a configuration file of each node in the dynamic resources.

[0019] The above technical solution writes the association relationship between dynamic resources into the configuration file of each node in the dynamic resources, so that when the shooting mode is switched, it is convenient to obtain the updated dependency relationship between nodes through the dynamic resource manager.

[0020] In a possible implementation, in response to setting the current shooting mode to one of the at least two shooting modes, loading the common resources and dynamic resources of the at least two shooting modes includes: in response to a user triggering an operation of a control or in response to a user performing a sliding operation on a camera application interface, controlling the camera application to display a preview interface corresponding to the control, determining the current shooting mode; if the current shooting mode is one of the at least two shooting modes, loading the common resources and dynamic resources of the at least two shooting modes; or if the current shooting mode is not any of the at least two shooting modes, loading the resources corresponding to the current shooting mode.

[0021] Through the above technical solution, the current shooting mode can be accurately determined, and when the current shooting mode is determined to be one of the at least two shooting modes, the common resources and dynamic resources of the at least two shooting modes are pre-loaded, so that when the shooting mode is switched, there is no need to reload the resources. When the current shooting mode is not any of the at least two shooting modes, the resources corresponding to the current shooting mode are loaded, thereby reducing the number of resources that need to be loaded and improving the resource loading efficiency.

[0022] In a possible implementation, the shooting operation is performed according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode, including: establishing a camera pipeline corresponding to the current shooting mode according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode; and processing the image data captured by the camera according to the camera pipeline corresponding to the current shooting mode.

[0023] Through the above technical scheme, a camera pipeline corresponding to the current shooting mode is established based on the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the predetermined current shooting mode, which effectively improves the generation efficiency of the camera pipeline and further improves the data processing efficiency during the shooting process.

[0024] In a possible implementation, in response to the current shooting mode being switched to another shooting mode among the at least two shooting modes, the shooting operation is performed according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the other shooting mode, including: in response to the operation of a user triggering a control or in response to the user controlling the camera application to display a preview interface corresponding to the control by performing a sliding operation, determining the switched shooting mode; if the switched shooting mode is another shooting mode among the at least two shooting modes, establishing a camera pipeline corresponding to the current shooting mode according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the other shooting mode; and processing the image data captured by the camera according to the camera pipeline corresponding to the other shooting mode.

[0025] Through the above technical solution, when the shooting mode is switched, it is only necessary to switch the dependencies between resources and establish a new camera pipeline without reloading resources, which effectively improves the generation efficiency of the camera pipeline and further improves the data processing efficiency during the shooting process.

[0026] In a possible implementation manner, the method further includes: if the switched shooting mode is not another shooting mode of the at least two shooting modes, loading resources corresponding to the switched shooting mode.

[0027] Through the above technical solution, when the switched shooting mode is not another shooting mode of at least two shooting modes, the resources corresponding to the switched shooting mode are loaded in time, thereby improving the data processing efficiency when shooting according to the switched shooting mode.

[0028] In a possible implementation, the method further includes: determining whether the number of dynamic resources of at least two shooting modes is greater than or equal to a preset number; when the number of dynamic resources of at least two shooting modes is less than the preset number, establishing an association relationship between the dynamic resources of at least two shooting modes.

[0029] Through the above technical solution, when the number of dynamic resources of at least two shooting modes is less than a preset number, it means that the dynamic resources of at least two shooting modes are relatively small, that is, the difference in node resources between at least two shooting modes is relatively small, and the number of resources that need to be loaded at the same time is relatively small. It is suitable for establishing an association relationship between the dynamic resources in at least two shooting modes, and there is no need to reload resources when the shooting mode is switched.

[0030] In a second aspect, the present application provides an electronic device, comprising a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device executes the above-mentioned camera resource management method.

[0031] In a third aspect, the present application provides a chip coupled to a memory in an electronic device, wherein the chip is used to control a processor of the electronic device to execute the above-mentioned camera resource management method.

[0032] In a fourth aspect, the present application provides a computer storage medium storing program instructions, which, when executed on an electronic device, causes a processor of the electronic device to execute the above-mentioned camera resource management method.

[0033] In addition, the technical effects brought about by the second to fourth aspects can be found in the descriptions related to the methods of each design in the above method part, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of a camera architecture of the prior art.

[0035] Figure 2 is another schematic diagram of a camera architecture of the prior art.

[0036] Figure 3 It is a software architecture diagram of an electronic device provided in one embodiment of the present application.

[0037] Figure 4 This is another software architecture diagram of an electronic device provided by an embodiment of the present application.

[0038] Figure 5 It is a flowchart of a camera resource management method provided by an embodiment of the present application.

[0039] Figure 6 It is a schematic diagram of a camera application program interface provided in one embodiment of the present application.

[0040] Figure 7is a schematic diagram of a camera architecture provided in an embodiment of the present application.

[0041] Figure 8 It is a schematic diagram of a dynamic resource manager provided in an embodiment of the present application.

[0042] Fig. 9 It is a flowchart of a camera resource management method provided by an embodiment of the present application.

[0043] Fig.10 This is a hardware architecture diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] The terms "first" and "second" involved in an embodiment of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in an embodiment of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field in this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can represent A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" refers to one or more. "Multiple" refers to two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c. In the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0046] For ease of understanding, some concepts related to the embodiments of the present application are exemplarily described for reference:

[0047] Node: The basic component unit in the camera module, which belongs to the camera resources and is responsible for performing specific functions, such as image capture, image processing, etc.

[0048] Pipeline: A data processing flow consisting of multiple nodes in a specific order, used to implement various camera functions or modes, such as preview, photo taking, and video recording.

[0049] Session: An interface for interacting with camera hardware, used to configure and control camera module parameters, such as resolution, exposure time, focus mode, etc. A session can contain one or more Pipelines.

[0050] Feature: Various functions and features provided by the camera module, such as autofocus, face recognition, HDR mode, etc.

[0051] Use case: The way and purpose of using the camera module in a specific scenario, such as taking photos, recording videos, real-time video transmission, etc. Different use cases may involve different combinations of Pipelines and Features.

[0052] Stream: A real-time data stream of continuous image data or video data from a camera sensor. Through the camera module's Pipeline, these real-time data streams can be processed, encoded, compressed, and then output to a display device, storage device, or other target device. Streams are usually used for real-time preview (for example, displaying the camera's captured image in a camera application), video recording, real-time analysis, and other application scenarios.

[0053] With the development of terminal technology, smart terminal devices such as smartphones and tablets are equipped with cameras, have shooting functions, and support different shooting modes, such as portrait mode, night scene mode, macro mode, etc. Each shooting mode corresponds to a camera pipeline, and the camera pipeline provides the camera resource collection required for the corresponding shooting mode. In actual use, the terminal device needs to switch between different shooting modes. Each time the shooting mode is switched, the camera pipeline needs to be switched accordingly, and the camera resource collection corresponding to the switched camera pipeline needs to be reloaded. The reloading of the camera resource collection can easily cause the camera preview interface to be blurred or stuck, thereby affecting the user experience.

[0054] The current mainstream camera architecture usually organizes camera resources by abstracting a certain photo or video recording behavior into a single or continuous request processing flow. The camera resources required for this process are usually created in the flow distribution stage. When the shooting mode is switched, in order to adapt to the camera resource requirements of the switched shooting mode, the underlying camera resources also need to be refreshed and recreated to complete a new photo or video recording request, and the above process is completed in the flow distribution stage. Among them, the flow distribution of the camera pipeline refers to processing and adjusting the camera input stream (sensor input) through a series of image processing and algorithm modules to generate the final camera output stream (camera output). This process includes various stages from input to output, such as: image sensor data capture, color correction, noise reduction, automatic exposure control, autofocus, etc. The flow distribution of the camera pipeline determines the order and parameter settings of each module, as well as the data flow and interaction between modules, thereby affecting the final generated image quality and functional performance.

[0055] There are many derivatives of the use case in the camera architecture Camx. Camx creates different use case objects for different streams to manage the selection of features, and to create sessions, pipelines, and corresponding nodes. In the current camera architecture, the upper-layer application configures the corresponding stream to the bottom layer according to the requirements. The bottom layer selects the corresponding use case based on the requested stream, and then completes the function selection. Different functions contain different camera pipelines. The process of creating a camera pipeline is the process of connecting a series of nodes in series, which can realize the processing of a specific request. The above process describes the flow distribution process of a camera pipeline, which is also the current general process of camera resource creation and management.

[0056] See also Figure 1 As shown, it is a schematic diagram of the camera architecture of the prior art. For example, a use case 1 includes function 1, function 1 includes session 1 and session 2, session 1 includes pipeline 1, session 2 includes pipeline 2, pipeline 1 includes sensor node SensorNode, image front-end engine node IFE Node, A node A Node, B node B Node, after the stream is processed by each node in pipeline 1, the image data or video data is output through the output node Sink Target, for example, it is output to a display screen for display, output to a memory for storage, output to other devices or applications, etc. Pipeline 2 includes sensor node SensorNode, image front-end engine node IFE Node, C node A Node, D node D Node, after the stream is processed by each node in pipeline 2, the image data or video data is output through the output node Sink Target.

[0057] See also Figure 2 As shown, it is another schematic diagram of the camera architecture of the prior art. Taking pipeline 1 as an example, the A node in pipeline 1 is the Bayer processing stage node BPS Node, the B node is the JPEG encoding node JPEG Node, the sensor node obtains the image data stream, and transmits the image data stream to the image front-end engine node, the image front-end engine node performs color correction, downsampling, de-mosaicing and other processing on the image data, and then transmits the image data stream to the Bayer processing stage node, the Bayer processing stage node removes bad pixels of the image data, performs HDR processing, noise reduction processing, etc. on the image data, and then transmits the image data stream to the JPEG encoding node, the JPEG encoding node encodes the image data into an image file in JPEG format and outputs it.

[0058] When the camera application of the electronic device adopts the shooting mode 1 corresponding to the pipeline 1 to shoot, the pipeline 1 is loaded with the sensor node, the image front-end engine node, the A node and the B node, and the preview screen under the shooting mode 1 is displayed on the display screen. When the camera application switches the shooting mode, for example, the shooting mode 1 is switched to the shooting mode 2, the pipeline 1 of the camera module is switched to the pipeline 2 accordingly. At this time, the pipeline 2 needs to reload the camera resources, that is, the sensor node, the image front-end engine node, the C node and the D node need to be loaded. It takes a certain amount of time to reload the camera resources. During this period, the camera application cannot obtain the image data stream and process the image data stream, resulting in the inability to display the preview screen on the display screen, which in turn causes the preview screen to be blurred or stuck, which makes the user feel visually disconnected, thereby affecting the user's experience.

[0059] In order to avoid blurring or freezing of the preview screen of the camera application due to the need to reload camera resources when switching shooting modes, an embodiment of the present application provides a camera resource management method, which can pre-load camera resources under different shooting modes and establish connections between different camera resources under different shooting modes, so that when switching shooting modes, only the camera resources need to be reassembled without reloading the camera resources, thereby achieving fast camera pipeline switching, making the preview screen of the camera application displayed smoothly, and providing users with a smooth visual experience.

[0060] See also Figure 3 As shown, it is a software architecture diagram of the electronic device provided in the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system is from top to bottom an application layer 101, a framework layer 102, an Android runtime (Android runtime) and a system library 103, a hardware abstraction layer 104, a kernel layer 105, and a hardware layer 106.

[0061] The application layer 101 may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, device control service and other applications.

[0062] The framework layer 102 provides an application programming interface (API) and a programming framework for the application programs of the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0063] Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to the application. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for electronic devices. For example, the management of call status (including connected, hung up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of an icon or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0064] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

[0065] The application layer 101 and the framework layer 102 run in a virtual machine. The virtual machine executes the Java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0066] The system library 103 may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (eg, OpenGL ES), a 2D graphics engine (eg, SGL), and the like.

[0067] Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple commonly used audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0068] The hardware abstraction layer 104 runs in the user space, encapsulates the kernel layer driver, and provides a calling interface to the upper layer.

[0069] The kernel layer 105 is a layer between hardware and software, and includes at least a display driver, a touch driver, an audio driver, and a sensor driver.

[0070] The kernel layer 105 is the core of the operating system of the electronic device. It is the first layer of software expansion based on hardware, provides the most basic functions of the operating system, is the basis of the operating system, is responsible for managing the system's processes, memory, device drivers, files and network systems, and determines the performance and stability of the system. For example, the kernel layer can determine the operating time of an application on a certain part of the hardware.

[0071] The kernel layer 105 includes programs closely related to the hardware, such as interrupt handlers, device drivers, etc., and also includes basic, common, and high-frequency modules, such as clock management modules, process scheduling modules, etc., and also includes key data structures. The kernel layer can be set in the processor or solidified in the internal memory.

[0072] The hardware layer 106 includes the hardware of the electronic device, such as a display screen, buttons, a camera, etc.

[0073] For the detailed implementation process of the camera resource management method, please refer to the description in each embodiment below.

[0074] See also Figure 4 As shown, it is another software architecture diagram of an electronic device provided by an embodiment of the present application. The application layer 101 includes a camera application (Camera App) 1011, the framework layer 102 includes a camera framework (Camera Framework) 1021, and the hardware abstraction layer 104 includes a camera service (Camera Service) 1041 and a camera provider (CameraProvider) 1042. The camera service 1041 encapsulates the Camera AIDL cross-process interface, provides it to the camera framework 1021 for calling, receives an image request from the camera framework 1021, and sends the image request to the camera provider 1042, and receives the data returned by the camera provider 1042, including partial metadata (partial metadata), metadata (metadata) and image data, and the partial metadata includes the shooting parameters of the image, including exposure time, sensitivity, white balance data, etc., and the metadata includes the shooting parameters and related information of the image, such as the related information including geographic location information, shooting time, shooting mode, etc., and the image data includes the image or video shot by the camera device, and the image data is transmitted by sending the image buffer (image buffer) information.

[0075] The camera provider 1042 is used to manage and provide access permissions and functions for camera devices. It is an intermediary between the camera hardware and the camera application, providing the application with camera-related functions and interfaces, specifically for managing the opening and closing of camera devices; providing a list of camera devices and property information, such as supported resolutions, frame rates, etc.; processing the configuration of camera devices, including setting parameters such as exposure, focal length, white balance, etc.; processing image capture requests, including starting preview, taking photos, and recording videos; handling errors and abnormal conditions of camera devices, such as connection problems or sensor failures.

[0076] The camera provider includes 1042, including a main program unit 1043 and a customization unit 1044. The main program unit 1043 includes the main program code of the camera provider 1042. The customization unit 1044 is a CamX-CHI architecture, including a CamX unit and a CHI unit. The CamX unit includes a basic service code, and the CHI unit includes an extension code and a customization code, which is convenient for adding customized extension functions. Specifically, the CamX unit mainly includes a hal (Hardware Abstraction Layer) module that implements the HAL3 entry, a csl (Chip Support Library) module that implements interaction with the V4L2 driver, an hwl that implements the hardware node, and a swl that implements the software node. The swl is a Node that does not have independent computing capabilities and needs to rely on the CPU to be implemented. The CHI unit abstracts the concepts of Usecase, Feature, Session, Pipeline, and Node, so that manufacturers can access their own algorithms by implementing the Node interface, and flexibly configure the structural relationship of Usecase, Pipeline, and Node through XML files (configuration files).

[0077] The kernel layer 105 includes a camera driver 1051 , and the hardware layer 106 includes a lens 1061 , a photoreceptor 1062 , and an image signal processor 1063 . The lens 1061 , the photoreceptor 1062 , and the image signal processor 1063 constitute a camera hardware module.

[0078] See also Figure 5 FIG. 1 is a flowchart of a camera resource management method provided by an embodiment of the present application. The method is applied in an electronic device, and the camera resource management method includes:

[0079] S101, obtaining node information of a plurality of nodes corresponding to at least two shooting modes respectively.

[0080] In one embodiment of the present application, the camera application of the electronic device can generally provide multiple shooting modes, for example, the multiple shooting modes may include eagle eye capture mode, smart focus tracking mode, night scene mode, macro mode, portrait mode, slow motion mode, panoramic mode, smile capture mode, etc. Each shooting mode corresponds to a camera pipeline, and the camera pipeline includes multiple nodes and dependencies between multiple nodes, which are used to implement the shooting function corresponding to the shooting mode. Among them, the dependency relationship between multiple nodes at least includes the connection relationship between multiple nodes.

[0081] In one embodiment of the present application, the node information of the node includes but is not limited to: necessary information, node attributes, and basic connection relationship. The necessary information of the node includes the necessary identification of the node in the current shooting mode, and the necessary identification includes the first identification and the second identification. If the node is a necessary node of a shooting mode, then in this shooting mode, the necessary identification of the node corresponding to the shooting mode is the first identification. If the node is a non-essential node of a shooting mode, then in this shooting mode, the necessary identification of the node corresponding to the shooting mode is the second identification. For example, the first identification is 1 and the second identification is 0. Node attributes include, but are not limited to: node name, node parameters, node status, processing function, for example, the node name is an image front-end engine node, the node parameters include image format, shooting parameters, etc., the node status is the working status of the node, including working and not working, and the processing function includes the algorithm, model, etc. used by the node when processing the image. The basic connection relationship includes the input and output ports of the node. The basic connection relationship can reflect the dependency relationship between nodes. For example, if the input port of node B is node A and the output port is node C, then node B has a dependency relationship with node A and node C. The basic connection relationship can also be included in the node attributes.

[0082] like Figure 4 As shown, in one embodiment of the present application, the customization unit 1044 includes a dynamic resource manager 1045, each node includes a configuration file for storing node information, and the dynamic resource manager 1045 can obtain the node information of each node from the configuration files of each node corresponding to at least two shooting modes. After the electronic device is turned on, the operating system of the electronic device initializes the camera provider, and after the camera provider is initialized, the node information of each node is obtained from the configuration files of each node corresponding to at least two shooting modes through the dynamic resource manager.

[0083] S102: Determine common resources and dynamic resources of at least two shooting modes according to node information of multiple nodes.

[0084] In one embodiment of the present application, the dynamic resource manager compares the node information of multiple nodes corresponding to at least two shooting modes, and regards nodes with the same node name and the same basic connection relationship as public resources, and regards nodes with different node names and nodes with the same node name but different basic connection relationships as dynamic resources. After determining that a node is a public resource or a dynamic resource, the dynamic resource manager writes the resource identifier of the node in the configuration file corresponding to the node to mark the node as a public resource or a dynamic resource under the at least two shooting modes. The resource identifier includes a third identifier and a fourth identifier, the third identifier is used to mark the node as a public resource under the at least two shooting modes, and the fourth identifier is used to mark the node as a dynamic resource under the at least two shooting modes. For example, the third identifier is 120, and the fourth identifier is 121, wherein, in the resource identifier, 1 represents shooting mode 1, 2 represents shooting mode 2, then 12 represents the combination of shooting mode 1 and shooting mode 2, 0 represents a public resource, and 1 represents a dynamic resource. Therefore, the embodiment of the present application marks whether the node belongs to a dynamic resource or a public resource under a certain shooting mode by adding relevant attribute descriptions to the node.

[0085] For example, Figure 1 As shown, the node name and basic connection relationship of the sensor node in pipeline 1 are the same as the node name and basic connection relationship of the sensor node in pipeline 2, so the sensor node can be a common resource for the shooting mode corresponding to pipeline 1 and the shooting mode corresponding to pipeline 2. The node name of the IFE node in pipeline 1 is the same as the node name of the IFE node in pipeline 2, but the basic connection relationship of the IFE node in pipeline 1 is different from the basic connection relationship of the IFE node in pipeline 2, and the node name and basic connection relationship of the A node and the B node in pipeline 1 are different from the node name and basic connection relationship of the Cnode and the D node in pipeline 2, so the IFE node, A node, B node, Cnode, and D node can be dynamic resources for the shooting mode corresponding to pipeline 1 and the shooting mode corresponding to pipeline 2.

[0086] S103: Establish an association relationship between at least two dynamic resources in shooting modes.

[0087] In one embodiment of the present application, the association relationship between dynamic resources includes the dependency relationship between nodes in the dynamic resources in at least two shooting modes. The dynamic resource manager sets the input port and output port of each node in the dynamic resource in at least two shooting modes, and establishes the association relationship between the dynamic resources in at least two shooting modes. Among them, the input port of a node in the dynamic resource includes a node that inputs media data (such as an image or video, etc.) into the node in each shooting mode, and the output node is a node that receives the media data output by the node in each shooting mode.

[0088] In one embodiment of the present application, an association relationship between dynamic resources in at least two shooting modes is established through a dynamic resource manager, and the association relationship between the dynamic resources is written into a configuration file of each node. For example, at least two shooting modes include a first shooting mode and a second shooting mode, and the input port of A node in the first shooting mode is set to IFE node, and the output port is set to B node; the input port of A node in the second shooting mode is set to IFEnode, and the output port is set to C node; the input port of B node in the first shooting mode is set to A node, and the output port is set to the output node sink target of media data; the input port of Cnode in the second shooting mode is set to IFE node, and the output port in the second shooting mode is set to D node; the input port of D node in the second shooting mode is set to C node, and the output port in the second shooting mode is set to the output node sinktarget of media data.

[0089] It can be understood that in the prior art, after the camera application is started and a shooting mode is selected, the camera pipeline corresponding to the selected shooting mode is allocated, and multiple nodes of the camera pipeline and the dependencies between the nodes in the single shooting mode are configured. In the embodiment of the present application, the dependencies between nodes corresponding to different shooting modes can be pre-configured by a dynamic resource manager.

[0090] S104: In response to setting the current shooting mode of the camera application to one of the at least two shooting modes, load common resources and dynamic resources of the at least two shooting modes.

[0091] In one embodiment of the present application, when a user needs to use a camera of an electronic device to shoot an image or video, the user clicks on an icon of a camera application on a desktop interface of the electronic device, or clicks on a quick-start control of a camera application on a lock screen interface of the electronic device to execute an operation of opening the camera application. In response to the user's operation of opening the camera application, the electronic device opens the camera application, displays an interface of the camera application, such as a preview interface of camera shooting, and provides multiple shooting modes.

[0092] See also Figure 6 FIG. 1 is a schematic diagram of a camera application program interface provided by an embodiment of the present application. In an embodiment of the present application, each shooting mode has a corresponding control, which may be the name of the shooting mode, for example, Figure 6 The "photograph" in the camera application interface shown is a control corresponding to the normal photographing mode, the "panorama" is a control corresponding to the panoramic mode, and the "recording" is a control corresponding to the normal recording mode. The user can control the camera application to display the preview interface corresponding to the control by triggering the control or by performing a sliding operation (such as a left sliding operation or a right sliding operation) on the camera application interface, and control the camera application to switch the current shooting mode to the shooting mode corresponding to the control. For example, when the user triggers the "panorama" control, the camera application sets the current shooting mode to the panoramic mode and displays the preview interface corresponding to the panoramic mode. In response to the user triggering the control operation or in response to the user controlling the camera application to display the preview interface corresponding to the control by performing a sliding operation on the camera application interface, the current shooting mode is determined, and it is determined whether the current shooting mode is one of the at least two shooting modes. If the current shooting mode is one of the at least two shooting modes, the common resources and dynamic resources of the at least two shooting modes are loaded. If the current shooting mode is not any of the at least two shooting modes, the resources corresponding to the current shooting mode are loaded, and there is no need to load the resources of multiple shooting modes.

[0093] In one embodiment of the present application, nodes belonging to common resources of at least two shooting modes and nodes belonging to dynamic resources of at least two shooting modes can be determined based on node information (such as required identifiers) and resource identifiers in a configuration file of each node provided by an extension unit in a camera provider.

[0094] For example, at least two shooting modes include normal shooting mode and panoramic mode. If the electronic device opens the camera application and the user selects the normal shooting mode by performing the corresponding operation, the common resources and dynamic resources of the normal shooting mode and the panoramic mode are loaded. If the user selects the portrait mode by performing the corresponding operation, the resources corresponding to the portrait shooting mode are loaded.

[0095] In one embodiment of the present application, after a user opens a camera application, at the application layer, the camera application detects user operations, and generates a shooting request (i.e., a first shooting request) by selecting a shooting mode so that the camera application displays a preview interface, or by triggering a shooting control to perform image shooting or video shooting. The first shooting request is sent to the camera framework of the framework layer, and the first shooting request is sent to the camera service of the hardware abstraction layer through the camera framework. The camera service responds to the first shooting request, creates a shooting request response thread, and sends the first shooting request to the shooting request response thread, responds to the first shooting request, and the shooting request response thread sends the first shooting request to the camera provider, providing a customized unit of the camera provider (i.e., the CamX-CHI architecture) to process the first shooting request. The camera provider triggers a use case, a session, and a camera pipeline for processing the first shooting request, performs flow distribution through the camera pipeline, determines common resources and dynamic resources of at least two shooting modes including the current shooting mode, and loads the determined common resources and dynamic resources of at least two shooting modes including the current shooting mode.

[0096] In another embodiment of the present application, the electronic device may automatically enter a shooting mode after the camera application is turned on. The shooting mode automatically entered may be a default shooting mode set by the user or a shooting mode corresponding to the last time the camera application performed a shooting operation.

[0097] S105, performing a shooting operation according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode.

[0098] In one embodiment of the present application, the dynamic resource manager establishes a camera pipeline corresponding to the current shooting mode based on the public resources corresponding to the current shooting mode, dynamic resources, and the association relationship between the dynamic resources, and processes the image data captured by the camera according to the camera pipeline corresponding to the current shooting mode, thereby executing the shooting operation corresponding to the current shooting mode.

[0099] See also Figure 7As shown, it is a schematic diagram of the camera architecture provided by an embodiment of the present application. For example, in the current shooting mode, the loaded public resources and dynamic resources include F node, E node, A node, B node, C node, and D node. The public resources corresponding to the current shooting mode include F node, and the dynamic resources include A node and B node. The association relationship between the dynamic resources includes that the input port of A node is F node and the output port is B node; the input port of Bnode is A node and the output port is sink target. The dynamic resource manager establishes the node sequence of pipeline 1 corresponding to the current shooting mode as F node, Anode, B node, and sink target according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode, and executes the shooting operation corresponding to the current shooting mode through pipeline 1.

[0100] In one embodiment of the present application, after the common resources and dynamic resources of at least two shooting modes including the current shooting mode are loaded, the first shooting request continues to be processed, the flow is distributed through the camera pipeline corresponding to the current shooting mode, multiple nodes of the camera pipeline and the dependencies between the multiple nodes are determined, and the establishment of the camera pipeline is completed. The image data captured by the camera is processed in sequence through the multiple nodes in the camera pipeline, and the processed image data is transmitted to the camera application via the camera provider, the camera service, and the camera framework. The processed image data is drawn and rendered by the camera application and displayed as a preview screen, thereby completing the response to the first shooting request.

[0101] In one embodiment of the present application, assuming that the current shooting mode is a normal shooting mode, the multiple nodes of the pipeline 1 corresponding to the normal shooting mode are sensor node, IFE node, BPS node, and Sink Target in sequence, and the execution process of the shooting operation includes: first, the sensor node collects media data (such as images or videos), and transmits the collected media data to the IFE node, and the IFE node pre-processes the media data, including white balance correction, color correction, noise reduction, etc., and transmits the pre-processed media data to the BPS node, and the BPS node optimizes the media data, including bad pixel removal, phase focus, de-mosaicing, downsampling, HDR processing, noise reduction, etc., and transmits the optimized media data to the SinkTarget for output, such as output to the camera application, and the media data is drawn and rendered by the camera application, so that the media data captured by the camera is displayed as a preview screen. The functions of the nodes in the above pipeline 1 are only used for illustration, and the current shooting mode can also include different nodes according to requirements, and the embodiments of the present application are not limited to this.

[0102] S106: In response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, a shooting operation is performed according to the public resources, dynamic resources and the association relationship of the dynamic resources corresponding to the other shooting mode.

[0103] In one embodiment of the present application, when the user needs to switch the shooting mode, the control corresponding to another shooting mode can be triggered to determine whether the shooting mode corresponding to the triggered control is another shooting mode of at least two shooting modes. If the shooting mode corresponding to the triggered control is another shooting mode of at least two shooting modes, a camera pipeline corresponding to another shooting mode is established according to the public resources, dynamic resources and the association relationship of the dynamic resources corresponding to the other shooting mode, and the image data captured by the camera is processed according to the camera pipeline corresponding to the other shooting mode, so as to perform the shooting operation corresponding to the other shooting mode. If the shooting mode corresponding to the triggered control is not another shooting mode of at least two shooting modes, that is, the switched shooting mode is not another shooting mode of at least two shooting modes, it is necessary to reload the resources corresponding to the currently switched shooting mode and perform the shooting operation.

[0104] like Figure 7As shown, the public resources corresponding to another shooting mode include F node, the dynamic resources include C node and Dnode, and the association relationship between the dynamic resources includes that the input port of C node is F node and the output port is Dnode; the input port of Dnode is C node and the output port is sink target. The dynamic resource manager establishes the node sequence of pipeline 2 corresponding to another shooting mode as F node, C node, D node and sink target according to the public resources corresponding to another shooting mode, the dynamic resources and the association relationship between the dynamic resources.

[0105] For example, assuming that another shooting mode among at least two shooting modes is a panoramic mode, the data flow of the pipeline 2 corresponding to the panoramic mode is sensor node, IFE node, distortion correction node, and Sink Target in sequence. The execution process of the shooting operation includes: first, the sensor node collects media data (such as images or videos), transmits the collected media data to the IFE node, the IFE node pre-processes the media data, including white balance correction, color correction, noise reduction, etc., transmits the pre-processed media data to the Distortion correction node, and the Distortion correction node performs distortion correction on the media data, and transmits the distortion-corrected media data to the Sink Target for output, such as output to a camera application, and the media data is rendered by the camera application, so that the media data captured by the camera is displayed as a preview screen. The functions of the nodes in the above pipeline 2 are only used for illustration, and another shooting mode among at least two shooting modes can also include different nodes according to needs, and the embodiments of the present application are not limited to this.

[0106] In one embodiment of the present application, when the camera application detects a user operation and switches the shooting mode so that the camera application displays a preview interface, a shooting request is generated again, and the shooting request (i.e., the second shooting request) is sent to the camera framework of the framework layer. The second shooting request is sent to the camera service of the hardware abstraction layer through the camera framework. The camera service responds to the second shooting request and sends the second shooting request to the shooting request response thread to respond to the second shooting request. The shooting request response thread sends the second shooting request to the camera provider, and the customized unit (i.e., CamX-CHI architecture) of the camera provider is provided to process the second shooting request. The flow is directly distributed through the camera pipeline without loading resources again. Multiple nodes of the camera pipeline corresponding to another shooting mode and the dependency relationship between the multiple nodes are determined to complete the establishment of the camera pipeline. The image data captured by the camera is processed in sequence by the multiple nodes in the camera pipeline, and the processed image data is transmitted to the camera application via the camera provider, the camera service, and the camera framework. The processed image data is drawn and rendered by the camera application and displayed as a preview screen, thereby completing the response to the second shooting request.

[0107] Through the above-mentioned embodiments of the present application, when controlling the camera application to enter the shooting mode, node resources corresponding to multiple shooting modes can be loaded, and the common resources, dynamic resources in the node resources and the dependencies between the node resources can be determined. When the shooting mode is switched to another shooting mode, only the dependencies between the node resources can be switched without reloading the node resources, thereby avoiding the time-consuming reloading of the node resources causing the preview interface of the camera application to become stuck.

[0108] See also Figure 8As shown, it is a schematic diagram of a dynamic resource manager provided by an embodiment of the present application. The dynamic resource manager includes an input adapter, multiple dynamic nodes and an output adapter. The adapter is a data interface, which can interact with a hardware device (such as a camera hardware module) or an application (such as a camera application) to realize data transmission. The multiple dynamic nodes include A node, B node, C node, D node, and E node. The extension unit of the image provider can input external static configuration information into the dynamic resource manager. The external static configuration information includes public resources, dynamic resources, and the association relationship between dynamic resources. The dynamic resource manager receives the input external static configuration information, parses the dependency relationship of the dynamic resources, and establishes camera pipelines corresponding to multiple shooting modes respectively. For example, the dynamic resource manager establishes the dependency relationship between A node and B node and the dependency relationship between C node and D node according to the input external static configuration information. A node and B node correspond to pipeline 1, C node and D node correspond to pipeline 2, and E node corresponds to pipeline 3. Pipeline 1, pipeline 2, and pipeline 3 correspond to a shooting mode respectively. When the shooting mode of the camera application is set to the shooting mode corresponding to pipeline 1, dynamic node resources A node, B node, C node, D node, and E node are loaded, and pipeline 1 is established according to the dependency relationship between dynamic node resources A node and B node. After the camera collects media data, it is input into the dynamic resource manager, and the media data is received through the input adapter, and the media data is input into the corresponding camera pipeline, that is, pipeline 1. After each node in pipeline 1 processes the media data, the processed media data is output through the output adapter, for example, output to the camera application, triggering the drawing and rendering process, and displaying the processed media data in the form of a preview screen. When the shooting mode of the camera application is switched to the shooting mode corresponding to pipeline 2, since the dynamic node resources Cnode and D node have been loaded, pipeline 2 can be directly established according to the dependency relationship between dynamic node resources C node and D node. After the camera collects media data, it is input into the dynamic resource manager, and the media data is received through the input adapter, and the media data is input into the corresponding camera pipeline, that is, pipeline 2. After each node in pipeline 2 processes the media data, the processed media data is output through the output adapter.When the shooting mode of the camera application is switched to the shooting mode corresponding to pipeline 3, since the dynamic node resource E node has been loaded, pipeline 3 can be established directly based on the dynamic node resource E node. After the camera collects the media data, it is input into the dynamic resource manager, the media data is received through the input adapter, and the media data is input into the corresponding camera pipeline, that is, pipeline 3. After the nodes in pipeline 3 process the media data, the processed media data is output through the output adapter.

[0109] See also Fig. 9 FIG. 1 is a flowchart of a camera resource management method provided by another embodiment of the present application. The method is applied in an electronic device, and the camera resource management method includes:

[0110] S201, obtaining node information of a plurality of nodes corresponding to at least two shooting modes respectively.

[0111] S202: Determine common resources and dynamic resources of at least two shooting modes according to node information of multiple nodes.

[0112] S203, determining whether the number of dynamic resources of at least two shooting modes is greater than or equal to a preset number. When the number of dynamic resources of at least two shooting modes is greater than or equal to the preset number, returning to S201; when the number of dynamic resources of at least two shooting modes is less than the preset number, executing S204.

[0113] In another embodiment of the present application, the preset number may be 3, 4, 5 or other numbers. It is understood that when the number of dynamic resources of at least two shooting modes is greater than or equal to the preset number, it means that there are more dynamic resources in at least two shooting modes, that is, the difference in node resources between at least two shooting modes is large, and the number of resources that need to be loaded at the same time is large, which is not suitable for establishing an association relationship between dynamic resources in at least two shooting modes, and resources are not reloaded when the shooting mode is switched. When the number of dynamic resources of at least two shooting modes is less than the preset number, it means that there are fewer dynamic resources in at least two shooting modes, that is, the difference in node resources between at least two shooting modes is small, and the number of resources that need to be loaded at the same time is small, which is suitable for establishing an association relationship between dynamic resources in at least two shooting modes, and resources do not need to be reloaded when the shooting mode is switched.

[0114] S204: Establish an association relationship between at least two dynamic resources in shooting modes.

[0115] S205 : In response to setting the current shooting mode to one of the at least two shooting modes, load common resources and dynamic resources of the at least two shooting modes.

[0116] S206: Execute a shooting operation according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode.

[0117] S207: In response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, a shooting operation is performed according to the public resources, dynamic resources, and the association relationship between the dynamic resources corresponding to the other shooting mode.

[0118] The embodiment of the present application can decompose the camera pipelines corresponding to the shooting modes with relatively close system resource dependencies, and combine the parts that need to be processed differently into a set of dynamic Node sets with associated relationships. The dynamic resource management module can obtain the Node static properties in advance according to the pre-configured static data for pre-processing, and at the same time, it does not affect the effectiveness of the native dependency mechanism. The existing dependencies can be abstracted and converted into Node dependencies through the dynamic resource manager, and the Node resources managed by the dynamic resource management module can be flexibly combined and quickly switched, so as to support the experience of seamless switching between some modes (without restarting the flow), and improve the smoothness of the user's viewing experience.

[0119] The present application also provides an electronic device 100. Fig.10 As shown, the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0120] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0121] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0122] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0123] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0124] The processor 110 may also be provided with a memory for storing instructions and data. In one embodiment of the present application, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use instructions or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0125] In one embodiment of the present application, the processor 110 may include one or more interfaces. The interface may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0126] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (Serial Data Line, SDA) and a serial clock line (Serial Clock Line, SCL). In one embodiment of the present application, the processor 110 may include multiple groups of I2C buses. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 can be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to realize the touch function of the electronic device 100.

[0127] The I2S interface can be used for audio communication. In one embodiment of the present application, the processor 110 may include multiple I2S buses. The processor 110 may be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In one embodiment of the present application, the audio module 170 may transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call via a Bluetooth headset.

[0128] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In one embodiment of the present application, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In one embodiment of the present application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering a call via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0129] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In one embodiment of the present application, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In one embodiment of the present application, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0130] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In one embodiment of the present application, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0131] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In one embodiment of the present application, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0132] The USB interface 130 is an interface that complies with the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices 100, such as AR devices, etc.

[0133] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0134] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.

[0135] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0136] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0137] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0138] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (Low Noise Amplifier, LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In one embodiment of the present application, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In one embodiment of the present application, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0139] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In one embodiment of the present application, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be arranged in the same device as the mobile communication module 150 or other functional modules.

[0140] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0141] In one embodiment of the present application, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a Beidou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and / or a Satellite Based Augmentation System (SBAS).

[0142] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for camera resource management, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0143] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, Microled, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment of the present application, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0144] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0145] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In one embodiment of the present application, ISP can be set in camera 193.

[0146] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment of the present application, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0147] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0148] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0149] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0150] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0151] Random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.;

[0152] Non-volatile memory may include disk storage devices and flash memory.

[0153] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle. It can be divided into single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level. It can be divided into universal flash storage (UFS), embedded Multi Media Card (eMMC), etc. according to the storage specification.

[0154] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0155] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0156] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0157] The internal memory 121 or the external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 110, the screen display detection method executed on the electronic device 100 in the above embodiment can be implemented to implement the screen display detection function of the electronic device 100.

[0158] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0159] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In one embodiment of the present application, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0160] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0161] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0162] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0163] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0164] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0165] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0166] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0167] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In one embodiment of the present application, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. The embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on the electronic device 100, the electronic device 100 executes the above-mentioned related method steps to implement the camera resource management method in the above-mentioned embodiment.

[0168] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the camera resource management method in the above-mentioned embodiment.

[0169] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the camera resource management method in the above-mentioned method embodiments.

[0170] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0171] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program code.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A camera resource management method, characterized in that: Applied to electronic equipment, the method comprises: Acquire node information of a plurality of nodes corresponding to at least two shooting modes respectively, wherein the node information includes node attributes and basic connection relationships, wherein the node attributes include node names, and the basic connection relationships include input ports and output ports of each node; Determining the common resources and dynamic resources of the at least two shooting modes according to the node information of the multiple nodes, including: comparing the node information of the multiple nodes corresponding to the at least two shooting modes, taking nodes with the same node name and the same basic connection relationship as the common resources, and taking nodes with different node names and nodes with the same node name but different basic connection relationships as the dynamic resources; Establishing an association relationship between the dynamic resources of the at least two shooting modes, including: setting an input port and an output port of each node in the dynamic resource in the at least two shooting modes through a dynamic resource manager; In response to setting the current shooting mode to one of the at least two shooting modes, loading common resources and dynamic resources of the at least two shooting modes; Execute a shooting operation according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode; In response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, the shooting operation is performed according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the other shooting mode.

2. The camera resource management method according to claim 1, characterized in that: Each of the at least two shooting modes corresponds to a camera pipeline, and the camera pipeline includes multiple nodes and dependencies between the multiple nodes, which are used to implement the shooting function corresponding to each shooting mode.

3. The camera resource management method according to claim 1, characterized in that: The node information also includes necessary information, and the necessary information includes a necessary identification of each node in the current shooting mode. The node attributes also include node parameters, node status, and processing functions.

4. The camera resource management method according to claim 3, characterized in that: Each node includes a configuration file, and the acquiring node information of a plurality of nodes corresponding to at least two shooting modes respectively includes: The node information of each node is obtained from the configuration file of each node through the dynamic resource manager.

5. The camera resource management method according to claim 1, characterized in that: The association relationship between the dynamic resources includes the dependency relationship between the nodes in the dynamic resources in the at least two shooting modes.

6. The camera resource management method according to claim 5, characterized in that: The establishing of the association relationship between the dynamic resources of the at least two shooting modes further includes: The association relationship between the dynamic resources is written into the configuration file of each node in the dynamic resources.

7. The camera resource management method according to claim 1, characterized in that: In response to setting the current shooting mode to one of the at least two shooting modes, loading common resources and dynamic resources of the at least two shooting modes includes: In response to a user triggering an operation of the control or in response to a user performing a sliding operation on the camera application interface, controlling the camera application to display a preview interface corresponding to the control, and determining the current shooting mode; If the current shooting mode is one of the at least two shooting modes, load common resources and dynamic resources of the at least two shooting modes; or If the current shooting mode is not any of the at least two shooting modes, load resources corresponding to the current shooting mode.

8. The camera resource management method according to claim 1, characterized in that: The performing the shooting operation according to the public resources, dynamic resources and the association relationship between the dynamic resources corresponding to the current shooting mode includes: Establishing a camera pipeline corresponding to the current shooting mode according to the public resources corresponding to the current shooting mode, the dynamic resources and the association relationship between the dynamic resources; The image data captured by the camera is processed according to the camera pipeline corresponding to the current shooting mode.

9. The camera resource management method according to claim 1, characterized in that: In response to the current shooting mode being switched to another shooting mode of the at least two shooting modes, performing the shooting operation according to the public resources, dynamic resources, and the association relationship between the dynamic resources corresponding to the another shooting mode, includes: In response to a user triggering an operation of the control or in response to a user controlling a camera application to display a preview interface corresponding to the control by performing a sliding operation, determining a switched shooting mode; If the switched shooting mode is another shooting mode among the at least two shooting modes, establishing a camera pipeline corresponding to the current shooting mode according to the public resources, dynamic resources and associations between the dynamic resources corresponding to the other shooting mode; The image data captured by the camera is processed according to the camera pipeline corresponding to the other shooting mode.

10. The camera resource management method according to claim 9, characterized in that: The method further comprises: If the switched shooting mode is not another shooting mode of the at least two shooting modes, resources corresponding to the switched shooting mode are loaded.

11. The camera resource management method according to claim 1, characterized in that: The method further comprises: Determining whether the number of dynamic resources of the at least two shooting modes is greater than or equal to a preset number; When the number of the dynamic resources of the at least two shooting modes is less than a preset number, an association relationship between the dynamic resources of the at least two shooting modes is established.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor: Wherein, the memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device executes the camera resource management method according to any one of claims 1 to 11.

13. A chip coupled to a memory in an electronic device, characterized in that: The chip is used to control the electronic device to execute the camera resource management method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that: The computer storage medium stores program instructions, and when the program instructions are executed on an electronic device, a processor of the electronic device executes the camera resource management method according to any one of claims 1 to 11.

Images