Camera algorithm configuration method and device

Abstract

The present application provides a camera algorithm configuration method and device, which can dynamically enable the algorithm according to the current shooting scene of the camera application (such as shooting mode and shooting parameters), that is, based on the shooting scene, a camera algorithm that matches the shooting scene is decided. Only necessary camera algorithms need to be loaded, and there is no need to load all camera algorithms, thereby shortening the time consumed when the electronic device is started, and also reducing the power consumption during the startup phase.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a camera algorithm configuration method and device. Background Art

[0002] In order to provide users with better shooting functions and shooting experience, digital cameras, mobile phones and other electronic devices have launched a variety of shooting modes adapted to specific shooting scenes, such as portrait mode, night scene mode, panoramic mode, video mode, large aperture mode, etc. Furthermore, users can also set different shooting parameters, such as photo ratio, video resolution, video frame rate, and enable beauty function in portrait shooting or video mode.

[0003] Different shooting modes may require different algorithms. For example, the algorithms used in the normal shooting mode and the portrait shooting mode are different. In the normal shooting mode, the tracking algorithm is required to optimize the focus effect, but the portrait mode does not require the tracking effect, but the beauty algorithm is required to meet the user's beauty needs. However, the current algorithm configuration method is to uniformly load all algorithms for camera applications during the system initialization phase, but most algorithms are usually not running in all scenarios. The process of loading all algorithms takes a certain amount of time and produces certain functions. Therefore, loading all the algorithms corresponding to the camera during the initialization phase reduces the system startup performance and increases power consumption. Summary of the invention

[0004] In view of this, the present application provides a camera algorithm configuration method and device to solve at least part of the above problems, and the disclosed technical solution is as follows:

[0005] In the first aspect, the present application provides a camera algorithm configuration method, which is applied to electronic devices, and the method includes: obtaining the current shooting scene of the camera application; determining a first target algorithm configuration file that matches the current shooting scene based on a pre-configured algorithm configuration file, the first target algorithm configuration file including the names and execution order of each camera algorithm required for the current shooting scene; constructing corresponding algorithm nodes based on the names of the camera algorithms in the first target algorithm configuration file, and establishing the connection relationship between each algorithm node according to the execution order of each camera algorithm, and obtaining a first algorithm operation topology map; running the first algorithm operation topology map. In this way, the necessary algorithms can be dynamically enabled based on the current shooting scene of the camera application, that is, for each shooting scene, only the algorithms required for the scene are configured and loaded, without loading all algorithms, thereby shortening the duration of the startup phase, while also reducing the power consumption of the startup phase and improving the startup performance.

[0006] In a possible implementation of the first aspect, the first target algorithm configuration file includes the operation frequency information of the first algorithm operation topology map, and the operation frequency information includes the operation frequency and the operation frame number of the first algorithm operation topology map; running the first algorithm operation topology map includes: running the algorithm operation topology map once every first preset value of image frames set by the operation frequency, and continuously running the second preset value of image frames set by the operation frame number each time. In this way, the operation frequency of the algorithm operation topology map can be independently controlled, the power consumption of the algorithm operation process can be reduced, and the operation performance of the electronic device can be improved.

[0007] In a possible implementation of the first aspect, the first target algorithm configuration file includes the operating frequency information of a single camera algorithm, and the operating frequency information includes the operating frequency and the number of operating frames of the single camera algorithm; running the first algorithm operation topology map includes: running a single camera algorithm once at intervals of a third preset value of image frames set by the operating frequency, and continuously running the fourth preset value of image frames set by the number of operating frames each time. In this way, the operating frequency can be configured for a single algorithm in the algorithm operation topology map, and the operating frequency can be configured for a larger algorithm, thereby reducing the algorithm operation power consumption and improving the operation performance.

[0008] In a possible implementation of the first aspect, determining a first target algorithm profile that matches the current shooting scene based on a preconfigured algorithm profile includes: searching for a first algorithm profile set that matches the current shooting mode in the current shooting scene, the first algorithm profile set including at least one algorithm profile; reading conditional expressions of each algorithm profile in the algorithm profile set; determining a target conditional expression that matches the current shooting scene from each conditional expression, and determining that the algorithm profile corresponding to the target conditional expression is the first target algorithm profile.

[0009] In a possible implementation manner of the first aspect, a corresponding algorithm node is constructed based on the name of a camera algorithm in a first target algorithm configuration file, including: parsing the first target algorithm configuration file to obtain first algorithm configuration information, where the first algorithm configuration information includes the names of each camera algorithm required for a current shooting scene; for the name of any camera algorithm included in the first algorithm configuration information, calling a builder that matches the name of any camera algorithm to construct an algorithm instance corresponding to any camera algorithm.

[0010] In a possible implementation of the first aspect, after parsing the first target algorithm configuration file to obtain the first algorithm configuration information, the method further includes: storing the first algorithm configuration information and a mapping relationship between the first algorithm configuration information and the first target algorithm configuration file. In this way, the next time the shooting scene is switched to, the corresponding algorithm configuration information can be directly read without parsing the target algorithm configuration file, which saves the time spent on parsing the configuration file and improves the algorithm configuration and operation efficiency.

[0011] In a possible implementation of the first aspect, the method further includes: during an electronic device initialization phase, obtaining all algorithm configuration files corresponding to the camera application, parsing all the algorithm configuration files, and obtaining algorithm configuration information corresponding to each algorithm configuration file; constructing a corresponding algorithm node based on the name of the camera algorithm in the first target algorithm configuration file, including: searching for first algorithm configuration information that matches the first target algorithm configuration file, the first algorithm configuration information including the names of each camera algorithm required for the current shooting scene; for the name of any camera algorithm contained in the first algorithm configuration information, calling a builder that matches the name of any camera algorithm to construct an algorithm instance corresponding to any camera algorithm.

[0012] In a possible implementation manner of the first aspect, the electronic device includes a database for storing shooting scene information of a camera application; obtaining a current shooting scene of the camera application includes: reading the current shooting scene of the camera application from the database when it is detected that the camera application is started.

[0013] In a possible implementation of the first aspect, the method further includes: after detecting that the camera application adjusts the shooting scene information, obtaining the adjusted shooting scene information; determining a second target algorithm configuration file that matches the adjusted shooting scene information based on a pre-configured algorithm configuration file, the second target algorithm configuration file including the names and execution order of each camera algorithm required for the adjusted shooting scene; constructing corresponding algorithm nodes based on the names of the camera algorithms in the second target algorithm configuration file, and establishing a connection relationship between each algorithm node according to the execution order of each camera algorithm, obtaining a second algorithm operation topology map; and running the second algorithm operation topology map. In this way, when the user switches the camera's shooting scene, the algorithm operation map corresponding to the new shooting scene can also be dynamically configured. For each shooting scene, only the algorithm required for the scene is configured and loaded, without loading all algorithms, thereby shortening the duration of the startup phase, while also reducing the power consumption of the startup phase and improving the startup performance.

[0014] In the second aspect, the present application also provides a camera algorithm configuration method, which is applied to electronic devices, and the method includes: obtaining the current shooting scene of the camera application; after determining that the current shooting scene is a multi-stream scene, determining a first target algorithm configuration file that matches the shooting scene of the main stream from a pre-configured mainstream algorithm configuration file; constructing corresponding algorithm nodes based on the name of the camera algorithm in the first target algorithm configuration file, and establishing a connection relationship between each algorithm node according to the execution order of each camera algorithm, and obtaining a first algorithm operation topology map; for the main stream data, running the first algorithm operation topology map; after determining that the current shooting scene is a multi-stream scene, determining a second target algorithm configuration file that matches the shooting scene of the auxiliary stream from a pre-configured auxiliary stream algorithm configuration file; constructing corresponding algorithm nodes based on the name of the camera algorithm in the second target algorithm configuration file, and establishing a connection relationship between each algorithm node according to the execution order of each camera algorithm, and obtaining a second algorithm operation topology map; for the auxiliary stream data, running the second algorithm operation topology map. In this way, for the main / auxiliary streams, only the algorithms necessary for the image processing process need to be loaded separately, thereby reducing the power consumption of the camera application and improving the operation performance.

[0015] In a third aspect, the present application also provides an electronic device, comprising: one or more processors, a memory and a touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device implements a camera algorithm configuration method as described in any one of the first aspects.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium having instructions stored thereon, which, when the instructions are executed on an electronic device, enables the electronic device to execute a camera algorithm configuration method as described in any one of the first aspects.

[0017] In a fifth aspect, the present application also provides a chip system, comprising: at least one processor and an interface, the interface being used to receive code instructions and transmit them to at least one processor; at least one processor runs the code instructions to implement any camera algorithm configuration method of the first aspect.

[0018] In a sixth aspect, the present application further provides a computer program product having an executable program stored thereon. When the computer program product is run on an electronic device, the electronic device implements a camera algorithm configuration method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of an interface of a camera application provided in an embodiment of the present application;

[0020] Figure 2 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 3 It is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a camera algorithm configuration process provided by an embodiment of the present application;

[0023] Figure 5 is a flowchart of a camera algorithm configuration method provided in an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of another camera algorithm configuration process provided by an embodiment of the present application;

[0025] Figure 7 It is a flowchart of another camera algorithm configuration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0027] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0028] See also Figure 1 , showing a schematic diagram of the interface of the camera application.

[0029] like Figure 1 The interface shown in (1) is the interface 10 of the camera application, which includes a setting bar 11 at the top, a preview window 14 with the largest area located below the setting bar, a menu bar 17 located below the preview window 14, and a button area located below the menu bar, which may include, for example, a capture button 19, a review button 18, and a camera reverse button 20.

[0030] The setting bar 11 may include a plurality of setting buttons for adjusting shooting parameters, such as a flash button 13 and a setting button 12. In addition, it may also include a filter button, an auto-follow button, etc. The present application does not specifically limit the button types included in the setting bar.

[0031] The preview window 14 is used to display the image captured by the camera in real time. In the embodiment of the present application, the bottom of the preview window 14 also includes a zoom multiple selection button 16 and a beauty button 15.

[0032] The menu bar 17 may include multiple shooting mode options, such as "portrait", "photo", "video", "night scene", "panorama", "movie" and other options. For example, interface 10 is the main interface of the video mode. The number of options that can be displayed in the menu bar 17 is limited, and some mode options cannot be displayed. You can slide the menu bar left and right to display other mode options.

[0033] The user can select and switch to other modes in the menu bar 17. Different shooting modes may require different algorithms. The algorithm configuration method provided in this application can load the algorithms required by different shooting modes.

[0034] In addition, the user clicks the setting button 12 to display the following Figure 1 The interface 21 shown in (2) includes shooting parameters when taking photos or recording videos, such as photo ratio 22, video resolution 23, video frame rate 24, voice-controlled photo taking, gesture photo taking, smile capture and other setting items. For example, the photo ratio may include 4:3, 1:1 or full screen, and the video resolution may include [16:9]4k, [16:9]1080p, [full screen]1080p, [21:9]1080p, [21:9]720p, [16:9]720p, etc. The video frame rate may include automatic, 30fps, 60fps, etc.

[0035] The camera algorithm configuration method provided in this application can dynamically enable the algorithm according to the current shooting mode and shooting parameters of the camera application, that is, based on the shooting scene (such as shooting mode and shooting parameters), decide the algorithm that matches the shooting scene, and only need to load the matching algorithm, without loading all camera algorithms, thereby shortening the startup time of the electronic device and reducing the power consumption of the startup phase. For example, in the front camera recording mode, usually only the face detection algorithm is needed, and other detection algorithms are not needed, such as cat and dog detection algorithms, sunrise / sunset detection algorithms, etc. This method can be used to load only the face detection algorithm in the front camera recording scene without loading other detection algorithms.

[0036] See also Figure 2 , shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device may include a processor 201 , a memory 202 , a camera 203 , and a display screen 204 .

[0037] The processor 201 may include one or more processing units. Different processing units may be independent devices or integrated into one or more processors.

[0038] The memory 202 can be used to store computer executable program codes, which include instructions. The processor 201 executes various functional applications and data processing of the terminal device by running the instructions stored in the memory. For example, in this embodiment, the processor can perform an image blur processing method by executing the instructions in the memory.

[0039] The camera 203 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras, where N is a positive integer greater than 1.

[0040] The display screen 204 is used to display images, videos, etc., and can also display a series of graphical user interfaces (GUIs), which are all the main screens of the electronic device. In the embodiment of the present application, the display screen can be a touch screen. In addition, the electronic device may include 1 or N display screens.

[0041] In some embodiments, the electronic device is a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, a digital camera, and the like.

[0042] In addition, on top of the above components, an operating system also runs in the electronic device, for example, operating system, operating system, Operating system, etc. Application programs can be installed and run on the operating system.

[0043] The operating system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces.

[0044] See also Figure 3 , shows a software structure block diagram of an electronic device provided in an embodiment of the present application.

[0045] In some embodiments, Figure 3 As shown, the Android system may include an application layer, an application framework layer, a hardware abstraction layer, and a kernel layer arranged in sequence from top to bottom.

[0046] The application layer can include a series of application packages. Figure 3As shown, the application package may include applications such as camera, gallery, etc.

[0047] The application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. For example, the application framework layer in the embodiment of the present application may include a camera service. The camera service provides an API and a programming framework for the camera application of the application layer.

[0048] The hardware abstraction layer is an interface layer between the application framework layer and the kernel layer, and provides a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can provide one or more virtual camera devices, which are virtual hardware of the camera. The operating system calls the corresponding camera through the virtual hardware of the camera.

[0049] In the embodiment of the present application, the hardware abstraction layer may include modules such as the camera HAL and the camera algorithm library, and the camera HAL includes camera devices 1 to N, a perception engine, etc. The perception engine may include a perception engine manager and an algorithm operation graph manager, wherein the algorithm operation graph manager may include modules such as an algorithm graph matching module (AlgoManager), an algorithm node management module (NodeManager), and an algorithm operation graph assembly module (Graph).

[0050] The perception engine manager is used to manage perception detection algorithms in electronic devices, such as face detection algorithms, cat / dog detection algorithms, human body detection algorithms, sunrise / sunset detection algorithms, etc.

[0051] The algorithm matching module (AlgoManager) is used to determine the matching algorithm configuration file (e.g., XML format file) based on the current shooting mode (e.g., photo mode) and shooting parameters (screen ratio, resolution, etc.), and to parse the algorithm configuration file to obtain the corresponding algorithm configuration information.

[0052] The algorithm node management module (NodeManager) is used to manage the builders (Builder) of all current algorithms. The Builder is used to create algorithm instances. When the required algorithm is determined, the corresponding builder of the algorithm is called to build a specific algorithm instance.

[0053] The algorithm operation graph assembly module (Graph) is used to create algorithm nodes according to the algorithm configuration information obtained by AlgoManager decision and analysis, and establish connection relationships between nodes according to the execution order between algorithms to form an algorithm operation topology graph.

[0054] The camera algorithm library includes all the algorithm modules required for camera applications, such as face detection algorithm, cat / dog detection algorithm, human detection algorithm, sunrise / sunset detection algorithm, focus tracking algorithm, beauty algorithm, etc.

[0055] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, etc.

[0056] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0057] See also Figure 4 , which shows a schematic diagram of a camera algorithm configuration process provided by an embodiment of the present application. Figure 4 As shown, the camera algorithm configuration process includes:

[0058] After the camera application is started, AlgoManager obtains the current shooting mode (such as portrait, photo, video, night scene, panorama) and shooting parameters (such as photo ratio, video resolution, video frame rate, etc.) of the camera application. Then, it searches for the algorithm configuration file that matches the current shooting mode, further obtains the conditional expression in the algorithm configuration file, and calls the expression parser (ExpressionParser) to parse the value of the conditional expression based on the shooting parameters to determine whether it is "correct (true)" or "wrong (false)" (that is, compare whether the shooting parameters are consistent with the conditional parameters contained in the conditional expression). If they are consistent, it indicates that the algorithm configuration file corresponding to the conditional expression is equal to the current shooting mode and shooting parameters.

[0059] After AlgoManager selects an algorithm configuration file, it parses the configuration file to obtain algorithm configuration information, which includes the names of the algorithms contained in the algorithm map, the execution order of the algorithms, and other configuration parameters, such as the running frequency and the number of running frames.

[0060] Furthermore, AlgoManager provides the parsed algorithm configuration information to Graph.

[0061] Graph calls the corresponding algorithm builder in NodeManager according to the parsed algorithm name to build a specific algorithm instance. For example, if the selected algorithm graph includes the beauty algorithm, Graph calls the builder corresponding to the beauty algorithm in NodeManager to build the beauty algorithm instance, that is, the beauty algorithm Node. Furthermore, Graph connects the created algorithm Nodes according to the execution order of each algorithm to form an algorithm operation topology graph and outputs it, so that the camera application can run each algorithm according to the execution order in the algorithm operation topology graph.

[0062] See also Figure 5 , shows a flowchart of a method for configuring a camera algorithm provided in an embodiment of the present application, the method may include:

[0063] S101, the perception engine manager creates an algorithm operation diagram manager.

[0064] The operating system of the electronic device creates a perception engine manager during the startup phase, and then the perception engine manager creates an algorithm run chart manager.

[0065] like Figure 4 As shown, the algorithm operation graph manager mainly includes an algorithm graph matching module (AlgoManager), an algorithm node management module (NodeManager) and an algorithm operation graph assembly module (Graph).

[0066] S102, AlgoManager obtains all algorithm configuration files of the camera application and parses them to obtain corresponding algorithm configuration information.

[0067] In an exemplary embodiment, after the algorithm run chart manager is created, the AlgoManager may read all algorithm configuration files of the camera application from a specified storage location.

[0068] Different shooting modes may require different algorithms, so program developers configure corresponding algorithm configuration files for each shooting mode in the camera application.

[0069] In addition, when a shooting mode includes multiple different shooting parameters, a corresponding algorithm configuration file is further configured for each shooting parameter in the same shooting mode. In other words, a shooting mode may include multiple different algorithm configuration files. For example, different video resolutions such as [16:9]4k, [16:9]720p can be set in the recording mode, or different video frame rates such as 30fps, 60fps, etc. can be set. Then, four algorithm configuration files corresponding to the four groups of shooting parameters [16:9]4k, 60fps; [16:9]720p, 60fps; [16:9]4k, 30fps; [16:9]720p, 60fps can be configured for the recording mode.

[0070] In an exemplary embodiment, the algorithm configuration file may be a file in XML format. AlgoManager is required to parse each algorithm configuration file into algorithm configuration information that can be recognized by the electronic device, such as the algorithm configuration information may include the algorithm name contained in the algorithm map, the execution order between the algorithms, and the running frequency and running frame number of the algorithm map.

[0071] The algorithm map corresponding to a scene mode usually includes multiple processing algorithms, and the multiple processing algorithms have a certain execution order. For example, a portrait scene requires a face detection algorithm, a beauty algorithm, a blur algorithm, etc., and the execution order of these algorithms is face detection algorithm → blur algorithm → beauty algorithm.

[0072] The running frequency refers to the interval between how many frames the algorithm in the algorithm map is run. The number of running frames refers to how many frames the algorithm map is run continuously. For example, if the algorithm running frequency is not set in a normal 60fps video scene, the algorithm in the corresponding algorithm map will run 60 frames per second, that is, the corresponding algorithm will run for every frame. In fact, it is not necessary for all algorithms to run at this frequency. This solution can configure a single algorithm Node or the entire algorithm map to run according to the set running frequency (how many frames it runs once and how many frames it runs continuously after starting to run). For example, a 60FPS scene can be configured to run the algorithm every other frame, running 1 frame each time, that is, running the algorithm 30 times per second, and running the algorithm 1 frame each time.

[0073] In some embodiments, some algorithms in the algorithm map may be configured to run at a lower frequency. For example, in a scenario where the video is recorded at 60fps, some algorithms may be configured to run at alternate frames.

[0074] In other embodiments, the operating frequency of the entire algorithm map can be configured, such as the operating frequency of the algorithm map corresponding to the 60fps video recording scene is configured to run every other frame, that is, all algorithms in the algorithm map run at the configured operating frequency. This reduces the power consumption of the algorithm running process and improves the performance of the electronic device.

[0075] It is understandable that when the video frame rate is low, the operating frequency may not be configured. For example, when the video frame rate is 30fps, the operating frequency may not be set, that is, the corresponding algorithm is run for each frame.

[0076] S103, AlgoManager obtains the current shooting scene of the camera application from the database.

[0077] A database is set up in the perception engine to store relevant information of the camera application, for example, information about the shooting scene of the camera application, such as shooting mode and shooting parameters, which is passed down from the application layer in sequence. AlgoManager can read the current shooting mode and shooting parameters of the camera application from the database, and decide on a matching algorithm map based on the current shooting mode and shooting parameters.

[0078] S104, AlgoManager searches for a matching first target algorithm configuration file based on the current shooting scene.

[0079] In an exemplary embodiment, an algorithm configuration file set matching the current shooting mode is first searched, and then a matching first target algorithm configuration file is searched from the algorithm configuration file set based on the current shooting parameters.

[0080] As mentioned before, different algorithm maps can be configured for different shooting parameters under the same shooting mode. For example, different frame rates and different video resolutions can be set in the normal recording mode. In this scenario, different algorithm maps can be configured for different frame rates, and different algorithm maps can be configured for different video resolutions.

[0081] Different algorithm maps correspond to different algorithm configuration files. In other words, different shooting parameters under the same shooting mode correspond to different algorithm configuration files, that is, one shooting mode may include multiple algorithm configuration files.

[0082] In an exemplary embodiment, the algorithm configuration file may be named based on the shooting mode. For example, the name of the algorithm configuration file corresponding to the portrait mode includes “portrait”. In this way, a matching algorithm configuration file set may be found directly according to the name of the shooting mode.

[0083] In order to distinguish the shooting parameters applicable to different algorithm profiles, each algorithm profile includes a conditional expression, which includes the conditions for the algorithm profile to be adapted. For example, in a normal video recording scenario, there are algorithm profiles corresponding to a frame rate of 60fps and an algorithm profile corresponding to a frame rate of 30fps. In this case, the camera application searches for two algorithm profiles corresponding to the mode based on the video recording mode in a 60fps video recording scenario, and further selects the final algorithm profile based on the frame rate of 60fps set in the video recording mode.

[0084] Exemplarily, the conditional expression in the algorithm configuration file may be "frame rate = 60fps". If the video frame rate of the camera application = 60fps, it is determined that the current shooting parameters of the camera application match the conditional expression, and the algorithm configuration file is determined to be the target algorithm configuration file that matches the scene.

[0085] S105, AlgoManager parses the first target algorithm configuration file to obtain first algorithm configuration information.

[0086] The first algorithm configuration information here refers to the algorithm configuration information obtained by parsing the first target algorithm configuration file.

[0087] It should be understood that in the scenario where all algorithm configuration files have been parsed in the method steps shown in S102, there is no need to execute S105. In other words, in this scenario, after the algorithm operation diagram manager is created, all algorithm configuration files corresponding to the camera application are read and parsed to obtain the corresponding algorithm configuration information, that is, all configuration files are parsed at one time. In this way, the algorithm configuration information can be used directly later without having to wait until it is used to parse it, which shortens the time consumption of the algorithm configuration process and improves the efficiency of the algorithm configuration.

[0088] In another scenario, the algorithm operation chart manager does not parse the algorithm configuration file after it is created. Instead, after determining the target algorithm configuration file based on the current shooting mode and shooting parameters of the camera application, only the target algorithm configuration file is parsed, that is, only the algorithm configuration file currently needed is parsed. In this way, it is not necessary to parse all the algorithm configuration files at once, which shortens the time consumption of the camera application startup phase. That is, this scenario does not execute the method steps shown in S102, and the method steps shown in S105 need to be executed at this time.

[0089] Furthermore, the parsed algorithm configuration file can be stored in the memory, so that the configuration information corresponding to the algorithm configuration file can be directly obtained next time without repeated parsing.

[0090] S106, AlgoManager transmits the first algorithm configuration information to the algorithm operation graph assembly module (Graph).

[0091] AlgoManager provides the first algorithm configuration information of the first target algorithm configuration file obtained by parsing to Graph, so that Graph calls the algorithm builder in NodeManager to create an algorithm node (ie, algorithm instance) according to the algorithm configuration information.

[0092] S107, Graph calls the corresponding algorithm builder in NodeManager to create a corresponding algorithm node according to each algorithm name in the first algorithm configuration information.

[0093] NodeManager is used to manage the builders of all algorithms. Each algorithm corresponds to a Builder. These builders are built and registered to NodeManager during the initialization phase of camera startup. Builder is used to create actual algorithm instances (i.e., algorithm nodes).

[0094] According to the names of the algorithms included in the first algorithm configuration information, Graph calls the Builder corresponding to the algorithm in NodeManager to build the corresponding algorithm instance. For example, if the algorithm configuration information includes the beauty algorithm, the Builder corresponding to the beauty algorithm is called to read the beauty algorithm from the camera algorithm library to build the beauty algorithm instance, and configure the running frequency and running frame number corresponding to the algorithm to the algorithm instance. For example, run once every other frame and run 1 frame each time, and configure the running parameters of the algorithm instance according to the configuration information.

[0095] It should be understood that the operating parameters (i.e., operating frequency, number of operating frames) of a single algorithm in the algorithm map can be configured, and the operating parameters of the entire algorithm map can also be configured.

[0096] S108, Graph establishes a connection relationship between the created algorithm nodes according to the execution order in the first algorithm configuration information to obtain a first algorithm operation topology graph.

[0097] The algorithm operation diagram assembly module obtains the execution order of each algorithm in the first algorithm configuration information, and then establishes the connection relationship between each algorithm node according to the execution order to form an algorithm operation topology diagram. That is, the algorithm operation topology diagram includes the algorithm nodes and the operation order between the algorithm nodes.

[0098] In an exemplary embodiment, the next algorithm node to be run is configured in the algorithm node, and when there is no next node, it can be configured as empty. For example, an algorithm running topology diagram includes algorithm node A→algorithm node B→algorithm node C. The next node configured in node A is node B. Similarly, the next node configured in node B is node C, and the next node configured in node C is empty. In this way, after the first node in the topology diagram is triggered to run, the corresponding algorithm can be automatically executed in the order of the algorithm nodes in the topology diagram.

[0099] S109, triggering the execution of the first algorithm to execute the topology graph.

[0100] After creating the algorithm running topology map, run the algorithm running topology map to process the image data, and execute the corresponding algorithm according to the configured running frequency and running frame number.

[0101] When the electronic device detects that the user switches the shooting mode or modifies the shooting parameters, the algorithm to be loaded can be dynamically adjusted according to the adjusted shooting mode or shooting parameters. The process of dynamically adjusting the loaded algorithm will be described below in conjunction with S100 to S116:

[0102] S110, AlgoManager obtains the adjusted shooting mode or shooting parameters corresponding to the camera application from the database.

[0103] For example, a user Figure 1 In the interface shown in (1), click on the "Portrait" mode, and the camera application switches from the video mode to the portrait mode. The switched shooting mode or shooting parameters will be transmitted to the perception engine and stored in the database, and the AlgoManager will be triggered to read the latest shooting scene from the database.

[0104] For example, users Figure 1 Click the Settings button 11 in the interface shown in (1) to display Figure 1 After the interface shown in (2) is displayed, the original video frame rate of the recording mode is adjusted from 30fps to 60fps. The perception engine receives the information that the video frame rate is changed to 60fps from the camera application and stores it in the database. AlgoManager can read the latest video frame rate from the database.

[0105] S111 , searching for a matching second target algorithm configuration file based on the adjusted shooting mode or the adjusted shooting parameters.

[0106] In one scenario, the user switches the shooting mode, such as switching from video mode to portrait (photographing) mode. In this scenario, the algorithm configuration file containing the name of the adjusted shooting mode is searched for. For example, in this example, after switching from video mode to portrait mode, the algorithm configuration file containing "portrait" is searched, and then the shooting parameters in portrait mode (such as picture ratio, etc.) are further searched for a target algorithm configuration file that matches the shooting parameters in portrait mode. In order to distinguish it from the target algorithm configuration file in S104, the target algorithm configuration file here may be referred to as a second target algorithm configuration file.

[0107] In another scenario, the user does not switch the shooting mode, but modifies the shooting parameters in the current shooting mode, for example, in the video recording mode, the video frame rate is changed from 30fps to 60fps. In this scenario, in the algorithm configuration file set corresponding to the video recording mode, the algorithm configuration file that matches the modified shooting parameters is searched, that is, the second target algorithm configuration file.

[0108] S112, AlgoManager parses the second target algorithm configuration file to obtain second algorithm configuration information.

[0109] The implementation process of S112 in this embodiment is the same as the implementation process of S105, which will not be repeated here. As with S105, in the scenario where S102 has parsed all algorithm configuration files, it is not necessary to execute S112 to parse the second target algorithm configuration file again. In the case where S102 is not executed, S112 will be executed to parse the second target algorithm configuration file.

[0110] S113, AlgoManager transmits the second algorithm configuration information to the algorithm operation graph assembly module (Graph).

[0111] S114, Graph calls the algorithm builder in NodeManager to create corresponding algorithm nodes according to the algorithm names in the second algorithm configuration information.

[0112] S115, Graph assembles the created algorithm nodes according to the execution order in the second algorithm configuration information to obtain a second algorithm operation topology graph.

[0113] S116, triggering the execution of the second algorithm to execute the topology graph.

[0114] The implementation process of S113 to S116 in this embodiment is completely the same as the implementation process of S106 to S109, and will not be repeated here.

[0115] The camera algorithm configuration method provided in this embodiment can dynamically enable the necessary algorithms based on the current shooting scene of the camera application, that is, for each shooting scene, only the algorithm required for the scene is configured and loaded, without loading all algorithms, thereby shortening the duration of the startup phase, while also reducing the power consumption of the startup phase and improving the startup performance.

[0116] In a multi-stream scenario (i.e., using different cameras to capture at least two images at the same time), for example, in a portrait scenario, the wide-angle camera and the main camera can be used to capture two image streams respectively. The image stream captured by the wide-angle camera is called the auxiliary stream, and the image stream captured by the main camera is called the main stream. Usually, the auxiliary stream needs to run various detection algorithms, such as face detection algorithms, smile detection, etc. The main stream needs to run blur algorithms, beauty algorithms, etc. It can be seen that the algorithms required for the auxiliary stream and the main stream are not exactly the same, and the auxiliary stream needs to run fewer algorithms, and there is no need to load a large number of algorithms like the main stream. The camera algorithm configuration method provided in this application supports configuring an algorithm operation diagram that matches the current scene for each image stream.

[0117] The following takes a dual-stream scenario as an example to illustrate the camera algorithm configuration process in a multi-stream scenario.

[0118] See also Figure 6 , showing a schematic diagram of another camera algorithm configuration process provided in an embodiment of the present application.

[0119] During the initialization phase of the camera application during the startup of the operating system of the electronic device, an algorithm operation diagram manager instance 1 is created, which is the system default algorithm operation diagram manager. After detecting that the camera application is in a multi-stream scenario (such as portrait blur photo / video, dual-view video, etc.), another algorithm operation diagram manager instance 2 is created to handle the algorithm configuration process of the auxiliary stream. That is, multiple operation diagram manager instances need to be created in a multi-stream scenario, and each operation diagram manager instance handles the algorithm configuration process of one image stream.

[0120] Algorithm operation diagram manager instance 1 is used to process the algorithm configuration process of the mainstream, and algorithm operation diagram manager instance 2 is used to process the algorithm configuration process of the auxiliary stream.

[0121] The AlgoManager in the algorithm operation diagram manager instance 1 obtains the shooting scene information of the main stream (e.g., the portrait video scene and shooting parameters, such as video resolution and frame rate, etc.), first searches for an algorithm configuration file set that matches the portrait video scene mode, and further calls the expression parser to search for a matching first target algorithm configuration file based on the shooting parameters under the scene. Furthermore, the AlgoManager parses the first target algorithm configuration file to obtain the corresponding algorithm configuration information and provides it to the NodeManager.

[0122] The NodeManager in the algorithm running graph manager instance 1 calls the corresponding algorithm builder according to the algorithm name to build a specific algorithm instance. For example, if the selected algorithm graph includes the beautification algorithm, the NodeManager calls the builder corresponding to the beautification algorithm to build a beautification algorithm instance, that is, the beautification algorithm node.

[0123] The Graph in Algorithm Operation Graph Manager Instance 1 connects the created algorithm nodes according to the execution order of each algorithm to form an algorithm operation topology graph and outputs it, so that the camera application can run each algorithm according to the execution order in the algorithm operation topology graph. For example, the algorithm operation graph corresponding to the mainstream in the portrait video scene includes blur algorithm → beauty algorithm.

[0124] Similarly, the AlgoManager in the algorithm operation diagram manager instance 2 determines a matching second target algorithm configuration file based on the shooting scene information of the auxiliary stream, and parses the configuration file to obtain the second algorithm configuration information.

[0125] The NodeManager in the algorithm operation graph manager instance 2 calls the corresponding algorithm builder according to the algorithm name to build a specific algorithm instance, such as a face detection algorithm instance.

[0126] The Graph in Algorithm Operation Graph Manager Instance 2 connects the created algorithm nodes according to the execution order of each algorithm to form an algorithm operation topology graph and outputs it, so that the camera application can run each algorithm according to the execution order in the algorithm operation topology graph. For example, the algorithm operation topology graph corresponding to the auxiliary stream in the portrait video scene includes face detection algorithm, human body detection algorithm, etc.

[0127] From the above process, it can be seen that the camera algorithm configuration method provided by this application, in a multi-stream scenario, creates algorithm operation diagram manager instances corresponding to the main and auxiliary streams respectively, and executes the algorithm configuration processes corresponding to the main and auxiliary streams respectively, so as to configure different algorithm operation diagrams for the main and auxiliary streams respectively.

[0128] The following will be combined Figure 7 The process of configuring the camera algorithm for dual-stream scenarios is described in detail. Figure 7 As shown, the method may include the following steps:

[0129] S200, during the camera application initialization phase, the perception engine manager creates an algorithm operation graph manager instance 1.

[0130] The electronic device creates an algorithm operation chart manager instance 1 during the camera application initialization phase, and the instance is the system default algorithm operation chart manager.

[0131] S201, after detecting that the camera application is in a multi-stream scenario, the perception engine manager creates an algorithm running graph manager instance 2.

[0132] After detecting that the camera application is in a multi-stream scenario (such as portrait blur photo / video, dual-view video, etc.), the perception engine manager creates another algorithm run chart manager instance 2 to handle the algorithm configuration process of the auxiliary stream.

[0133] S202, the AlgoManager in the algorithm operation diagram manager instance 1 (which may be referred to as the first AlgoManager) obtains all algorithm configuration files of the camera application and parses them to obtain corresponding algorithm configuration information.

[0134] S203, the first AlgoManager obtains dual-stream and main-stream shooting parameters of the camera application from the database.

[0135] In an exemplary embodiment, the image data captured by the main camera carries a data identifier, such as the identifier of the main camera, or a preset ID. Similarly, the auxiliary stream also carries a corresponding data identifier, such as the identifier of the camera that captured the auxiliary stream, or a preset ID. The first AlgoManager finds the specific scene mode and shooting parameters of the mainstream according to the data identifier of the main stream.

[0136] S204: The first AlgoManager searches for a matching first target algorithm configuration file based on the dual-stream scene mode and the shooting parameters corresponding to the primary stream.

[0137] In an exemplary embodiment, the R&D personnel may separately configure the algorithm configuration file corresponding to the main flow and the algorithm configuration file corresponding to the auxiliary flow.

[0138] In some embodiments, the first AlgoManager obtains a specific multi-stream scene, such as a portrait recording scene or a dual-view recording mode, and searches for a matching mainstream algorithm configuration file set according to the name of the multi-stream scene. Further, the target algorithm configuration file is searched from the algorithm configuration file set according to the specific shooting parameters under the scene mode, which can be called the first target algorithm configuration file.

[0139] S205: The first AlgoManager parses the first target algorithm configuration file to obtain first algorithm configuration information.

[0140] S206: The first AlgoManager transmits the first algorithm configuration information to the first Graph.

[0141] S207: The first Graph calls the corresponding algorithm builder in the NodeManager to create a corresponding algorithm node according to each algorithm name in the first algorithm configuration information.

[0142] S208, the first algorithm operation graph assembly module (Graph) assembles the created algorithm nodes according to the execution order in the first algorithm configuration information to obtain a first algorithm operation topology graph.

[0143] S209, triggering the execution of the first algorithm to execute the topology graph.

[0144] The implementation process of S205 to S209 in this embodiment is similar to Figure 4 The implementation process of S105 to S109 in the illustrated embodiment is the same and will not be repeated here.

[0145] S210, the AlgoManager in the algorithm operation diagram manager instance 2 (which may be referred to as the second AlgoManager) obtains all algorithm configuration files and parses them to obtain corresponding algorithm configuration information, including each algorithm, the execution order between algorithms, and other configuration parameters.

[0146] This step is optional. In some embodiments, any one of the first AlgoManager and the second AlgoManager can parse all the algorithm configuration files corresponding to the camera application, without the need for both AlgoManagers to parse all the algorithm configuration files. In addition, in some embodiments, the first AlgoManager and the second AlgoManager do not need to parse all the algorithm configuration files. After deciding on the target algorithm configuration file, the target algorithm configuration file can be parsed separately, that is, only the algorithm configuration file currently needed is parsed. Furthermore, the parsed algorithm configuration file can be stored in the memory, so that the configuration information corresponding to the algorithm configuration file can be directly obtained next time without repeated parsing.

[0147] S211, the second AlgoManager obtains the dual-stream scene information of the camera application from the database.

[0148] Similarly, the second AlgoManager searches the database for shooting scene information corresponding to the auxiliary stream according to the data identifier of the auxiliary stream.

[0149] In some embodiments, the second AlgoManager obtains a specific multi-stream scene, such as a portrait blur scene or a dual-view recording mode, and searches for a matching auxiliary stream algorithm configuration file set according to the name of the multi-stream scene. Further, the target algorithm configuration file is searched from the algorithm configuration file set according to the specific shooting parameters under the scene mode, which can be called a second target algorithm configuration file.

[0150] S212: The second AlgoManager searches for a matching second target algorithm configuration file based on the multi-stream scene mode and the shooting parameters corresponding to the auxiliary stream.

[0151] S213: The second AlgoManager parses the second target algorithm configuration file to obtain second algorithm configuration information.

[0152] S214: The second AlgoManager transmits the first algorithm configuration information to the second Graph.

[0153] S215, the second Graph calls the corresponding algorithm builder in the NodeManager to create a corresponding algorithm node according to each algorithm name in the second algorithm configuration information.

[0154] S216, the second algorithm operation graph assembly module (Graph) assembles the created algorithm nodes according to the execution order in the second algorithm configuration information to obtain a second algorithm operation topology graph.

[0155] S217, triggering the execution of the second algorithm to execute the topology graph.

[0156] Since the implementation process of the above-mentioned S210-S217 is similar to that of S202-S209, and the execution order of each step is the same as the execution order of the same step in S202-S209, that is, S202-S209 and S210-S217 are respectively executed synchronously by different threads, such as Figure 7 shown.

[0157] The camera algorithm configuration method provided in this embodiment configures different algorithm operation diagrams for each image stream for multi-stream scenarios. When the camera operates in a multi-stream scenario, a set of mainstream algorithm configuration files corresponding to the scene is determined based on the shooting scene and shooting parameters corresponding to the main stream, and a target mainstream algorithm configuration file is further selected from the mainstream algorithm configuration file set based on the shooting parameters of the mode. Similarly, a target auxiliary stream algorithm configuration file is determined based on the shooting scene and shooting parameters of the auxiliary stream. It can be seen that using this method, only the necessary algorithms for the image processing process need to be loaded separately for the main / auxiliary streams, thereby reducing the power consumption of the camera application and improving the operating performance.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A camera algorithm configuration method, characterized in that: Applied to electronic equipment, the method comprises: Get the current shooting scene of the camera application; Determine a first target algorithm configuration file matching the current shooting scene based on a pre-configured algorithm configuration file, wherein the first target algorithm configuration file includes the names and execution order of each camera algorithm required for the current shooting scene; Constructing corresponding algorithm nodes based on the names of the camera algorithms in the first target algorithm configuration file, and establishing connection relationships between the algorithm nodes according to the execution order of the camera algorithms, to obtain a first algorithm operation topology map, wherein the first target algorithm configuration file further includes first operation frequency information or second operation frequency information, wherein the first operation frequency information includes the operation frequency and the number of operation frames of the first algorithm operation topology map, and the second operation frequency information includes the operation frequency and the number of operation frames of a single camera algorithm in the first algorithm operation topology map; The first algorithm operation topology map is run once every first preset value of image frames set by the first operation frequency information, and the operation is continuously run for a second preset value of image frames set by the operation frame number each time; or, the single camera algorithm is run once every third preset value of image frames set by the second operation frequency information, and the operation is continuously run for a fourth preset value of image frames set by the operation frame number each time.

2. The method according to claim 1, characterized in that The determining, based on the pre-configured algorithm configuration file, a first target algorithm configuration file matching the current shooting scene comprises: searching for a first algorithm configuration file set that matches a current shooting mode in the current shooting scene, wherein the first algorithm configuration file set includes at least one algorithm configuration file; Reading a conditional expression of each algorithm configuration file in the algorithm configuration file set; A target conditional expression matching the current shooting scene is determined from each conditional expression, and an algorithm configuration file corresponding to the target conditional expression is determined as a first target algorithm configuration file.

3. The method according to claim 1 or 2, characterized in that: The step of constructing a corresponding algorithm node based on the name of the camera algorithm in the first target algorithm configuration file includes: Parsing the first target algorithm configuration file to obtain first algorithm configuration information, where the first algorithm configuration information includes the names of various camera algorithms required for the current shooting scene; For the name of any camera algorithm included in the first algorithm configuration information, a builder matching the name of any camera algorithm is called to build an algorithm instance corresponding to the any camera algorithm.

4. The method according to claim 3, characterized in that After parsing the first target algorithm configuration file to obtain first algorithm configuration information, the method further includes: storing the first algorithm configuration information and a mapping relationship between the first algorithm configuration information and the first target algorithm configuration file.

5. The method according to claim 1, 2 or 4, characterized in that: The method further comprises: During the electronic device initialization phase, all algorithm configuration files corresponding to the camera application are obtained, and all the algorithm configuration files are parsed to obtain algorithm configuration information corresponding to each algorithm configuration file; The step of constructing a corresponding algorithm node based on the name of the camera algorithm in the first target algorithm configuration file includes: Searching for first algorithm configuration information matching the first target algorithm configuration file, where the first algorithm configuration information includes names of various camera algorithms required for a current shooting scene; For the name of any camera algorithm included in the first algorithm configuration information, a builder matching the name of any camera algorithm is called to build an algorithm instance corresponding to the any camera algorithm.

6. The method according to claim 1, 2 or 4, characterized in that: The electronic device includes a database for storing shooting scene information of a camera application; The obtaining of the current shooting scene of the camera application includes: reading the current shooting scene of the camera application from the database after detecting that the camera application is started.

7. The method according to claim 1, 2 or 4, characterized in that: The method further comprises: After detecting that the camera application adjusts the shooting scene information, acquiring the adjusted shooting scene information; Determining a second target algorithm configuration file that matches the adjusted shooting scene information based on a pre-configured algorithm configuration file, wherein the second target algorithm configuration file includes the names and execution order of each camera algorithm required for the adjusted shooting scene; Constructing corresponding algorithm nodes based on the names of the camera algorithms in the second target algorithm configuration file, and establishing connection relationships between the algorithm nodes according to the execution order of the camera algorithms, to obtain a second algorithm operation topology graph; Running the second algorithm runs the topology graph.

8. A camera algorithm configuration method, characterized in that: Applied to electronic equipment, the method comprises: Get the current shooting scene of the camera application; After determining that the current shooting scene is a multi-stream scene, determining a first target algorithm configuration file that matches the shooting scene of the main stream from pre-configured mainstream algorithm configuration files; Constructing corresponding algorithm nodes based on the names of the camera algorithms in the first target algorithm configuration file, and establishing connection relationships between the algorithm nodes according to the execution order of the camera algorithms, to obtain a first algorithm operation topology map, wherein the first target algorithm configuration file further includes first operation frequency information or second operation frequency information, wherein the first operation frequency information includes the operation frequency and the number of operation frames of the first algorithm operation topology map, and the second operation frequency information includes the operation frequency and the number of operation frames of a single camera algorithm in the first algorithm operation topology map; For the main stream data, the first algorithm operation topology map is run once every first preset value of image frames set by the first operation frequency information, and the operation is continuously run for a second preset value of image frames set by the operation frame number each time, or the single camera algorithm is run once every third preset value of image frames set by the second operation frequency information, and the operation is continuously run for a fourth preset value of image frames set by the operation frame number each time; After determining that the current shooting scene is a multi-stream scene, determining a second target algorithm configuration file that matches the shooting scene of the auxiliary stream from the pre-configured auxiliary stream algorithm configuration files; Based on the name of the camera algorithm in the second target algorithm configuration file, a corresponding algorithm node is constructed, and a connection relationship between each algorithm node is established according to the execution order of each camera algorithm to obtain a second algorithm operation topology map, wherein the second target algorithm configuration file also includes third operation frequency information or fourth operation frequency information, the third operation frequency information includes the operation frequency and the number of operation frames of the second algorithm operation topology map, and the fourth operation frequency information includes the operation frequency and the number of operation frames of a single camera algorithm in the second algorithm operation topology map; For the auxiliary flow data, the second algorithm operation topology map is run once every fifth preset value of image frames set by the third operating frequency information, and the operation is continuously run for a sixth preset value of image frames set by the operating frame number each time, or the single camera algorithm is run once every seventh preset value of image frames set by the fourth operating frequency information, and the operation is continuously run for an eighth preset value of image frames set by the operating frame number each time.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors, a memory and a touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device implements the camera algorithm configuration method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Instructions are stored thereon, and when the instructions are executed on an electronic device, the electronic device executes the camera algorithm configuration method as described in any one of claims 1 to 8.

11. A chip system, characterized in that: include: at least one processor and an interface, the interface being used to receive code instructions and transmit them to the at least one processor; The at least one processor runs the code instructions to implement the camera algorithm configuration method according to any one of claims 1 to 8.

Images