Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method and a related device, and relates to the technical field of terminals. The method comprises the following steps: inserting one or more to-be-inserted algorithms into a proper position of a processing flow based on information such as input data and output data of the one or more to-be-inserted algorithms, so that in the processing flow, the output of a previous node of the to-be-inserted algorithm is matched with the input of the to-be-inserted algorithm; and matching the output of the to-be-inserted algorithm with the input of the next node of the to-be-inserted algorithm. In this way, the electronic device can flexibly determine the insertion position of one or more algorithms, and can also support the insertion of one or more algorithms at the intermediate node position of the processing flow.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to image processing methods and related devices. Background Art

[0002] An electronic device may be installed with a variety of applications, among which some applications may support functions such as taking photos, recording videos, beautifying pictures, and beautifying the face, such as camera applications, beautifying face applications, or beautifying picture applications.

[0003] However, in some scenarios, the image processing process of electronic devices takes a long time, which reduces the user experience. Summary of the Invention

[0004] The image processing method and related apparatus provided in the embodiments of the present application can insert one or more algorithms to be inserted into appropriate locations in a processing flow based on information such as the input data and output data of the one or more algorithms to be inserted, so that in the processing flow, the output of the previous node of the algorithm to be inserted matches the input of the algorithm to be inserted, and the output of the algorithm to be inserted matches the input of the next node of the algorithm to be inserted. In this way, the electronic device can flexibly determine the insertion location of one or more algorithms and can also support the insertion of one or more algorithms at intermediate nodes in the processing flow.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, applied to an electronic device, the method comprising:

[0006] Obtain information about an image to be processed, the information including an identifier of an image processing flow and identifiers of one or more preset algorithms, the one or more preset algorithms not being fixed in the image processing flow, and the image processing flow including at least one of the following flows set in a hardware abstraction layer: a photo capture flow, a preview flow, or a video recording flow; partially or fully setting the one or more preset algorithms in the image processing flow to obtain an updated image processing flow; wherein, for a target algorithm in the image processing flow set in the hardware abstraction layer, the output data format requirements of the algorithm preceding the target algorithm meet the input data format requirements of the target algorithm, and the output data format requirements of the algorithm following the target algorithm meet the input data format requirements of the algorithm following the target algorithm; and processing the image to be processed based on the updated image processing flow. In this way, in the image processing flow, the output of the previous node of the preset algorithm matches the input of the preset algorithm, and the output of the preset algorithm matches the input of the next node of the preset algorithm. The electronic device can flexibly determine the insertion position of the one or more preset algorithms and can also support the insertion of one or more preset algorithms at intermediate nodes in the image processing flow.

[0007] In one possible implementation, the input data format requirements include requirements for the quantity and data type of input data, and the output data format requirements include requirements for the quantity and data type of output data. Thus, inserting the algorithm based on the input and output data format requirements ensures normal data processing within the image processing workflow, reduces data processing anomalies, and allows the algorithm to be inserted at the appropriate location within the image processing workflow.

[0008] In one possible implementation, some or all of one or more preset algorithms are set in the image processing process, including: if the image processing process identifier in the target algorithm information is an identifier corresponding to the photo shooting scene, the target algorithm is set in the photo shooting process; if the image processing process identifier in the target algorithm information is an identifier corresponding to the preview scene, the target algorithm is set in the preview process; if the image processing process identifier in the target algorithm information is an identifier corresponding to the video recording scene, the target algorithm is set in the video recording process. In this way, using the image processing process identifier makes it easier to distinguish different image processing processes and conveniently determine which image processing process the target algorithm should be inserted into. This simplifies the code execution process in the electronic device, improves the code readability and flexibility, and can concisely represent the process to which the target algorithm is to be inserted.

[0009] In one possible implementation, part or all of one or more preset algorithms are set in the image processing process, including: for the target algorithm, traversing one or more algorithms fixedly set in the image processing process; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, the target algorithm is set after the first algorithm in the image processing process; wherein the first algorithm is an algorithm fixedly set in the image processing process. In this way, based on traversing one or more algorithms fixedly set in the image processing process and judging the format requirements, the electronic device can determine whether it is appropriate to insert the preset algorithm in the image processing process, and the location of the insertion. The electronic device can flexibly insert the preset algorithm according to the actual situation in different image processing processes, without the need to fix the algorithm, thereby improving the flexibility of algorithm processing.

[0010] In one possible implementation, setting the target algorithm before the first algorithm in the image processing flow also includes setting target information and creating an execution node for the target algorithm, where the target information includes dependency information between the target algorithm and other algorithms in the image processing flow. Setting the target algorithm after the first algorithm in the image processing flow also includes executing the execution node of the target algorithm after the first algorithm in the image processing flow based on the dependency information. In this way, the electronic device can continue to execute the target algorithm based on the results of the first algorithm based on the target information, ensuring orderly execution of the algorithms.

[0011] In one possible implementation, an electronic device includes a first module, a second module, a third module, and a fourth module, and obtains information of an image to be processed, including: obtaining information of the image to be processed from an application layer through a hardware abstraction layer interface definition language (HIDL) interface based on the first module; obtaining information of the image to be processed from the application layer through a camera service (CameraService) based on the second module, and passing it to the third module; after obtaining the information of the image to be processed, further including: determining an image processing process based on an identifier of the image processing process in the information of the image to be processed based on the third module, and instructing the fourth module to create and initialize the image processing process; creating and initializing the image processing process based on the fourth module, and loading one or more algorithms fixedly set in the image processing process; setting part or all of the one or more preset algorithms in the image processing process, including: traversing one or more algorithms fixedly set in the image processing process based on the first module for a target algorithm; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, setting the target algorithm after the first algorithm in the image processing process based on the first module. In this way, electronic devices can use dynamic insertion modules to determine the processing flow and position of each algorithm to be inserted in the processing flow, so that each algorithm can freely decide whether to insert it into the current processing flow during runtime, thereby improving the flexibility of code execution and better adapting to the functions provided by upper-level applications.

[0012] In one possible implementation, the method further includes: in the process of traversing one or more algorithms fixedly set in the image processing process, if there is no output data format requirement of any algorithm that meets the input data format requirement of the first preset algorithm, and the output data format requirement of the first preset algorithm meets the input data format requirement of any algorithm adjacent to the next algorithm, then the updated image processing process does not include the first preset algorithm, and the first preset algorithm is an algorithm among the one or more preset algorithms; after processing the image to be processed based on the updated image processing process, it also includes: passing the processed image to be processed to the application layer; executing the first preset algorithm on the processed image to be processed at the application layer. In this way, for algorithms that are not successfully inserted into the image processing process, the electronic device can use the previous image processing method to implement it. Thereby, the function corresponding to the unsuccessfully inserted algorithm can still be reflected in the image, thereby improving the user experience.

[0013] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device or a chip or chip system within an electronic device. The device may include a processing unit. The processing unit is used to implement any processing-related method performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to enable the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to enable the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, a cache, etc.) or a storage unit within the electronic device located outside the chip (e.g., a read-only memory, a random access memory, etc.).

[0014] Exemplarily, the processing unit is used to obtain information of the image to be processed; it is also used to set part or all of one or more preset algorithms in the image processing process to obtain an updated image processing process, and is also used to process the image to be processed based on the updated image processing process.

[0015] In a possible implementation, the input data format requirement includes a quantity requirement of the input data and a data type requirement of the input data, and the output data format requirement includes a quantity requirement of the output data and a data type requirement of the output data.

[0016] In a possible implementation, the processing unit is configured to set the target algorithm in a photographing process if the identification of the image processing process in the information of the target algorithm is an identification corresponding to a photographing scenario; set the target algorithm in a preview process if the identification of the image processing process in the information of the target algorithm is an identification corresponding to a preview scenario; and set the target algorithm in a video recording process if the identification of the image processing process in the information of the target algorithm is an identification corresponding to a video recording scenario.

[0017] In a possible implementation, the processing unit is configured to traverse one or more algorithms fixed in the image processing process, and set the target algorithm after a first algorithm in the image processing process if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of a next algorithm adjacent to the first algorithm.

[0018] In a possible implementation, the processing unit is configured to set target information, and create an execution node of the target algorithm, and execute the execution node of the target algorithm.

[0019] In a possible implementation, the processing unit is configured to acquire, by a first module, information of a to-be-processed image from an application layer based on a hardware abstraction layer interface definition language (HIDL) interface, acquire, by a second module, the information of the to-be-processed image from the application layer based on a camera service (CameraService), and deliver the information to a third module, determine, by the third module, an image processing process according to an identification of the image processing process in the information of the to-be-processed image, instruct a fourth module to create and initialize the image processing process, create and initialize the image processing process based on the fourth module, and load one or more algorithms fixed in the image processing process, traverse the one or more algorithms fixed in the image processing process based on the first module, and set, by the first module, a target algorithm after a first algorithm in the image processing process if an output data format requirement of the first algorithm meets an input data format requirement of the target algorithm, and an output data format requirement of the target algorithm meets an input data format requirement of a next algorithm adjacent to the first algorithm.

[0020] In a possible implementation, the processing unit is configured to deliver the processed to-be-processed image to the application layer, and execute a first preset algorithm on the processed to-be-processed image in the application layer.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including one or more processors and a memory, the memory being coupled to the one or more processors, the memory being configured to store computer program codes, the computer program codes including computer instructions, and the one or more processors being configured to invoke the computer instructions to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when the computer program runs on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.

[0024] In a sixth aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is configured to execute a computer program or instruction to perform the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0025] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0026] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of module interaction of an image processing process provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a process for determining an algorithm to be inserted into an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a method for determining the position of an algorithm to be inserted in a processing flow according to an embodiment of the present application;

[0033] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0036] 1. Terminology

[0037] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0038] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0039] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0040] 2. Electronic devices

[0041] The electronic device of the embodiment of the present application may also be a terminal device in any form. For example, the electronic device may include: a mobile phone, a tablet computer, a PDA, a laptop computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a wearable device, an electronic device in a 5G network or a future evolved public land mobile communication network (PLMN) Mobile network, PLMN) and other electronic devices, and the embodiments of the present application are not limited to this.

[0042] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0043] In addition, in the embodiment of the present application, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0044] The electronic devices in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0045] In the embodiments of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0046] For example, Figure 1 A schematic structural diagram of an electronic device is shown.

[0047] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0048] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0050] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that have just been used or are recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thereby improving the efficiency of the system. For example, in an embodiment of the present application, the processor 110 may be used to create and initialize processing flows such as a preview flow, a photo taking flow, or a video recording flow, as well as to load and execute algorithms used in the processing flow.

[0051] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface.

[0052] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0053] The internal memory 121 can be used to store computer executable program code, and the executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. For example, in an embodiment of the present application, the internal memory 121 can be used to store relevant codes of media algorithms and universal algorithms, etc.

[0054] The electronic device implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information. The electronic device can implement camera functionality through an ISP, camera 193, video codec, GPU, display screen 194, and application processor.

[0055] In some embodiments, the electronic device may include 1 or N cameras 193 , where N is a positive integer greater than 1. The camera 193 may be used to capture still images or videos.

[0056] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than one.

[0057] Figure 2 This is a block diagram of the software structure of the electronic device in an embodiment of the present application. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer (HAL), and the kernel layer.

[0058] The application layer can also be called the application layer, which can include a series of application packages. Figure 2As shown, the application package can include applications such as phone, music, calendar, camera, game, memo, video, etc. Applications can include system applications and third-party applications.

[0059] The application framework layer, also known as the Framework layer, provides an application programming interface (API) and programming framework for applications in the application layer. The Framework layer may include some predefined functions.

[0060] like Figure 2 As shown, the Framework layer may include a camera service (CameraService), a window manager, a resource manager, and a view system.

[0061] The camera service provides a standardized interface that allows applications at the application layer to access and control camera hardware. The camera service can capture image and video data and perform image processing operations such as autofocus, exposure control, and white balance adjustment.

[0062] The window manager is responsible for managing window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, take screenshots, etc.

[0063] The resource manager can provide various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0064] The view system is responsible for drawing the application interface and handling events.

[0065] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for controlling and managing the Android system.

[0066] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0067] The application layer and framework layer run in a virtual machine. The virtual machine executes the Java files in the application and framework layers as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiments of the present application, the virtual machine can be used to create and initialize processing flows such as the preview process, the photo capture process, or the video recording process, as well as load and execute algorithms used in the processing flow.

[0068] The system library can also be called the Native layer. The Native layer can include multiple functional modules, such as the media library, function library, and graphics processing library.

[0069] The HAL layer is an abstract structure between the kernel layer and the Android runtime. The hardware abstraction layer can be a package for hardware drivers, providing a unified interface for upper-layer applications to call. In the embodiment of the present application, the HAL layer can also include a perception engine, a collection engine, a decision engine, and a processing engine.

[0070] The perception engine can obtain relevant data of the camera application based on the camera service of the Framework layer, and can also pass the data to the decision engine.

[0071] The acquisition engine can obtain images captured by the camera hardware based on the camera driver, and the acquisition engine can also pass the images to the decision engine.

[0072] The decision engine can decide the appropriate process to be executed based on the data transmitted by the perception engine and the acquisition engine, and pass the decided process to the processing engine.

[0073] The processing engine can create and initialize processing processes such as preview process, photo process or video process. The processing engine can also load and execute the algorithms used in the processing process.

[0074] The kernel layer is the layer between hardware and software. The kernel layer may include display driver, camera driver, audio driver, battery driver, Bluetooth driver, CPU driver, USB driver, etc.

[0075] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0076] An electronic device may be installed with a variety of applications, among which some applications may support functions such as taking photos, recording videos, beautifying pictures, and beautifying the face, such as camera applications, beautifying face applications, or beautifying picture applications.

[0077] Taking the camera application as an example, the camera application can implement functions such as preview, photo taking, and video recording. Specifically, when the camera application is previewing, the electronic device can initialize and execute the preview process; when the camera application is taking a photo, the electronic device can initialize and execute the photo taking process; when the camera application is recording a video, the electronic device can initialize and execute the video recording process.

[0078] It is understandable that in the processing processes such as the preview process, the photo process or the video recording process, one or more algorithms are included, and these algorithms can respectively implement different functions, for example, they can include watermark algorithms, filter algorithms, skin beautification algorithms, portrait segmentation algorithms, portrait enhancement algorithms, etc. Among them, the watermark algorithm can be used to implement the watermarking function, the filter algorithm can be used to implement the filter function, the skin beautification algorithm can be used to implement the skin beautification function, the portrait segmentation algorithm can be used to implement the portrait segmentation function, and the portrait enhancement algorithm can be used to implement the portrait enhancement function. When the electronic device executes the processing processes such as the preview process, the photo process or the video recording process, it can execute the one or more algorithms according to the needs of different processing processes to implement the corresponding functions.

[0079] like Figure 3 As shown, the Framework layer serves as a bridge between the camera application and the camera hardware device, and needs to provide various hardware capabilities for the upper-layer camera application. The functions developed by the upper-layer camera application also need to be nested in the process of the hardware platform. For example, the camera application can interact with the perception engine in the HAL layer based on the camera service of the Framework layer. In the HAL layer, the perception engine can obtain relevant information passed by the camera application based on the camera service, such as brightness value, exposure parameters and other information, and pass the information to the decision engine. The acquisition engine can obtain the image data collected by the camera hardware based on the camera driver, and pass the image data to the decision engine.

[0080] The decision engine can determine the appropriate processing flow to be executed based on the information transmitted by the perception engine and the image data transmitted by the acquisition engine, and pass the determined processing flow to the processing engine. The processing engine can create and initialize processing flows such as the preview flow, the photo taking flow, or the video recording flow, and can also load and execute the algorithms used in the processing flow. Figure 3 The source node in each processing flow can represent the head node of the processing flow, representing the beginning of the processing flow, and the sink node can represent the tail node of the processing flow, representing the end of the processing flow.

[0081] In some implementations, the application layer can be pre-configured with certain algorithms, such as watermarking algorithms, filter algorithms, and skin beautification algorithms. After the HAL layer completes image processing, it can return the image to the application layer, where the application can then apply various algorithms to the returned image. However, since applications in the application layer have difficulty utilizing the capabilities of hardware devices, such as GPUs for accelerated processing, the HAL layer can more easily utilize the accelerated processing capabilities provided by the GPU to process images. Therefore, compared to the HAL layer, the application performs algorithmic processing at a relatively slow speed.

[0082] In addition, in a possible implementation, the HAL layer can pre-configure different processing flows when processing an image. For example, the HAL layer can configure different processing flows in an extensible markup language (XML) file, and each processing flow is configured with a different algorithm. If the HAL layer needs to process a certain algorithm, the HAL layer can execute the processing flow configured with the algorithm. If the HAL layer does not need to process the algorithm, the HAL layer can execute the processing flow not configured with the algorithm. In this way, when the HAL layer switches from a scene where the algorithm needs to be processed to a scene where the algorithm does not need to be processed, the HAL layer needs to first delete the processing flow configured with the algorithm, and then recreate a preview flow that is not configured with the algorithm, which may cause the image processing flow to be time-consuming and the execution flow of image processing is not flexible enough.

[0083] In other implementations, because the head and tail nodes of a process flow are relatively fixed and easily identifiable—for example, the head node has no input but only output, while the tail node has only input but no output—the electronic device can insert the algorithm after the head node or before the tail node of the process flow. However, inserting the algorithm at the head or tail node limits the location where the electronic device can insert the algorithm.

[0084] In view of this, the image processing method provided in the embodiment of the present application can insert one or more algorithms to be inserted into the appropriate position of the processing flow based on information such as the input data and output data of one or more algorithms to be inserted, so that in the processing flow, the output of the previous node of the algorithm to be inserted matches the input of the algorithm to be inserted, and the output of the algorithm to be inserted matches the input of the next node of the algorithm to be inserted. In this way, the electronic device can flexibly determine the insertion position of one or more algorithms, and can also support the insertion of one or more algorithms at the intermediate node position of the processing flow, reducing the situation where the electronic device recreates the processing flow and improving the processing speed of the algorithm.

[0085] Figure 4 A schematic diagram of module interaction of the image processing method according to an embodiment of the present application is shown.

[0086] and Figure 3 Similar to the corresponding embodiments, the camera application can interact with the perception engine in the HAL layer based on the camera service in the Framework layer. In the HAL layer, the perception engine can obtain relevant information transmitted by the camera application based on the camera service and pass this information to the decision engine. The acquisition engine can obtain image data captured by the camera hardware based on the camera driver and pass this image data to the decision engine.

[0087] The decision engine determines the appropriate processing flow to execute based on the information transmitted by the perception engine and the image data transmitted by the acquisition engine, and passes the determined processing flow to the processing engine. The processing engine can create and initialize processing flows such as the preview flow, the photo taking flow, or the video recording flow, and can also load and execute the algorithms used in the processing flow.

[0088] In the embodiment of the present application, the nodes in the processing flow can be understood as the execution steps or algorithms in the processing flow. Figure 4 As shown, the photographing process may include multiple nodes, such as a source node, a node for algorithm A, a node for algorithm B, a node for algorithm C, a sink node, etc. Among them, algorithm A, algorithm B, and algorithm C can be understood as preset algorithms in the photographing process, or as algorithms that will be used in the photographing process. Algorithm A, algorithm B, and algorithm C can correspond to different algorithms respectively.

[0089] Similarly, the preview process can include multiple nodes, such as a source node, a node for Algorithm D, a node for Algorithm E, a node for Algorithm F, and a sink node. Algorithm D, Algorithm E, and Algorithm F can be understood as preset algorithms in the preview process or as algorithms used in the preview process. Algorithm D, Algorithm E, and Algorithm F can correspond to different algorithms.

[0090] The recording process may include multiple nodes, such as a source node, a node for algorithm G, a node for algorithm H, a node for algorithm I, a sink node, etc. Among them, algorithm G, algorithm H, and algorithm I can be understood as preset algorithms in the recording process, or as algorithms that will be used in the recording process. Algorithm G, algorithm H, and algorithm I can correspond to different algorithms respectively.

[0091] It can be understood that Algorithm A, Algorithm B, Algorithm C in the photo-taking process and Algorithm D, Algorithm E and Algorithm F in the preview process can be the same algorithm or different algorithms, and Algorithm D, Algorithm E and Algorithm F in the preview process and Algorithm G, Algorithm H and Algorithm I in the video recording process can be the same algorithm or different algorithms, and the embodiments of this application do not limit this.

[0092] and Figure 3The difference between the corresponding embodiments is that the HAL layer in the electronic device can also include a dynamic insertion position decision module. The electronic device can use the dynamic insertion position decision module to determine the processing flow to be inserted into each algorithm and the position to be inserted in the processing flow, so that each algorithm can freely decide whether to insert it into the current processing flow during operation, thereby better adapting to the functions provided by the upper-level application. Among them, the dynamic insertion position decision module can also be called a dynamic insertion module, which can support the insertion of the HAL layer algorithm into the processing flow, and can also support the insertion of the algorithm in the application into the processing flow through the hardware abstraction layer interface definition language (HAL interface definition language, HIDL) interface.

[0093] The algorithms that can be processed by the dynamic insertion module include media algorithms and general algorithms. Among them, the media algorithm can be understood as the algorithm preset in the application package (Android application package, APK) of each application in the application layer, such as Figure 4 In the camera application, the pre-set media algorithms may include watermarking algorithms, filter algorithms, and / or skin beautification algorithms. The general algorithms can be understood as some pre-set algorithms in the HAL layer for image processing, such as Figure 4 In the perception engine, general algorithms that can be preset may include portrait segmentation algorithm, portrait enhancement algorithm, etc.

[0094] Optionally, the dynamic insertion module can also be implemented in the decision engine, which is not limited in the embodiments of the present application.

[0095] Figure 5 A schematic diagram shows a processing flow in which a dynamic insertion module determines the algorithm to be inserted.

[0096] S501: The dynamic insertion module obtains relevant information of the algorithms to be inserted, thereby determining the number of algorithms to be inserted, the enabling scenarios of the algorithms to be inserted, and the output formats of the algorithms to be inserted.

[0097] The dynamic insertion module can obtain the relevant information of the algorithm to be inserted passed by the camera application, and can also obtain the relevant information of the algorithm to be inserted passed by the perception engine module. For example, the relevant information of the algorithm to be inserted includes the function pointer corresponding to the algorithm to be inserted, the number of input parameters of the function, the data type of the input parameters of the function, the output format of the function, and other related information, as well as the enabling scenario of the algorithm to be inserted. Among them, the enabling scenario of the algorithm to be inserted can be used to indicate in which processing flow the algorithm to be inserted is used, for example, whether the algorithm to be inserted is used in the photo-taking process, the preview process, or the video recording process.

[0098] Based on the acquired information about the algorithms to be inserted, the dynamic insertion module can determine the number of algorithms to be inserted, the enabling scenarios of the algorithms to be inserted, the required output format, and the like.

[0099] After obtaining the relevant information of the algorithm to be inserted, the dynamic insertion module may execute step S502.

[0100] S502: The dynamic insertion module determines whether all algorithms to be inserted have been processed.

[0101] In a possible implementation, the dynamic insertion module can process one or more algorithms to be inserted in a loop and determine the processing flow to be inserted for each algorithm to be inserted. When the last algorithm to be inserted is traversed and processed, it means that all algorithms to be inserted have been processed.

[0102] If the algorithms to be inserted have not all been processed, the dynamic insertion module may execute step S503 to process each algorithm to be inserted.

[0103] If all algorithms to be inserted have been processed, the loop traversal ends.

[0104] S503: The dynamic insertion module determines whether the algorithm to be inserted can be inserted into the processing flow.

[0105] The dynamic insertion module can determine whether the algorithm to be inserted can be inserted into a processing flow such as a photo-taking process, a preview process, or a video recording process based on the enabling scenario of the algorithm to be inserted.

[0106] If the enabled scenario for the algorithm to be inserted is the photo shooting scenario, the algorithm can be inserted into the photo shooting process; if the enabled scenario for the algorithm to be inserted is the preview scenario, the algorithm can be inserted into the preview process; if the enabled scenario for the algorithm to be inserted is the video recording scenario, the algorithm can be inserted into the video recording process. Different scenarios can use different enabled scenario identifiers.

[0107] For example, in response to a user trigger operation, the camera application may determine an enabling scenario for the algorithm to be inserted. The user trigger operation may include touching, clicking, long pressing, or sliding the screen of the electronic device, pressing a button of the electronic device, or controlling the electronic device by voice input, etc., which is not limited in the embodiments of the present application.

[0108] Taking a scenario where a watermark needs to be added as an example, when a user triggers a photo-taking operation, the camera application can, in response to this operation, pass relevant information about the watermark algorithm to the dynamic insertion module, including, for example, the function pointer corresponding to the watermark algorithm, the number of input parameters of the function, the data type of the input parameters of the function, the output format of the function, and other related information. It can also pass that the enabling scenario of the watermark algorithm is the photo-taking scenario, etc. Based on the enabling scenario of the watermark algorithm, the dynamic insertion module can determine whether the watermark algorithm is used in the photo-taking process.

[0109] If a watermark algorithm needs to be inserted into both the preview and capture processes, the camera application can pass the watermark algorithm information twice to the dynamic insertion module. The information in these two passes can include enabling scenarios for preview and capture, respectively. In other words, the dynamic insertion module can insert the same algorithm multiple times, and into different processing flows.

[0110] It is understandable that for algorithms with the same function, different formats need to be processed in different processing flows. For example, taking the watermark algorithm as an example, in the photo-taking process, adding a watermark is to process images in the YUV format; in the thumbnail generation process, adding a watermark is to process images in the RGB format; in the preview process, adding a watermark is to process images in the UBWC format.

[0111] The use of image processing process identifiers makes it easy to distinguish different image processing processes and to determine into which image processing process the target algorithm should be inserted. This simplifies the execution process of the code in the electronic device and improves the readability and flexibility of the code, thereby making it possible to concisely represent the process into which the target algorithm is to be inserted.

[0112] After determining the processing flow that needs to be inserted into the algorithm to be inserted, the dynamic insertion module may execute step S504.

[0113] If the processing flow that the algorithm to be inserted needs to be inserted into is a processing flow that the electronic device does not support, or the data corresponding to the enabling scenario of the algorithm to be inserted is incorrect, the dynamic insertion module can end this loop, execute step S502, and continue to judge and process the next algorithm to be inserted.

[0114] It is understood that for algorithms that are not successfully inserted into the processing flow, the dynamic insertion module can transmit information to the camera application at the application layer or the perception engine at the HAL layer, indicating that the algorithm was not successfully inserted. After the processing engine performs image processing, it can transmit the processed image to the camera application or perception engine. If the camera application or perception engine determines that the function corresponding to the algorithm still needs to be implemented on the image, the camera application or perception engine can execute the algorithm on the processed image to implement the algorithm's function.

[0115] That is, for algorithms that are not successfully inserted into the processing flow, the electronic device can use the previous image processing method to implement them. In this way, the function corresponding to the algorithm that was not successfully inserted can still be reflected in the image, thereby improving the user experience.

[0116] In a possible implementation, the dynamic insertion module can save the obtained relevant information of the algorithm to be inserted into a data structure such as an array to facilitate subsequent use.

[0117] S504: The dynamic insertion module determines whether the algorithm to be inserted passes parameter verification.

[0118] The dynamic insertion module can verify the relevant parameters of the algorithm to be inserted. For example, it can verify the function pointer corresponding to the algorithm to be inserted, the number of input parameters of the function, the data type of the input parameters of the function, the output format of the function, etc., to determine whether the parameters passed by the algorithm to be inserted can be used in the subsequent processing flow.

[0119] If the number or format of the parameters passed by the algorithm to be inserted is incorrect, it means that it cannot be used in the subsequent processing flow, that is, the parameter verification fails, then the dynamic insertion module can end this loop and execute step S502 to continue judging and processing the next algorithm to be inserted.

[0120] If the number or format of the parameters passed by the algorithm to be inserted is incorrect, it means that the parameter information passed by the algorithm to be inserted can be used in the subsequent processing flow, that is, the parameter verification passes, and the dynamic insertion module can execute step S505.

[0121] The dynamic insertion module determines whether the algorithm to be inserted has passed parameter verification. If the algorithm fails parameter verification, it will not be inserted. This can reduce unnecessary code execution and reduce errors in subsequent execution processes, allowing the electronic device to operate stably.

[0122] S505 : The dynamic insertion module sets meta information and uses the algorithm to be inserted as the node to be inserted, thereby creating an execution node of the algorithm to be inserted.

[0123] It can be understood that the meta information can include some information related to the image, which can provide additional information about the image. For example, the meta information can include one or more of the following information: shooting parameters, device information, geographical location, timestamp, author information, copyright information, image description, etc.

[0124] The shooting parameters can include exposure time, aperture value, ISO sensitivity, focal length, etc. The device information can include the brand and model of the shooting device, such as detailed information of an electronic device such as a camera or a mobile phone. If the electronic device supports positioning function, the geographical location can be used to record the geographical coordinates, such as longitude and latitude, when the image is taken. The timestamp can include the specific date and time of the photo shooting. The author information can include the name of the photographer. The copyright information can include the copyright statement or usage restrictions. The image description can include a brief description or title of the image content.

[0125] In the embodiments of the present application, the dynamic insertion module can set the dependency information between algorithms in the meta information, and after inserting the execution node of the to-be-inserted algorithm in the processing flow, the processing engine can execute the algorithm based on the dependency information. For example, assuming that algorithm A executes slowly in the photographing process, the electronic device can start a new thread to execute algorithm A. Since the execution of algorithm B depends on the result of algorithm A, the dynamic insertion module can set the dependency information between algorithm A and algorithm B in the meta information. In this way, the electronic device can continue to execute algorithm B based on the result of algorithm A, so that the algorithms can be executed in order.

[0126] The dynamic insertion module can also insert the to-be-inserted algorithm as a to-be-inserted node, thereby creating an execution node of the to-be-inserted algorithm to load the information of the to-be-inserted algorithm, so as to facilitate the initialization preparation for subsequent execution of the to-be-inserted algorithm. In a possible implementation, the dynamic insertion module can create the execution node of the to-be-inserted algorithm in the createPlugin process, and the embodiments of the present application are not limited thereto.

[0127] It can be understood that after the dynamic insertion module determines the processing flow to which the to-be-inserted algorithm is to be inserted, the dynamic insertion module can further determine the position of the to-be-inserted algorithm to be inserted in the processing flow.

[0128] Figure 6 A schematic diagram of the dynamic insertion module determining the position of the to-be-inserted algorithm to be inserted in the processing flow is shown.

[0129] S601, the dynamic insertion module determines whether the to-be-inserted algorithm has been processed.

[0130] In a possible implementation, in the case that the dynamic insertion module determines that the to-be-inserted algorithm has been processed, the dynamic insertion module can further determine the position of the to-be-inserted algorithm to be inserted in the processing flow. Figure 5After the loop traversal of the corresponding embodiment is completed, the dynamic insertion module can process one or more algorithms to be inserted again through the loop traversal method and determine the position of the one or more algorithms to be inserted in the processing flow. When the last algorithm to be inserted is traversed and processed, it means that all algorithms to be inserted have been processed.

[0131] If the algorithms to be inserted have not all been processed, the dynamic insertion module may execute step S602 to process each algorithm to be inserted.

[0132] If all algorithms to be inserted have completed processing, this loop traversal ends.

[0133] It is understandable that the algorithm to be inserted in the loop traversal in step S601 is Figure 5 The algorithms to be inserted that pass the parameter verification in the corresponding embodiment, that is, the number of algorithms to be inserted that are traversed in a loop in step S502 is greater than or equal to the number of algorithms to be inserted that are traversed in a loop in step S601.

[0134] For the convenience of description, the loop traversal in step S502 is referred to as the first loop traversal, and the loop traversal in step S601 is referred to as the second loop traversal.

[0135] If one or more algorithms to be inserted in the first loop traversal all pass parameter verification, the number of algorithms to be inserted in the first loop traversal is equal to the number of algorithms to be inserted in the second loop traversal. If some of the one or more algorithms to be inserted in the first loop traversal pass parameter verification, the number of algorithms to be inserted in the first loop traversal is greater than the number of algorithms to be inserted in the second loop traversal.

[0136] For example, if there are five algorithms to be inserted in the first loop, if all five algorithms pass parameter verification, then in the second loop, the dynamic insertion module will determine the positions to be inserted for these five algorithms. If four of these five algorithms pass parameter verification, then in the second loop, the dynamic insertion module will determine the positions to be inserted for these four algorithms that passed parameter verification.

[0137] S602: The dynamic insertion module obtains the input data type and output data type of the execution node of the algorithm to be inserted.

[0138] In a possible implementation, the dynamic insertion module obtains the input data type and the output data type from a data structure such as an array storing relevant information of the algorithm to be inserted, and executes step S603.

[0139] S603: The dynamic insertion module determines whether all nodes in the current processing flow have been traversed.

[0140] The dynamic insertion module can loop through each node in the current processing flow to determine whether the algorithm to be inserted is suitable for insertion into the currently traversed node.

[0141] It is understood that the dynamic insertion module can perform the loop traversal process of step S603 during the second loop traversal of step S601. In other words, the dynamic insertion module can perform a double loop traversal. Thus, for each algorithm to be inserted, the dynamic insertion module will perform the loop traversal of step S603 to determine whether the current algorithm to be inserted is suitable for insertion into the currently traversed node.

[0142] If the traversal of the nodes of the current processing flow has not been completed, the dynamic insertion module may execute steps S604 and S605 to find a node suitable for insertion for the algorithm to be inserted.

[0143] If all nodes of the current processing flow have been traversed, indicating that no node suitable for insertion has been found for the algorithm to be inserted, the dynamic insertion module can end the loop of step S603 and execute step S601 to continue looking for a suitable insertion position for the next algorithm to be inserted.

[0144] In an embodiment of the present application, based on traversing each node in the processing flow and judging the format requirements, the dynamic insertion module can determine whether it is appropriate to insert the algorithm to be inserted in the processing flow, as well as the insertion position. In this way, the dynamic insertion module can flexibly insert the algorithm to be inserted according to the actual situation of different processing flows, without the need to fix the algorithm in a fixed position, thereby improving the flexibility of algorithm processing.

[0145] S604: The dynamic insertion module determines whether the output data of the currently traversed node meets the input data requirements of the algorithm node to be inserted.

[0146] S605: The dynamic insertion module determines whether the input data of the next node of the currently traversed node meets the output data requirements of the node to be inserted into the algorithm.

[0147] The output data of the currently traversed node meets the input data requirements of the node to be inserted into the algorithm, which may include the number of output data of the currently traversed node being consistent with the number of input data of the node to be inserted into the algorithm, and the data type of the output data of the currently traversed node being consistent with the data type of the input data of the node to be inserted into the algorithm.

[0148] The input data of the next node of the currently traversed node meets the output data requirements of the node to be inserted into the algorithm, which may include the number of input data of the next node of the currently traversed node being consistent with the number of output data of the node to be inserted into the algorithm, and the data type of the input data of the next node of the currently traversed node being consistent with the data type of the output data of the node to be inserted into the algorithm.

[0149] If the output data of the currently traversed node meets the input data requirements of the node to be inserted into the algorithm, and the input data of the next node of the currently traversed node meets the output data requirements of the node to be inserted into the algorithm, it means that the algorithm is suitable for insertion between the currently traversed node and the next node of the currently traversed node, then the dynamic insertion module can execute step S606.

[0150] In this way, data can be processed normally in the image processing process, abnormalities in data processing can be reduced, and the algorithm can be inserted into the appropriate position in the image processing process.

[0151] If the output data of the currently traversed node does not meet the input data requirements of the node to be inserted into the algorithm, or the input data of the next node of the currently traversed node does not meet the output data requirements of the node to be inserted into the algorithm, it means that the algorithm is not suitable for insertion between the currently traversed node and the next node of the currently traversed node. The dynamic insertion module can execute step S603, traverse the next node, and find a suitable insertion position for the algorithm to be inserted.

[0152] It is understandable that step S604 and step S605 may be executed in any order.

[0153] S606: The dynamic insertion module inserts the node to be inserted into the algorithm after the currently traversed node, and updates the node list of the processing flow.

[0154] For example, taking the photo-taking process as an example, if the current traversal reaches the node of algorithm A, and the next node of the node of algorithm A is the node of algorithm B, when the output data of the node of algorithm A meets the input data requirements of the node of the algorithm to be inserted, and the input data of the node of algorithm B meets the output data requirements of the node of the algorithm to be inserted, the dynamic insertion module can insert the node of the algorithm to be inserted after the node of algorithm A and update the node list of the photo-taking process.

[0155] It is understood that for a particular algorithm to be inserted, if the dynamic insertion module has found a node suitable for inserting the algorithm before the loop traversal of step S603 ends, the loop of step S603 can be terminated and step S601 can be executed to continue searching for a suitable insertion location for the next algorithm to be inserted. This can reduce unnecessary execution logic in the electronic device, save computing power, and improve the performance of the electronic device.

[0156] The following describes the method of the embodiment of the present application in detail through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.

[0157] Figure 7 The image processing method of an embodiment of the present application is shown. The method includes:

[0158] S701. Obtain information about an image to be processed, where the information includes an identifier of an image processing process and identifiers of one or more preset algorithms. The one or more preset algorithms are not fixed in the image processing process. The image processing process includes at least one of the following processes set in a hardware abstraction layer: a photo taking process, a preview process, or a video recording process.

[0159] In the embodiment of the present application, the image to be processed can be understood as an image that needs to execute one or more preset algorithms. The preset algorithm can be understood as Figure 5 The algorithm to be inserted in the corresponding embodiment. The information of the image to be processed includes relevant information about the algorithm to be inserted. For example, the relevant information about the algorithm to be inserted includes the function pointer corresponding to the algorithm to be inserted, the number of input parameters of the function, the data type of the input parameters of the function, the output format of the function, and other related information, as well as the enabling scenarios of the algorithm to be inserted.

[0160] The identification of the image processing process can be understood as Figure 5 The enabling scenario of the algorithm to be inserted in the corresponding embodiment can be used to indicate in which processing flow the algorithm to be inserted is used, for example, whether the algorithm to be inserted is used in the photo process, the preview process, or the video process.

[0161] It is understood that the one or more preset algorithms may include algorithms set in an application program at the application layer, such as Figure 5 The media algorithm set in the camera application in the corresponding embodiment may also include a general algorithm set in the perception engine module of the hardware abstraction layer.

[0162] S702. Part or all of one or more preset algorithms are set in the image processing flow to obtain an updated image processing flow; wherein, for the target algorithm of the image processing flow set in the hardware abstraction layer in one or more preset algorithms, the output data format requirement of the algorithm adjacent to the target algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the target algorithm.

[0163] In the embodiment of the present application, the image processing process can also be understood as Figure 6 The insertion into the processing flow in the corresponding embodiment, for example, inserts the node corresponding to the algorithm to be inserted into the node list of the processing flow.

[0164] It is understood that if one or more preset algorithms meet the setup conditions, then all of the one or more preset algorithms can be set in the image processing flow; if one or more preset algorithms do not meet the setup conditions, then the one or more preset algorithms can be partially set in the image processing flow. The setup conditions include that the output data format requirements of the algorithm preceding the target algorithm meet the input data format requirements of the target algorithm, and that the output data format requirements of the target algorithm meet the input data format requirements of the algorithm following the target algorithm.

[0165] S703: Process the image to be processed based on the updated image processing flow.

[0166] In the embodiment of the present application, the updated image processing flow can also be understood as Figure 6 The node list of the update processing flow in the corresponding embodiment.

[0167] It is understood that based on information such as the input and output data of one or more preset algorithms, the one or more preset algorithms are inserted into appropriate locations in the image processing process, such that, in the image processing process, the output of the previous node of the preset algorithm matches the input of the preset algorithm, and the output of the preset algorithm matches the input of the next node of the preset algorithm. In this way, the electronic device can flexibly determine the insertion location of one or more preset algorithms and can also support the insertion of one or more preset algorithms at intermediate nodes in the image processing process.

[0168] Optional, in Figure 7 Based on the corresponding embodiment, the input data format requirement includes the quantity requirement of the input data and the data type requirement of the input data, and the output data format requirement includes the quantity requirement of the output data and the data type requirement of the output data.

[0169] In the embodiment of this application, the specific input data format requirements and output data format requirements can be referred to Figure 6The relevant description in step S605 of the corresponding embodiment is omitted for brevity.

[0170] It can be understood that inserting the algorithm according to the input data format requirements and the output data format requirements can enable the data to be processed normally in the image processing process, reduce the occurrence of abnormal data processing, and enable the algorithm to be inserted at the appropriate position in the image processing process.

[0171] Optional, in Figure 7 On the basis of the corresponding embodiments, part or all of one or more preset algorithms are set in the image processing process, which may include: if the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to the photo-taking scene, the target algorithm is set in the photo-taking process; if the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to the preview scene, the target algorithm is set in the preview process; if the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to the video recording scene, the target algorithm is set in the video recording process.

[0172] In the embodiment of the present application, the process of setting the corresponding process according to the identification of the image processing process can be referred to Figure 5 The relevant description in step S503 of the corresponding embodiment is omitted for brevity.

[0173] The use of image processing process identifiers makes it easy to distinguish different image processing processes and to determine into which image processing process the target algorithm should be inserted. This simplifies the execution process of the code in the electronic device and improves the readability and flexibility of the code, thereby making it possible to concisely represent the process into which the target algorithm is to be inserted.

[0174] Optional, in Figure 7 On the basis of the corresponding embodiments, part or all of one or more preset algorithms are set in the image processing process, which may include: for the target algorithm, traversing one or more algorithms fixedly set in the image processing process; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, the target algorithm is set after the first algorithm in the image processing process; wherein the first algorithm is an algorithm fixedly set in the image processing process.

[0175] In the embodiment of the present application, the implementation of one or more algorithms fixed in the image processing flow can be referred to Figure 6 The implementation of looping through each node in the current processing flow in step S603 of the corresponding embodiment will not be described in detail. Figure 6 The relevant descriptions of step S604 and step S605 of the corresponding embodiment are omitted for brevity.

[0176] According to the traversal of the one or more algorithms fixedly set in the image processing flow and the judgment of the format requirement, the electronic device can determine whether the preset algorithm is suitable to be inserted in the image processing flow and the position of the insertion, so that the electronic device can flexibly insert the preset algorithm according to the actual situation in different image processing flows, without fixing the algorithm, thereby improving the flexibility of algorithm processing.

[0177] Optionally, in the corresponding embodiment, Figure 7 On the basis of the corresponding embodiment, the target algorithm is set before the first algorithm of the image processing flow, and the method can further include: setting target information and creating an execution node of the target algorithm, wherein the target information includes dependency information between the target algorithm and other algorithms in the image processing flow; and the target algorithm is set after the first algorithm of the image processing flow, and the method can further include: based on the dependency information, executing the execution node of the target algorithm after the first algorithm of the image processing flow.

[0178] In the embodiment of the present application, the target information can be understood as Figure 5 The meta information in step S505 of the corresponding embodiment. The execution node of the target algorithm can also refer to the related description in Figure 5 The corresponding embodiment of step S505, which will not be repeated. In this way, the electronic device can continue to execute the target algorithm based on the target information according to the result of the first algorithm, so that the algorithms can be executed in order.

[0179] Optionally, in the corresponding embodiment, Figure 7 On the basis of the corresponding embodiment, the electronic device includes a first module, a second module, a third module, and a fourth module, and the method of obtaining the information of the image to be processed can include: obtaining the information of the image to be processed from an application layer based on the first module through a hardware abstraction layer interface definition language (HIDL) interface; obtaining the information of the image to be processed from the application layer based on the second module through a camera service (CameraService) and delivering the information to the third module; after obtaining the information of the image to be processed, the method can further include: determining the image processing flow based on the identifier of the image processing flow in the information of the image to be processed based on the third module, and instructing the fourth module to create and initialize the image processing flow; creating and initializing the image processing flow based on the fourth module, and loading one or more algorithms fixedly set in the image processing flow; and setting part or all of the one or more preset algorithms in the image processing flow can include: traversing the one or more algorithms fixedly set in the image processing flow based on the first module for a target algorithm; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, setting the target algorithm after the first algorithm of the image processing flow based on the first module.

[0180] In the embodiment of the present application, the first module can be understood as Figure 4 The second module can be understood as the dynamic insertion module in the corresponding embodiment. Figure 4 The third block of the corresponding perception engine in the embodiment can be understood as Figure 4 The fourth module can be understood as the decision engine in the corresponding embodiment. Figure 4 For the interaction between the specific modules of the processing engine in the corresponding embodiment, please refer to Figure 4 The relevant descriptions in the corresponding embodiments are omitted here.

[0181] Electronic devices can use dynamic insertion modules to determine the processing flow and position of each algorithm to be inserted in the processing flow, so that each algorithm can freely decide whether to insert it into the current processing flow during runtime, thereby improving the flexibility of code execution and better adapting to the functions provided by upper-level applications.

[0182] Optional, in Figure 7 On the basis of the corresponding embodiment, the method may further include: in the process of traversing one or more algorithms fixedly set in the image processing process, if the output data format requirement of any algorithm does not meet the input data format requirement of the first preset algorithm, and the output data format requirement of the first preset algorithm meets the input data format requirement of any algorithm adjacent to the subsequent algorithm, then the updated image processing process does not include the first preset algorithm, and the first preset algorithm is an algorithm among the one or more preset algorithms; after processing the image to be processed based on the updated image processing process, it may also include: passing the processed image to be processed to the application layer; and executing the first preset algorithm on the processed image to be processed at the application layer.

[0183] In the embodiment of this application, if the format requirements are not met, you can refer to Figure 4 The relevant descriptions in the corresponding embodiments will not be repeated here. It is understood that for algorithms that are not successfully inserted into the image processing flow, the electronic device can use the previous image processing method to implement them. This allows the function corresponding to the unsuccessfully inserted algorithm to still be reflected in the image, thereby improving the user experience.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0185] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0187] like Figure 8 FIG. 8 is a schematic diagram of a chip structure provided by an embodiment of the present application. The chip 800 includes one or more (including two) processors 801 , a communication circuit 802 , a communication interface 803 , and a memory 804 .

[0188] In some implementations, the memory 804 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0189] The method described in the above embodiment of the present application can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 801 or an instruction in the form of software. The above-mentioned processor 801 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 801 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0190] The steps of the method disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 804, and the processor 801 reads the information in the memory 804 and completes the steps of the above method in combination with its hardware.

[0191] The processor 801 , the memory 804 , and the communication interface 803 may communicate with each other via a communication line 802 .

[0192] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0193] The present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)).

[0194] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0195] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers.

[0196] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An image processing method, applied to electronic equipment, characterized in that: The method comprises: Obtaining information about an image to be processed, the information including an identifier of an image processing process and identifiers of one or more preset algorithms, wherein the one or more preset algorithms are not fixed in the image processing process, and the image processing process includes at least one of the following processes set in a hardware abstraction layer: a photo taking process, a preview process, or a video recording process; Setting part or all of the one or more preset algorithms in the image processing flow to obtain an updated image processing flow; wherein, for a target algorithm of the image processing flow set in the hardware abstraction layer among the one or more preset algorithms, an output data format requirement of an algorithm preceding the target algorithm meets an input data format requirement of the target algorithm, and an output data format requirement of the target algorithm meets an input data format requirement of an algorithm following the target algorithm; The image to be processed is processed based on the updated image processing flow.

2. The method according to claim 1, characterized in that The input data format requirements include the quantity requirements of input data and the data type requirements of input data, and the output data format requirements include the quantity requirements of output data and the data type requirements of output data.

3. The method according to claim 1 or 2, characterized in that The setting of part or all of the one or more preset algorithms in the image processing flow includes: If the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to a photographing scene, setting the target algorithm in the photographing process; If the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to the preview scene, setting the target algorithm in the preview process; If the identifier of the image processing process in the information of the target algorithm is an identifier corresponding to a video recording scene, the target algorithm is set in the video recording process.

4. The method according to any one of claims 1 to 3, characterized in that The setting of part or all of the one or more preset algorithms in the image processing flow includes: For the target algorithm, traverse one or more algorithms fixedly set in the image processing flow; If the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, the target algorithm is set after the first algorithm in the image processing process; wherein, the first algorithm is an algorithm fixedly set in the image processing process.

5. The method according to claim 4, characterized in that The step of setting the target algorithm before the first algorithm in the image processing flow further includes: Setting target information and creating an execution node of the target algorithm, wherein the target information includes dependency information between the target algorithm and other algorithms in the image processing flow; The step of setting the target algorithm after the first algorithm in the image processing flow further includes: Based on the dependency information, an execution node of the target algorithm is executed after the first algorithm in the image processing flow.

6. The method according to claim 4 or 5, characterized in that The electronic device includes a first module, a second module, a third module, and a fourth module, and the acquiring information of the image to be processed includes: Based on the first module, the information of the image to be processed is obtained from the application layer through the hardware abstraction layer interface definition language HIDL interface; Based on the second module, the information of the image to be processed is obtained from the application layer through the camera service CameraService, and is passed to the third module; After obtaining the information of the image to be processed, the method further includes: Determining the image processing process based on the identifier of the image processing process in the information of the image to be processed by the third module, and instructing the fourth module to create and initialize the image processing process; Creating and initializing the image processing process based on the fourth module, and loading the one or more algorithms fixedly set in the image processing process; The setting of part or all of the one or more preset algorithms in the image processing flow includes: For the target algorithm, traverse the one or more algorithms fixedly set in the image processing flow based on the first module; If the output data format requirements of the first algorithm meet the input data format requirements of the target algorithm, and the output data format requirements of the target algorithm meet the input data format requirements of the algorithm adjacent to the first algorithm, the target algorithm is set after the first algorithm in the image processing process based on the first module.

7. The method according to any one of claims 4 to 6, characterized in that: The method further comprises: In the process of traversing the one or more algorithms fixedly set in the image processing flow, if the output data format requirement of no algorithm meets the input data format requirement of the first preset algorithm, and the output data format requirement of the first preset algorithm meets the input data format requirement of the algorithm immediately adjacent to the one or more algorithms, then the updated image processing flow does not include the first preset algorithm, and the first preset algorithm is an algorithm among the one or more preset algorithms; After processing the image to be processed based on the updated image processing flow, the method further includes: Passing the processed image to be processed to the application layer; The first preset algorithm is executed on the processed image to be processed at the application layer.

8. An electronic device, characterized in that: The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, and is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 7.

Images