Image processing method and related apparatus

Abstract

The image processing method and the related device provided by the embodiments of the present application relate to the technical field of terminals. The method comprises the following steps: one or more to-be-inserted algorithms can be inserted into a suitable position of a processing flow based on information such as input data and output data of the one or more to-be-inserted algorithms, so that in the processing flow, the output of a previous node of the to-be-inserted algorithm matches the input of the to-be-inserted algorithm, and the output of the to-be-inserted algorithm matches the input of a next node of the to-be-inserted algorithm. In this way, the electronic device can flexibly determine the insertion position of one or more algorithms, and can also support the insertion of one or more algorithms at the intermediate node position of the processing flow.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to image processing methods and related devices. Background Technology

[0002] Electronic devices can have a variety of applications installed. Some of these applications support functions such as taking photos, recording videos, editing images, and beautifying them, such as camera apps, beautification apps, or image editing apps.

[0003] However, in some scenarios, the image processing workflow of electronic devices is time-consuming, which reduces the user experience. Summary of the Invention

[0004] The image processing method and related apparatus provided in this application can insert one or more algorithms into appropriate positions in the processing flow based on information such as the input and output data of the algorithms to be inserted. This ensures that, in the processing flow, the output of the preceding node of the algorithm to be inserted matches the input of the algorithm to be inserted, and the output of the algorithm to be inserted matches the input of the following node. In this way, the electronic device can flexibly determine the insertion position of one or more algorithms and can also support the insertion of one or more algorithms at intermediate nodes in the processing flow.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device, the method comprising:

[0006] The process involves acquiring information about the image to be processed, including an identifier for the image processing flow and identifiers for one or more preset algorithms. These preset algorithms are not fixed within the image processing flow, which includes at least one of the following flows located at the hardware abstraction layer: a capture flow, a preview flow, or a recording flow. The process then involves incorporating some or all of the preset algorithms into the image processing flow to obtain an updated image processing flow. Specifically, for the target algorithm within the image processing flow located at the hardware abstraction layer, the output data format of the algorithm preceding the target algorithm must satisfy the input data format requirements of the target algorithm, and the output data format of the target algorithm must satisfy the input data format requirements of the algorithm following the target algorithm. The image to be processed is then processed based on the updated image processing flow. This ensures that, within the image processing flow, the output of the preceding node of the preset algorithm matches the input of the preset algorithm, and the output of the preset algorithm matches the input of the next node of the preset algorithm. The electronic device can flexibly determine the insertion position of one or more preset algorithms and can also support the insertion of one or more preset algorithms at intermediate nodes within the image processing flow.

[0007] In one possible implementation, the input data format requirements include the quantity and data type requirements of the input data, and the output data format requirements include the quantity and data type requirements of the output data. By inserting the algorithm according to these input and output data format requirements, the data can be processed normally in the image processing workflow, reducing the occurrence of data processing anomalies and allowing the algorithm to be inserted at an appropriate position within the image processing flow.

[0008] In one possible implementation, some or all of one or more preset algorithms are set in the image processing flow, including: if the identifier of the image processing flow in the target algorithm's information corresponds to the identifier of the shooting scene, the target algorithm is set in the shooting flow; if the identifier of the image processing flow in the target algorithm's information corresponds to the identifier of the preview scene, the target algorithm is set in the preview flow; if the identifier of the image processing flow in the target algorithm's information corresponds to the identifier of the recording scene, the target algorithm is set in the recording flow. In this way, using the identifier of the image processing flow makes it easy to distinguish different image processing flows, facilitates the determination of which image processing flow the target algorithm should be inserted into, simplifies the execution flow of code in electronic devices, improves code readability and flexibility, and thus concisely represents the flow in which the target algorithm should be inserted.

[0009] In one possible implementation, one or more preset algorithms are set in the image processing flow, including: for a target algorithm, traversing one or more algorithms fixedly set in the image processing flow; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, then the target algorithm is set after the first algorithm in the image processing flow; wherein the first algorithm is an algorithm fixedly set in the image processing flow. In this way, based on traversing one or more algorithms fixedly set in the image processing flow and judging the format requirements, the electronic device can determine whether it is suitable to insert a preset algorithm in the image processing flow, and the insertion position. The electronic device can flexibly insert preset algorithms according to the actual situation in different image processing flows, without needing to fix the algorithm settings, thereby improving the flexibility of algorithm processing.

[0010] In one possible implementation, placing the target algorithm before the first algorithm in the image processing flow further includes: setting target information and creating an execution node for the target algorithm, wherein the target information includes dependency information between the target algorithm and other algorithms in the image processing flow; placing the target algorithm after the first algorithm in the image processing flow further includes: executing the execution node of the target algorithm after the first algorithm in the image processing flow based on the dependency information. In this way, the electronic device can continue to execute the target algorithm based on the target information and the result of the first algorithm, enabling the algorithms to be executed in an orderly manner.

[0011] In one possible implementation, the electronic device includes a first module, a second module, a third module, and a fourth module. Acquiring information about the image to be processed includes: obtaining the image information from the application layer via a Hardware Abstraction Layer Interface Definition Language (HIDL) interface based on the first module; obtaining the image information from the application layer via a Camera Service based on the second module and passing it to the third module; after acquiring the image information, the device further includes: determining the image processing flow based on the identifier of the image processing flow in the image information based on the third module, and instructing the fourth module to create and initialize the image processing flow; creating and initializing the image processing flow based on the fourth module, and loading one or more algorithms fixedly set in the image processing flow; setting some or all of the one or more preset algorithms in the image processing flow, including: for a target algorithm, traversing one or more algorithms fixedly set in the image processing flow based on the first module; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, setting the target algorithm after the first algorithm in the image processing flow based on the first module. In this way, electronic devices can use dynamic insertion modules to determine the processing flow in which each algorithm is to be inserted and the position in the processing flow to be inserted. This allows each algorithm to freely decide whether to be inserted into the current processing flow during runtime, improving the flexibility of code execution and thus better adapting to the functions provided by upper-layer applications.

[0012] In one possible implementation, the method further includes: during the process of traversing one or more algorithms fixed in the image processing flow, if no algorithm's output data format requirement satisfies the input data format requirement of the first preset algorithm, and the first preset algorithm's output data format requirement satisfies the input data format requirement of any algorithm's subsequent adjacent algorithm, then the updated image processing flow does not include the first preset algorithm, which is one of the one or more preset algorithms; after processing the image to be processed based on the updated image processing flow, the method further includes: passing the processed image to be processed to the application layer; and executing the first preset algorithm on the processed image to be processed at the application layer. In this way, for algorithms that are not successfully inserted into the image processing flow, the electronic device can use the previous image processing method. This allows the image to still reflect the function corresponding to the unsuccessfully inserted algorithm, improving the user experience.

[0013] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. The apparatus may include a processing unit. The processing unit is used to implement any processing-related method performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the apparatus is an electronic device, the processing unit may be a processor. The apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. When the apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0014] For example, the processing unit is used to acquire information about the image to be processed; it is also used to set some or all of one or more preset algorithms in the image processing flow to obtain an updated image processing flow; and it is also used to process the image to be processed based on the updated image processing flow.

[0015] In one possible implementation, the input data format requirements include the quantity requirements and data type requirements of the input data, and the output data format requirements include the quantity requirements and data type requirements of the output data.

[0016] In one possible implementation, the processing unit is configured to set the target algorithm in the photo-taking process if the identifier of the image processing flow in the target algorithm information corresponds to the identifier of the photo-taking scene; set the target algorithm in the preview process if the identifier of the image processing flow in the target algorithm information corresponds to the identifier of the preview scene; and set the target algorithm in the recording process if the identifier of the image processing flow in the target algorithm information corresponds to the identifier of the recording scene.

[0017] In one possible implementation, the processing unit is configured to traverse one or more algorithms fixedly arranged in the image processing flow; and to set the target algorithm after the first algorithm in the image processing flow if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm.

[0018] In one possible implementation, the processing unit is used to set target information and create execution nodes for the target algorithm; it is also used to execute the execution nodes for the target algorithm.

[0019] In one possible implementation, the processing unit is configured to: obtain information about the image to be processed from the application layer via the Hardware Abstraction Layer Interface Definition Language (HIDL) interface based on the first module; obtain information about the image to be processed from the application layer via the Camera Service via the second module and pass it to the third module; specifically, determine the image processing flow based on the identifier of the image processing flow in the information about the image to be processed by the third module, and instruct the fourth module to create and initialize the image processing flow; create and initialize the image processing flow based on the fourth module, and load one or more algorithms fixedly set in the image processing flow; traverse one or more algorithms fixedly set in the image processing flow based on the first module; and if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, set the target algorithm after the first algorithm in the image processing flow based on the first module.

[0020] In one possible implementation, a processing unit is used to pass the processed image to be processed to the application layer; and the application layer executes a first preset algorithm on the processed image to be processed.

[0021] Thirdly, embodiments of this application provide an electronic device including one or more processors and a memory, the memory being coupled to one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and one or more processors being used to invoke the computer instructions to perform the method described in the first aspect or any possible implementation of the first aspect.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0023] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0024] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0025] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0026] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram of module interaction for an image processing flow provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram of module interaction for an image processing method provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram illustrating an insertion process for determining the algorithm to be inserted, provided in an embodiment of this application.

[0032] Figure 6 This is a schematic diagram illustrating the insertion position of the algorithm in a determination process provided by an embodiment of this application.

[0033] Figure 7 A schematic diagram illustrating an image processing method provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0035] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0036] 1. Terminology

[0037] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0038] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0040] 2. Electronic equipment

[0041] The electronic devices in this application embodiment can also be any form of terminal device. For example, electronic devices may include: mobile phones, tablet computers, handheld computers, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, electronic devices in 5G networks, or future evolved public land mobile communication networks (PLANs). The embodiments of this application do not limit the scope of electronic devices in a mobile network (PLMN).

[0042] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0043] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0044] The electronic equipment in the embodiments of this application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0045] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0046] For example, Figure 1 A schematic diagram of the electronic device is shown.

[0047] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0048] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0049] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0050] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. For example, in the embodiments of this application, the processor 110 can be used to create and initialize processing flows such as preview processes, photo capture processes, or video recording processes, as well as load and execute algorithms used in the processing flows.

[0051] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0052] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0053] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor. For example, in this embodiment, internal memory 121 may be used to store code related to media algorithms and general algorithms, etc.

[0054] The electronic device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The electronic device can implement shooting functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0055] In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1. The cameras 193 may be used to capture still images or videos.

[0056] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, an electronic device may include one or N displays screens 194, where N is a positive integer greater than 1.

[0057] Figure 2 This is a software structure block diagram of an electronic device according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware adaptation layer (HAL), and the kernel layer.

[0058] The application layer, also known as the application layer, can include a series of application packages. For example... Figure 2As shown, the application package can include applications such as phone, music, calendar, camera, games, notes, and video. Applications can include system applications and third-party applications.

[0059] The application framework layer, also known as the framework layer, provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer can include some predefined functions.

[0060] like Figure 2 As shown, the Framework layer can include CameraService, WindowManager, ResourceManager, and ViewSystem, etc.

[0061] The camera service provides a standardized interface that allows application-layer applications to access and control camera hardware. It can capture image and video data and perform image processing operations such as autofocus, exposure control, and white balance adjustment.

[0062] The window manager is responsible for managing windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, allow screen touch, drag the screen, and capture the screen, among other things.

[0063] The file explorer can provide applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0064] The view system can be responsible for drawing the application's interface and handling events.

[0065] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the control and management of the Android system.

[0066] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0067] The application layer and framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiments of this application, the virtual machine can be used to execute processing flows such as creating and initializing preview processes, photo-taking processes, or video recording processes, as well as loading and executing algorithms used in the processing flows.

[0068] The system library, also known as the native layer, can include multiple functional modules. Examples include media libraries, function libraries, and graphics processing libraries.

[0069] The Hardware Abstraction Layer (HAL) is an abstraction layer situated between the kernel layer and the Android runtime. The HAL can be a wrapper around hardware drivers, providing a unified interface for calls from upper-layer applications. In this embodiment, the HAL layer may further include a perception engine, a data acquisition engine, a decision engine, and a processing engine.

[0070] The perception engine can obtain relevant data from the camera application based on the camera service in the Framework layer, and the perception engine can also pass the data to the decision engine.

[0071] The acquisition engine can acquire images captured by the camera hardware based on the camera driver, and the acquisition engine can also pass the images to the decision engine.

[0072] The decision engine can determine the appropriate process to be executed based on the data transmitted by the perception engine and the acquisition engine, and then pass the determined process to the processing engine.

[0073] The processing engine can create and initialize processing flows such as preview flows, photo-taking flows, or video-recording flows. The processing engine can also load and execute the algorithms used in the processing flows.

[0074] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, camera drivers, audio drivers, battery drivers, Bluetooth drivers, CPU drivers, USB drivers, etc.

[0075] It should be noted that the embodiments of this application are only illustrated using the Android system. In other operating systems (such as Windows system, iOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of this application, the solution of this application can also be implemented.

[0076] Electronic devices can have a variety of applications installed. Some of these applications support functions such as taking photos, recording videos, editing images, and beautifying them, such as camera apps, beautification apps, or image editing apps.

[0077] Taking a camera app as an example, a camera app can perform functions such as previewing, taking photos, and recording videos. Specifically, when the camera app is previewing, the electronic device can initialize and execute the preview process; when the camera app is taking a photo, the electronic device can initialize and execute the photo-taking process; and when the camera app is recording video, the electronic device can initialize and execute the recording process.

[0078] It is understandable that processing flows such as previewing, taking photos, or recording videos involve one or more algorithms, each capable of implementing different functions. These algorithms may include watermarking algorithms, filter algorithms, skin-smoothing algorithms, portrait segmentation algorithms, and portrait enhancement algorithms. Specifically, watermarking algorithms can be used to add watermarks, filter algorithms to implement filter functions, skin-smoothing algorithms to implement skin-smoothing functions, portrait segmentation algorithms to implement portrait segmentation functions, and portrait enhancement algorithms to implement portrait enhancement functions. When executing previewing, taking photos, or recording videos, electronic devices can execute one or more of these algorithms as needed for different processing flows to achieve the corresponding functions.

[0079] like Figure 3 As shown, the Framework layer acts as a bridge between the camera application and the camera hardware, providing various hardware capabilities to the upper-layer camera application. The functionalities developed for the upper-layer camera application also need to be nested within the hardware platform's workflow. For example, the camera application can interact with the perception engine in the HAL layer based on the camera service in the Framework layer. In the HAL layer, the perception engine can obtain relevant information transmitted by the camera application based on the camera service, such as brightness values ​​and exposure parameters, and then pass this information to the decision engine. The acquisition engine can acquire image data captured by the camera hardware based on the camera driver and then pass the image data to the decision engine.

[0080] The decision engine can determine the appropriate processing flow to be executed based on the information transmitted by the perception engine and the image data transmitted by the acquisition engine, and then pass the determined processing flow to the processing engine. The processing engine can create and initialize processing flows such as preview, photo capture, or video recording, and can also load and execute the algorithms used in those processing flows. Figure 3 In each processing flow, the source node can represent the head node of the processing flow, indicating the start of the processing flow, and the sink node can represent the tail node of the processing flow, indicating the end of the processing flow.

[0081] In some implementations, applications in the application layer can pre-configure algorithms, such as watermarking, filtering, and skin-smoothing algorithms. After image processing is completed in the HAL layer, the image can be returned to the application layer, where it can be further processed using various algorithms. However, because applications in the application layer have limited access to hardware capabilities, such as GPUs, for accelerated processing, while the HAL layer can more easily utilize the acceleration capabilities provided by GPUs, the processing speed of applications is relatively slower compared to the HAL layer when performing algorithmic processing.

[0082] Furthermore, in a possible implementation, the HAL layer can pre-configure different processing flows when processing images. For example, the HAL layer can configure different processing flows in an Extensible Markup Language (XML) file, with each flow using a different algorithm. If the HAL layer needs to process a certain algorithm, it can execute the processing flow configured with that algorithm; if it doesn't need to process that algorithm, it can execute a processing flow without that algorithm. However, when the HAL layer switches from a scenario requiring algorithm processing to one where it doesn't, it needs to first delete the processing flow configured with that algorithm and then recreate a preview flow without it. This can lead to potentially time-consuming image processing flows and a lack of flexibility in the execution process.

[0083] In other implementations, the head and tail nodes of the processing flow are relatively fixed and easily identifiable; for example, the head node has no input but only output, and the tail node has only input but no output. Therefore, electronic devices can insert algorithms after the head node or before the tail node of the processing flow. However, inserting algorithms at the head or tail node limits the placement of algorithms in the electronic device.

[0084] In view of this, the image processing method provided in this application can insert one or more algorithms into appropriate positions in the processing flow based on information such as the input and output data of the algorithms to be inserted. This ensures that the output of the preceding node of the algorithm to be inserted matches its input, and the output of the algorithm to be inserted matches its input. In this way, the electronic device can flexibly determine the insertion position of one or more algorithms, and can also support the insertion of one or more algorithms at intermediate nodes in the processing flow, reducing the need for the electronic device to recreate the processing flow and improving the processing speed of the algorithms.

[0085] Figure 4 A schematic diagram illustrating the module interaction of an image processing method according to an embodiment of this application is shown.

[0086] and Figure 3 Similar to the corresponding implementation, the camera application can interact with the perception engine in the HAL layer based on the camera service in the Framework layer. In the HAL layer, the perception engine can obtain relevant information transmitted by the camera application based on the camera service and pass the information to the decision engine. The acquisition engine can acquire image data captured by the camera hardware based on the camera driver and pass the image data to the decision engine.

[0087] The decision engine can determine the appropriate processing flow to be executed based on the information transmitted by the perception engine and the image data transmitted by the acquisition engine, and then pass the determined processing flow to the processing engine. The processing engine can create and initialize processing flows such as preview, photo capture, or video recording, and can also load and execute the algorithms used in the processing flow.

[0088] In this embodiment of the application, nodes in the processing flow can be understood as execution steps or algorithms within the processing flow. For example... Figure 4 As shown, the photo-taking process can include multiple nodes, such as a source node, a node for Algorithm A, a node for Algorithm B, a node for Algorithm C, and a sink node. Algorithms A, B, and C can be understood as pre-set algorithms in the photo-taking process, or as algorithms used during the process. Algorithms A, B, and C can each correspond to different algorithms.

[0089] Similarly, the preview process can include multiple nodes, such as the source node, the node for Algorithm D, the node for Algorithm E, the node for Algorithm F, and the sink node. Algorithms D, E, and F can be understood as pre-set algorithms in the preview process, or as algorithms used in the preview process. Algorithms D, E, and F can each correspond to different algorithms.

[0090] The recording process can include multiple nodes, such as a source node, a node for algorithm G, a node for algorithm H, a node for algorithm I, and a sink node. Algorithms G, H, and I can be understood as pre-set algorithms in the recording process, or as algorithms used during the recording process. Algorithms G, H, and I can each correspond to different algorithms.

[0091] It is understood that algorithms A, B, and C in the photo-taking process and algorithms D, E, and F in the preview process can be the same or different algorithms. Similarly, algorithms D, E, and F in the preview process and algorithms G, H, and I in the video recording process can be the same or different algorithms. This application does not limit the implementation of these algorithms.

[0092] and Figure 3The corresponding embodiment differs in that the HAL layer in the electronic device may also include a dynamic insertion position decision module. The electronic device can use this module to determine the processing flow in which each algorithm is to be inserted and its insertion position within that flow. This allows each algorithm to freely decide whether to insert itself into the current processing flow during runtime, thus better adapting to the functionality provided by the upper-layer application. This dynamic insertion position decision module, also known as a dynamic insertion module, supports inserting HAL layer algorithms into the processing flow and also supports inserting algorithms from the application into the processing flow via the Hardware Abstraction Layer Interface Definition Language (HIDL) interface.

[0093] The algorithms that the dynamically inserted module can handle can include media algorithms and general algorithms. Media algorithms can be understood as algorithms pre-built into the application packages (Android application packages, APKs) of various applications in the application layer, for example... Figure 4 In camera applications, pre-configured media algorithms can include watermarking algorithms, filter algorithms, and / or skin-smoothing algorithms. General algorithms can be understood as pre-configured image processing algorithms within the HAL layer, such as those used in... Figure 4 The pre-configurable general algorithms in the perception engine can include human face segmentation algorithms, human face enhancement algorithms, etc.

[0094] Optionally, the dynamic insertion module can also be implemented in the decision engine, and this application embodiment does not limit it.

[0095] Figure 5 This diagram illustrates the processing flow of the dynamic insertion module in determining which algorithm to insert.

[0096] S501, The dynamic insertion module obtains relevant information about the algorithm to be inserted, thereby determining the number of algorithms to be inserted, the enabling scenarios of the algorithms to be inserted, and the output format of the algorithms to be inserted.

[0097] The dynamic insertion module can obtain information about the algorithm to be inserted from both the camera application and the perception engine module. This information includes the function pointer, the number of input parameters, the data types of those parameters, the output image format, and the enabling scenario for the algorithm. The enabling scenario indicates the processing flow in which the algorithm is used, such as whether it's used in the photo capture, preview, or video recording process.

[0098] Based on the relevant information of the algorithms to be inserted, the dynamic insertion module can determine the number of algorithms to be inserted, the enabling scenarios of the algorithms to be inserted, and the required output format, etc.

[0099] After obtaining the relevant information of the algorithm to be inserted, the dynamic insertion module can execute step S502.

[0100] S502, The dynamic insertion module determines whether all algorithms to be inserted have been processed.

[0101] In a possible implementation, the dynamic insertion module can process one or more algorithms to be inserted by looping through them, determining the processing flow for each algorithm to be inserted. When the last algorithm to be inserted is reached and processed, it means that all algorithms to be inserted have been processed.

[0102] If not all algorithms to be inserted have been processed, the dynamic insertion module can execute step S503 to process each algorithm to be inserted.

[0103] If all algorithms for insertion have been processed, then the current iteration ends.

[0104] S503, The dynamic insertion module determines whether the algorithm to be inserted can be inserted into the processing flow.

[0105] The dynamic insertion module can determine whether the algorithm to be inserted can be inserted into the processing flow such as the photo taking process, the preview process, and the video recording process based on the enabling scenario of the algorithm to be inserted.

[0106] If the enabling scenario for the algorithm to be inserted is a photo-taking scenario, the algorithm can be inserted into the photo-taking process; if the enabling scenario is a preview scenario, the algorithm can be inserted into the preview process; if the enabling scenario is a video recording scenario, the algorithm can be inserted into the video recording process. Different enabling scenario identifiers can be used for different scenarios.

[0107] For example, in response to a user's triggering action, the camera application can determine the enabling scenario for the algorithm to be inserted. The user's triggering action may include touching, clicking, long-pressing, or swiping the screen of an electronic device, pressing a button on the electronic device, or using voice input to control the electronic device, etc., and this application embodiment does not limit the scope of the action.

[0108] Taking a scenario requiring watermarking as an example, when a user triggers a photo-taking action, the camera application can, in response to this action, pass relevant information about the watermarking algorithm to the dynamic insertion module. This information includes the function pointer corresponding to the watermarking algorithm, the number of input parameters for the function, the data type of the input parameters, and the output image format. It can also pass information such as the enabling scenario for the watermarking algorithm being a photo-taking scenario. Based on the enabling scenario of the watermarking algorithm, the dynamic insertion module can determine whether the watermarking algorithm is used in the photo-taking process.

[0109] If the watermarking algorithm needs to be inserted in both the preview and photo capture processes, the camera application can pass the relevant information for the watermarking algorithm twice to the dynamic insertion module. This information can include enabling scenarios for both the preview and photo capture scenarios. In other words, the dynamic insertion module can insert the same algorithm multiple times into different processing flows.

[0110] It is understandable that for algorithms with the same function, different formats need to be processed in different processing flows. For example, taking watermarking algorithms as an example, in the photo-taking process, adding a watermark involves processing YUV format images; in the thumbnail generation process, adding a watermark involves processing RGB format images; and in the preview process, adding a watermark involves processing UBWC format images.

[0111] Using image processing flow identifiers makes it easier to distinguish different image processing flows, facilitates the determination of which image processing flow the target algorithm should be inserted into, simplifies the execution flow of code in electronic devices, improves code readability and flexibility, and thus can concisely represent the flow into which the target algorithm should be inserted.

[0112] After determining the processing flow that the algorithm to be inserted needs to be processed, the dynamic insertion module can execute step S504.

[0113] If the processing flow to be inserted by the algorithm to be inserted is a processing flow that is not supported by the electronic device, or if the data corresponding to the enabling scenario of the algorithm to be inserted is incorrect, the dynamic insertion module can end the current loop, execute step S502, and continue to judge and process the next algorithm to be inserted.

[0114] Understandably, for algorithms that fail to be successfully inserted into the processing flow, the dynamic insertion module can send information to the camera application at the application layer or the perception engine at the HAL layer. This information indicates that the algorithm failed to be inserted. After the processing engine processes the image, it can pass the processed image to the camera application or the perception engine. Then, if the camera application or the perception engine determines that the function corresponding to the algorithm still needs to be implemented on the image, it can execute the algorithm on the processed image to achieve the algorithm's function.

[0115] In other words, for algorithms that fail to be successfully inserted into the processing flow, electronic devices can use existing image processing methods. This allows the image to still reflect the functionality corresponding to the failed algorithm, thereby improving the user experience.

[0116] In a possible implementation, the dynamic insertion module can save the relevant information of the algorithm to be inserted into an array or other data structure for easy use later.

[0117] S504. The dynamic insertion module determines whether the algorithm to be inserted has passed parameter validation.

[0118] The dynamic insertion module can validate the relevant parameters of the algorithm to be inserted. This includes validating the function pointer corresponding to the algorithm, the number of input parameters, the data types of the input parameters, and the output format of the function, to determine whether the parameters passed to the algorithm can be used in subsequent processing.

[0119] If the number or format of the parameters passed to the algorithm to be inserted is incorrect, it means that it cannot be used in the subsequent processing flow. In other words, the parameter validation fails, and the dynamic insertion module can end the current loop, execute step S502, and continue to judge and process the next algorithm to be inserted.

[0120] If the number or format of the parameters passed by the algorithm to be inserted is incorrect, it means that the parameter information passed by the algorithm to be inserted can be used in the subsequent processing flow. In other words, if the parameter verification passes, the dynamic insertion module can execute step S505.

[0121] The dynamic insertion module determines whether the algorithm to be inserted has passed parameter validation. Algorithms that fail validation are not inserted. This reduces unnecessary code execution and minimizes errors in subsequent execution processes, ensuring stable operation of the electronic device.

[0122] S505. The dynamic insertion module sets meta information and uses the algorithm to be inserted as the node to be inserted, thereby creating the execution node of the algorithm to be inserted.

[0123] Understandably, meta information can include information related to the image, providing additional information about it. For example, meta information may include one or more of the following: shooting parameters, device information, geographic location, timestamp, author information, copyright information, image description, etc.

[0124] Shooting parameters may include exposure time, aperture value, ISO sensitivity, focal length, etc. Device information may include the brand and model of the shooting device, such as detailed information about a camera or mobile phone. If the electronic device supports location services, the geographic location can be used to record the geographical coordinates at the time the image was taken, such as longitude and latitude. The timestamp may include the specific date and time the photo was taken. Author information may include the photographer's name. Copyright information may include a copyright notice or usage restrictions. The image description may include a brief description or title of the image content.

[0125] In this embodiment, the dynamic insertion module can set dependency information between algorithms in the meta information. After inserting the execution node of the algorithm to be inserted into the processing flow, the processing engine can execute the algorithm based on the dependency information. For example, assuming that algorithm A executes slowly in the photo-taking process, the electronic device can start a separate thread to execute algorithm A. Since the execution of algorithm B depends on the result of algorithm A, the dynamic insertion module can set the dependency information between algorithm A and algorithm B in the meta information. In this way, the electronic device can continue to execute algorithm B based on the result of algorithm A, so that the algorithms can be executed in an orderly manner.

[0126] The dynamic insertion module can also use the algorithm to be inserted as a node to be inserted, thereby creating an execution node for the algorithm to be inserted to load its information, facilitating initialization preparation for subsequent execution of the algorithm. In a possible implementation, the dynamic insertion module may create the execution node for the algorithm to be inserted during the createPlugin process; this embodiment of the application does not impose such limitations.

[0127] Understandably, after the dynamic insertion module determines the processing flow in which the algorithm to be inserted is to be inserted, the dynamic insertion module can further determine the position in the processing flow where the algorithm to be inserted is to be inserted.

[0128] Figure 6 This diagram illustrates how the dynamic insertion module determines the insertion position of the algorithm to be inserted in the processing flow.

[0129] S601, The dynamic insertion module determines whether all algorithms to be inserted have been processed.

[0130] In possible implementations, Figure 5After the loop traversal in the corresponding embodiment is completed, the dynamic insertion module can process one or more algorithms to be inserted again by loop traversal and determine the position of the one or more algorithms to be inserted in the processing flow. When the last algorithm to be inserted is traversed and processed, it means that all algorithms to be inserted have been processed.

[0131] If not all algorithms to be inserted have been processed, the dynamic insertion module can execute step S602 to process each algorithm to be inserted.

[0132] If all algorithms for insertion have been processed, then the current iteration ends.

[0133] It is understandable that the insertion algorithm traversed in step S601 is... Figure 5 In the corresponding embodiment, the algorithm to be inserted is verified by parameters, that is, the number of algorithms to be inserted that are traversed in step S502 is greater than or equal to the number of algorithms to be inserted that are traversed in step S601.

[0134] For ease of description, the loop traversal in step S502 will be referred to as the first loop traversal, and the loop traversal in step S601 will be referred to as the second loop traversal.

[0135] If all or more algorithms to be inserted in the first iteration pass parameter validation, then the number of algorithms to be inserted in the first iteration is equal to the number of algorithms to be inserted in the second iteration. If some of the algorithms to be inserted in the first iteration pass parameter validation, then the number of algorithms to be inserted in the first iteration is greater than the number of algorithms to be inserted in the second iteration.

[0136] For example, if there are 5 algorithms to be inserted in the first iteration, and all 5 algorithms pass the parameter validation, the dynamic insertion module will determine the insertion position for each of the 5 algorithms in the second iteration. If 4 of the 5 algorithms pass the parameter validation, the dynamic insertion module will determine the insertion position for each of the 4 valid algorithms in the second iteration.

[0137] S602, The dynamic insertion module obtains the input data type and output data type of the execution node of the algorithm to be inserted.

[0138] In a possible implementation, the dynamic insertion module obtains the input data type and output data type from a data structure such as an array that stores information about the algorithm to be inserted, and then executes step S603.

[0139] S603, The dynamic insertion module determines whether all nodes in the current processing flow have been traversed.

[0140] The dynamic insertion module can iterate through each node in the current processing flow to determine whether the algorithm to be inserted is suitable for insertion into the currently traversed node.

[0141] Understandably, the dynamic insertion module can execute the loop traversal process of step S603 during the second loop traversal of step S601. In other words, the dynamic insertion module can perform a double loop traversal. Thus, for each algorithm to be inserted, the dynamic insertion module will execute the loop traversal of step S603 to determine whether the current algorithm is suitable for insertion into the currently traversed node.

[0142] If the nodes in the current processing flow have not been traversed, the dynamic insertion module can execute steps S604 and S605 to find suitable nodes for insertion for the algorithm to be inserted.

[0143] If all nodes in the current processing flow have been traversed, it means that no suitable node has been found for the algorithm to be inserted. In this case, the dynamic insertion module can end the loop of step S603 and execute step S601 to continue to find a suitable insertion position for the next algorithm to be inserted.

[0144] In this embodiment, based on the traversal of each node in the processing flow and the judgment format requirements, the dynamic insertion module can determine whether it is suitable to insert the algorithm to be inserted in the processing flow, and the insertion position. In this way, the dynamic insertion module can flexibly insert the algorithm to be inserted according to the actual situation of different processing flows, without having to set the algorithm in a fixed position, thereby improving the flexibility of algorithm processing.

[0145] S604. The dynamic insertion module determines whether the output data of the currently traversed node meets the input data requirements of the node to be inserted by the algorithm.

[0146] S605. The dynamic insertion module determines whether the input data of the next node of the currently traversed node meets the output data requirements of the node to be inserted by the algorithm.

[0147] The output data of the currently traversed node must meet the input data requirements of the node to be inserted by the algorithm. This can include the number of output data of the currently traversed node being the same as the number of input data of the node to be inserted by the algorithm, and the data type of the output data of the currently traversed node being the same as the data type of the input data of the node to be inserted by the algorithm.

[0148] The input data of the next node after the currently traversed node must satisfy the output data requirements of the node to be inserted by the algorithm. This can include the number of input data of the next node after the currently traversed node being the same as the number of output data of the node to be inserted by the algorithm, and the data type of the input data of the next node after the currently traversed node being the same as the data type of the output data of the node to be inserted by the algorithm.

[0149] If the output data of the currently traversed node meets the input data requirements of the node to be inserted by the algorithm, and the input data of the next node of the currently traversed node meets the output data requirements of the node to be inserted by the algorithm, it means that the current traversed node and the next node of the current traversed node are suitable for the insertion algorithm, and the dynamic insertion module can execute step S606.

[0150] This ensures that data is processed normally in the image processing flow, reduces the occurrence of data processing anomalies, and allows the algorithm to be inserted into the appropriate position in the image processing flow.

[0151] If the output data of the currently traversed node does not meet the input data requirements of the node to be inserted by the algorithm, or if the input data of the next node to the currently traversed node does not meet the output data requirements of the node to be inserted by the algorithm, it means that the current node and the next node to the currently traversed node are not suitable for the insertion algorithm. In this case, the dynamic insertion module can execute step S603 to traverse the next node and find a suitable insertion position for the algorithm to be inserted.

[0152] It is understandable that the order in which steps S604 and S605 are executed is not important.

[0153] S606. The dynamic insertion module inserts the node to be inserted into the algorithm after the currently traversed node and updates the node list of the processing flow.

[0154] For example, taking the photo-taking process as an example, if the current traversal reaches the node of algorithm A, and the next node of the node of algorithm A is the node of algorithm B, and the output data of the node of algorithm A meets the input data requirements of the node of the algorithm to be inserted, and the input data of the node of algorithm B meets the output data requirements of the node of the algorithm to be inserted, the dynamic insertion module can insert the node of the algorithm to be inserted after the node of algorithm A and update the node list of the photo-taking process.

[0155] Understandably, for a given algorithm to be inserted, if the dynamic insertion module has already found a suitable node before the loop traversal in step S603 ends, then the loop in step S603 can end, and step S601 can be executed to continue searching for a suitable insertion position for the next algorithm to be inserted. This reduces unnecessary execution logic in electronic devices, saves computing power, and improves the performance of electronic devices.

[0156] The methods of this application will be described in detail below through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0157] Figure 7 An image processing method according to an embodiment of this application is illustrated. The method includes:

[0158] S701. Obtain information about the image to be processed. The information about the image to be processed includes the identifier of the image processing flow and the identifier of one or more preset algorithms. The one or more preset algorithms are not fixed in the image processing flow. The image processing flow includes at least one of the following flows set in the hardware abstraction layer: photo capture flow, preview flow, or video recording flow.

[0159] In this embodiment of the application, the image to be processed can be understood as an image that needs to undergo one or more preset algorithms. The preset algorithms can be understood as... Figure 5 The corresponding embodiment describes the algorithm to be inserted. The information of the image to be processed includes relevant information about the algorithm to be inserted. For example, the relevant information about the algorithm to be inserted includes the function pointer corresponding to the algorithm to be inserted, the number of input parameters of the function, the data type of the input parameters of the function, the output image format of the function, and the enabling scenario of the algorithm to be inserted.

[0160] The identifier of the image processing flow can be understood as Figure 5 The enabling scenario for the algorithm to be inserted in the corresponding embodiment. The enabling scenario for the algorithm to be inserted can be used to indicate in which processing flow the algorithm to be inserted is used, such as whether the algorithm to be inserted is used in the photo taking process, the preview process, or the video recording process, etc.

[0161] It is understood that the one or more preset algorithms may include algorithms set in the application layer, such as... Figure 5 The media algorithm set in the camera application of the corresponding embodiment may also include a general algorithm set in the perception engine module of the hardware abstraction layer.

[0162] S702. Set some or all of one or more preset algorithms in the image processing flow to obtain an updated image processing flow; wherein, for the target algorithm in the image processing flow set in the hardware abstraction layer of one or more preset algorithms, the output data format requirement of the algorithm adjacent to the target algorithm before the target algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the target algorithm after the target algorithm.

[0163] In this embodiment of the application, the setting in the image processing flow can also be understood as... Figure 6 In the corresponding embodiment, insertion into the processing flow means, for example, inserting the node corresponding to the algorithm to be inserted into the node list of the processing flow.

[0164] It is understandable that if one or more preset algorithms meet the setting conditions, then all of these preset algorithms can be set in the image processing flow; if any of the preset algorithms does not meet the setting conditions, then only a portion of these preset algorithms can be set in the image processing flow. The setting conditions include that the output data format requirements of the algorithm immediately preceding the target algorithm meet the input data format requirements of the target algorithm, and that the output data format requirements of the target algorithm meet the input data format requirements of the algorithm immediately following the target algorithm.

[0165] S703, Process the image to be processed based on the updated image processing workflow.

[0166] In this embodiment of the application, the updated image processing flow can also be understood as follows: Figure 6 The node list of the update process in the corresponding embodiment.

[0167] Understandably, based on the input and output data of one or more preset algorithms, these preset algorithms are inserted into appropriate positions within the image processing flow. This ensures that the output of the preceding node matches the input of the preset algorithm, and the output matches the input of the following node. In this way, the electronic device can flexibly determine the insertion position of one or more preset algorithms and can also support the insertion of one or more preset algorithms at intermediate nodes in the image processing flow.

[0168] Optional, in Figure 7 Based on the corresponding embodiments, the input data format requirements include the quantity requirements and data type requirements of the input data, and the output data format requirements include the quantity requirements and data type requirements of the output data.

[0169] In this embodiment, the specific input data format requirements and output data format requirements can be referred to... Figure 6The relevant descriptions in step S605 of the corresponding embodiment will not be repeated here.

[0170] Understandably, inserting algorithms according to the input and output data format requirements ensures that data can be processed normally in the image processing workflow, reduces the occurrence of data processing anomalies, and allows the algorithm to be inserted into the appropriate position in the image processing workflow.

[0171] Optional, in Figure 7 Based on the corresponding embodiments, setting some or all of one or more preset algorithms in the image processing flow may include: if the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the shooting scene, setting the target algorithm in the shooting flow; if the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the preview scene, setting the target algorithm in the preview flow; if the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the recording scene, setting the target algorithm in the recording flow.

[0172] In this embodiment of the application, the specific process of setting the corresponding process according to the identifier of the image processing flow can be referred to... Figure 5 The relevant descriptions in step S503 of the corresponding embodiment will not be repeated here.

[0173] Using image processing flow identifiers makes it easier to distinguish different image processing flows, facilitates the determination of which image processing flow the target algorithm should be inserted into, simplifies the execution flow of code in electronic devices, improves code readability and flexibility, and thus can concisely represent the flow into which the target algorithm should be inserted.

[0174] Optional, in Figure 7 Based on the corresponding embodiments, setting some or all of one or more preset algorithms in the image processing flow may include: for the target algorithm, traversing one or more algorithms fixedly set in the image processing flow; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm adjacent to the first algorithm, setting the target algorithm after the first algorithm in the image processing flow; wherein, the first algorithm is an algorithm fixedly set in the image processing flow.

[0175] In this embodiment of the application, the implementation of one or more algorithms fixedly set in the image processing flow can be referred to... Figure 6 The implementation details of step S603 in the corresponding embodiment, which iterates through each node in the current processing flow, will not be repeated here. For specific format requirements, please refer to... Figure 6 The relevant descriptions of steps S604 and S605 in the corresponding embodiments will not be repeated.

[0176] Based on the traversal of one or more algorithms fixed in the image processing flow and the judgment of format requirements, the electronic device can determine whether it is appropriate to insert a preset algorithm in the image processing flow and the insertion position. In this way, the electronic device can flexibly insert preset algorithms according to the actual situation in different image processing flows, without having to fix the algorithm settings, thereby improving the flexibility of algorithm processing.

[0177] Optional, in Figure 7 Based on the corresponding embodiments, setting the target algorithm before the first algorithm in the image processing flow may further include: setting target information and creating an execution node for the target algorithm, wherein the target information includes dependency information between the target algorithm and other algorithms in the image processing flow; setting the target algorithm after the first algorithm in the image processing flow may further include: executing the execution node of the target algorithm after the first algorithm in the image processing flow based on the dependency information.

[0178] In this embodiment of the application, the target information can be understood as... Figure 5 The meta information in step S505 of the corresponding embodiment. The execution node of the target algorithm can also be referenced. Figure 5 The relevant descriptions in step S505 of the corresponding embodiment will not be repeated here. In this way, the electronic device can continue to execute the target algorithm based on the target information and the result of the first algorithm, enabling the algorithms to be executed in an orderly manner.

[0179] Optional, in Figure 7 Based on the corresponding embodiments, the electronic device includes a first module, a second module, a third module, and a fourth module. Acquiring information about the image to be processed can include: obtaining the information from the application layer via the Hardware Abstraction Layer Interface Definition Language (HIDL) interface through the first module; obtaining the information from the application layer via the Camera Service through the second module and passing it to the third module; after acquiring the information, it can further include: determining the image processing flow based on the image processing flow identifier in the information of the image to be processed, and instructing the fourth module to create and initialize the image processing flow; creating and initializing the image processing flow through the fourth module, and loading one or more algorithms fixedly set in the image processing flow; setting some or all of the one or more preset algorithms in the image processing flow can include: for a target algorithm, traversing one or more algorithms fixedly set in the image processing flow through the first module; if the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, setting the target algorithm after the first algorithm in the image processing flow through the first module.

[0180] In this embodiment of the application, the first module can be understood as Figure 4 In the corresponding embodiment of the dynamic insertion module, the second module can be understood as... Figure 4 The third part of the perception engine in the corresponding embodiment can be understood as... Figure 4 In the corresponding embodiment of the decision engine, the fourth module can be understood as... Figure 4 For the processing engine in the corresponding embodiment, the interaction between specific modules can be found in the following examples. Figure 4 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0181] Electronic devices can use dynamic insertion modules to determine the processing flow to which each algorithm should be inserted and its position within that flow. This allows each algorithm to freely decide whether to insert itself into the current processing flow during runtime, improving the flexibility of code execution and thus better adapting to the functionalities provided by upper-layer applications.

[0182] Optional, in Figure 7 Based on the corresponding embodiments, the method may further include: during the process of traversing one or more algorithms fixedly set in the image processing flow, if there is no output data format requirement of any algorithm that satisfies the input data format requirement of the first preset algorithm, and the output data format requirement of the first preset algorithm satisfies the input data format requirement of any algorithm adjacent to the first preset algorithm, then the updated image processing flow does not include the first preset algorithm, and the first preset algorithm is an algorithm among one or more preset algorithms; after processing the image to be processed based on the updated image processing flow, the method may further include: passing the processed image to be processed to the application layer; and executing the first preset algorithm on the processed image to be processed in the application layer.

[0183] In this embodiment of the application, for cases where the format requirements are not met, please refer to... Figure 4 The relevant descriptions in the corresponding embodiments will not be repeated. It is understood that for algorithms that are not successfully inserted into the image processing flow, the electronic device can use previous image processing methods. This allows the image to still reflect the function corresponding to the unsuccessfully inserted algorithm, thereby improving the user experience.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0185] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] This application embodiment can divide the apparatus for implementing the method into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0187] like Figure 8 The diagram shows a chip structure according to an embodiment of this application. The chip 800 includes one or more processors 801, communication lines 802, communication interfaces 803, and memory 804.

[0188] In some implementations, memory 804 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0189] The methods described in the embodiments of this application can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 801 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0190] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 804, and processor 801 reads the information in memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0191] The processor 801, memory 804 and communication interface 803 can communicate with each other through communication line 802.

[0192] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0193] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0194] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0195] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0196] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. An image processing method applied to electronic devices, characterized in that, The method includes: Obtain information about the image to be processed, the information of the image to be processed includes the identifier of the image processing flow and the identifier of one or more preset algorithms. The one or more preset algorithms are not fixed in the image processing flow. The image processing flow includes at least one of the following flows set in the hardware abstraction layer: a photo capture flow, a preview flow, or a video recording flow. By setting some or all of the one or more preset algorithms in the image processing flow, an updated image processing flow is obtained; wherein, for the target algorithm of the image processing flow set in the hardware abstraction layer among the one or more preset algorithms, the output data format requirements of the algorithm adjacent to the target algorithm before the target algorithm meet the input data format requirements of the target algorithm, and the output data format requirements of the target algorithm meet the input data format requirements of the algorithm adjacent to the target algorithm after the target algorithm; The image to be processed is processed based on the updated image processing flow; The step of setting some or all of the one or more preset algorithms in the image processing flow includes: For the target algorithm, iterate through one or more algorithms that are fixedly set in the image processing flow; If the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, the target algorithm is set after the first algorithm in the image processing flow; wherein, the first algorithm is an algorithm that is fixedly set in the image processing flow.

2. The method according to claim 1, characterized in that, The input data format requirements include the quantity requirements and data type requirements of the input data, and the output data format requirements include the quantity requirements and data type requirements of the output data.

3. The method according to claim 1, characterized in that, The step of setting some or all of the one or more preset algorithms in the image processing flow includes: If the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the shooting scene, then the target algorithm is set in the shooting flow; If the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the preview scene, then the target algorithm is set in the preview flow; If the identifier of the image processing flow in the information of the target algorithm is the identifier corresponding to the recording scene, then the target algorithm is set in the recording flow.

4. The method according to any one of claims 1-3, characterized in that, The step of setting the target algorithm before the first algorithm in the image processing flow further includes: Set target information and create execution nodes for the target algorithm, wherein the target information includes dependency information between the target algorithm and other algorithms in the image processing flow; The step of setting the target algorithm after the first algorithm in the image processing flow further includes: Based on the dependency information, the execution node of the target algorithm is executed after the first algorithm in the image processing flow.

5. The method according to any one of claims 1-3, characterized in that, The electronic device includes a first module, a second module, a third module, and a fourth module. The acquisition of information about the image to be processed includes: Based on the first module, information about the image to be processed is obtained from the application layer through the Hardware Abstraction Layer Interface Definition Language (HIDL) interface. Based on the second module, the information of the image to be processed is obtained from the application layer through the CameraService and then passed to the third module; After obtaining the information of the image to be processed, the process further includes: Based on the identifier of the image processing flow in the information of the image to be processed, the third module determines the image processing flow and instructs the fourth module to create and initialize the image processing flow. The image processing flow is created and initialized based on the fourth module, and one or more algorithms that are fixedly set in the image processing flow are loaded. The step of setting some or all of the one or more preset algorithms in the image processing flow includes: For the target algorithm, the first module iterates through one or more algorithms fixedly set in the image processing flow; If the output data format requirement of the first algorithm meets the input data format requirement of the target algorithm, and the output data format requirement of the target algorithm meets the input data format requirement of the algorithm immediately following the first algorithm, the target algorithm is set after the first algorithm in the image processing flow based on the first module.

6. The method according to claim 1, characterized in that, The method further includes: During the process of traversing the one or more algorithms fixed in the image processing flow, if there is no output data format requirement of any algorithm that satisfies the input data format requirement of the first preset algorithm, and the output data format requirement of the first preset algorithm satisfies the input data format requirement of the algorithm adjacent to the first preset algorithm, then the updated image processing flow does not include the first preset algorithm, and the first preset algorithm is an algorithm among the one or more preset algorithms. After processing the image to be processed based on the updated image processing flow, the process further includes: The processed image to be processed is then passed to the application layer. At the application layer, the first preset algorithm is executed on the processed image to be processed.

7. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-6.

8. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-6.

Images