Image processing method and electronic device

Abstract

The present application provides an image processing method and an electronic device, which relate to the field of terminal technology, and the electronic device includes an electronic device that responds to a first operation of a user to obtain first image information to be processed. The electronic device generates a first algorithm call request based on the first image information; wherein the first algorithm call request indicates the first image processing algorithm required for the first image information under the corresponding image processing requirements. According to the first algorithm call request, the electronic device calls the first image processing algorithm from the preset image processing algorithms to perform image processing on the first image information. Among them, the preset image processing algorithms include multiple image processing algorithms, and different image processing algorithms correspond to different image processing requirements. This solution can realize flexible customization of image algorithms according to different image processing requirements of image information, thereby improving image processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and electronic device. Background Art

[0002] The anti-shake effect is one of the important indicators for evaluating the quality of video recording. The Global Motion Estimation (GME) module is provided in the native platform of electronic devices, which can be used to help achieve inter-frame anti-shake during video recording. It is an important functional module for achieving the anti-shake effect. At present, the GME module is used in ordinary video scenes, professional video scenes, movie mode scenes, etc. The native platform refers to the platform provided by the chip manufacturer of the electronic device and has not been modified by a third-party manufacturer.

[0003] However, in actual use and debugging, since the built-in algorithms in the GME module of the native platform are fixed, it is difficult to adapt the parameters for specific image processing scenarios, resulting in the inability to meet the needs of users in different image processing scenarios. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and an electronic device, which can realize flexible customization of image algorithms according to different image processing requirements and improve image processing efficiency.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which can be applied to an electronic device, and the method includes: the electronic device obtains first image information to be processed in response to a first operation of a user. The electronic device generates a first algorithm call request based on the first image information. The first algorithm call request indicates the first image processing algorithm required for the first image information under the corresponding image processing requirement. The electronic device calls the first image processing algorithm from the preset image processing algorithms to perform image processing on the first image information according to the first algorithm call request. The preset image processing algorithms include multiple image processing algorithms, and different image processing algorithms correspond to different image processing requirements.

[0007] In this solution, the electronic device starts to acquire the image information to be processed in response to the user's operation. The user's operation here may be an operation triggered by the user to enter video recording. The electronic device may generate an algorithm call request based on the acquired image information to be processed, and indicate the image processing algorithm corresponding to the image information in the algorithm call request, that is, the algorithm call request may include identification information corresponding to the image processing algorithm. By adopting the above technical solution, the electronic device can flexibly call the corresponding graphics processing algorithm according to the actual image processing requirements of the image information to be processed, thereby realizing flexible customization of the image processing algorithm and improving the image processing efficiency. The image processing requirement here may be an anti-shake requirement, and the image processing algorithm is an anti-shake algorithm.

[0008] In a possible implementation of the first aspect, the electronic device includes a GME module, an adaptation layer and an algorithm layer, the algorithm layer includes a preset image processing algorithm, and the adaptation layer is used to receive an algorithm call request from the GME module and adapt to the image processing algorithm corresponding to the algorithm call request in the algorithm layer. The electronic device calls a first image processing algorithm from a preset image processing algorithm to perform image processing on the first image information according to the first algorithm call request, including: the GME module sends the first algorithm call request to the algorithm layer by calling the adaptation layer. The algorithm layer calls the first image processing algorithm to perform image processing on the first image information according to the first algorithm call request.

[0009] In this solution, the electronic device can construct an adaptation layer and an algorithm layer for the GME module. The algorithm layer can be used to receive the call of the adaptation layer and implement the call of the corresponding image processing algorithm. The mapping relationship between image information and image processing algorithms is pre-stored in the GME module, and the mapping relationship indicates the image processing algorithm required for each image information under the corresponding image processing requirements.

[0010] Among them, the GME module, the adaptation layer and the algorithm layer are in the hardware abstraction layer of the electronic device. After receiving the image information to be processed, the GME module of the electronic device generates the above-mentioned algorithm call request, and calls the GME module, the adaptation layer and the algorithm layer in sequence according to the algorithm call request, and calls the first image processing algorithm in the algorithm layer to perform image processing on the first image information.

[0011] In a possible implementation of the first aspect, a first interface layer is included between the GME module and the adaptation layer, and a second interface layer is included between the adaptation layer and the algorithm layer. The GME module sends a first algorithm call request to the algorithm layer by calling the adaptation layer, including: the GME module calls the first interface layer according to the algorithm identifier, sends the first algorithm call request to the adaptation layer, the adaptation layer receives the first algorithm call request, and calls the second interface layer to send the first algorithm call request to the algorithm layer. The GME module communicates with the adaptation layer through the first interface layer, and the adaptation layer communicates with the algorithm layer through the second interface layer.

[0012] In a possible implementation of the first aspect, the first algorithm call request includes an algorithm identifier corresponding to the first image processing algorithm, the GME module includes a first native interface, the adaptation layer includes a second native interface, an adapter and a first algorithm interface, and the algorithm layer includes a second algorithm interface. Among them, the first native interface is connected to the second native interface and belongs to the first interface layer; the first algorithm interface is connected to the second algorithm interface and belongs to the second interface layer. Among them, the above-mentioned GME module calls the first interface layer according to the algorithm identifier, sends the first algorithm call request to the adaptation layer, the adaptation layer receives the first algorithm call request, and calls the second interface layer to send the first algorithm call request to the algorithm layer, including: the GME module calls the first target native interface in the first native interface according to the algorithm identifier to send the first algorithm call request to the adapter; the adapter adapts the first target algorithm interface in the first algorithm interface to the second target algorithm interface in the second algorithm interface through the second target native interface in the second native interface; the adapter sends the first algorithm call request to the algorithm layer through the second target algorithm interface. Among them, the first target native interface, the second target native interface, the first target algorithm interface and the second target algorithm interface all correspond to the algorithm identifier.

[0013] It can be understood that a first native interface, a second native interface, a first algorithm interface and a second algorithm interface corresponding to a first image processing algorithm. The first native interface, the second native interface, the first algorithm interface and the second algorithm interface can be respectively provided with an interface identifier. If the interface identifier is the same as or corresponds to the algorithm identifier, then the first native interface, the second native interface, the first algorithm interface and the second algorithm interface corresponding to the interface identifier are the first native interface, the second native interface, the first algorithm interface and the second algorithm interface corresponding to the first image processing algorithm corresponding to the algorithm identifier.

[0014] It can be understood that the first native interface and the second native interface are the same interface, and the first algorithm interface and the second algorithm interface are the same interface. The above-mentioned GME module calls the first interface layer according to the algorithm identifier, sends a first algorithm call request to the adaptation layer, the adaptation layer receives the first algorithm call request, and calls the second interface layer to send the first algorithm call request to the algorithm layer, including: the GME module calls the first target native interface in the first native interface according to the algorithm identifier to send the first algorithm call request to the adapter, the adapter adapts to the first target algorithm interface in the first algorithm interface; the adapter sends the first algorithm call request to the algorithm layer through the first target algorithm interface.

[0015] In a possible implementation of the first aspect, the preset image processing algorithm includes an image processing algorithm and / or an algorithm data package; the algorithm data package includes at least two image processing algorithms. The image processing algorithm in the algorithm layer is determined according to the image processing requirements, and the image information can call one or more image processing algorithms according to different image processing requirements. The preset image processing algorithm in the algorithm layer may include a single encapsulated image processing algorithm, or an algorithm data package integrating multiple image processing algorithms.

[0016] In a possible implementation of the first aspect, after the electronic device calls the first image processing algorithm from the preset image processing algorithms to perform image processing on the first image information according to the first algorithm call request, the method further includes: the electronic device obtains the second image information and sends the second image information to the image processing engine. The second image information is the image information obtained by the electronic device calling the first image processing algorithm to perform image processing on the first image information. The electronic device displays the second image information processed by the image processing engine. The second image information includes the image processing calculation result after calling the first image processing algorithm to perform image processing on the first image information, and the first image information. After the image processing engine performs image processing on the first image information according to the image processing calculation result, the processed second image information is obtained.

[0017] In a possible implementation manner of the first aspect, first image information to be processed is obtained, and the method includes: the electronic device obtains the first image information to be processed after performing an original image acquisition operation, an image signal front-end processing operation, and a spatial alignment transformation processing operation.

[0018] In a possible implementation of the first aspect, the method further includes: the GME module of the electronic device pre-stores a mapping relationship. The mapping relationship is used to indicate the relationship between the image information and the image processing algorithm. Before the electronic device leaves the factory, the method in the prior art can be used to determine the corresponding image processing requirements according to the image information, and determine which image processing algorithms need to be used to perform image processing on the image information according to the image processing requirements.

[0019] In a possible implementation of the first aspect, the electronic device generates a first algorithm call request based on the first image information, including: a GME module of the electronic device queries a mapping relationship based on the first image information. In response to the query result, the first algorithm call request is generated. The query result indicates that the first image information corresponds to the first image processing algorithm.

[0020] In a second aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory and the processor are coupled; wherein the memory is used to store computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes any one of the methods described in the first aspect above.

[0021] In a possible implementation of the second aspect, an electronic device provided by the present application includes: a camera, a processor, a display screen, a memory and a communication module; the camera, the display screen, the memory, the communication module and the processor are coupled; wherein the memory is used to store computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes any one of the methods described in the first aspect above.

[0022] In a third aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods described in the first aspect.

[0023] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0024] It can be understood that the electronic device described in the second aspect provided above, the computer-readable storage medium described in the third aspect and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a shooting scene of a mobile phone provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0028] Figure 4 A flowchart for implementing image processing of an electronic device provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the architecture of an anti-shake algorithm module provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the architecture of another anti-shake algorithm module provided in an embodiment of the present application;

[0031] Figure 7 It is a schematic diagram of the adaptation of a customized algorithm and a GME module provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The following terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features.

[0034] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0035] like Figure 1 As shown in the shooting scene diagram, in the shooting interface of the mobile phone, the mobile phone responds to the user's touch operation on the record button and starts recording the video. At present, the anti-shake effect is one of the important indicators for evaluating the quality of video recording. In order to achieve inter-frame anti-shake during video recording, a GME module is provided in the native platform of the mobile phone. The GME module can output the motion compensated temporal filter (Motion Compensated Temporal Filter, MCTF) alignment matrix and distortion correction grid. The recorded video is anti-shake processed with the help of the MCTF alignment matrix and distortion correction grid output by GME. The GME module is an important functional module for achieving anti-shake effect. The GME module is used in ordinary video scenes, professional video scenes, movie mode scenes, etc.

[0036] In actual applications, since the algorithms built into the native GME modules in electronic devices are fixed, they are generally a whole algorithm data package. When electronic devices call the GME module for image processing, they call the whole algorithm data package built into the GME module, which makes it difficult to adapt parameters for specific image processing scenarios, resulting in the inability to call the algorithm according to the actual needs of different image processing scenarios. Therefore, in some image processing scenarios, it may only be necessary to call a part of the algorithm, and it is not necessary to call the entire algorithm data package. Then, the above-mentioned overall call to the algorithm data package will cause a waste of algorithm calls and lead to low image processing efficiency.

[0037] The embodiment of the present application provides an image processing method, which can be applied to electronic devices. In the image processing method, the electronic device can pre-set the corresponding algorithm according to different image processing requirements, and in subsequent applications, determine and execute the corresponding algorithm according to the acquired image processing requirements. As a result, the electronic device can flexibly call the corresponding algorithm according to the actual image processing requirements, realize flexible customization of the algorithm, and improve image processing efficiency.

[0038] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a smart watch, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, and other devices including a camera. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0039] The execution subject of the image processing method provided in the embodiment of the present application may be an image processing device, and the execution device may be Figure 2 The electronic device shown. At the same time, the execution device can also be a central processing unit (CPU) of the electronic device, or a control module for image processing in the electronic device. In the embodiment of the present application, the image processing method provided by the embodiment of the present application is described by taking the execution of the image processing method by the electronic device as an example.

[0040] The following will describe in detail the implementation of the embodiment of the present application in conjunction with the accompanying drawings. Figure 2The figure shows a schematic diagram of the structure of an electronic device 200 provided in an embodiment of the present application. The electronic device 200 may include a processor 210, a memory 220, a camera 230, a display screen 240, a communication module 250, a motor 260, an audio module 270, a sensor module 280, a button 290, an external memory interface, etc. The sensor module 280 may include a gyroscope sensor 280A, an acceleration sensor 280B, a distance sensor 280C, a proximity light sensor 280D, an ambient light sensor 280E, etc.

[0041] The structure illustrated in the embodiment of the present application does not constitute a limitation on the electronic device 200. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0043] The memory 220 is used to store instructions and data. The memory 220 can be set in the processor 210, or can be deployed independently from the processor 210 without limitation. In the embodiment where the memory 220 is set in the processor 210, the memory 220 in the processor 210 is a cache memory, which can save instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it can be directly called from the memory 220. Repeated access is avoided, the waiting time of the processor 210 is reduced, and the efficiency of the system is improved.

[0044] The memory 220 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory. The memory 220 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the memory 220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash memory (Universal Flash Storage, UFS), etc.

[0045] In some embodiments, the processor 210 may include an interface. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc. The specific functions of these interfaces and the connected modules can be referred to the existing ones and will not be described in detail.

[0046] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 200. The electronic device 200 may adopt different interface connection methods or a combination of multiple interface connection methods in the embodiment of the present invention.

[0047] The electronic device 200 can implement a shooting function through an ISP, a camera 230, a video codec, a GPU, a display screen 240, and an application processor.

[0048] The ISP is used to process the data fed back by the camera 230. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and chromaticity of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 230.

[0049] The camera 230 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 230, where N is a positive integer greater than 1.

[0050] The electronic device 200 implements the display function through a GPU, a display screen 240, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0051] The display screen 240 is used to display images, videos, etc. The display screen 240 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0052] The wireless communication function of the electronic device 200 can be implemented by an antenna, a radio frequency module, a communication module 250, a modem, and a baseband processor. The antenna is used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna.

[0053] Motor 260 can generate vibration prompts. Motor 260 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations acting on different areas of the display screen 240 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization. In addition, the motor can also be used in the camera zoom shooting scene to achieve lens switching of different focal lengths by driving the movement of the lens.

[0054] The electronic device 200 can implement audio functions, such as voice-controlled photography, music playback and recording, etc., through the speaker, receiver, microphone, headphone interface, and application processor of the audio module 270.

[0055] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0056] The sensor module 280 may include a gyro sensor 280A, an acceleration sensor 280B, a distance sensor 280C, a proximity light sensor 280D, and an ambient light sensor 280E. The gyro sensor 280A and the acceleration sensor 280B may be used for posture correction or horizontal positioning during the photographing process, the distance sensor 280C may be used for depth of field determination or image processing during the photographing process, and the proximity light sensor 280D and the ambient light sensor 280E may be used for sensing ambient light and adjusting image brightness values ​​during the photographing process.

[0057] The key 290 includes a power key, a volume key, etc. The key 290 may be a mechanical key or a touch key. The electronic device 200 receives the key 290 input and generates a key signal input related to the user settings and function control of the electronic device 200.

[0058] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface to implement a data storage function, such as storing music, video and other files in the external memory card.

[0059] The image processing methods in the aforementioned embodiments can all be implemented in the electronic device 200 having the aforementioned hardware structure.

[0060] like Figure 3 The figure shows the software architecture of the electronic device. The following will explain the image processing process of the electronic device in detail in combination with the internal architecture of the electronic device.

[0061] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the internal architecture of the electronic device can be divided into four layers, from top to bottom, namely, the application layer (Application, App), the application framework layer (Framework, FWK), the hardware abstraction layer (HardwareAbstraction Layer, HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included without limitation.

[0062] The application layer can include a series of application packages, such as camera applications, gallery applications, applications with camera functions, etc. Of course, when other applications need to use the shooting function, they can also call the camera application to implement the shooting function. The application package can also include applications such as calls, calendars, maps, navigation, music, videos, and short messages.

[0063] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0064] In the embodiments of the present application, Figure 3 As shown, taking the camera application as an example, a camera service (Camera Service) can be set in the application framework layer. The camera application can start the camera service by calling a preset API to implement functions related to shooting. During operation, the camera service can interact with the camera hardware call module (Camera HAL) in the hardware abstraction layer (HAL).

[0065] In addition, the application framework layer can also include window managers, content providers, view systems, resource managers, notification managers, etc.

[0066] The kernel layer is a layer between hardware (e.g., camera) and software (e.g., hardware abstraction layer). The kernel layer includes at least a camera driver. The camera driver can be used to drive a hardware module with a shooting function, such as a camera sensor (CameraSensor). In other words, the camera driver is responsible for data interaction with the camera sensor in the camera. The kernel layer may also include a display driver, an audio driver, a sensor driver, etc., and the embodiments of the present application do not impose any restrictions on this.

[0067] The hardware abstraction layer can encapsulate the drivers in the kernel layer and provide a calling interface to the framework layer, shielding the implementation details of the underlying hardware. Figure 3 As shown, the hardware abstraction layer may include a camera hardware call module (Camera HAL), etc. Among them, the Camera HAL is responsible for interacting with the hardware device (such as a camera) that implements the shooting function in the mobile phone. On the one hand, the Camera HAL hides the implementation details of the relevant hardware devices (such as specific image processing algorithms), and on the other hand, it can provide the system with an interface for calling the relevant hardware devices.

[0068] Camera HAL is the core software framework of the camera, which includes an interface module, a sensor node, and a data processing module. In one implementation, the data processing module may include an image signal front-end processing (IFE) module, a spatial alignment transform (SAT) processing module, a global motion estimation module, and an image processing engine. The sensor node, data processing module, and interface module are components in the image data and control instruction transmission pipeline in the camera call processing module.

[0069] Specifically, the sensor node can be a control node for the camera sensor, and the sensor node can control the camera sensor through the camera driver. The interface module can be a software interface for the application framework layer, which is used to interact with the application framework layer for data. Of course, the interface module can also interact with the data processing module, sensor node, etc. in the camera call processing module. The data processing module can process the original image data returned by the camera sensor, wherein the image signal front-end processing module is used to perform white balance, de-mosaicing, Gamma correction and other operations on the preview image collected by the camera sensor, and output the YUV image. The spatial alignment transformation processing module is used to perform spatial alignment on the image data according to the spatial alignment transformation processing algorithm, analyze and calculate the image data to obtain a multi-camera consistency matrix, which can be used for smooth switching of multiple cameras to make the preview image and video image smoother. The global motion estimation module is used to call the corresponding algorithm interface according to the business requirements of the received image data, adapt to the interface in the corresponding custom algorithm through the adapter, and then execute the corresponding algorithm to output the algorithm calculation results and image data. The image processing engine is used to align the image according to the image data output by the global motion estimation module and the algorithm calculation results, and perform operations such as twisting and cropping to form the final display image and return it to the camera application.

[0070] Understandably, Figure 3 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0071] In addition, it is understood that in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0072] In the above Figure 3 Based on the internal architecture diagram shown, Figure 4 The camera application and related supporting modules in the electronic device are retained, illustrating the image processing process of the electronic device.

[0073] For example, when the camera application is running, the relevant control commands issued by the user (such as preview, zoom, photo, and video instructions) can be sent to the camera service. On the one hand, the camera service can send the received control commands to the Camera HAL, so that the Camera HAL can call the camera driver in the kernel layer according to the received control commands, and the camera driver drives the camera and other hardware devices to respond to the control commands to collect image data. For example, the camera can pass each frame of image data collected to the Camera HAL through the camera driver at a certain frame rate. Among them, the transmission process of control commands within the operating system can be seen in Figure 4 The specific transmission process of control flow.

[0074] Among them, the process of image processing by electronic equipment can be mainly divided into two parts: top-down signal flow ( Figure 4 The electronic device receives the preset operation of the user and generates a corresponding control signal to control the relevant modules to perform the image acquisition operation to obtain the image. The preset operation here can refer to the user's touch operation on the recording button. Figure 4 The camera sensor of the electronic device acquires the original image and obtains the anti-shake processed image after image processing, and sends the anti-shake processed image to the camera application of the application layer for display.

[0075] like Figure 4 The signal flow shown by the solid arrow in the middle mainly includes: the user applies the recording operation by clicking, sliding or other predefined operation methods in the shooting preview interface of the electronic device. The touch module of the electronic device responds to the recording touch operation and generates a corresponding touch event and sends it to the camera application of the application layer. After receiving the touch event, the camera application sends the touch event to the camera service of the application framework layer. The camera service generates a recording request and sends the recording request to the camera hardware call module in the hardware abstraction layer. The recording request is sent to the sensor node. According to the recording request, the sensor node controls the camera sensor through the camera driver to perform the image output operation.

[0076] like Figure 4The data flow shown by the dotted arrow in the figure mainly includes: after the camera sensor completes the image output operation, the raw image data corresponding to the recording request is sent to the image signal front-end processing module through the camera driver. The image signal front-end processing module performs preliminary cropping processing on the raw image data and sends the processed image data to the spatial alignment transformation processing module. Among them, the image signal front-end processing module can output a YUV image so that the spatial alignment transformation processing module can perform spatial alignment processing on the image. The spatial alignment transformation processing module performs image spatial alignment processing according to the relevant spatial alignment transformation processing algorithm, analyzes and calculates the image data to obtain a multi-camera consistency matrix. The multi-camera consistency matrix can be used to output the smoothness of the image when multiple cameras are switched. The spatial alignment transformation processing module sends the image data to the global motion estimation module. The global motion estimation module calls the corresponding algorithm interface according to the image data output by the spatial alignment transformation processing module, adapts to the interface in the corresponding custom algorithm through the adapter, and then executes the corresponding algorithm and outputs the algorithm calculation result. The image processing engine is used to align the image according to the image data output by the global motion estimation module and the algorithm calculation results, and perform operations such as twisting and cropping to form the final display image, and return it to the camera application at the application layer for preview display.

[0077] It should be noted that in the process of the original image output from the camera sensor passing through the image signal front-end processing module, the spatial alignment transformation processing module, the global motion estimation module and the image processing engine to the display, in addition to the data stream of image data transmission, a signal stream containing various data processing instructions and parameters is also transmitted.

[0078] The methods in the following embodiments can all be implemented in an electronic device having the above hardware structure and the above software architecture. An image processing method provided by an embodiment of the present application is described in detail below. The image processing method can specifically include steps S501-S503.

[0079] S501: The electronic device obtains first image information to be processed in response to a first operation of a user.

[0080] In the embodiment of the present application, the electronic device is taken as a mobile phone as an example. The mobile phone responds to the user's operation on the camera application and opens the camera application. After the mobile phone opens the camera application, it can enter the video recording scene in response to the user's touch operation on the video recording button in the camera application interface. Among them, the user's first operation may include the user's opening operation of the camera application and the touch operation of the video recording button. The first operation may also be other operations of the user, as long as it can put the electronic device in the video recording scene. In other words, the first operation is an operation that triggers the electronic device to enter video recording.

[0081] First, before the video recording starts, the mobile phone needs to complete the application and initialization operations of related resources. For example, the related resources here can include memory. Specifically, the image-related data will apply for some memory to complete the reversal of the entire image processing process. Among them, the initialization operation refers to setting initial values ​​for the above-mentioned related resources. For example, if the mobile phone applies for a piece of memory to save picture information, then the memory is initialized, that is, all the contents of the memory are set to the initial value, such as 0, to facilitate the subsequent image processing process.

[0082] After completing the application and initialization operations of the above-mentioned related resources, exemplarily, the electronic device sends a control instruction to the camera application in response to the user's first operation. The camera application sends the received control instruction to the application framework layer, the hardware abstraction layer and the kernel layer in sequence. The control instruction here refers to the one generated according to the user's first operation. For example, the electronic device generates a control instruction corresponding to the operation in response to the user triggering the electronic device to enter video recording. The control instruction is used to control the electronic device to enter video recording. Specifically, the camera application sends the control instruction to the camera service of the application framework layer, and the camera service can send the received control command to the CameraHAL of the hardware abstraction layer, so that the Camera HAL can call the camera driver in the kernel layer according to the received control command, and the camera driver drives the camera and other hardware devices to respond to the control command to collect image data.

[0083] Then, the camera sensor Sensor of the kernel layer starts to output images. After the camera sensor completes the image output operation, the original Raw image information is sent to the image signal front-end processing module of the hardware abstraction layer through the camera driver.

[0084] The image signal front-end processing module performs white balance, de-mosaicing, gamma correction and other operations on the original image information, and outputs the YUV image to the spatial alignment transformation module. The image signal front-end processing module performs white balance, de-mosaicing, and gamma correction operations on the original image information. The image can be white balanced, de-mosaiced, and gamma corrected by methods in the prior art, which will not be described in detail in this application.

[0085] After receiving the YUV image, the spatial alignment transformation module analyzes and calculates the YUV image to obtain a multi-camera consistency matrix, which can be used for smooth switching of multiple cameras. Then, the spatial alignment transformation module passes the output image information to the GME module. Among them, the above-mentioned spatial alignment transformation module analyzes and calculates the YUV image to obtain a multi-camera consistency matrix. The YUV image can be analyzed and calculated by the method in the prior art to obtain a multi-camera consistency matrix, which is not repeated in this application.

[0086] In the embodiment of the present application, the first image information refers to image information processed by the image signal front-end processing module and the space alignment transformation processing module.

[0087] Further, in S501, after the electronic device obtains the first image information, the first image information is input into the global motion estimation module for subsequent operations. The global motion estimation module is a native module in the electronic device, and has a native algorithm data package built in. The native module refers to a module set in the native platform of the electronic device and has not been modified by a third-party manufacturer.

[0088] S502: The electronic device generates a first algorithm calling request based on the first image information.

[0089] The first algorithm calling request indicates the first image processing algorithm required for the first image information under the corresponding image processing requirement. The electronic device can call the corresponding first image processing algorithm according to the first algorithm calling request.

[0090] It is understandable that different image information may correspond to different image processing requirements, and image processing requirements refer to which image processing algorithms need to be used to perform image processing on the image information. The image processing requirement may be an anti-shake requirement. In an embodiment of the present application, before the electronic device leaves the factory, the method in the prior art may be used to pre-calculate which image processing algorithms need to be used to perform image processing on the image information based on the image information. Since the global motion estimation module is applied to video stabilization, the image processing algorithms here mainly include anti-shake algorithms. Among them, the anti-shake algorithm may include an alignment matrix algorithm, a correction grid algorithm, and the like. It should be noted that the anti-shake algorithm may be any anti-shake algorithm in the prior art, or an anti-shake algorithm developed by the manufacturer.

[0091] It is understandable that in the prior art, there is already a complete judgment logic that can calculate what anti-shake requirements the image information will have and what corresponding anti-shake algorithms need to be called, and this application will not go into details here.

[0092] In some embodiments, before the electronic device leaves the factory, a large amount of image information and the corresponding anti-shake requirements can be obtained by performing preliminary image tests. The anti-shake requirement refers to which anti-shake algorithms need to be called to obtain the preset anti-shake effect. This preset anti-shake effect can be obtained through data statistics, and the anti-shake effect is higher than a preset anti-shake threshold. Through the above-mentioned image test results, a mapping relationship between image information and anti-shake algorithms is constructed. The electronic device can obtain the mapping relationship between the above-mentioned image information and the anti-shake algorithm before leaving the factory, and these anti-shake algorithms can be separately packaged and stored in the electronic device according to different anti-shake requirements. When the GME module of the electronic device receives the image information, it can call the anti-shake algorithm corresponding to the image information according to the above-mentioned mapping relationship, so as to meet different anti-shake requirements.

[0093] In some embodiments, the first image information may also include quality requirement information. For example, different selection controls corresponding to quality requirement information are preset in the mobile phone interface. The quality requirement information here may indicate the image processing algorithm that the user wants to perform on the shot video, that is, the algorithm identifier of the corresponding image processing algorithm is included in the quality requirement information. When shooting a video, the mobile phone can also obtain the corresponding quality requirement information in response to the user's touch operation on a selection control on the mobile phone interface. The GME module of the mobile phone can call the corresponding image processing algorithm according to the algorithm identifier of the image processing algorithm contained in the quality requirement information.

[0094] Among them, the above-mentioned calculated image information and the corresponding anti-shake algorithm can be constructed into a one-to-one mapping relationship. Before the electronic device leaves the factory, the mapping relationship between the pre-acquired image information and the anti-shake algorithm can be stored in the global motion estimation module of the electronic device. The mapping relationship between the above-mentioned image information and the anti-shake algorithm can be, for example, image information 1 corresponds to anti-shake algorithm 1, and image information 2 corresponds to anti-shake algorithm 2. It can be understood that if the first image information received by the global motion estimation module is image information 1, then the corresponding algorithm to be called is anti-shake algorithm 1. Among them, the embodiment of the present application does not limit the storage form and storage location of the above-mentioned mapping relationship. For example, the storage form can be as shown in Table 1 below.

[0095] First image information Anti-shake algorithm name Image information 1 Anti-shake algorithm 1 Image information 2 Anti-shake algorithm 2 Image information 3 Anti-shake algorithm 3 (anti-shake algorithm 1 + anti-shake algorithm 2) Image information 4 Anti-shake algorithm 4

[0096] Table 1

[0097] The first image information may correspond to an independent anti-shake algorithm or an algorithm data packet composed of multiple anti-shake algorithms. For example, in Table 1, the anti-shake algorithm 3 corresponding to the image information 3 is actually an algorithm data packet composed of the anti-shake algorithm 1 and the anti-shake algorithm 2.

[0098] In the embodiment of the present application, after the GME module of the electronic device obtains the first image information, the anti-shake algorithm corresponding to the first image information is determined according to the mapping relationship between the above image information and the anti-shake algorithm pre-stored in the GME module. Then, the GME module of the electronic device can generate an algorithm call request according to the first image information and the corresponding anti-shake algorithm.

[0099] S503: The electronic device calls a first image processing algorithm from preset image processing algorithms according to the first algorithm calling request to perform image processing on the first image information.

[0100] In an embodiment of the present application, an algorithm layer may be set in the electronic device, and the algorithm layer includes the above-mentioned preset image processing algorithm, that is, the preset anti-shake algorithm. Among them, the preset anti-shake algorithm includes multiple anti-shake algorithms, and different anti-shake algorithms correspond to different anti-shake algorithms. The GME module of the electronic device calls the first image processing algorithm from the preset image processing algorithms according to the first algorithm call request, that is, calls the first image processing algorithm in the preset image processing algorithms in the algorithm layer.

[0101] Furthermore, an adaptation layer may be provided in the electronic device to adapt the GME module to the algorithm layer, that is, to adapt to the corresponding anti-shake algorithm in the algorithm layer according to the algorithm call request after receiving the algorithm call request of the GME module, and call the algorithm layer. The algorithm layer is used to call the corresponding anti-shake algorithm after receiving the algorithm call request of the adaptation layer.

[0102] In an embodiment of the present application, in order to meet different anti-shake requirements corresponding to different image information, and on the basis of minimizing changes to the native architecture, the electronic device can build an adaptation module on the HAL software path to complete the seamless replacement of the internal algorithm of the GME module. That is, the first image information received by the GME module is calculated by the anti-shake algorithm preset in the algorithm layer. Among them, the anti-shake algorithm preset in the algorithm layer may be referred to as a customized algorithm in the embodiment of the present application. A customized algorithm refers to an anti-shake algorithm set in the algorithm layer according to different anti-shake requirements. It can be understood that a customized algorithm can be a separate anti-shake algorithm or an algorithm data packet integrated with multiple anti-shake algorithms. The embodiment of the present application does not limit the algorithm composition in the customized algorithm. The native architecture is the architecture provided by the native platform of the electronic device.

[0103] See also Figure 5 , Figure 5 Schematic diagram of the architecture of an anti-shake algorithm module provided in an embodiment of the present application. Figure 5As shown, the dotted box includes a GME module, an adaptation layer and an algorithm layer. Among them, the adaptation layer is used to receive the call of the GME module. The algorithm layer may include multiple independent anti-shake algorithms for receiving the call of the adaptation layer. When the algorithm layer receives the algorithm call request of the adaptation layer, it can call the anti-shake algorithm corresponding to the algorithm call request to perform image processing on the received image information. The algorithm call request may include the identification information of the anti-shake algorithm, and the algorithm layer can determine the anti-shake algorithm to be called according to the identification information in the algorithm call request.

[0104] Figure 5 In the figure, the direction of the arrow is the transmission direction of the data flow. Specifically, after receiving the first image information, the GME module transmits the first image information to the algorithm layer through the adaptation layer; the algorithm layer calls the algorithm corresponding to the first image information to perform image processing on the first image information to obtain the algorithm execution result; then, the algorithm execution result and the first image information are transmitted back to the GME module through the adaptation layer.

[0105] In some embodiments of the present application, a first interface layer may be provided between the GME module and the adaptation layer, for providing a calling interface to the GME module and calling the adaptation layer. The GME module communicates with the adaptation layer through the first interface layer. A second interface layer may be provided between the adaptation layer and the algorithm layer, for providing a calling interface to the adaptation layer and calling the algorithm layer. The adaptation layer communicates with the algorithm layer through the second interface layer. For a schematic diagram of the relationship between the GME module, the first interface layer, the adaptation layer, the second interface layer and the algorithm layer, please refer to Figure 6 .

[0106] like Figure 6 As shown in a, a first interface layer is set between the GME module and the adaptation layer, and a second interface layer is set between the adaptation layer and the algorithm layer. Figure 6 The direction of the arrow in a is the flow direction of the first image information.

[0107] like Figure 6 As shown in b, the first interface layer includes a first native interface and a second native interface. The second interface layer includes a first algorithm interface and a second algorithm interface. Among them, the first native interface is the interface of the GME module, the second native interface is the interface of the adaptation layer, and the first native interface corresponds to the second native interface. The first algorithm interface is the interface of the adaptation layer, the second algorithm interface is the interface of the algorithm layer, and the first algorithm interface corresponds to the second algorithm interface. Among them, the number of the first native interface, the second native interface, the first algorithm interface and the second algorithm interface is the same as the number of anti-shake algorithms in the algorithm layer, Figure 6The number of interfaces in b is only an example. Each anti-shake algorithm in the algorithm layer has a corresponding first native interface, a second native interface, a first algorithm interface and a second algorithm interface. Among them, the electronic device can set interface identifiers for the first native interface, the second native interface, the first algorithm interface and the second algorithm interface, respectively, and set the algorithm identifier for the anti-shake algorithm. If the algorithm identifier of the anti-shake algorithm corresponds to the interface identifier of the first native interface, the interface identifier of the second native interface, the interface identifier of the first algorithm interface and the interface identifier of the second algorithm interface, then the anti-shake algorithm corresponds to the first native interface, the second native interface, the first algorithm interface and the second algorithm interface. For example, anti-shake algorithm 1 corresponds to the first native interface 1, the second native interface 1, the first algorithm interface 1 and the second algorithm interface 1, the algorithm identifier of anti-shake algorithm 1 is 1, and the interface identifiers of the first native interface, the second native interface, the first algorithm interface and the second algorithm interface are all 1. The embodiment of the present application does not limit the specific form of the algorithm identifier and the interface identifier.

[0108] It can be understood that the first native interface and the second native interface can be substantially the same interface, and the first algorithm interface and the second algorithm interface can be substantially the same interface.

[0109] Among them, the electronic device calls the GME module according to the algorithm call request, that is, calls the corresponding first native interface (for example, the first native interface 1) in the GME module, and then adapts to the corresponding second algorithm interface (for example, the second algorithm interface 1) of the algorithm layer through the adaptation layer. Specifically, the electronic device calls the corresponding first native interface (for example, the first native interface 1) in the GME module, and reaches the adapter in the adaptation layer through the corresponding second native interface (for example, the second native interface 1) in the adaptation layer. The adapter determines the corresponding first algorithm interface (for example, the first algorithm interface 1) according to the second native interface (for example, the second native interface 1), and then determines the corresponding second algorithm interface (for example, the second algorithm interface 1) in the algorithm layer according to the first algorithm interface (for example, the first algorithm interface 1). As a result, the electronic device executes the corresponding anti-shake algorithm (for example, the anti-shake algorithm 1) to perform image processing on the first image information by calling the second algorithm interface (for example, the second algorithm interface 1) to obtain the algorithm execution result.

[0110] It can be understood that the anti-shake algorithm to be called can be known according to the algorithm call request. In the algorithm layer, the anti-shake algorithm is called by the electronic device in sequence according to the algorithm call request by calling the corresponding first native interface, second native interface, first algorithm interface and second algorithm interface, and finally calls the anti-shake algorithm. It can be seen that different anti-shake algorithms correspond to different native interfaces and algorithm interfaces. The mapping relationship between the anti-shake algorithm and the native interface and the algorithm interface is also pre-stored in the GME module of the electronic device, for example, as shown in Table 2 below.

[0111] Anti-shake algorithm name First native interface Second native interface First algorithm interface Second algorithm interface Anti-shake algorithm 1 Native interface 1 Native interface 1 Algorithm interface 1 Algorithm interface 1 Anti-shake algorithm 2 Native interface 2 Native interface 2 Algorithm interface 2 Algorithm interface 2 Anti-shake algorithm 3 Native interface 3 Native interface 3 Algorithm interface 3 Algorithm interface 3 Anti-shake algorithm 4 Native interface 4 Native interface 4 Algorithm interface 4 Algorithm interface 4

[0112] Table 2

[0113] After receiving the first image information, the GME module of the electronic device can determine the corresponding anti-shake algorithm through the pre-stored Table 1 above, and determine the corresponding first native interface, second native interface, first algorithm interface and second algorithm interface through the pre-stored Table 2 above. Therefore, after receiving the first image information, when the GME module of the electronic device generates an algorithm call request according to the first image information and the corresponding anti-shake algorithm, the algorithm call request may include the interface identifiers of the native interface and the algorithm interface corresponding to the anti-shake algorithm, and the interface identifier is the parameter indicating the interface. The electronic device can call the corresponding first native interface, second native interface, first algorithm interface and second algorithm interface in sequence according to the algorithm call request, and finally call the anti-shake algorithm corresponding to the first image information. Therefore, the electronic device can select the corresponding anti-shake algorithm for image processing according to the received image information to meet different anti-shake requirements corresponding to different image information.

[0114] In some embodiments, corresponding identification information can also be set in advance for the anti-shake algorithm, and the corresponding first algorithm interface, second algorithm interface, first native interface, and second native interface. When the electronic device generates an algorithm call request based on the first image information and the corresponding anti-shake algorithm, the algorithm call request can include the identification information corresponding to the anti-shake algorithm. Then, the electronic device can determine the corresponding first native interface, second native interface, first algorithm interface, and second algorithm interface based on the identification information included in the algorithm call request.

[0115] The identification information of the anti-shake algorithm is also pre-set, and the first image information, the anti-shake algorithm and the identification information can be stored in the GME module in correspondence. The identification information of the anti-shake algorithm can also be called an algorithm identification, as shown in Table 3 below.

[0116]

[0117]

[0118] Table 3

[0119] Among them, the identification information of the anti-shake algorithm can be the letters in Table 3 above, or it can be other forms such as numbers, text, etc., and this application does not impose any restrictions on this.

[0120] Similarly, in the GME module of the electronic device, the first native interface, the second native interface, the first algorithm interface, and the second algorithm interface corresponding to each identification information are pre-stored in the adaptation layer and the algorithm layer, as shown in Table 4 below.

[0121] Identification Information First native interface Second native interface First algorithm interface Second algorithm interface A Native interface 1 Native interface 1 Algorithm interface 1 Algorithm interface 1 B Native interface 2 Native interface 2 Algorithm interface 2 Algorithm interface 2 C Native interface 3 Native interface 3 Algorithm interface 3 Algorithm interface 3 D Native interface 4 Native interface 4 Algorithm interface 4 Algorithm interface 4

[0122] Table 4

[0123] It is understandable that the mapping relationship between the above-mentioned first image information and the corresponding anti-shake algorithm, native interface, and algorithm interface are all set before the electronic device leaves the factory. When the electronic device is in a video shooting scene, it can determine the corresponding anti-shake algorithm and the called interface based on the received image information and the above-mentioned mapping relationship stored in advance, and call the corresponding anti-shake algorithm to perform image processing on the image information by calling the corresponding interface. Among them, the storage forms of the above-mentioned Tables 1 to 4 are only examples, and this application does not limit the storage form of the first image information and the corresponding anti-shake algorithm, native interface, and algorithm interface.

[0124] In an embodiment of the present application, if there are multiple first native interfaces, the first native interface may include the original interface in the GME module and the newly added interface in the GME module. If it is a native interface in the GME module, then when calling the interface, you can call the native built-in algorithm of the GME module, or you can call the algorithm in the corresponding algorithm layer. Generally speaking, the algorithm processing capability of the algorithm in the algorithm layer is better than the algorithm processing capability of the original algorithm in the GME module. Therefore, when the electronic device calls the native interface according to the algorithm call request, it will call the corresponding algorithm in the algorithm layer through the adaptation layer.

[0125] In each iteration and upgrade of electronic devices, if new anti-shake requirements are added, a new anti-shake algorithm can be added to the algorithm layer. Then, the electronic device needs to add a new native interface in the GME module to correspond to the newly added anti-shake algorithm. In this way, the electronic device can increase the ability to undertake new anti-shake requirements and meet various customized needs.

[0126] See also Figure 7 , Figure 7 A schematic diagram of a customized algorithm and a GME module adapted to an embodiment of the present application is shown in FIG. Figure 7 As shown, an adaptation layer is set up to adapt the GME module and the algorithm layer. The adaptation layer includes an adapter, a native interface and a customized interface (i.e., the above-mentioned algorithm interface). The GME module includes a native interface, and the algorithm layer includes a customized interface. Among them, let the native interface of the GME module be the first native interface, the native interface of the adaptation layer be the second native interface, the customized interface of the adaptation layer be the first customized interface, and the customized interface of the algorithm layer be the second customized interface. The second native interface of the adaptation layer is connected to the first native interface of the GME module, and the first customized interface of the adaptation layer is connected to the second customized interface of the algorithm layer.

[0127] Among them, the first native interface is adapted to the second customized interface in the corresponding customized algorithm in the algorithm layer through the adapter. Specifically, the first native interface in the GME module is connected to the adapter through the second native interface of the adaptation layer. Then, through the first customized interface of the adapter, it is connected to the second customized interface of the algorithm layer, and the second customized interface of the algorithm layer corresponds to each customized algorithm respectively. Figure 7 It can be seen that the native interface 1 (first native interface) of the GME module is connected to the adapter through the native interface 1 (second native interface) of the adaptation layer. Then it is connected to the customized interface 1 (second customized interface) of the algorithm layer through the customized interface 1 (second customized interface) of the adapter. The customized interface 1 of the algorithm layer corresponds to the customized algorithm 1.

[0128] Among them, multiple independent algorithms can be loaded in the adapter. An independent algorithm can refer to a single algorithm or an algorithm data packet. The adapter receives the call information of the second native interface, and the call information refers to the algorithm call request generated above. The adapter adapts to the second customized interface of the corresponding algorithm layer through the first customized interface of the corresponding algorithm in the algorithm call request. For example, the adapter receives the call information through the native interface 1 in the second native interface, then, according to the first customized interface corresponding to the algorithm 1 corresponding to the native interface 1, it adapts to the second customized interface of the corresponding algorithm layer. That is, the adapter sends the call information to the algorithm 1 corresponding to the algorithm layer, and the electronic device calls the algorithm 1 to perform image processing.

[0129] After the custom interface of the algorithm layer is called, the corresponding custom algorithm begins to execute. Specifically, the algorithm layer of the electronic device can calculate the anti-shake algorithm corresponding to the corresponding business needs based on the incoming image information to obtain the calculation result of the custom algorithm. The calculation result of the custom algorithm can be used for subsequent inter-frame anti-shake of the image. The image information here refers to the first image information output by the above-mentioned spatial alignment transformation module. Among them, the GME module also needs to use other data information when calling the algorithm to process the image information. The electronic device can call the interface to obtain the above data information from the hardware device, such as gyroscope (gyro), optical image stabilization (Optical Image Stabilization, OIS) data, feature point (Oriented FAST and Rotated BRIEF, ORB) data, etc.

[0130] For example, it is assumed that the customized algorithms stored in the algorithm layer include: customized algorithm 1: correction grid algorithm (intra-frame); customized algorithm 2: alignment matrix algorithm (inter-frame); customized algorithm 3: correction grid algorithm (intra-frame) + alignment matrix algorithm (inter-frame). The GME module stores the mapping relationship between the first image information and the customized algorithm. For example, the anti-shake requirement corresponding to image information 1 is to output the anti-shake grid within the frame separately, which corresponds to customized algorithm 1; the anti-shake requirement corresponding to image information 2 is to output the anti-shake matrix between frames separately, which corresponds to customized algorithm 2; the anti-shake requirement corresponding to image information 3 is to output the data of the anti-shake information between frames and within frames comprehensively, which corresponds to customized algorithm 3. If the first image information received by the GME module is image information 1, then according to the mapping relationship between the first image information and the customized algorithm, it can be determined that the corresponding customized algorithm is customized algorithm 1. Then, according to the mapping relationship between the pre-stored customized algorithm and each interface, the electronic device sequentially calls the first native interface in the GME module, the second native interface in the adaptation layer, the first customized interface in the adaptation layer, and the second customized interface in the algorithm layer to realize the calling of customized algorithm 1 to perform image processing on the image information. For image information 1, the corresponding image processing requirement is that only the correction grid algorithm within the frame needs to be called for anti-shake image processing to achieve a good anti-shake effect. Therefore, the electronic device does not need to call all anti-shake algorithms for processing, thereby improving the efficiency of image processing.

[0131] In some embodiments, if the first image information received by the GME module is image information 3, then the corresponding customized algorithm can be determined to be customized algorithm 3 according to the mapping relationship between the first image information and the customized algorithm. Alternatively, since customized algorithm 3 is composed of algorithms in customized algorithm 1 and customized algorithm 2, the customized algorithm corresponding to image information 3 can also be customized algorithm 1 and customized algorithm 2. Then, according to the mapping relationship between the pre-stored customized algorithms and each interface, the electronic device can sequentially call the first native interface in the GME module corresponding to customized algorithm 3, the second native interface in the adaptation layer, the first customized interface in the adaptation layer, and the second customized interface in the algorithm layer to implement image processing of the image information by calling customized algorithm 3. Alternatively, the electronic device can also sequentially call the first native interface in the GME module corresponding to customized algorithm 1 and customized algorithm 2, the second native interface in the adaptation layer, the first customized interface in the adaptation layer, and the second customized interface in the algorithm layer to implement image processing of the image information by calling customized algorithm 3.

[0132] Then, the calculation result of the customized algorithm is fed back to the GME module in sequence through the customized interface, the adapter, and the native interface. The GME module sends the calculation result of the customized algorithm to the image processing engine. Thus, after the electronic device calls the first algorithm corresponding to the first algorithm call request to perform image processing on the first image information, it obtains the calculation result of the first algorithm and sends it to the image processing engine.

[0133] After the image processing engine receives the calculation result of the custom algorithm in the GME module and the first image information sent by the GME module, it aligns the first image information according to the calculation result of the custom algorithm, and performs operations such as twisting and cropping to form the final display picture, and returns it to the camera application. Among them, the calculation result of the custom algorithm and the first image information can be set as the second image information. Among them, the image processing engine can use methods in the prior art to align, twist, crop and other operations on the image, which will not be repeated in this application.

[0134] In an image processing method provided in an embodiment of the present application, an electronic device can pre-set a corresponding anti-shake algorithm according to different image processing requirements, and in subsequent applications, determine and execute the corresponding algorithm according to the image processing requirements corresponding to the acquired image information. Thus, the electronic device can flexibly call the corresponding algorithm according to the actual image processing requirements, realize flexible customization of the algorithm, and improve image processing efficiency.

[0135] In the embodiment of the present application, during actual use and debugging, on the one hand, due to the poor robustness of the algorithm in the platform-native GME module, it is highly sensitive to abnormal data and has poor fault tolerance. When data is lost or the value is abnormal, it may cause anti-shake failure at the least and screen freezing at the worst. On the other hand, the anti-shake capability of the algorithm in the platform-native GME module itself is poor, and there is a phenomenon of poor anti-shake effect in anti-shake. Therefore, the embodiment of the present application can solve the problem of poor anti-shake effect of the algorithm in the above-mentioned platform-native GME module by replacing the algorithm in the platform-native GME module with an algorithm with better image processing effect.

[0136] The present application also provides a chip system, such as Figure 8As shown, the chip system 90 includes at least one processor 901 and at least one interface circuit 902. The processor 901 and the interface circuit 902 can be interconnected via lines. For example, the interface circuit 902 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 902 can be used to send signals to other devices (such as processor 901). Exemplarily, the interface circuit 902 can read instructions stored in the memory and send the instructions to the processor 901. When the instructions are executed by the processor 901, the electronic device can execute the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0137] An embodiment of the present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0138] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0139] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0144] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device comprises a global motion estimation GME module, an adaptation layer and an algorithm layer, the algorithm layer comprises a preset image processing algorithm corresponding to the GME module, the adaptation layer is used to receive an algorithm call request of the GME module, and adapt to the image processing algorithm corresponding to the algorithm call request in the algorithm layer; the method comprises: The electronic device acquires first image information to be processed in response to a first operation of a user; The electronic device generates a first algorithm calling request based on the first image information; wherein the first algorithm calling request indicates a first image processing algorithm required for the first image information under a corresponding image processing requirement; The GME module sends the first algorithm call request to the algorithm layer by calling the adaptation layer; the algorithm layer calls the first image processing algorithm to perform image processing on the first image information according to the first algorithm call request; wherein the preset image processing algorithm includes multiple image processing algorithms, and different image processing algorithms correspond to different image processing requirements.

2. The image processing method according to claim 1, characterized in that: A first interface layer is included between the GME module and the adaptation layer, and a second interface layer is included between the adaptation layer and the algorithm layer; wherein the GME module sends the first algorithm call request to the algorithm layer by calling the adaptation layer, including: The GME module calls the first interface layer and sends the first algorithm calling request to the adaptation layer. The adaptation layer receives the first algorithm calling request and calls the second interface layer to send the first algorithm calling request to the algorithm layer.

3. The image processing method according to claim 2, characterized in that: The first algorithm call request includes an algorithm identifier corresponding to the first image processing algorithm, the GME module includes a first native interface, the adaptation layer includes a second native interface, an adapter and a first algorithm interface, and the algorithm layer includes a second algorithm interface; wherein the first native interface is connected to the second native interface and belongs to the first interface layer; the first algorithm interface is connected to the second algorithm interface and belongs to the second interface layer; The GME module calls the first interface layer to send the first algorithm call request to the adaptation layer, the adaptation layer receives the first algorithm call request, and calls the second interface layer to send the first algorithm call request to the algorithm layer, including: The GME module calls the first target native interface in the first native interface according to the algorithm identifier to send the first algorithm calling request to the adapter; The adapter is adapted to the second target algorithm interface in the second algorithm interface through the second target native interface in the second native interface and the first target algorithm interface in the first algorithm interface; The adapter sends the first algorithm call request to the algorithm layer through the second target algorithm interface; The first target native interface, the second target native interface, the first target algorithm interface and the second target algorithm interface all correspond to the algorithm identifier.

4. The image processing method according to any one of claims 1 to 3, characterized in that: The preset image processing algorithm includes an image processing algorithm and / or an algorithm data package; the algorithm data package includes at least two image processing algorithms.

5. The image processing method according to any one of claims 1 to 4, characterized in that: After the electronic device calls the first image processing algorithm from a preset image processing algorithm to perform image processing on the first image information according to the first algorithm calling request, the method further includes: The electronic device acquires second image information, and sends the second image information to the image processing engine; wherein the second image information is image information obtained by the electronic device performing image processing on the first image information by calling the first image processing algorithm; The electronic device displays the second image information processed by the image processing engine.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The obtaining of first image information to be processed includes: The electronic device obtains first image information to be processed after performing original image acquisition operations, image signal front-end processing operations and space alignment transformation processing operations.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The method further comprises: The GME module of the electronic device pre-stores a mapping relationship; the mapping relationship is used to indicate the relationship between the image information and the image processing algorithm; The electronic device generates a first algorithm calling request based on the first image information, including: The GME module of the electronic device queries the mapping relationship based on the first image information; In response to the query result, the first algorithm calling request is generated; wherein the query result indicates that the first image information corresponds to the first image processing algorithm.

8. An electronic device, characterized in that: It comprises a processor and a memory; the memory and the processor are coupled; wherein the memory is used to store computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed in an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images