Video processing method and device and storage medium

Abstract

The invention discloses a video processing method and device and a storage medium, and belongs to the technical field of videos. The method comprises the following steps: processing a video by adopting a first electronic anti-shake algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix; processing the video by adopting a global motion estimation algorithm to obtain a first registration matrix; processing the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix to obtain a second registration matrix; and performing MCTF processing on the video according to the first inter-frame offset data and the second registration matrix. According to the method and the device, the influence on the MCTF processing effect after a native electronic anti-shake algorithm of a chip system is replaced by a self-developed electronic anti-shake algorithm can be improved, so that the self-developed algorithm is more adaptive to an MCTF processing scene.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a video processing method, device and storage medium. Background Art

[0002] With the rapid development of mobile Internet and smart phones, mobile phone photography has become an indispensable part of people's lives. In order to meet users' demand for a better shooting experience, mobile phone manufacturers have launched mobile phones equipped with chip platforms that support video stabilization.

[0003] For example, the chip platform that supports video stabilization integrates a visual analysis engine (EVA), an image processing engine (IPE), an electronic stabilization module, and a global motion estimation module. The electronic stabilization module is configured with a native electronic stabilization algorithm provided by the chip platform, and the global motion estimation module is configured with a native global motion estimation algorithm provided by the chip platform. For videos shot by mobile phones, the electronic stabilization module, the global motion estimation module, the visual analysis engine, and the image processing engine can be used to perform motion compensated temporal filtering (MCTF) on the video to improve the video display effect.

[0004] In some scenarios, in order to meet specific video processing requirements, such as achieving better video stabilization effects, mobile phone manufacturers usually replace the native electronic stabilization algorithm configured in the electronic stabilization module with the electronic stabilization algorithm developed by the mobile phone manufacturer, and replace the native global motion estimation algorithm configured in the global motion estimation module with the global motion estimation algorithm developed by the mobile phone manufacturer. However, after replacing the native algorithm with the self-developed algorithm, some unpredictable problems may occur in the MTCF processing effect, such as the problem of texture suspension in the video screen after processing, affecting the video display effect. Summary of the invention

[0005] The present application provides a video processing method, device and storage medium, which can improve the impact of replacing the native algorithm with a self-developed algorithm on the MCTF processing effect, making the self-developed algorithm more adaptable to the MCTF processing scenario.

[0006] In a first aspect, a video processing method is provided, which is applied in an electronic device, and the method includes:

[0007] The video captured by the camera is processed by a first electronic anti-shake algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix; the video is processed by a global motion estimation algorithm to obtain a first registration matrix; the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix are processed to obtain a second registration matrix; and the video is subjected to MCTF processing according to the first inter-frame offset data and the second registration matrix.

[0008] In this way, the registration matrix used in the MCTF processing process calculated based on the inter-frame offset data and the registration matrix output by the global motion estimation algorithm in the native process can be modified to the registration matrix used in the MCTF processing process calculated based on the inter-frame offset data output by the electronic image stabilization algorithm and the registration matrix output by the global motion estimation algorithm, so that the inter-frame offset data used in calculating the registration matrix is ​​consistent with the inter-frame offset data used in the MCTF processing process, thereby reducing or eliminating the texture suspension problem of the video screen after the self-developed algorithm is replaced, reducing the impact of the self-developed algorithm replacing the native algorithm on the MCTF processing effect, and making the self-developed algorithm more adapted to the MCTF processing scenario.

[0009] Among them, the first inter-frame offset data is used to indicate the offset between the front and rear video frames in the video. For example, the first inter-frame offset data can be inter-frame grid data, which is used to indicate the offset of the grid points of the front and rear video frames in the video. The inter-frame perspective transformation matrix is ​​used to indicate the projection mapping relationship between the front and rear video frames in the video. The first registration matrix is ​​used to indicate the transformation relationship for aligning the front and rear video frames in the video. That is, the registration matrix is ​​used to describe the transformation relationship such as translation, rotation, and scaling between the front and rear video frames, and the transformation relationship is used to align the front and rear video frames.

[0010] In one embodiment, the above video is a preview video shot by a camera, that is, the preview video shot by the mobile phone can be processed by MCTF according to the video processing method provided in the embodiment of the present application to reduce the impact of the replacement of the algorithm on the preview video display effect.

[0011] In one embodiment, a second electronic anti-shake algorithm may be used to process the recorded video recorded by the camera to obtain second inter-frame offset data, where the second inter-frame offset data is used to indicate the offset between the front and rear video frames in the recorded video; and MCTF processing may be performed on the recorded video according to the second inter-frame offset data and the second registration matrix. By performing MCTF processing on the recorded video, the video display effect of the recorded video may be improved.

[0012] In one embodiment, the operating system of the electronic device includes a first electronic anti-shake module, a global motion estimation module, a visual analysis engine, and a first image processing engine. The first electronic anti-shake module can be used to process the video using a first electronic anti-shake algorithm to obtain first inter-frame offset data; the global motion estimation module can be used to process the video using a global motion estimation algorithm to obtain a first registration matrix; the visual analysis engine can be used to process the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix to obtain a second registration matrix; and the first image processing engine can be used to perform MCTF processing on the video based on the first inter-frame offset data and the second registration matrix.

[0013] In one embodiment, the operating system also includes a second electronic anti-shake module and a second image processing engine. The second electronic anti-shake module can also be used to process the recorded video of the camera using a second electronic anti-shake algorithm to obtain second inter-frame offset data; and the second image processing engine can be used to perform MCTF processing on the recorded video according to the second inter-frame offset data and the second alignment matrix.

[0014] In one embodiment, the first electronic anti-shake algorithm and the second electronic anti-shake algorithm are electronic anti-shake algorithms provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device, rather than a native algorithm provided by the chip platform. The first electronic anti-shake algorithm and the second electronic anti-shake algorithm may be the same or different, and this embodiment of the application does not limit this.

[0015] In a second aspect, an electronic device is provided, the operating system of which includes a first electronic anti-shake module, a global motion estimation module, a visual analysis engine and a first image processing engine; the first electronic anti-shake module uses a first electronic anti-shake algorithm to process the video taken by the camera, obtains first inter-frame offset data and an inter-frame perspective transformation matrix, sends the first inter-frame offset data and the inter-frame perspective transformation matrix to the visual analysis engine, and sends the first inter-frame offset data to the first image processing engine; the global motion estimation module uses a global motion estimation algorithm to process the video, obtains a first alignment matrix, and sends the first configuration matrix to the visual analysis engine; the visual analysis engine processes the first inter-frame offset data, the inter-frame perspective transformation matrix and the first alignment matrix to obtain a second alignment matrix, and sends the second alignment matrix to the first image processing engine; the first image processing engine performs motion compensated time domain filtering MCTF processing on the video according to the first inter-frame offset data and the second alignment matrix.

[0016] Among them, the first inter-frame offset data is used to indicate the offset of the corresponding grid area in the front and rear two video frames in the video, the inter-frame perspective transformation matrix is ​​used to indicate the projection mapping relationship between the front and rear two video frames in the video, and the first alignment matrix is ​​used to indicate the transformation relationship for aligning the front and rear video frames in the video.

[0017] In the embodiment of the present application, the data path between the modules of the native process is modified, that is, a new path is added for the preview electronic anti-shake module to send the inter-frame offset data to the visual analysis engine, and the path for the global motion estimation module to send the inter-frame offset data to the visual analysis engine is cut off, so that the preview electronic anti-shake module can send its calculated inter-frame offset data to the visual analysis engine and the preview image processing engine, and then the visual analysis engine can calculate a new alignment matrix from the native process according to the inter-frame offset data output by the global motion estimation module, and modify it to generate a new alignment matrix according to the inter-frame offset data output by the preview electronic anti-shake module, so as to ensure that the inter-frame offset data used by the visual analysis engine and the inter-frame offset data used by the preview image processing engine can be consistent. After the path is modified, the texture suspension problem of the MCTF processing effect in the bright light environment is reduced or eliminated, thus reducing the impact of the self-developed algorithm replacing the native algorithm on the video display effect to a certain extent, making the self-developed algorithm more adaptable to the chip platform and MCTF processing scenarios.

[0018] In one embodiment, the operating system also includes a second electronic anti-shake module and a second image processing engine, and the video is a preview video; the visual analysis engine sends the second alignment matrix to the second image processing engine; the second anti-shake module uses a second electronic anti-shake algorithm to process the recorded video recorded by the camera to obtain second inter-frame offset data, and sends the second inter-frame offset data to the second image processing engine; the second image processing engine performs MCTF processing on the recorded video according to the second inter-frame offset data and the second alignment matrix.

[0019] That is, the preview video can be processed by MCTF through the first electronic anti-shake module, the global motion estimation module, the visual analysis engine and the first image processing engine, and the recorded video can be processed by MCTF through the second electronic anti-shake module and the second image processing engine.

[0020] In one embodiment, the first electronic anti-shake module may also send the inter-frame perspective transformation matrix to the first image processing engine.

[0021] In one embodiment, the global motion estimation module may also send the first inter-frame offset data to the second electronic anti-shake module.

[0022] In one embodiment, the first electronic anti-shake algorithm and the second electronic anti-shake algorithm are electronic anti-shake algorithms provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device, rather than a native algorithm provided by the chip platform. That is, the native anti-shake algorithm in the electronic anti-shake module can be replaced in advance with the electronic anti-shake algorithm developed by the device manufacturer, and the native global motion estimation algorithm in the global motion estimation module can be replaced with the global motion estimation algorithm developed by the device manufacturer. Among them, the first electronic anti-shake algorithm and the second electronic anti-shake algorithm can be the same or different, and this embodiment of the present application does not limit this.

[0023] In a third aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method described in the first aspect or the second aspect. The video processing device includes at least one module, wherein the at least one module is used to implement the video processing method described in the first aspect.

[0024] In a fourth aspect, a video processing device is provided, wherein the structure of the video processing device includes a processor and a memory, wherein the memory is used to store a program that supports the video processing device to execute the video processing method provided in the first aspect or the second aspect, and to store data involved in implementing the video processing method described in the first aspect or the second aspect. The processor is configured to execute the program stored in the memory. The video processing device may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0025] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the video processing method described in the first aspect or the second aspect.

[0026] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video processing method described in the first aspect.

[0027] In the seventh aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the video processing method described in the first aspect or the second aspect above.

[0028] The technical effects obtained by the third, fourth, fifth, sixth and seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first or second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0030] Figure 2 is a block diagram of a software system of a terminal provided in an embodiment of the present application;

[0031] Figure 3 This is a schematic diagram of a process of processing a video using a native algorithm based on a chip platform provided in an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of a video processing flow after replacing the native algorithm of the chip platform with a self-developed algorithm of an electronic device provided in an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram comparing video images with normal and abnormal MCTF processing effects provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of an optimized video processing flow after replacing the native algorithm of the chip platform with the self-developed algorithm of the electronic device provided in an embodiment of the present application;

[0035] Figure 7 It is a flowchart of a video processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0037] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0038] Before describing in detail the video processing method provided in the embodiment of the present application, the terms involved in the embodiment of the present application are first explained.

[0039] Video jitter: Video jitter means that the position of the object image is deviated, which is mainly caused by the high-frequency vibration of the camera. As the camera shakes up and down, left and right, the image will produce pixel displacement in the video image. The video shot by a stable camera is continuous and smooth, while for a camera with jitter, the video shot often appears irregular and discontinuous. Therefore, eliminating camera jitter means eliminating the discontinuous changes between adjacent frames in the video.

[0040] Electronic image stabilization (EIS) is a technology that uses software algorithms to achieve image stabilization. It can estimate the motion between consecutive video frames at the digital image algorithm level and perform reverse compensation to obtain a stable video image.

[0041] Global motion estimation: Estimate the global motion model between the previous and next video frames. Commonly used global motion estimation methods include optical flow method, feature matching method, block matching method, grayscale projection method or gyroscope estimation method.

[0042] Each frame of the image sequence is divided into many non-overlapping macroblocks, and the displacement of all pixels in the macroblock is assumed to be the same. Then, for each macroblock, the block most similar to the current block is found within a given specific search range of the reference frame according to certain matching criteria, that is, the matching block. The relative displacement between the matching block and the current block is the motion vector.

[0043] Engine of visual analytics (EVA): In this application, EVA is mainly used to achieve image alignment and correction through visual analysis algorithms.

[0044] Image Process Engine (IPE): IPE is a hardware processing unit on the chip, which is mainly responsible for secondary processing of image data processed by the image front end (IFE), including image denoising, sharpening, contrast adjustment, color mapping and other functions, and finally outputs the processed image data to the display or encoder.

[0045] Motion compensated temporal filter (MCTF): MCTF is an image processing technique that can be used to reduce motion blur, align images, and denoise videos. MCTF can restore clear images by analyzing motion information in a video frame sequence.

[0046] Before explaining the video processing method provided in the embodiment of the present application in detail, the electronic device involved in the embodiment of the present application is first described.

[0047] The video processing method provided in the embodiment of the present application can be applied to an electronic device with video shooting and video processing functions, and the electronic device can be a terminal such as a mobile phone, a tablet computer, a smart wearable device or a computer.

[0048] Figure 1 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 1 The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0049] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0051] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0052] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0053] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0054] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0055] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0056] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0057] In some embodiments, antenna 1 of terminal 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that terminal 100 can communicate with the network and other devices through wireless communication technology.

[0058] The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0059] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0060] The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0061] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193. In some embodiments, the algorithm module corresponding to ISP can include an electronic anti-shake module and a global motion estimation (GME) module.

[0062] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0063] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0064] Video codecs are used to compress or decompress digital videos. Terminal 100 may support one or more video codecs. Thus, terminal 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0065] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0066] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0067] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the terminal 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0068] The terminal 100 can implement audio functions, such as music playing and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0069] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0070] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the terminal 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a pressure threshold acts on a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0071] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be arranged on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the terminal 100, which is different from the position of the display screen 194.

[0072] Next, the software system of the terminal 100 will be described.

[0073] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the terminal 100.

[0074] Figure 2 1 is a block diagram of a software system of a terminal 100 provided in an embodiment of the present application. Figure 2 , the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android Runtime) and the system layer, the kernel layer, and the hardware abstraction layer (HAL).

[0075] The application layer can include a series of application packages. Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0076] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application. The display interface can be composed of one or more views, for example, including a view for displaying a text message notification icon, a view for displaying text, and a view for displaying pictures. The phone manager is used to provide communication functions of the terminal 100, such as management of call status (including connected, hung up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a reminder sound, an electronic device vibrating, an indicator light flashing, etc.

[0077] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0078] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0079] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0080] The HAL layer is an interface layer between the kernel layer and the hardware layer, and its purpose is to abstract the hardware. The hardware abstraction layer is a programming layer that allows the operating system to interact with hardware devices at the logical layer rather than the hardware layer. For example, the HAL layer includes an EVA hardware abstraction layer corresponding to the EVA of the hardware layer, and an IPE hardware abstraction layer corresponding to the IPE. In addition, the HAL layer may also include an image processing module, and the image processing module includes a global motion estimation module and at least one electronic anti-shake module. For example, the at least one electronic anti-shake module includes a first electronic anti-shake module and a second electronic anti-shake module. For example, Figure 2 As shown, the first electronic anti-shake module and the second electronic anti-shake module can be a preview electronic anti-shake module and a video electronic anti-shake module respectively. The preview electronic anti-shake module is used to perform video anti-shake processing on the preview video, and the video electronic anti-shake module is used to perform video anti-shake processing on the recorded video. The EVA hardware abstraction layer includes a visual analysis engine. The IPE hardware abstraction layer includes at least one image processing engine, such as a first image processing engine and a second image processing engine. For example, Figure 2 As shown, the first image processing engine and the second image processing engine may be a preview image processing engine and a video image processing engine, respectively, the preview image processing engine is used to perform video processing on the preview video, and the video image processing engine is used to perform video processing on the recorded video. As an example, each image processing engine may include an image correction and alignment module (ICA), and the image correction and alignment module may be configured with a native MCTF algorithm of the chip platform to perform MCTF processing on the video.

[0081] The hardware layer includes at least a processor, a display, an audio device, a camera group, an EVA and an IPE. The ISP, EVA and IPE can be integrated on a chip platform carried by the electronic device.

[0082] The following is an illustrative description of the software and hardware workflow of the terminal 100 in conjunction with a video recording scenario.

[0083] The camera application in the application layer can be displayed on the screen of the electronic device in the form of an icon. When the icon of the camera application is clicked by the user to be triggered, the electronic device starts running the camera application. When the recording option in the camera application interface is clicked by the user to be triggered, the camera application calls the interface corresponding to the camera application in the application framework layer to start the camera driver, turns on one or more cameras in the camera group, and records through the turned-on cameras to obtain a preview video stream and a recorded video stream. For the preview video stream, the preview electronic anti-shake module, the global motion estimation module, the visual analysis engine, and the preview image processing engine in the hardware abstraction layer can be called to perform video processing on the preview video stream. For the recorded video stream, the video electronic anti-shake module and the video image processing engine in the hardware abstraction layer can be called to process the recorded video stream.

[0084] Next, the inventive concept of the embodiments of the present application is explained in detail.

[0085] Please refer to Figure 3 , Figure 3 This is a flow chart of a video processing process based on a native algorithm of a chip platform provided in an embodiment of the present application. When the global motion estimation module is configured with a native global motion estimation algorithm provided by the chip platform, and the electronic anti-shake module is configured with a native electronic anti-shake algorithm provided by the chip platform, Figure 2 The process of MCTF processing of videos by various modules in the hardware abstraction layer can be as follows: Figure 3 As shown. The chip platform can be Chip platform, etc., which is not limited in the embodiments of the present application.

[0086] like Figure 3As shown, the preview video shot by the camera of the electronic device can be input to the preview electronic anti-shake module and the global motion estimation module respectively. On the one hand, the preview electronic anti-shake module uses the native electronic anti-shake algorithm to calculate the preview video, obtain the inter-frame offset data 1 and the inter-frame perspective transformation matrix, send the inter-frame perspective transformation matrix to the visual analysis engine, and send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module uses the native global motion estimation algorithm to calculate the preview video, obtain the inter-frame offset data 2 and the registration matrix 1, send the inter-frame offset data 2 and the registration matrix 1 to the visual analysis engine, and send the inter-frame offset data 1 to the video electronic anti-shake module. The visual analysis engine can generate the registration matrix 2 according to the inter-frame perspective transformation matrix sent by the preview electronic anti-shake module, and the registration matrix 1 and the inter-frame offset data 2 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic anti-shake module and the registration matrix 2 sent by the visual analysis engine.

[0087] In addition, the recorded video shot by the camera can be input to the video electronic image stabilization module, which uses the native electronic image stabilization algorithm to calculate the recorded video to obtain the inter-frame offset data 3, and sends the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0088] Among them, the inter-frame offset data is used to indicate the offset between the front and rear video frames in the corresponding video. For example, the inter-frame offset data can be inter-frame grid data, which is used to indicate the offset of the grid points of the front and rear video frames in the video. The inter-frame perspective transformation matrix is ​​used to indicate the projection mapping relationship between the front and rear video frames in the corresponding video. The registration matrix is ​​used to indicate the transformation relationship for aligning the front and rear video frames. That is, the registration matrix is ​​used to describe the transformation relationship such as translation, rotation, and scaling between the front and rear video frames, and the transformation relationship is used to align the front and rear video frames.

[0089] Currently, the native algorithm based on the chip platform can be Figure 3The video processing flow shown in the figure processes the video to achieve video stabilization. The stabilization effect is one of the important indicators for evaluating video quality. In actual use and debugging, it is found that the native electronic stabilization algorithm provided by the chip platform may have poor robustness, and there are phenomena such as poor stabilization effect, stabilization failure, and even frozen screen. To solve the above problems, equipment manufacturers of electronic equipment can use self-developed electronic stabilization algorithms to replace the native electronic stabilization algorithms of the chip platform, and use self-developed global motion estimation algorithms to replace the native global motion estimation algorithms of the chip platform to achieve better video stabilization effects.

[0090] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a video processing flow after the native algorithm of the chip platform is replaced by the self-developed algorithm of the electronic device provided by the embodiment of the present application. After the native global motion estimation algorithm of the global motion estimation module is replaced by the self-developed global motion estimation algorithm provided by the device manufacturer of the electronic device, and the native electronic anti-shake algorithm of the electronic anti-shake module is replaced by the self-developed electronic anti-shake algorithm provided by the device manufacturer, Figure 2 The process of MCTF processing of videos by various modules in the hardware abstraction layer can be as follows: Figure 4 shown.

[0091] like Figure 4 As shown, the preview video captured by the camera can be input into the preview electronic anti-shake module and the global motion estimation module respectively. On the one hand, the preview electronic anti-shake module uses the first electronic anti-shake algorithm developed by the device to calculate the preview video, obtain the inter-frame offset data 1 and the inter-frame perspective transformation matrix, send the inter-frame perspective transformation matrix to the visual analysis engine, and send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module uses the global motion estimation algorithm developed by the device to calculate the preview video, output the registration matrix 1 and the inter-frame offset data 2, send the inter-frame offset data 2 and the registration matrix 1 to the visual analysis engine, and send the inter-frame offset data 1 to the video electronic anti-shake module. The visual analysis engine can generate the registration matrix 2 according to the inter-frame perspective transformation matrix sent by the preview electronic anti-shake module, and the registration matrix 1 and the inter-frame offset data 2 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic anti-shake module and the registration matrix sent by the visual analysis engine.

[0092] In addition, the recorded video captured by the camera can be input to the video electronic image stabilization module, which uses the second electronic image stabilization algorithm developed by the device to calculate the recorded video, obtain the inter-frame offset data 3, and send the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix sent by the visual analysis engine.

[0093] However, replacing the native algorithm with a self-developed algorithm may cause some unpredictable problems in the processed video. For example, the inventor found that the texture of the processed video screen appeared suspended, which greatly affected the video display effect. Figure 5 , Figure 5 It is a schematic diagram comparing video images with normal and abnormal MCTF processing effects provided by an embodiment of the present application. Figure 5 Figure (a) is a schematic diagram of a video image with normal MCTF processing effect. Figure 5 Figure (b) is a schematic diagram of a video screen with abnormal MCTF processing effect after the original algorithm is replaced by the self-developed algorithm. Figure 5 As shown in Figure (b), after the original algorithm was replaced by the self-developed algorithm, the video screen showed obvious texture suspension phenomenon. Texture suspension phenomenon refers to the area being obviously dark and blurred, and having obvious shadow suspension phenomenon.

[0094] In response to the problem of texture suspension appearing in the video screen after the native algorithm is replaced by the self-developed algorithm, the inventor has conducted a series of analyses and studies on the problem to find out the causes of this problem, so as to solve the problem in a targeted manner based on the causes found, thereby reducing or eliminating the impact of replacing the native algorithm with the self-developed algorithm on the video display effect.

[0095] First, the inventors obtained the processing logs before and after the replacement of the self-developed algorithms (self-developed electronic image stabilization algorithm and global motion estimation algorithm). By carefully comparing the processing logs before and after the replacement of the self-developed algorithms, it was found that the data calculated by the self-developed algorithm (hereinafter referred to as the self-developed process) and the native algorithm of the chip platform (hereinafter referred to as the native process) had large differences in order of magnitude. For example, data of different orders of magnitude were used to describe the same offset. Therefore, it was initially suspected that the texture suspension problem of the above-mentioned video screen was caused by the difference in order of magnitude. However, after consulting the chip manufacturer about this suspicion, the chip manufacturer responded that the difference in order of magnitude would not cause the problem.

[0096] Afterwards, the inventor re-analyzed and studied other reasons that caused the above problems. By comparing the logs of the self-developed process and the native process, it was found that the inter-frame offset data calculated by the preview electronic anti-shake module and the global motion estimation module in the native process are consistent, that is, the inter-frame offset data 1 calculated by the preview electronic anti-shake module is the same as the inter-frame offset data 2 calculated by the global motion estimation module. However, the inter-frame offset data calculated by the preview electronic anti-shake module and the global motion estimation module in the self-developed process are inconsistent, that is, the inter-frame offset data 1 calculated by the preview electronic anti-shake module is different from the inter-frame offset data 2 calculated by the global motion estimation module. In addition, by analyzing the logs of the native process, it is found that the alignment matrix 2 received by the image processing engine (preview image processing engine and video image processing engine) is numerically inconsistent with the alignment matrix 1 calculated by the global motion estimation module. Therefore, it can be inferred that the alignment matrix 2 received by the image processing engine is recalculated, that is, after the alignment matrix 1 calculated by the global motion estimation module is output to the visual analysis engine, the visual analysis engine will calculate a new alignment matrix 2 based on the inter-frame perspective transformation matrix sent by the preview electronic anti-shake module, and the alignment matrix 1 and inter-frame offset data 2 sent by the global motion estimation module, and then send the new alignment matrix 2 to the image processing engine for MCTF processing. In addition, by analyzing the logs of the native process, the inventor identified that the inter-frame offset data used by the visual analysis engine is the inter-frame offset data 2 calculated by the global motion estimation module, and the inter-frame offset data 2 used by the visual analysis engine is consistent with the inter-frame offset data 1 used by the preview image processing engine. However, by analyzing the logs of the self-developed process, it is identified that the inter-frame offset data 1 used by the preview image processing engine and the inter-frame offset data 2 used by the visual analysis engine are inconsistent. Based on this, the inventor suspects that the texture suspension problem in the above-mentioned video screen is caused by the inconsistency between the inter-frame offset data calculated by the preview electronic anti-shake module and the global motion estimation module in the self-developed process, which in turn causes the inter-frame offset data used by the visual analysis engine and the inter-frame offset data used by the preview image processing engine to be inconsistent.

[0097] In an embodiment of the present application, in order to ensure that the inter-frame offset data used by the visual analysis engine and the inter-frame offset data used by the preview image processing engine are consistent, a video processing method is provided. In this method, after the video is processed using a first electronic anti-shake algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix, and the video is processed using a global motion estimation algorithm to obtain a first registration matrix, the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix can be processed to obtain a second registration matrix, and then the video is subjected to MCTF processing based on the first inter-frame offset data and the second registration matrix.

[0098] In this way, the registration matrix used in the MCTF processing process calculated based on the inter-frame offset data and the registration matrix output by the global motion estimation algorithm in the native process can be modified to the registration matrix used in the MCTF processing process calculated based on the inter-frame offset data output by the electronic image stabilization algorithm and the registration matrix output by the global motion estimation algorithm, so that the inter-frame offset data used in calculating the registration matrix is ​​consistent with the inter-frame offset data used in the MCTF processing process, thereby reducing or eliminating the texture suspension problem of the video screen after the self-developed algorithm is replaced, and reducing the impact of the self-developed algorithm replacing the native algorithm on the video display effect.

[0099] Please refer to Figure 6 , Figure 6 This is a schematic diagram of an optimized video processing flow provided by an embodiment of the present application after replacing the native algorithm of the chip platform with the self-developed algorithm of the electronic device. After replacing the native global motion estimation algorithm of the global motion estimation module with the self-developed global motion estimation algorithm provided by the device manufacturer of the electronic device, and replacing the native electronic anti-shake algorithms of the preview electronic anti-shake module and the video electronic anti-shake module with the self-developed electronic anti-shake algorithm provided by the device manufacturer, the video processing method provided by the embodiment of the present application can be used to Figure 2 The modules in the hardware abstraction layer in the MCTF process of the video are optimized. The optimized video processing process can be as follows Figure 6 shown.

[0100] like Figure 6As shown, the preview video captured by the camera can be input into the preview electronic anti-shake module and the global motion estimation module respectively. On the one hand, the preview electronic anti-shake module uses the first electronic anti-shake algorithm developed by the device to process the preview video, obtain the inter-frame offset data 1 and the inter-frame perspective transformation matrix, send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the visual analysis engine, and send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module uses the global motion estimation processing method developed by the device to process the preview video, obtain the inter-frame offset data 2 and the registration matrix 1, send the registration matrix 1 to the visual analysis engine, and send the inter-frame offset data 1 to the video electronic anti-shake module. The visual analysis engine can generate a new registration matrix 2 based on the inter-frame offset data 1 and the inter-frame perspective transformation matrix sent by the preview electronic anti-shake module, and the registration matrix 1 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic anti-shake module and the alignment matrix 2 sent by the visual analysis engine. In addition, the recorded video captured by the camera can be input into the video electronic anti-shake module, which uses the second electronic anti-shake algorithm developed by the device to process the recorded video, obtain the inter-frame offset data 3, and send the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic anti-shake module and the alignment matrix 2 sent by the visual analysis engine.

[0101] Will Figure 6 and Figure 4 A comparative analysis shows that Figure 6 Compared with the native process, the data path between modules has been modified, that is, a new path has been added for the preview electronic anti-shake module to send inter-frame offset data to the visual analysis engine, and the path for the global motion estimation module to send inter-frame offset data to the visual analysis engine has been cut off, so that the preview electronic anti-shake module can send its calculated inter-frame offset data to the visual analysis engine and the preview image processing engine, and then the visual analysis engine can calculate a new alignment matrix based on the inter-frame offset data output by the global motion estimation module from the native process, and modify it to generate a new alignment matrix based on the inter-frame offset data output by the preview electronic anti-shake module to ensure that the inter-frame offset data used by the visual analysis engine is consistent with the inter-frame offset data used by the preview image processing engine. After the path has been modified, the texture suspension problem of the MCTF processing effect in bright light environments has been reduced or disappeared, thus reducing the impact of the self-developed algorithm replacing the native algorithm on the video display effect to a certain extent, making the self-developed algorithm more compatible with the MCTF processing scene.

[0102] In order to further reduce or eliminate the texture suspension problem of MCTF processing effects in dark light environments, the inventors further analyzed and studied the native process and the self-developed process. By comparing the logs of the self-developed process and the native process, the inventors found that there are differences in the data received by the visual analysis engine in the two processes. In the self-developed process, the registration matrix received by the visual analysis engine is a unit matrix, that is, a default value that has not been calculated. It can be inferred that the registration matrix 1 received by the visual analysis engine is not the registration matrix calculated by the global motion estimation module using the self-developed algorithm, but a pre-set initial value. Therefore, the electronic anti-shake module and the global motion estimation module in the self-developed process will only output inter-frame offset data and perspective transformation matrix, and output less registration matrix than the native process. By analyzing and studying the logs of the self-developed process, it was found that the self-developed algorithm merged the relevant data of the registration matrix into the inter-frame offset data.

[0103] like Figure 6 As shown, in the embodiment of the present application, the self-developed algorithm also splits the fused registration-related information in the inter-frame offset data according to the category of the native algorithm data, and can output the inter-frame offset data and the registration matrix respectively, and study the change trend of the native platform registration matrix data for adaptation. In this way, the texture suspension problem of the MCTF processing effect in the dark light environment can be alleviated or eliminated to a certain extent.

[0104] In addition, subsequent chip manufacturers have reported that the difference in the order of magnitude of the output results will also affect the MCTF effect. Therefore, the embodiment of the present application can also compress the order of magnitude of the self-developed algorithm output data according to the order of magnitude of the native algorithm output data to reduce the order of magnitude difference between the self-developed algorithm output data and the native algorithm output data. In this way, the texture suspension problem of the MCTF processing effect in a dark light environment can be further reduced or eliminated.

[0105] Next, combine Figure 2 and Figure 6 The video processing method provided in the embodiment of the present application is described in detail.

[0106] Figure 7 is a flow chart of a video processing method provided in an embodiment of the present application, taking an electronic mobile phone as an example. Figure 7 As shown, the method comprises the following steps:

[0107] Step 701: The user starts the camera application.

[0108] For example, a user can tap the application icon of a camera application to launch the camera application.

[0109] Step 702: In response to a user's start operation, the camera application starts.

[0110] After the camera application is started, a preview interface of the camera application may be displayed, and the preview interface may include a recording option.

[0111] Step 703: The user clicks on the recording option in the application interface of the camera application.

[0112] Step 704: In response to the user's click operation, the camera application sends a call instruction to the camera.

[0113] The calling instruction is used to call a camera to shoot. The called camera can be one or more cameras in the camera group, and the embodiment of the present application does not limit this.

[0114] Step 705: The camera shoots video according to the calling instruction and generates a preview video stream and a recording video stream.

[0115] After the camera shoots a video, a preview video stream and a recorded video stream can be generated respectively. The preview video stream is used for video preview, including continuous video frames to be previewed and displayed, and the recorded video stream is used for video storage, including continuous video frames to be stored. The preview video stream and the recorded video stream can also be referred to as preview stream and video stream respectively, which is not limited in the embodiments of the present application.

[0116] After generating the preview video stream, the camera can send the preview video to the preview electronic anti-shake module and the global motion estimation module respectively. After generating the recording video stream, the camera can send the recording video stream to the video electronic anti-shake module.

[0117] Step 706: The camera sends a preview video stream to the preview electronic image stabilization module.

[0118] The preview electronic anti-shake module is equipped with the first electronic anti-shake algorithm developed by the mobile phone manufacturer, rather than the native electronic anti-shake algorithm of the chip platform. That is, the native electronic anti-shake algorithm of the chip platform originally configured in the preview electronic anti-shake module is replaced by the first electronic anti-shake algorithm developed by the mobile phone manufacturer.

[0119] It should be noted that the first electronic image stabilization algorithm developed by mobile phone manufacturers has the same algorithm principle as the native electronic image stabilization algorithm of the chip platform, and only has certain differences in the details of the algorithm implementation, such as the specific parameter settings in the algorithm and the order of magnitude of the data.

[0120] Step 707: The camera sends a preview video stream to the global motion estimation module.

[0121] Among them, the global motion estimation module is configured with a global motion estimation algorithm developed by the mobile phone manufacturer, rather than the native global motion estimation algorithm of the chip platform. That is, the global motion estimation algorithm of the chip platform originally configured in the global motion estimation module is replaced by the global motion estimation algorithm developed by the mobile phone manufacturer.

[0122] It should be noted that the global motion estimation algorithm developed by mobile phone manufacturers and the native global motion estimation algorithm of the chip platform have the same algorithm principle, and only have certain differences in the details of the algorithm implementation, such as the specific parameter settings in the algorithm and the order of magnitude of the data.

[0123] Step 708: The camera sends the recorded video stream to the video electronic image stabilization module.

[0124] Among them, the video electronic anti-shake module is equipped with a second electronic anti-shake algorithm developed by the mobile phone manufacturer, rather than the native electronic anti-shake algorithm of the chip platform. In other words, the native electronic anti-shake algorithm of the chip platform originally configured in the video electronic anti-shake module is replaced by the second electronic anti-shake algorithm developed by the mobile phone manufacturer.

[0125] It should be noted that the second electronic image stabilization algorithm developed by mobile phone manufacturers has the same algorithm principle as the native electronic image stabilization algorithm of the chip platform, and only has certain differences in the details of the algorithm implementation, such as the specific parameter settings in the algorithm and the order of magnitude of the data.

[0126] It should also be noted that the second electronic anti-shake algorithm configured in the video electronic anti-shake module and the first electronic anti-shake algorithm configured in the preview electronic anti-shake module can be the same electronic anti-shake algorithm or different electronic anti-shake algorithms, which is not limited in this embodiment of the present application.

[0127] As an example, the electronic anti-shake algorithm configured in the video electronic anti-shake module and the preview electronic anti-shake module is the same, that is, the first electronic anti-shake algorithm and the second electronic anti-shake algorithm are the same.

[0128] Step 709: The preview electronic anti-shake module uses the first electronic anti-shake algorithm developed by the mobile phone to calculate the video data corresponding to the preview video stream to obtain the inter-frame offset data 1 and the inter-frame perspective transformation matrix.

[0129] By using the first electronic anti-shake algorithm to calculate the video data corresponding to the preview video stream, motion information of continuous video frames in the preview video stream can be obtained. The motion information may include inter-frame offset data 1 and an inter-frame perspective transformation matrix.

[0130] The inter-frame offset data 1 is used to indicate the offset between the previous and next video frames in the preview video stream. For example, the inter-frame offset data 1 may be inter-frame grid data, which is used to indicate the offset of the grid points between the previous and next video frames, such as the offset of the grid points between the previous and next two video frames on the X-axis and the Y-axis.

[0131] The inter-frame perspective transformation matrix is ​​used to indicate the projection mapping relationship between the previous and next video frames in the preview video stream, such as the projection mapping relationship of projecting the previous video frame onto the next video frame.

[0132] Step 710: The preview electronic anti-shake module sends the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the visual analysis engine.

[0133] It should be noted that the embodiments of the present application are different from the above Figure 3 What is different from the native process of the chip platform shown is that after the preview electronic image stabilization module calculates the inter-frame offset data 1 and the inter-frame perspective transformation matrix, it needs to send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the visual analysis engine and the preview image processing engine at the same time to ensure that the inter-frame offset data obtained by the visual analysis engine and the preview image processing engine are consistent.

[0134] Step 711: The preview electronic image stabilization module sends the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine.

[0135] Step 712: The global motion estimation module uses the global motion estimation method developed by the mobile phone to calculate the video data corresponding to the preview video stream to obtain the inter-frame offset data 2 and the alignment matrix 1.

[0136] By using a global motion estimation method to calculate the video data corresponding to the preview video stream, a global motion model of continuous video frames in the preview video stream can be obtained. The global motion model includes inter-frame offset data 2 and a registration matrix 1.

[0137] Among them, the inter-frame offset data 2 is also used to indicate the offset between the previous and next video frames in the preview video stream, but due to different internal algorithms, the inter-frame offset data 2 and the inter-frame offset data 1 output by the above-mentioned preview electronic image stabilization module may be different in value.

[0138] Among them, the registration matrix 1 is used to indicate the transformation relationship for aligning the previous and next video frames in the preview video stream, that is, the previous and next video frames in the preview video stream can be aligned according to the change relationship, such as aligning the previous video frame with the current video frame.

[0139] Step 713: The global motion estimation module sends the registration matrix 1 to the visual analysis engine.

[0140] It should be noted that the embodiments of the present application are different from the above Figure 3 The difference from the native process of the chip platform shown is that the global motion estimation module does not need to send the inter-frame offset data 2 to the visual analysis engine to avoid inconsistency between the inter-frame offset data obtained by the visual analysis engine and the inter-frame offset data obtained by the preview image processing engine.

[0141] Step 714: The global motion estimation module sends the inter-frame offset data 2 to the video electronic anti-shake module.

[0142] Step 715: The visual analysis engine generates a registration matrix 2 based on the inter-frame offset data 1, the inter-frame perspective transformation matrix and the registration matrix 1.

[0143] The visual analysis engine can use the native visual analysis model of the chip platform on which it is configured to calculate the inter-frame offset data 1, the inter-frame perspective transformation matrix and the registration matrix 1, thereby outputting a new registration matrix, namely, the registration matrix 2.

[0144] Among them, the chip platform can be Chip platform, etc.

[0145] Step 716: The visual analysis engine sends the registration matrix 2 to the preview image processing engine.

[0146] The visual analysis engine calculates the registration matrix 2 and can send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively.

[0147] Step 717: The visual analysis engine sends the registration matrix 2 to the video image processing engine.

[0148] Step 718: The preview image processing engine performs MCTF processing on the video data corresponding to the preview video stream according to the inter-frame offset data 1 and the registration matrix 2 sent by the visual analysis engine to obtain preview video data.

[0149] For example, the preview image processing engine includes an image correction and alignment module, which can perform MCTF processing on the video data corresponding to the preview video stream according to the inter-frame offset data 1 sent by the preview electronic image stabilization module and the alignment matrix 2 sent by the visual analysis engine.

[0150] For example, a native MCTF algorithm of a chip platform configured with an image correction and alignment module may be used to perform MCTF processing on video data corresponding to the preview video stream.

[0151] Step 719: The preview image processing engine sends the preview video data to the camera application.

[0152] Step 720: The camera application displays the corresponding preview video on the preview interface according to the preview video data.

[0153] Step 721: The video electronic anti-shake module uses the second electronic anti-shake algorithm developed by the mobile phone to calculate the video data corresponding to the recorded video stream to obtain the inter-frame offset data 3.

[0154] By using the second electronic anti-shake algorithm to calculate the video data corresponding to the recorded video stream, motion information of continuous video frames in the recorded video stream can be obtained, and the motion information may include inter-frame offset data 3.

[0155] The inter-frame offset data 3 is used to indicate the offset between the previous and next video frames in the recorded video stream. For example, the inter-frame offset data 3 may be inter-frame grid data, which is used to indicate the offset of the grid points between the previous and next video frames, such as the offset of the grid points between the previous and next two video frames on the X-axis and the Y-axis.

[0156] Step 722: The video electronic image stabilization module sends the inter-frame offset data 3 to the video image processing engine.

[0157] After the video electronic anti-shake module calculates the inter-frame offset data 3, the inter-frame offset data 3 may be sent to the video image processing engine for processing.

[0158] Step 723: The video image processing engine performs MCTF processing on the video data corresponding to the recorded video stream according to the inter-frame offset data 3 and the registration matrix 2 to obtain the recorded video data.

[0159] For example, the video image processing engine includes an image correction and alignment module, through which MCTF processing can be performed on the video data corresponding to the recorded video stream according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the alignment matrix 2 sent by the visual analysis engine.

[0160] For example, the native MCTF algorithm of the chip platform configured with the image correction and alignment module may be used to perform MCTF processing on the video data corresponding to the recorded video stream.

[0161] Step 724: The video image processing engine sends the recorded video data to the camera application.

[0162] Step 725: The camera application stores the recorded video according to the recorded video data.

[0163] In the embodiment of the present application, by modifying the path of the native process of the chip platform, that is, adding a path for the preview electronic anti-shake module to send inter-frame offset data to the visual analysis engine, and cutting off the path for the global motion estimation module to send inter-frame offset data to the visual analysis engine, the preview electronic anti-shake module can send its calculated inter-frame offset data to the visual analysis engine and the preview image processing engine, so that the visual analysis engine can calculate a new alignment matrix from the native process according to the inter-frame offset data output by the global motion estimation module, and modify it to generate a new alignment matrix according to the inter-frame offset data output by the preview electronic anti-shake module to ensure that the inter-frame offset data used by the visual analysis engine is consistent with the inter-frame offset data used by the preview image processing engine. By modifying the path, the texture suspension problem of the MCTF processing effect is improved, and the impact of the self-developed algorithm replacing the native algorithm on the video display effect is reduced to a certain extent, making the self-developed algorithm more compatible with the MCTF processing scene.

[0164] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0165] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0166] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0167] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program codes. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0168] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0169] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0171] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: Applied in an electronic device, the method comprises: Using a first electronic anti-shake algorithm to process a video captured by a camera, obtaining first inter-frame offset data and an inter-frame perspective transformation matrix, wherein the first inter-frame offset data is used to indicate an offset between front and rear video frames in the video, and the inter-frame perspective transformation matrix is ​​used to indicate a projection mapping relationship between front and rear video frames in the video; Processing the video using a global motion estimation algorithm to obtain a first registration matrix, where the first registration matrix is ​​used to indicate a transformation relationship for aligning previous and next video frames in the video; Processing the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix to obtain a second registration matrix; According to the first inter-frame offset data and the second registration matrix, a motion compensated time domain filtering MCTF process is performed on the video.

2. The method according to claim 1, characterized in that The video is a preview video; the method further includes: Processing the recorded video recorded by the camera using a second electronic anti-shake algorithm to obtain second inter-frame offset data, where the second inter-frame offset data is used to indicate an offset between previous and next video frames in the recorded video; Perform MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

3. The method according to claim 1 or 2, characterized in that The first electronic anti-shake algorithm is an electronic anti-shake algorithm provided by a device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by a device manufacturer of the electronic device.

4. The method according to claim 2, characterized in that The second electronic anti-shake algorithm is an electronic anti-shake algorithm provided by a device manufacturer of the electronic device.

5. A video processing method, characterized in that: Applied in an electronic device, the operating system of the electronic device comprises a first electronic anti-shake module, a global motion estimation module, a visual analysis engine and a first image processing engine; The first electronic anti-shake module processes the video captured by the camera using the first electronic anti-shake algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix, sends the first inter-frame offset data and the inter-frame perspective transformation matrix to the visual analysis engine, and sends the first inter-frame offset data to the first image processing engine; wherein the first inter-frame offset data is used to indicate the offset of the corresponding grid area in the two preceding and following video frames in the video, and the inter-frame perspective transformation matrix is ​​used to indicate the projection mapping relationship between the two preceding and following video frames in the video; The global motion estimation module processes the video using a global motion estimation algorithm to obtain a first registration matrix, and sends the first registration matrix to the visual analysis engine, wherein the first registration matrix is ​​used to indicate a transformation relationship for aligning front and rear video frames in the video; The visual analysis engine processes the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix to obtain a second registration matrix, and sends the second registration matrix to the first image processing engine; The first image processing engine performs motion compensated time domain filtering MCTF processing on the video according to the first inter-frame offset data and the second registration matrix.

6. The method according to claim 5, characterized in that The operating system further includes a second electronic anti-shake module and a second image processing engine, and the video is a preview video; the method further includes: The visual analysis engine sends the second registration matrix to the second image processing engine; The second electronic anti-shake module processes the recorded video recorded by the camera using a second electronic anti-shake algorithm to obtain second inter-frame offset data, and sends the second inter-frame offset data to the second image processing engine, wherein the second inter-frame offset data is used to indicate an offset between previous and next video frames in the recorded video; The second image processing engine performs MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

7. The method according to claim 5 or 6, characterized in that The method further comprises: The first electronic anti-shake module sends the inter-frame perspective transformation matrix to the first image processing engine.

8. The method according to claim 6, characterized in that The method further comprises: The global motion estimation module sends the first inter-frame offset data to the second electronic anti-shake module.

9. The method according to any one of claims 5 to 8, characterized in that: The first electronic anti-shake algorithm is an electronic anti-shake algorithm provided by a device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by a device manufacturer of the electronic device.

10. The method according to claim 6 or 8, characterized in that: The second electronic anti-shake algorithm is an electronic anti-shake algorithm provided by a device manufacturer of the electronic device.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1-4 or claims 5-10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 4 or claims 5 to 10.

Images