Video Processing Method, Device, and Storage Medium

Abstract

The present application discloses a video processing method, device and storage medium, belonging to the field of video technology. The method includes: processing the video by using a first electronic anti-shake algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix; processing the video by using a global motion estimation algorithm to obtain a first registration matrix; processing the first inter-frame offset data, the inter-frame perspective transformation matrix and the first registration matrix to obtain a second registration matrix; and performing MCTF processing on the video according to the first inter-frame offset data and the second registration matrix. The present application can improve the influence on the MCTF processing effect after replacing the native electronic anti-shake algorithm of the chip system with the self-developed electronic anti-shake algorithm, so that the self-developed algorithm is more adapted to the MCTF processing scenario.

Description

Technical Field

[0001] This application relates to the field of video technology, and particularly to a video processing method, device, and storage medium. Background Art

[0002] With the rapid development of the mobile Internet and smartphones, mobile phone shooting has become an indispensable part of people's lives. To meet users' demands for a better shooting experience, mobile phone manufacturers have successively launched mobile phones equipped with chip platforms supporting video anti-shake.

[0003] For example, a chip platform supporting video anti-shake integrates a visual analytics engine (EVA), an image process engine (IPE), an electronic anti-shake module, and a global motion estimation module. The electronic anti-shake module is configured with a native electronic anti-shake algorithm provided by the chip platform, and the global motion estimation module is configured with a native global motion estimation algorithm provided by the chip platform. For a video shot by a mobile phone, the video can be processed by the electronic anti-shake module, the global motion estimation module, the visual analytics engine, and the image process engine for motion-compensated temporal filtering (MCTF) to improve the video display effect.

[0004] In some scenarios, to meet specific video processing requirements, such as to achieve a better video anti-shake effect, mobile phone manufacturers usually replace the native electronic anti-shake algorithm configured in the electronic anti-shake module with a self-developed electronic anti-shake algorithm by the mobile phone manufacturer, and replace the native global motion estimation algorithm configured in the global motion estimation module with a self-developed global motion estimation algorithm by the mobile phone manufacturer. However, after replacing the native algorithm with the self-developed algorithm, some unpredictable problems may occur in the MTCF processing effect, such as the problem of texture suspension in the video frame of the processed video, which affects the video display effect. Summary of the Invention

[0005] This application provides a video processing method, device, and storage medium, which can improve the influence of replacing the native algorithm with the self-developed algorithm on the MCTF processing effect, and make the self-developed algorithm more adaptable to the MCTF processing scenario.

[0006] In a first aspect, a video processing method is provided, which is applied to an electronic device. The method includes:

[0007] The video captured by the camera is processed using the first electronic image stabilization algorithm to obtain the first inter-frame offset data and the inter-frame perspective transformation matrix; the video is processed using the global motion estimation algorithm to obtain the first registration matrix; the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix are processed to obtain the second registration matrix; the video is subjected to MCTF processing according to the first inter-frame offset data and the second registration matrix.

[0008] In this way, calculating the registration matrix used in the MCTF processing process based on the inter-frame offset data and the registration matrix output by the global motion estimation algorithm in the original process can be modified to calculating the registration matrix used in the MCTF processing process based on the inter-frame offset data output by the electronic image stabilization algorithm and the registration matrix output by the global motion estimation algorithm, so that the inter-frame offset data used when calculating the registration matrix is consistent with the inter-frame offset data used in the MCTF processing process, thereby reducing or eliminating the texture suspension problem of the video picture that appears after replacing the self-developed algorithm, reducing the impact of replacing the self-developed algorithm on the MCTF processing effect, and making the self-developed algorithm more adaptable to the MCTF processing scenario.

[0009] Among them, the first inter-frame offset data is used to indicate the offset between the front and rear video frames in the video. For example, the first inter-frame offset data can be inter-frame grid data, which is used to indicate the offset of the grid points of the front and rear video frames in the video. The inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between the front and rear video frames in the video. The first registration matrix is used to indicate the transformation relationship for aligning the front and rear video frames in the video. That is to say, the registration matrix is used to describe the transformation relationships such as translation, rotation, and scaling between the front and rear video frames, and this transformation relationship is used to align the front and rear video frames.

[0010] In one embodiment, the above video is the preview video captured by the camera. That is to say, the preview video captured by the mobile phone can be subjected to MCTF processing according to the video processing method provided in the embodiments of the present application, reducing the impact of replacing the research algorithm on the display effect of the preview video.

[0011] In one embodiment, the second electronic image stabilization algorithm can also be used to process the recorded video captured by the camera to obtain the second inter-frame offset data, and the second inter-frame offset data is used to indicate the offset between the front and rear video frames in the recorded video; the recorded video is subjected to MCTF processing according to the second inter-frame offset data and the second registration matrix. By performing MCTF processing on the recorded video, the video display effect of the recorded video can be improved.

[0012] In one embodiment, the operating system of the electronic device includes a first electronic image stabilization module, a global motion estimation module, a visual analysis engine, and a first image processing engine. The first electronic image stabilization module can process the video using a first electronic image stabilization algorithm to obtain first inter-frame offset data; the global motion estimation module can process the video using a global motion estimation algorithm to obtain a first registration matrix; the visual analysis engine processes the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix to obtain a second registration matrix; and the first image processing engine performs MCTF processing on the video according to the first inter-frame offset data and the second registration matrix.

[0013] In one embodiment, the operating system further includes a second electronic image stabilization module and a second image processing engine. The second electronic image stabilization module can also process the recorded video captured by the camera using a second electronic image stabilization algorithm to obtain second inter-frame offset data; and the second image processing engine performs MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

[0014] In one embodiment, the first electronic image stabilization algorithm and the second electronic image stabilization algorithm are electronic image stabilization algorithms provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device, rather than a native algorithm provided by the chip platform. Among them, the first electronic image stabilization algorithm and the second electronic image stabilization algorithm can be the same or different, and the embodiments of the present application do not limit this.

[0015] In a second aspect, an electronic device is provided. The operating system of the electronic device includes a first electronic image stabilization module, a global motion estimation module, a visual analysis engine, and a first image processing engine; the first electronic image stabilization module processes the video captured by the camera using a first electronic image stabilization algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix, sends the first inter-frame offset data and the inter-frame perspective transformation matrix to the visual analysis engine, and sends the first inter-frame offset data to the first image processing engine; the global motion estimation module processes the video using a global motion estimation algorithm to obtain a first registration matrix, and sends the first configuration matrix to the visual analysis engine; the visual analysis engine processes the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix to obtain a second registration matrix, and sends the second registration matrix to the first image processing engine; the first image processing engine performs motion compensation-based temporal filtering MCTF processing on the video according to the first inter-frame offset data and the second registration matrix.

[0016] Among them, the first inter-frame offset data is used to indicate the offset of the corresponding grid regions in two consecutive video frames in the video, the inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between two consecutive video frames in the video, and the first registration matrix is used to indicate the transformation relationship for aligning two consecutive video frames in the video.

[0017] In the embodiments of the present application, the data path between modules in the original process is transformed. Specifically, a new path for the preview electronic image stabilization module to send the inter-frame offset data to the visual analysis engine is added, and the path for the global motion estimation module to send the inter-frame offset data to the visual analysis engine is truncated. As a result, the preview electronic image stabilization module can send the inter-frame offset data it calculates to the visual analysis engine and the preview image processing engine. Furthermore, the visual analysis engine can calculate a new registration matrix based on the inter-frame offset data output by the preview electronic image stabilization module instead of the inter-frame offset data output by the global motion estimation module in the original process, so as to ensure that the inter-frame offset data used by the visual analysis engine is consistent with the inter-frame offset data used by the preview image processing engine. After the transformation of the path, the texture suspension problem in the MCTF processing effect in a bright environment is alleviated or eliminated. Therefore, to a certain extent, the impact of replacing the original algorithm with the self-developed algorithm on the video display effect is reduced, making the self-developed algorithm more adaptable to the chip platform and the MCTF processing scenario.

[0018] In one embodiment, the operating system further includes a second electronic image stabilization module and a second image processing engine, and the video is a preview video; the visual analysis engine sends the second registration matrix to the second image processing engine; the second image stabilization module processes the recorded video captured by the camera using the second electronic image stabilization algorithm to obtain the second inter-frame offset data, and sends the second inter-frame offset data to the second image processing engine; the second image processing engine performs MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

[0019] That is to say, the preview video can be subjected to MCTF processing through the first electronic image stabilization module, the global motion estimation module, the visual analysis engine, and the first image processing engine, and the recorded video can be subjected to MCTF processing through the second electronic image stabilization module and the second image processing engine.

[0020] In one embodiment, the first electronic image stabilization module can also send the inter-frame perspective transformation matrix to the first image processing engine.

[0021] In one embodiment, the global motion estimation module can also send the first inter-frame offset data to the second electronic image stabilization module.

[0022] In one embodiment, the first electronic image stabilization algorithm and the second electronic image stabilization algorithm are electronic image stabilization algorithms provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device, rather than a native algorithm provided by the chip platform. That is, the native image stabilization algorithm in the electronic image stabilization module can be replaced with the self-developed electronic image stabilization algorithm of the device manufacturer in advance, and the native global motion estimation algorithm in the global motion estimation module can be replaced with the self-developed global motion estimation algorithm of the device manufacturer. Among them, the first electronic image stabilization algorithm and the second electronic image stabilization algorithm can be the same or different, and the embodiments of the present application do not limit this.

[0023] In a third aspect, a video processing device is provided, and the video processing device has a function of implementing the behavior of the video processing method in the first aspect or the second aspect above. The video processing device includes at least one module, and the at least one module is used to implement the video processing method provided in the first aspect above.

[0024] In a fourth aspect, a video processing device is provided. The structure of the video processing device includes a processor and a memory. The memory is used to store a program that supports the video processing device to execute the video processing method provided in the first aspect or the second aspect above, and to store data related to implementing the video processing method described in the first aspect or the second aspect above. The processor is configured to execute the program stored in the memory. The video processing device may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

[0025] In a fifth aspect, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium, and when it runs on a computer, it causes the computer to execute the video processing method described in the first aspect or the second aspect above.

[0026] In a sixth aspect, a computer program product including instructions is provided. When it runs on a computer, it causes the computer to execute the video processing method described in the first aspect above.

[0027] In a seventh aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to cause the electronic device to execute the video processing method described in the first aspect or the second aspect above.

[0028] The technical effects obtained in the third aspect, the fourth aspect, the fifth aspect, the sixth aspect, and the seventh aspect above are similar to the technical effects obtained by the corresponding technical means in the first aspect or the second aspect above, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic structural diagram of a terminal provided by an embodiment of the present application;

[0030] Figure 2 It is a block diagram of a software system of a terminal provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic flowchart of processing a video by a native algorithm based on a chip platform provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of a video processing flow after replacing the native algorithm of a chip platform with a self-developed algorithm of an electronic device provided by an embodiment of the present application;

[0033] Figure 5 It is a comparative schematic diagram of normal and abnormal video pictures of MCTF processing effects provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of an optimized video processing flow after replacing the native algorithm of a chip platform with a self-developed algorithm of an electronic device provided by an embodiment of the present application;

[0035] Figure 7 It is a flowchart of a video processing method provided by an embodiment of the present application. Specific embodiments

[0036] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0037] It should be understood that the "multiple" mentioned in the present application refers to two or more. In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can represent A or B; the "and / or" herein is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, for the convenience of clearly describing the technical solutions of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily limit being different.

[0038] Before elaborating on the video processing method provided by the embodiments of the present application in detail, the terms involved in the embodiments of the present application will be explained first.

[0039] Video Jitter: Video jitter means that the imaging position of an object deviates, mainly caused by the high-frequency vibration of the camera. As the camera shakes up, down, left, and right, pixel displacement will occur in the video image. The video captured by a stable camera is characterized by continuity and smoothness, while for a camera with jitter, the captured video often shows an irregular and discontinuous state. Therefore, eliminating camera jitter is to eliminate the discontinuous changes between adjacent frames in the video.

[0040] Electronic Image Stabilization: That is, electronic image stabilization (EIS). EIS is a technology that realizes image stabilization through software algorithms. It can estimate the motion between consecutive video frames at the digital image algorithm level and perform reverse compensation to obtain a stable video image.

[0041] Global Motion Estimation: Estimate the global motion model between the previous and subsequent video frames of the video. Common global motion estimation methods include optical flow method, feature matching method, block matching method, gray projection method, or gyroscope estimation method, etc.

[0042] Each frame of the image sequence is divided into many non-overlapping macroblocks, and it is considered that the displacement of all pixels within the macroblock is the same. Then, for each macroblock within a certain given search range in the reference frame, the block that is most similar to the current block, that is, the matching block, is found according to a certain matching criterion. The relative displacement between the matching block and the current block is the motion vector.

[0043] Engine of Visual Analytics (EVA): In this application, EVA is mainly used to achieve image alignment and correction through visual analysis algorithms.

[0044] Image Process Engine (IPE): IPE is a hardware processing unit on the chip, mainly responsible for secondary processing of the image data processed by the Image Front End (IFE), including functions such as image denoising, sharpening, contrast adjustment, color mapping, etc. Finally, the processed image data is output to the display screen or encoder.

[0045] Motion Compensated Temporal Filter (MCTF): MCTF is an image processing technology that can be used to reduce motion blur for image alignment and video denoising. MCTF can restore a clear image by analyzing the motion information in the video frame sequence.

[0046] Before providing a detailed explanation of the video processing method provided in the embodiments of the present application, the electronic devices involved in the embodiments of the present application will be described first.

[0047] The video processing method provided in the embodiments of the present application can be applied to an electronic device with video shooting and video processing functions. The electronic device can be a terminal such as a mobile phone, a tablet computer, a smart wearable device, or a computer.

[0048] Figure 1 is a schematic structural diagram of a terminal provided in the embodiments of the present application. Refer to Figure 1 , the terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0049] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0051] Among them, the controller may be the nerve center and command center of the terminal 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0052] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0053] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0054] The wireless communication function of the terminal 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor, baseband processor, etc.

[0055] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the terminal 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be provided in the same device.

[0056] The wireless communication module 160 can provide solutions for wireless communications such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the terminal 100. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through antenna 2 and radiate them out.

[0057] In some embodiments, antenna 1 of the terminal 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technologies.

[0058] The terminal 100 implements the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0059] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0060] The terminal 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor, etc.

[0061] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP may be set in the camera 193. In some embodiments, the algorithm modules corresponding to the ISP may include an electronic image stabilization module and a global motion estimate (GME) module.

[0062] The camera 193 is used to capture static images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the terminal 100 may include one or N cameras 193, where N is an integer greater than 1.

[0063] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0064] The video codec is used to compress or decompress digital videos. The terminal 100 can support one or more video codecs. In this way, the terminal 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0065] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the terminal 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0066] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0067] The internal memory 121 can be used to store computer-executable program code, and the computer-executable program code includes instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the terminal 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0068] The terminal 100 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0069] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0070] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal 100 detects the intensity of the touch operation according to the pressure sensor 180A. The terminal 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0071] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch control screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor 180K can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the terminal 100, at a different position from the display screen 194.

[0072] Next, the software system of the terminal 100 will be described.

[0073] The software system of the terminal 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, the Android system with a layered architecture is taken as an example to exemplarily describe the software system of the terminal 100.

[0074] Figure 2 It is a block diagram of a software system of a terminal 100 provided by an embodiment of the present application. Refer to Figure 2 In the layered architecture, the software is divided into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom are the application layer, the application framework layer, the Android Runtime and the system layer, the kernel layer, and the hardware abstraction layer (HAL).

[0075] The application layer may include a series of application packages. As Figure 2 shown, the application packages may include applications such as a camera, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, a short message, etc.

[0076] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. As Figure 2As shown in the figure, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make this data accessible to application programs. This data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of an application program. The display interface can be composed of one or more views. For example, it includes a view for displaying a text message notification icon, a view for displaying text, and a view for displaying pictures. The telephone manager is used to provide the communication function of the terminal 100, such as the management of call status (including connection, disconnection, etc.). The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables application programs to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of a download, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application program. The notification manager can also be a notification that appears on the screen in the form of a dialogue window, such as prompting text information in the status bar, emitting a prompt tone, vibrating the electronic device, flashing the indicator light, etc.

[0077] Android Runtime includes a core library and a virtual machine. Android Runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0078] The system library can include multiple functional modules, such as: surface manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The Media Libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The Media Libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is the drawing engine for 2D drawing.

[0079] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0080] The HAL layer is the interface layer between the kernel layer and the hardware layer, and its purpose is to abstract the hardware. The hardware abstraction layer is a programming layer that allows the operating system to interact with hardware devices at the logical layer rather than the hardware layer. For example, the HAL layer includes the EVA hardware abstraction layer corresponding to the EVA in the hardware layer, and the IPE hardware abstraction layer corresponding to the IPE. In addition, the HAL layer can also include an image processing module, and the image processing module includes a global motion estimation module and at least one electronic image stabilization module. For example, this at least one electronic image stabilization module includes a first electronic image stabilization module and a second electronic image stabilization module. For example, as Figure 2 shown, the first electronic image stabilization module and the second electronic image stabilization module can be a preview electronic image stabilization module and a video electronic image stabilization module respectively. The preview electronic image stabilization module is used to perform video stabilization processing on the preview video, and the video electronic image stabilization module is used to perform video stabilization processing on the recorded video. The EVA hardware abstraction layer includes a visual analysis engine. The IPE hardware abstraction layer includes at least one image processing engine, such as including a first image processing engine and a second image processing engine. For example, as Figure 2 shown, the first image processing engine and the second image processing engine can be a preview image processing engine and a video image processing engine respectively. The preview image processing engine is used to perform video processing on the preview video, and the video image processing engine is used to perform video processing on the recorded video. As an example, each image processing engine can include an image correction and alignment module (image correction and alignment, ICA), and the image correction and alignment module can be configured with the native MCTF algorithm of the chip platform to perform MCTF processing on the video.

[0081] The hardware layer at least includes a processor, a display, an audio device, a camera group, an EVA, and an IPE. The ISP, EVA, and IPE can be integrated on the chip platform carried by the electronic device.

[0082] Next, in combination with the video recording scenario, the working processes of the software and hardware of the terminal 100 will be exemplarily described.

[0083] The camera application in the application layer can be displayed on the screen of the electronic device in the form of an icon. When the icon of the camera application is clicked and triggered by the user, the electronic device starts to run the camera application. When the recording option in the camera application interface is clicked and triggered by the user, the camera application calls the corresponding interface of the camera application in the application framework layer to start the camera driver, turns on one or more cameras in the camera group, and performs recording through the turned-on cameras to obtain a preview video stream and a recorded video stream. For the preview video stream, the preview electronic image stabilization module, global motion estimation module, visual analysis engine, and preview image processing engine in the hardware abstraction layer can be called to perform video processing on the preview video stream. For the recorded video stream, the video electronic image stabilization module and video image processing engine in the hardware abstraction layer can be called to process the recorded video stream.

[0084] Next, the inventive concept of the embodiments of the present application will be introduced in detail.

[0085] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of processing a video based on the native algorithm of the chip platform provided by the embodiments of the present application. In the case where the global motion estimation module is configured with the native global motion estimation algorithm provided by the chip platform and the electronic image stabilization module is configured with the native electronic image stabilization algorithm provided by the chip platform, the process of performing MCTF processing on the video by each module in the hardware abstraction layer in Figure 2 can be as shown in Figure 3 . The chip platform can be a chip platform, etc., and the embodiments of the present application do not make any limitations in this regard.

[0086] As shown in Figure 3As shown, the preview video captured by the camera of the electronic device can be input into the preview electronic image stabilization module and the global motion estimation module respectively. On the one hand, the preview electronic image stabilization module uses the native electronic image stabilization algorithm to calculate the preview video, obtaining the inter-frame offset data 1 and the inter-frame perspective transformation matrix, sending the inter-frame perspective transformation matrix to the visual analysis engine, and sending the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module uses the native global motion estimation algorithm to calculate the preview video, obtaining the inter-frame offset data 2 and the registration matrix 1, sending the inter-frame offset data 2 and the registration matrix 1 to the visual analysis engine, and sending the inter-frame offset data 1 to the video electronic image stabilization module. The visual analysis engine can generate the registration matrix 2 according to the inter-frame perspective transformation matrix sent by the preview electronic image stabilization module, the registration matrix 1 and the inter-frame offset data 2 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0087] In addition, the recorded video captured by the camera can be input into the video electronic image stabilization module. The video electronic image stabilization module uses the native electronic image stabilization algorithm to calculate the recorded video, obtaining the inter-frame offset data 3, and sending the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0088] Among them, the inter-frame offset data is used to indicate the offset between the front and rear video frames in the corresponding video. For example, the inter-frame offset data can be inter-frame grid data, which is used to indicate the offset of the grid points of the front and rear video frames in the video. The inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between the front and rear video frames in the corresponding video. The registration matrix is used to indicate the transformation relationship for aligning the front and rear video frames. That is to say, the registration matrix is used to describe the transformation relationships such as translation, rotation, and scaling between the front and rear video frames, and this transformation relationship is used to align the front and rear video frames.

[0089] Currently, the native algorithm based on the chip platform can be in accordance with Figure 3The video processing flow shown processes a video to achieve video anti-shake. The anti-shake effect is one of the important indicators for evaluating video quality. During actual use and debugging, it is found that the native electronic anti-shake algorithm provided by the chip platform may have poor robustness, with problems such as poor anti-shake effect, anti-shake failure, or even frozen screens. To solve the above problems, device manufacturers of electronic devices can replace the native electronic anti-shake algorithm of the chip platform with a self-developed electronic anti-shake algorithm and replace the native global motion estimation algorithm of the chip platform with a self-developed global motion estimation algorithm to achieve a better video anti-shake effect.

[0090] Please refer to Figure 4 , Figure 4 which is a schematic diagram of a video processing flow after replacing the native algorithm of the chip platform with a self-developed algorithm of the electronic device provided by an embodiment of the present application. After replacing the native global motion estimation algorithm of the global motion estimation module with the self-developed global motion estimation algorithm provided by the device manufacturer of the electronic device and replacing the native electronic anti-shake algorithm of the electronic anti-shake module with the self-developed electronic anti-shake algorithm provided by the device manufacturer, through Figure 2 the process of performing MCTF processing on the video by each module in the hardware abstraction layer can be as shown in Figure 4 .

[0091] As shown in Figure 4 , the preview video captured by the camera can be input to the preview electronic anti-shake module and the global motion estimation module respectively. On the one hand, the preview electronic anti-shake module uses the first self-developed electronic anti-shake algorithm of the device to calculate the preview video, obtains the inter-frame offset data 1 and the inter-frame perspective transformation matrix, sends the inter-frame perspective transformation matrix to the visual analysis engine, and sends the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module uses the self-developed global motion estimation algorithm of the device to calculate the preview video, outputs the registration matrix 1 and the inter-frame offset data 2, sends the inter-frame offset data 2 and the registration matrix 1 to the visual analysis engine, and sends the inter-frame offset data 1 to the video electronic anti-shake module. The visual analysis engine can generate the registration matrix 2 based on the inter-frame perspective transformation matrix sent by the preview electronic anti-shake module, the registration matrix 1 and the inter-frame offset data 2 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic anti-shake module and the registration matrix sent by the visual analysis engine.

[0092] In addition, the recorded video captured by the camera can be input into the video electronic image stabilization module. The video electronic image stabilization module calculates the recorded video using a second electronic image stabilization algorithm developed by the device itself to obtain the inter-frame offset data 3, and sends the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix sent by the visual analysis engine.

[0093] However, after replacing the native algorithm with the self-developed algorithm, some unpredictable problems may occur in the processed video. For example, the inventor found that there is a problem of texture suspension in the video picture of the processed video, which greatly affects the video display effect. Please refer to Figure 5 , Figure 5 FIG. is a comparison schematic diagram of video pictures with normal and abnormal MCTF processing effects provided by an embodiment of the present application. Figure 5 In FIG. (a), it is a schematic diagram of a video picture with normal MCTF processing effect, Figure 5 In FIG. (b), it is a schematic diagram of a video picture with abnormal MCTF processing effect after replacing the native algorithm with the self-developed algorithm. As Figure 5 shown in FIG. (b), after replacing the native algorithm with the self-developed algorithm, obvious texture suspension phenomenon appears in the video picture. The texture suspension phenomenon refers to that the area is significantly darker and blurred, and there is an obvious shadow suspension phenomenon.

[0094] Regarding the problem of texture suspension in the video picture after replacing the native algorithm with the self-developed algorithm, the inventor conducted a series of analyses and studies on this problem to find out the reason for this problem, so as to solve this problem targeted according to the found reason, thereby reducing or eliminating the impact of replacing the native algorithm with the self-developed algorithm on the video display effect.

[0095] First of all, the inventor respectively obtained the processing logs before and after replacing the native algorithm with the self-developed algorithm (the self-developed electronic image stabilization algorithm and the global motion estimation algorithm). By carefully comparing the processing logs before and after replacing the native algorithm with the self-developed algorithm, it was found that there are significant differences in the order of magnitude of the data calculated by the self-developed algorithm (hereinafter referred to as the self-developed process) and the native algorithm of the chip platform (hereinafter referred to as the native process). For example, for the same offset, different orders of magnitude of data are used for description. Therefore, it is initially suspected that the above-mentioned texture suspension problem of the video picture is caused by the difference in the order of magnitude. However, after consulting the chip manufacturer about this suspicion, the chip manufacturer feedback that the difference in the order of magnitude will not cause this problem.

[0096] After that, the inventor re-analyzed and studied other causes of the above problems. By comparing the logs of the self-developed process and the native process, it was found that the inter-frame offset data calculated by the preview electronic image stabilization module and the global motion estimation module in the native process was consistent, that is, the inter-frame offset data 1 calculated by the preview electronic image stabilization module was the same as the inter-frame offset data 2 calculated by the global motion estimation module. However, the inter-frame offset data calculated by the preview electronic image stabilization module and the global motion estimation module in the self-developed process was inconsistent, that is, the inter-frame offset data 1 calculated by the preview electronic image stabilization module was different from the inter-frame offset data 2 calculated by the global motion estimation module. In addition, by analyzing the logs of the native process, it was found that the registration matrix 2 received by the image processing engine (preview image processing engine and video image processing engine) was numerically inconsistent with the registration matrix 1 calculated by the global motion estimation module. Therefore, it can be speculated that the registration matrix 2 received by the image processing engine was recalculated. That is, after the registration matrix 1 calculated by the global motion estimation module was output to the visual analysis engine, the visual analysis engine would calculate a new registration matrix 2 based on the inter-frame perspective transformation matrix sent by the preview electronic image stabilization module, as well as the registration matrix 1 and the inter-frame offset data 2 sent by the global motion estimation module, and then send the new registration matrix 2 to the image processing engine for MCTF processing. In addition, the inventor identified through analyzing the logs of the native process that the inter-frame offset data used by the visual analysis engine was the inter-frame offset data 2 calculated by the global motion estimation module, and moreover, the inter-frame offset data 2 used by the visual analysis engine was consistent with the inter-frame offset data 1 used by the preview image processing engine. However, through analyzing the logs of the self-developed process, it was identified that the inter-frame offset data 1 used by the preview image processing engine was inconsistent with the inter-frame offset data 2 used by the visual analysis engine. Based on this, the inventor suspected that the texture suspension problem of the above video picture was caused by the inconsistency of the inter-frame offset data calculated by the preview electronic image stabilization module and the global motion estimation module in the self-developed process, which in turn caused the inconsistency between the inter-frame offset data used by the visual analysis engine and the inter-frame offset data used by the preview image processing engine.

[0097] In an embodiment of the present application, in order to ensure that the inter-frame offset data used by the visual analysis engine is consistent with the inter-frame offset data used by the preview image processing engine, a video processing method is provided. In this method, after processing the video using the first electronic image stabilization algorithm to obtain the first inter-frame offset data and the inter-frame perspective transformation matrix, and processing the video using the global motion estimation algorithm to obtain the first registration matrix, the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix can be processed to obtain a second registration matrix, and then the video can be subjected to MCTF processing according to the first inter-frame offset data and the second registration matrix.

[0098] In this way, the registration matrix used in the MCTF processing can be modified from being calculated based on the inter-frame offset data and the registration matrix output by the global motion estimation algorithm in the original process to being calculated based on the inter-frame offset data output by the electronic image stabilization algorithm and the registration matrix output by the global motion estimation algorithm, so that the inter-frame offset data used when calculating the registration matrix is consistent with the inter-frame offset data used in the MCTF processing, thereby reducing or eliminating the texture suspension problem of the video picture that appears after the replacement of the self-developed algorithm, and reducing the impact of the replacement of the self-developed algorithm for the original algorithm on the video display effect.

[0099] Please refer to Figure 6 , Figure 6 which is a schematic diagram of an optimized video processing flow after replacing the original algorithm of the chip platform with the self-developed algorithm of the electronic device. After replacing the original global motion estimation algorithm of the global motion estimation module with the self-developed global motion estimation algorithm provided by the device manufacturer of the electronic device, and replacing the original electronic image stabilization algorithms of the preview electronic image stabilization module and the video electronic image stabilization module with the self-developed electronic image stabilization algorithm provided by the device manufacturer, the video processing method provided by the embodiments of the present application can be used to Figure 2 optimize the process of the MCTF processing of the video by each module in the hardware abstraction layer in Figure 6 as shown.

[0100] As Figure 6As shown, the preview video captured by the camera can be input into the preview electronic image stabilization module and the global motion estimation module respectively. On the one hand, the preview electronic image stabilization module processes the preview video using the first electronic image stabilization algorithm developed by the device itself, obtaining the inter-frame offset data 1 and the inter-frame perspective transformation matrix, and sending the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the visual analysis engine, and also sending the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine. On the other hand, the global motion estimation module processes the preview video using the global motion estimation method developed by the device itself, obtaining the inter-frame offset data 2 and the registration matrix 1, sending the registration matrix 1 to the visual analysis engine, and sending the inter-frame offset data 1 to the video electronic image stabilization module. The visual analysis engine can generate a new registration matrix 2 based on the inter-frame offset data 1 and the inter-frame perspective transformation matrix sent by the preview electronic image stabilization module, and the registration matrix 1 sent by the global motion estimation module, and send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively. The preview image processing engine can perform MCTF processing on the preview video through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine. In addition, the recorded video captured by the camera can be input into the video electronic image stabilization module, and the video electronic image stabilization module processes the recorded video using the second electronic image stabilization algorithm developed by the device itself, obtaining the inter-frame offset data 3, and sending the inter-frame offset data 3 to the video image processing engine. The video image processing engine can perform MCTF processing on the recorded video through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0101] Comparing Figure 6 with Figure 4 and analyzing, it can be seen that Figure 6 the data path between modules has been transformed compared to the original process. That is, a new path for the preview electronic image stabilization module to send the inter-frame offset data to the visual analysis engine has been added, and the path for the global motion estimation module to send the inter-frame offset data to the visual analysis engine has been truncated. This enables the preview electronic image stabilization module to send the inter-frame offset data it calculates to the visual analysis engine and the preview image processing engine. Furthermore, it allows the visual analysis engine to calculate a new registration matrix based on the inter-frame offset data output by the preview electronic image stabilization module instead of the inter-frame offset data output by the global motion estimation module in the original process, ensuring that the inter-frame offset data used by the visual analysis engine is consistent with that used by the preview image processing engine. After the transformation of the path, the texture suspension problem in the MCTF processing effect in the bright light environment is alleviated or eliminated. Therefore, to a certain extent, the impact of replacing the original algorithm with the self-developed algorithm on the video display effect is reduced, making the self-developed algorithm more adaptable to the MCTF processing scenario.

[0102] In order to further reduce or eliminate the texture suspension problem in the MCTF processing effect in low-light environments, the inventors further analyzed and studied the native process and the self-developed process. By comparing the logs of the self-developed process and the native process, the inventors found that there are differences in the data received by the visual analysis engine in the two processes. In the self-developed process, the registration matrix received by the visual analysis engine is the identity matrix, that is, the default value without calculation. From this, it can be speculated that the registration matrix 1 received by the visual analysis engine is not the registration matrix calculated by the global motion estimation module using the self-developed algorithm, but the preset initial value. Therefore, in the self-developed process, the electronic image stabilization module and the global motion estimation module only output the inter-frame offset data and the perspective transformation matrix, and output one less registration matrix than the native process. By analyzing and studying the logs of the self-developed process, it is found that the self-developed algorithm integrates the relevant data of the registration matrix into the inter-frame offset data.

[0103] As Figure 6 shown, in the embodiments of the present application, the self-developed algorithm also splits the registration-related information integrated in the inter-frame offset data according to the category of the native algorithm data, and can output the inter-frame offset data and the registration matrix respectively, and study the change trend of the registration matrix data of the native platform for adaptation. In this way, the texture suspension problem in the MCTF processing effect in low-light environments can be reduced or eliminated to a certain extent.

[0104] In addition, subsequent chip manufacturers also reported that the difference in the order of magnitude of the output results would also affect the MCTF effect. Therefore, in the embodiments of the present application, the order of magnitude of the data output by the self-developed algorithm can also be compressed according to the order of magnitude of the data output by the native algorithm, so as to narrow the difference in the order of magnitude between the data output by the self-developed algorithm and the data output by the native algorithm. In this way, the texture suspension problem in the MCTF processing effect in low-light environments can be further reduced or eliminated.

[0105] Next, in combination with Figure 2 and Figure 6 a detailed description of the video processing method provided by the embodiments of the present application will be given.

[0106] Figure 7 is a flowchart of a video processing method provided by the embodiments of the present application. Taking a mobile phone as an example, as Figure 7 shown, the method includes the following steps:

[0107] Step 701: The user starts the camera application.

[0108] For example, the user can click on the application icon of the camera application to start the camera application.

[0109] Step 702: In response to the user's start operation, the camera application starts.

[0110] After the camera application is launched, a preview interface of the camera application can be displayed, and the preview interface includes a recording option.

[0111] Step 703: The user clicks on the recording option in the application interface of the camera application.

[0112] Step 704: In response to the user's click operation, the camera application sends a call instruction to the camera.

[0113] The call instruction is used to call the camera for shooting. The camera(s) called can be one or more in the camera group, and this application embodiment does not limit this.

[0114] Step 705: The camera performs video shooting according to the call instruction, generating a preview video stream and a recording video stream.

[0115] After the camera performs video shooting, it can generate a preview video stream and a recording video stream respectively. The preview video stream is used for video preview and includes consecutive video frames to be previewed and displayed. The recording video stream is used for video storage and includes consecutive video frames to be stored. The preview video stream and the recording video stream can also be respectively referred to as the preview stream and the video stream, and this application embodiment does not limit this.

[0116] After generating the preview video stream, the camera can send the preview video to the preview electronic image stabilization module and the global motion estimation module respectively. After generating the recording video stream, the camera can send the recording video stream to the video electronic image stabilization module.

[0117] Step 706: The camera sends the preview video stream to the preview electronic image stabilization module.

[0118] Among them, the preview electronic image stabilization module is configured with a first electronic image stabilization algorithm independently developed by the mobile phone manufacturer, rather than the native electronic image stabilization algorithm of the chip platform. That is, the native electronic image stabilization algorithm of the chip platform originally configured in the preview electronic image stabilization module is replaced by the first electronic image stabilization algorithm independently developed by the mobile phone manufacturer.

[0119] It should be noted that the first electronic image stabilization algorithm independently developed by the mobile phone manufacturer and the native electronic image stabilization algorithm of the chip platform have the same algorithm principle, and only have certain differences in the algorithm implementation details, such as the specific parameter settings and the order of magnitude of data in the algorithm.

[0120] Step 707: The camera sends the preview video stream to the global motion estimation module.

[0121] Among them, the global motion estimation module is configured with a globally motion estimation algorithm developed by the mobile phone manufacturer itself, rather than the native globally motion estimation algorithm of the chip platform. That is to say, the globally motion estimation algorithm of the chip platform originally configured in the global motion estimation module is replaced by the globally motion estimation algorithm developed by the mobile phone manufacturer itself.

[0122] It should be noted that the globally motion estimation algorithm developed by the mobile phone manufacturer itself has the same algorithm principle as the native globally motion estimation algorithm of the chip platform, and there are only some differences in the algorithm implementation details, such as the specific parameter settings in the algorithm and the order of magnitude of the data, etc.

[0123] Step 708: The camera sends the recorded video stream to the video electronic image stabilization module.

[0124] Among them, the video electronic image stabilization module is configured with a second electronic image stabilization algorithm developed by the mobile phone manufacturer itself, rather than the native electronic image stabilization algorithm of the chip platform. That is to say, the native electronic image stabilization algorithm of the chip platform originally configured in the video electronic image stabilization module is replaced by the second electronic image stabilization algorithm developed by the mobile phone manufacturer itself.

[0125] It should be noted that the second electronic image stabilization algorithm developed by the mobile phone manufacturer itself has the same algorithm principle as the native electronic image stabilization algorithm of the chip platform, and there are only some differences in the algorithm implementation details, such as the specific parameter settings in the algorithm and the order of magnitude of the data, etc.

[0126] It should also be noted that the second electronic image stabilization algorithm configured in the video electronic image stabilization module and the first electronic image stabilization algorithm configured in the preview electronic image stabilization module can be the same electronic image stabilization algorithm or different electronic image stabilization algorithms. The embodiments of the present application do not limit this.

[0127] As an example, the electronic image stabilization algorithms configured in the video electronic image stabilization module and the preview electronic image stabilization module are the same, that is, the first electronic image stabilization algorithm and the second electronic image stabilization algorithm are the same.

[0128] Step 709: The preview electronic image stabilization module uses the first electronic image stabilization algorithm developed by the mobile phone itself to calculate the video data corresponding to the preview video stream, and obtains the inter-frame offset data 1 and the inter-frame perspective transformation matrix.

[0129] By using the first electronic image stabilization algorithm to calculate the video data corresponding to the preview video stream, the motion information of consecutive video frames in the preview video stream can be obtained, and the motion information can include the inter-frame offset data 1 and the inter-frame perspective transformation matrix.

[0130] Among them, the inter-frame offset data 1 is used to indicate the offset between the front and rear video frames in the preview video stream. For example, the inter-frame offset data 1 can be inter-frame grid data, which is used to indicate the offset of grid points between the front and rear video frames, such as the offset of grid points on the X-axis and Y-axis between two consecutive video frames.

[0131] Among them, the inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between the front and rear video frames in the preview video stream, such as the projection mapping relationship of projecting the previous video frame onto the next video frame.

[0132] Step 710: The preview electronic image stabilization module sends the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the visual analysis engine.

[0133] It should be noted that, in the embodiments of the present application, different from the native process of the chip platform shown above Figure 3 After calculating the inter-frame offset data 1 and the inter-frame perspective transformation matrix, the preview electronic image stabilization module needs to send the inter-frame offset data 1 and the inter-frame perspective transformation matrix to both the visual analysis engine and the preview image processing engine simultaneously to ensure that the inter-frame offset data obtained by the visual analysis engine and the preview image processing engine is consistent.

[0134] Step 711: The preview electronic image stabilization module sends the inter-frame offset data 1 and the inter-frame perspective transformation matrix to the preview image processing engine.

[0135] Step 712: The global motion estimation module calculates the video data corresponding to the preview video stream using the self-developed global motion estimation algorithm of the mobile phone, and obtains the inter-frame offset data 2 and the registration matrix 1.

[0136] By calculating the video data corresponding to the preview video stream using the global motion estimation algorithm, the global motion model of consecutive video frames in the preview video stream can be obtained. The global motion module includes the inter-frame offset data 2 and the registration matrix 1.

[0137] Among them, the inter-frame offset data 2 is also used to indicate the offset between the front and rear video frames in the preview video stream. However, due to different internal algorithms, the numerical value of the inter-frame offset data 2 may be different from the inter-frame offset data 1 output by the above preview electronic image stabilization module.

[0138] Among them, the registration matrix 1 is used to indicate the transformation relationship for aligning the front and rear video frames in the preview video stream. That is, according to this transformation relationship, the front and rear video frames in the preview video stream can be aligned, such as aligning the previous video frame with the current video frame.

[0139] Step 713: The global motion estimation module sends the registration matrix 1 to the visual analysis engine.

[0140] It should be noted that, in the embodiments of the present application, different from the aboveFigure 3 Different from the native process of the chip platform shown, the global motion estimation module does not need to send the inter-frame offset data 2 to the visual analysis engine to avoid the inconsistency between the inter-frame offset data obtained by the visual analysis engine and the inter-frame offset data obtained by the preview image processing engine.

[0141] Step 714: The global motion estimation module sends the inter-frame offset data 2 to the video electronic image stabilization module.

[0142] Step 715: The visual analysis engine generates a registration matrix 2 based on the inter-frame offset data 1, the inter-frame perspective transformation matrix, and the registration matrix 1.

[0143] The visual analysis engine can use the native visual analysis model of the chip platform it is configured with to calculate the inter-frame offset data 1, the inter-frame perspective transformation matrix, and the registration matrix 1, so as to output a new registration matrix, that is, the registration matrix 2.

[0144] Among them, the chip platform can be a chip platform, etc.

[0145] Step 716: The visual analysis engine sends the registration matrix 2 to the preview image processing engine.

[0146] After calculating the registration matrix 2, the visual analysis engine can send the registration matrix 2 to the preview image processing engine and the video image processing engine respectively.

[0147] Step 717: The visual analysis engine sends the registration matrix 2 to the video image processing engine.

[0148] Step 718: The preview image processing engine performs MCTF processing on the video data corresponding to the preview video stream according to the inter-frame offset data 1 and the registration matrix 2 sent by the visual analysis engine to obtain the preview video data.

[0149] For example, the preview image processing engine includes an image correction and alignment module, and can perform MCTF processing on the video data corresponding to the preview video stream through the image correction and alignment module according to the inter-frame offset data 1 sent by the preview electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0150] For example, the native MCTF algorithm of the chip platform configured by the image correction and alignment module can be used to perform MCTF processing on the video data corresponding to the preview video stream.

[0151] Step 719: The preview image processing engine sends the preview video data to the camera application.

[0152] Step 720: The camera application displays the corresponding preview video on the preview interface according to the preview video data.

[0153] Step 721: The video electronic image stabilization module calculates the video data corresponding to the recorded video stream using a second electronic image stabilization algorithm developed by the mobile phone itself, and obtains the inter-frame offset data 3.

[0154] By calculating the video data corresponding to the recorded video stream using the second electronic image stabilization algorithm, the motion information of consecutive video frames in the recorded video stream can be obtained, and the motion information may include the inter-frame offset data 3.

[0155] Among them, the inter-frame offset data 3 is used to indicate the offset amount between the front and rear video frames in the recorded video stream. For example, the inter-frame offset data 3 may be inter-frame grid data, which is used to indicate the offset amount of grid points between the front and rear video frames, such as the offset amounts of grid points in the front and rear two video frames on the X-axis and Y-axis.

[0156] Step 722: The video electronic image stabilization module sends the inter-frame offset data 3 to the video image processing engine.

[0157] After the video electronic image stabilization module calculates the inter-frame offset data 3, it can send the inter-frame offset data 3 to the video image processing engine for processing.

[0158] Step 723: The video image processing engine performs MCTF processing on the video data corresponding to the recorded video stream according to the inter-frame offset data 3 and the registration matrix 2, and obtains the recorded video data.

[0159] For example, the video image processing engine includes an image correction and alignment module, and can perform MCTF processing on the video data corresponding to the recorded video stream through the image correction and alignment module according to the inter-frame offset data 3 sent by the video electronic image stabilization module and the registration matrix 2 sent by the visual analysis engine.

[0160] For example, the native MCTF algorithm of the chip platform configured by the image correction and alignment module can be used to perform MCTF processing on the video data corresponding to the recorded video stream.

[0161] Step 724: The video image processing engine sends the recorded video data to the camera application.

[0162] Step 725: The camera application stores the recorded video according to the recorded video data.

[0163] In the embodiments of the present application, by modifying the path of the native process of the chip platform, that is, adding a path for the preview electronic image stabilization module to send inter-frame offset data to the visual analysis engine, and truncating the path for the global motion estimation module to send inter-frame offset data to the visual analysis engine, the preview electronic image stabilization module can send the calculated inter-frame offset data to the visual analysis engine and the preview image processing engine. Furthermore, the visual analysis engine can calculate a new registration matrix from the inter-frame offset data output by the global motion estimation module in the native process, and it is modified to generate a new registration matrix based on the inter-frame offset data output by the preview electronic image stabilization module, so as to ensure that the inter-frame offset data used by the visual analysis engine is consistent with the inter-frame offset data used by the preview image processing engine. By modifying the path, the texture suspension problem of the MCTF processing effect is improved, and to a certain extent, the impact of replacing the native algorithm with the self-developed algorithm on the video display effect is reduced, making the self-developed algorithm more adaptable to the MCTF processing scenario.

[0164] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0165] The present application also provides an electronic device, which includes a chip that is used to read and execute computer programs or instructions stored in a memory, so that the methods in the embodiments are executed.

[0166] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions run on an electronic device, the electronic device is caused to execute the above-related method steps to implement the methods in the above embodiments.

[0167] This embodiment also provides a computer program product. The computer-readable storage medium stores program codes. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the methods in the above embodiments.

[0168] In addition, the embodiments of the present application also provide a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip executes the methods in the above method embodiments.

[0169] Among them, the electronic device, the computer-readable storage medium, the computer program product or the chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a Digital Versatile Disc (DVD)), or a semiconductor medium (such as a Solid State Disk (SSD)), etc.

[0171] The above are the optional embodiments provided by the present application, which are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that, Applied to an electronic device, the method includes: Processing a video captured by a camera using a first electronic image stabilization algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix, where the first inter-frame offset data is used to indicate the offset amount between consecutive video frames in the video, and the inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between consecutive video frames in the video; Processing the video using a global motion estimation algorithm to obtain a first registration matrix, where the first registration matrix is used to indicate the transformation relationship for aligning consecutive video frames in the video; Processing the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix to obtain a second registration matrix; Performing motion compensation-based temporal filtering (MCTF) processing on the video according to the first inter-frame offset data and the second registration matrix.

2. The method according to claim 1, wherein The video is a preview video; the method further includes: Processing a recorded video captured by the camera using a second electronic image stabilization algorithm to obtain second inter-frame offset data, where the second inter-frame offset data is used to indicate the offset amount between consecutive video frames in the recorded video; Performing MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

3. The method according to claim 1 or 2, characterized in that The first electronic image stabilization algorithm is an electronic image stabilization algorithm provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device.

4. The method according to claim 2, wherein The second electronic image stabilization algorithm is an electronic image stabilization algorithm provided by the device manufacturer of the electronic device.

5. A video processing method, characterized in that, Applied to an electronic device, the operating system of the electronic device includes a first electronic image stabilization module, a global motion estimation module, a visual analysis engine, and a first image processing engine; The first electronic image stabilization module processes a video captured by a camera using a first electronic image stabilization algorithm to obtain first inter-frame offset data and an inter-frame perspective transformation matrix, sends the first inter-frame offset data and the inter-frame perspective transformation matrix to the visual analysis engine, and sends the first inter-frame offset data to the first image processing engine; where the first inter-frame offset data is used to indicate the offset amount of corresponding grid regions between two consecutive video frames in the video, and the inter-frame perspective transformation matrix is used to indicate the projection mapping relationship between two consecutive video frames in the video; The global motion estimation module processes the video using a global motion estimation algorithm to obtain a first registration matrix, and sends the first configuration matrix to the visual analysis engine, where the first registration matrix is used to indicate the transformation relationship for aligning consecutive video frames in the video; The visual analysis engine processes the first inter-frame offset data, the inter-frame perspective transformation matrix, and the first registration matrix to obtain a second registration matrix, and sends the second registration matrix to the first image processing engine; The first image processing engine performs motion compensation-based temporal filtering (MCTF) processing on the video according to the first inter-frame offset data and the second registration matrix.

6. The method according to claim 5, wherein The operating system further includes a second electronic image stabilization module and a second image processing engine, and the video is a preview video; the method further includes: The vision analysis engine sends the second registration matrix to the second image processing engine; The second electronic image stabilization module processes the recorded video captured by the camera using a second electronic image stabilization algorithm to obtain second inter-frame offset data, and sends the second inter-frame offset data to the second image processing engine, where the second inter-frame offset data is used to indicate the offset between consecutive video frames in the recorded video; The second image processing engine performs MCTF processing on the recorded video according to the second inter-frame offset data and the second registration matrix.

7. The method according to claim 5 or 6, characterized in that, The method further includes: The first electronic image stabilization module sends the inter-frame perspective transformation matrix to the first image processing engine.

8. The method according to claim 6, wherein The method further includes: The global motion estimation module sends the first inter-frame offset data to the second electronic image stabilization module.

9. The method according to any one of claims 5-8, characterized in that, The first electronic image stabilization algorithm is an electronic image stabilization algorithm provided by the device manufacturer of the electronic device, and the global motion estimation algorithm is a global motion estimation algorithm provided by the device manufacturer of the electronic device.

10. The method according to claim 6 or 8, characterized in that, The second electronic image stabilization algorithm is an electronic image stabilization algorithm provided by the device manufacturer of the electronic device.

11. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1-4 or claims 5-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when running on an electronic device, cause the electronic device to execute the method according to any one of claims 1-4 or claims 5-10.

Images