Video processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a video processing method and device, electronic equipment and a computer readable storage medium. The video quality and the video effect of a slow action video can be achieved. The method comprises the following steps: under the condition that video acquisition is started, storing an original video stream of a first frame rate into a target storage space through a target transmission channel by an image sensor; after acquisition is completed, performing frame insertion processing on the original video data at a second frame rate, and determining intermediate video data; the second frame rate is greater than the first frame rate; encoding the intermediate video data at a target encoding frame rate, and determining a target video; the target coding frame rate is smaller than the second frame rate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, due to limitations in device computing power and camera hardware, high frame rate slow-motion solutions from film-grade equipment require significant additional hardware investment to implement on mobile devices. Related technologies utilizing the hardware capabilities of mobile terminals primarily achieve high frame rate slow-motion effects by boosting the frame rate during video recording and then attenuating the speed by a factor of two during playback. However, these methods consume substantial system resources and computing power, and are prone to issues such as stuttering, blocking, and frame drops during processing, thereby reducing the video quality and overall effect of the slow-motion video. Summary of the Invention

[0003] This application aims to provide a video processing method, apparatus, electronic device, and computer-readable storage medium.

[0004] The technical solution of this application is implemented as follows: Firstly, a video processing method is provided, including: When video capture is started, the raw video stream of the first frame rate is stored in the target storage space through the target transmission channel via the image sensor. After acquisition, the original video data is interpolated at a second frame rate to determine intermediate video data; the second frame rate is greater than the first frame rate. The intermediate video data is encoded at a target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

[0005] Secondly, a video processing apparatus is provided, comprising: The recording module is used to store the raw video stream of the first frame rate into the target storage space through the target transmission channel when video capture is started, via an image sensor. The frame interpolation module is used to perform frame interpolation processing on the original video data at a second frame rate after the acquisition is completed, to determine intermediate video data; the second frame rate is greater than the first frame rate. An encoding module is used to encode the intermediate video data at a target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

[0006] Thirdly, a video processing system is provided, including a terminal and a server, wherein: The terminal is used to, when video acquisition is started, store the raw video stream of the first frame rate into the target storage space through the target transmission channel via an image sensor; after acquisition is completed, send the raw video data in the target storage space to the server. The server is configured to, after acquisition, perform frame interpolation on the original video data at a second frame rate to determine intermediate video data; the second frame rate is greater than the first frame rate; encode the intermediate video data at a target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

[0007] Fourthly, an electronic device is provided, comprising: a memory and a processor; wherein, The memory is used to store executable instructions. The processor is used to implement the video processing method provided in this application when executing executable instructions stored in the memory.

[0008] Fifthly, a computer-readable storage medium is provided, storing executable instructions for inducing a processor to execute and implement the video processing method provided in this application.

[0009] In a sixth aspect, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the video processing method provided in this application.

[0010] This application provides a video processing method, apparatus, electronic device, and computer-readable storage medium. Upon initiation of video capture, the raw video stream at the first frame rate is directly stored in the target storage space via a target transmission channel. After capture is complete, the raw video data in the target storage space undergoes unified frame rate upscaling and encoding. This application decouples the computationally intensive and resource-intensive frame interpolation process from the video capture process in time. This allows the video capture and frame interpolation processes to be completed at different times by utilizing system resources, reducing the risk of stuttering, blocking, and frame loss due to system overload and resource contention during video capture, and improving the quality and integrity of the captured raw video data. Thus, the frame interpolation process can not only fully utilize system resources to implement high-precision algorithms but also execute based on complete and continuous raw video data, resulting in more reliable and smoother intermediate video data. Consequently, it improves the video quality and visual experience of the target video generated based on the intermediate video data, and enhances the video quality and effects of slow-motion videos. Attached Figure Description

[0011] Figure 1 A flowchart illustrating a video processing method provided in this application embodiment. Figure 1 ; Figure 2 A schematic diagram of the interface display when motion detection is not enabled, provided as an embodiment of this application; Figure 3 An exemplary process diagram of the video processing method provided in the embodiments of this application; Figure 4 A flowchart illustrating a video processing method provided in this application embodiment. Figure 2 ; Figure 5 A schematic diagram of a target area provided in an embodiment of this application; Figure 6 A schematic diagram of an interface display for a stabilization prompt provided in an embodiment of this application; Figure 7 Another exemplary process diagram of the video processing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the composition structure of a video processing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0013] In the field of video processing, there are many application scenarios that require the creation of slow-motion videos. These include time-freezing effects in film and television, slow-motion states in video games, capturing exciting moments in sports competitions, and creative image creation that artistically presents natural phenomena such as flowing water and flames. High frame rate slow-motion solutions for film-grade equipment typically utilize a surround camera array to capture dynamic moments, achieving time freeze or strong slow-motion effects. However, due to limitations in device computing power and camera hardware, applying high frame rate slow-motion solutions from film-grade equipment to mobile terminals requires significant additional hardware costs. For example, directly migrating film-grade solutions to mobile terminals requires using a circular camera array to simultaneously capture the target and synthesize smooth slow-motion videos; or, it requires relying on a mechanical gimbal to precisely control the trajectory of a panoramic camera, rotating it around the target to capture images, using inertial sensor data (gyroscope / accelerometer) to calculate the coordinates of the rotation center; and generating slow-motion videos through a spherical perspective model. It can be seen that all of the above methods require high additional hardware costs and are only suitable for shooting in fixed scenes, unable to be deployed mobilely, and difficult to match the imaging requirements of mobile terminals. Furthermore, implementing the above solution requires algorithms that demand high computing power, making it difficult to implement on mobile terminals.

[0014] Currently, high frame rate slow-motion solutions implemented on mobile terminals primarily involve acquiring video data at high frame rates (e.g., 240fps or higher), temporarily storing the high frame rate video data output from the sensor in a terminal cache, such as Dynamic Random Access Memory (DRAM). Simultaneously with video acquisition and writing to the cache, a Frame Rate Conversion (FRC) algorithm is used to interpolate frames to a higher frame rate (e.g., 960fps or higher). Based on the interpolated video data, a slow-motion video with a speed reduction of 8 to 64 times is encoded and output. It can be seen that these high frame rate slow-motion solutions rely on the DRAM of the mobile terminal to cache the high frame rate data output from the sensor. However, the high frame rate video data output from the sensor requires high write bandwidth (e.g., 240fps video frame rate data requires 768MB / s write bandwidth), while DRAM typically only has 8GB to 16GB, which cannot meet the requirements for long-term continuous writing. Therefore, it can only support very short recording durations (usually less than or equal to 1 second). To achieve long recording times, repeated recording triggers are required, leading to cumbersome operations and data fragmentation. This makes it unsuitable for slow-motion video scenarios requiring continuous recording over extended periods, such as capturing high-speed moving objects continuously. Furthermore, the related technologies perform highly complex frame interpolation calculations while recording video data, continuously consuming significant CPU and / or GPU resources and constantly reading and writing memory. This can easily cause system overheating, power consumption spikes, and even frame drops and stuttering due to insufficient processing time, resulting in discontinuous video data and stuttering. Moreover, real-time frame interpolation during video capture requires processing one frame within a very short timeframe while simultaneously competing for processor resources with video capture, which can easily lead to decreased accuracy in the frame interpolation algorithm and reduced video quality. In conclusion, the related technologies reduce the video quality and effect of slow-motion videos.

[0015] This application provides a video processing method, apparatus, electronic device, and computer-readable storage medium capable of producing slow-motion video effects. The video processing method of this application is applied to an electronic device. In some embodiments, the electronic device can be implemented as various types of user terminals such as a laptop computer, tablet computer, desktop computer, set-top box, and mobile terminal (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device). Exemplarily, the electronic device may include smartphones, tablet computers, laptop computers, desktop computers, smart speakers, smartwatches, etc., running intelligent operating systems; the specific choice depends on the actual situation, and this application does not limit the scope.

[0016] See Figure 1 , Figure 1 A flowchart illustrating a video processing method provided in this application embodiment. Figure 1 ,include: S101. When video capture is started, the original video stream of the first frame rate is stored in the target storage space through the target transmission channel via the image sensor.

[0017] This application applies to scenarios involving the recording and generation of slow-motion videos on mobile terminals. In some embodiments, the mobile terminal receives a video capture command, such as when a user clicks the video record button on the terminal's camera interface, or when it receives a video record command from a control device connected to the terminal, such as a computer host, and then initiates video capture, acquiring a video stream through the terminal's image sensor.

[0018] In this embodiment of the application, when video acquisition is initiated, the image sensor acquires and outputs the raw video stream at a first frame rate. In some embodiments, the first frame rate can be preset according to factors such as video quality and the capabilities of the image sensor. For example, the first frame rate can be the highest frame rate that the image sensor can output, such as 240fps, or it can be set to other values ​​according to actual needs. The specific selection is based on the actual situation, and this embodiment of the application does not limit it.

[0019] In this embodiment, the image sensor stores the raw video stream at the first frame rate, acquired in real time, into the target storage space of the terminal via the target transmission channel. The target storage space has a larger storage capacity than volatile storage media and can store video data generated during long-term recording. In some embodiments, the target storage space may include a non-volatile storage medium built into the terminal or an external non-volatile storage medium connected to the terminal via a hardware interface, and may also include network storage resources remotely connected to the terminal.

[0020] For example, the non-volatile storage medium built into the terminal may include flash memory or disk on the terminal, such as Universal Flash Storage (UFS), Embedded MultiMediaCard (eMMC), etc.; the non-volatile storage medium connected externally to the terminal may include removable memory cards, such as SD cards, portable hard drives, etc.; the network storage resources remotely connected to the terminal may include network storage nodes such as cloud storage services, network attached storage devices, or storage arrays of remote servers; the specific selection is based on the actual situation, and this application embodiment does not limit it.

[0021] In some embodiments, the volatile storage medium corresponding to the image sensor includes a resource pool such as memory and cache managed or allocated by the terminal's operating system, as well as a cache built into the image sensor. The volatile storage medium can take forms including, but is not limited to, SRAM, eDRAM, and DRAM. For example, the volatile storage medium corresponding to the image sensor may include the DRAM built into the image sensor, the terminal's system memory (DRAM or SRAM), the GPU's frame buffer, and the L1 / L2 cache managed by the CPU. In this embodiment, the raw video stream output by the image sensor does not pass through or is stored in any of the aforementioned volatile storage media.

[0022] It should be noted that, in the embodiments of this application, the volatile storage medium corresponding to the image sensor does not include the register group inside the storage device controller used for data pipeline processing, such as that required for protocol encapsulation.

[0023] In some embodiments, the target transmission channel connects the image sensor and the target storage space point-to-point, providing a dedicated data transmission path between the image sensor and the target storage space. The target transmission channel does not contain the volatile storage medium corresponding to the image sensor; that is, the raw video stream output by the image sensor does not pass through the volatile storage medium corresponding to the image sensor but is directly stored in the target storage space via the target transmission channel. Therefore, the video recording duration is not limited by the capacity of the volatile storage medium.

[0024] S102. After the acquisition is completed, the original video data is interpolated at the second frame rate to determine the intermediate video data; the second frame rate is greater than the first frame rate.

[0025] In this embodiment, the image sensor stores the real-time acquired raw video stream into the target storage space in real time through the target transmission channel. Thus, upon completion of acquisition, the raw video data is obtained in the target storage space. In some embodiments, the completion of acquisition can be triggered by the user actively ending video acquisition, or by the target storage space used to store the raw video stream becoming full.

[0026] In this embodiment, since the frame interpolation process is initiated after the video acquisition process is completed, the risk of resource contention during the video acquisition process is reduced, allowing the video acquisition process to make full use of system resources and record complete and continuous original video data in the target storage space, providing a stable and reliable data input source for the subsequent frame interpolation process.

[0027] In this embodiment, after acquisition is completed, for example, when the user clicks to view the video or the terminal's processor is idle, the original video data can be read from the target storage space, and a frame interpolation algorithm, such as the FRC algorithm, can be called to perform frame interpolation processing. That is, in this embodiment, video stream storage and frame interpolation processing are performed asynchronously. This separates the processing of video acquisition and storage from the computational power required by the frame interpolation algorithm, avoiding processor overload, reducing real-time computing power requirements and peak processor utilization, and lowering peak processor temperature. In actual testing, this can reduce power consumption by 62% and temperature by 9°C during video acquisition. This demonstrates that compared to related technologies, this embodiment consumes less system resources and computing power during video acquisition, reducing stuttering, blocking, and frame drops caused by resource contention or processing bottlenecks during video acquisition.

[0028] In this embodiment of the application, a frame interpolation algorithm is used to perform frame interpolation processing on the original video data in the target storage space. Video frames calculated by the frame interpolation algorithm are inserted between the original video frames of the original video data, thereby interpolating the original video data of the first frame rate to the second frame rate to obtain intermediate video data.

[0029] It is understandable that interpolating frames from the original video data at a first frame rate to a second frame rate, where the second frame rate is higher than the first, results in the intermediate video data having more frames than the original video data. For example, a terminal records one second of original video data at a first frame rate of 240fps (240 frames per second), containing 240 frames. After interpolating this original video data to a second frame rate, such as 960fps, the resulting intermediate video data contains 960 frames. In other words, the number of frames in the intermediate video data, i.e., the number of frames corresponding to the second frame rate, is greater than the number of frames in the original video data.

[0030] In this embodiment of the application, after the acquisition is completed, the video acquisition process and the data writing process release system resources. This allows the frame interpolation process to make full use of system resources to run high-precision frame interpolation algorithms that require intensive computation, and to perform data analysis and calculation on the preceding and following frames based on complete and continuous original video data, thereby generating high-quality interpolated video frames and obtaining intermediate video data.

[0031] S103. Encode the intermediate video data at the target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

[0032] In this embodiment, intermediate video data with a second frame rate is encoded according to a target encoding frame rate, wherein the target encoding frame rate is lower than the first frame rate. The target video is configured to have a duration greater than that of the original video data during playback, based on the number of frames corresponding to the second frame rate and the target encoding frame rate. Since the number of frames in the intermediate video data is greater than the number of frames in the original video data, encoding the intermediate video data using a target encoding frame rate lower than the first frame rate results in a target video with a playback duration greater than the original duration of the original video data and a smooth, flowing slow-motion visual effect.

[0033] For example, the image sensor can output the original video stream at its highest output frame rate, such as 240fps, as the first frame rate. The intermediate video data can be obtained by interpolating frames from the original 240fps video data to 960fps (the second frame rate). The target encoding frame rate can be 60fps. Encoding the intermediate video data at 60fps yields the target video, i.e., the slow-motion effect video. It can be seen that encoding the video frames contained in the 960fps intermediate video data at a frame rate of 60fps results in a playback duration four times longer than the original duration. Furthermore, based on the higher frame rate of the intermediate video data compared to the original video data, a smooth visual effect of slow-motion transitions can be achieved.

[0034] It is understood that the embodiments of this application decouple the computationally intensive and resource-intensive frame interpolation process from the video acquisition process in terms of time. This allows the video acquisition process and the frame interpolation process to be completed at different times by calling system resources, reducing the risk of stuttering, blocking, and frame loss caused by system overload and resource contention during video acquisition, and improving the quality and integrity of the acquired raw video data. In this way, the frame interpolation process can not only fully utilize system resources to implement high-precision algorithms, but also be executed based on complete and continuous raw video data, resulting in more reliable and smoother intermediate video data. Consequently, the video quality and visual experience of the target video generated based on the intermediate video data are improved, as are the video quality and effects of slow-motion videos.

[0035] In some embodiments, the target transmission channel includes: a storage device controller and a non-volatile storage bus; the target storage space includes a non-volatile storage medium; storing the raw video stream at a first frame rate acquired by an image sensor into the target storage space via the target transmission channel includes: The image sensor outputs the raw video stream to the storage device controller, which then uses the storage device controller to store the raw video stream into a non-volatile storage medium via a non-volatile storage bus.

[0036] In this embodiment, the storage device controller controls the transmission of the raw video stream output in real time by the image sensor to a non-volatile storage medium via a non-volatile storage bus. The non-volatile storage bus is independent of the connection path of the volatile storage medium corresponding to the image sensor.

[0037] In some embodiments, the image sensor acquires and generates a raw video stream at a first frame rate, and sends the raw video stream at the first frame rate to the storage device controller. The storage device controller encapsulates and processes the received raw video stream according to the communication protocol of the non-volatile storage medium, and writes it directly and continuously into the physical storage unit of the non-volatile storage medium through the non-volatile storage bus, thus completing persistent storage. In this way, by using the transmission control of the storage device controller and the non-volatile storage bus as the data writing path for the raw video stream, it is possible to directly store the raw video stream into the non-volatile storage medium without passing through the volatile storage medium corresponding to the image sensor.

[0038] For example, the storage device controller may include a Direct Memory Access Controller (DMA), and the non-volatile storage bus may include a UFS 3.1 bus, a UFS 4.0 bus, an eMMC 5.1 bus, etc. The specific selection is based on the actual terminal hardware configuration or architecture, and this application embodiment does not limit it.

[0039] For example, taking UFS 3.1 as an example, UFS 3.1's sustained write speed can reach or exceed 800MB / s, supporting long-term recording of raw video streams output by image sensors at 4K resolution and 240fps. For instance, a 512GB UFS, coupled with a UFS 3.1 bus with a sustained write speed of 1.2GB / s, supports a maximum recording time of 38 seconds. A 128GB eMMC, coupled with an eMMC 5.1 bus with a sustained write speed of 400MB / s, supports a maximum recording time of 12 seconds. In contrast, related technologies that store raw video streams in DRAM only support a maximum recording time of 1 second. It can be seen that the solution in this application significantly improves the recording time.

[0040] It is understood that the embodiments of this application construct a physical write-through path from the image sensor to the storage device controller and then to the non-volatile storage medium, thereby avoiding the limitation of recording time on the capacity bottleneck of the sensor's built-in cache, increasing the recording time, and supporting continuous capture of long-term high-speed motion scenes (such as an athlete's 100-meter sprint, bird flight paths, etc.), thus achieving long-term high frame rate video recording and improving scene applicability. Furthermore, it greatly improves the utilization rate of storage resources and avoids data fragmentation caused by repeated recording due to insufficient built-in cache capacity, thereby improving the resource utilization efficiency of video processing.

[0041] In some embodiments, the process of outputting the raw video stream to the storage device controller via the image sensor may include: The original video stream is output through the image sensor; the original video stream is divided into segments according to a preset time interval to determine the original video blocks; the original video blocks are transmitted to the storage device controller, so that the storage device controller can store the original video blocks into the non-volatile storage medium through the non-volatile storage bus.

[0042] In this embodiment, during the process of storing the raw video stream continuously output by the image sensor into a non-volatile storage medium, the raw video stream can be segmented according to a preset time interval. For example, the preset time interval can be 1 second, meaning it is segmented once per second. The data obtained from each segment (such as a 1-second raw video file) is used as a raw video block, and the raw video block is transmitted to the storage device controller, which then writes the raw video block into the non-volatile storage medium. In practical applications, the preset time interval can also be set to other values ​​as needed; no specific limitation is made here.

[0043] In some embodiments, the original video chunks can be time-stamped; for example, a timestamp can be added to the header of the original video chunk file. This allows the original video chunks to be organized and processed using time-stamping, avoiding transmission or file storage errors.

[0044] Understandably, writing the original video stream into non-volatile storage media in chunks can avoid I / O bottlenecks caused by excessively large single files and improve storage efficiency.

[0045] In some embodiments, the target storage space includes a first non-volatile storage medium built into the terminal and a second non-volatile storage medium connected externally to the terminal. For example, the first non-volatile storage medium may include an eMMC, UFS, or similar device built into the terminal, and the second non-volatile storage medium may include an external SD card. During the video stream acquisition phase, the original video stream at a first frame rate can be stored in the first non-volatile storage medium via the target transmission channel using an image sensor. If the available storage space of the first non-volatile storage medium is less than or equal to a preset available threshold, such as when the first non-volatile storage medium is full, the storage location of the original video stream is switched, and the acquired original video stream is stored in the second non-volatile storage medium.

[0046] It is understandable that, compared to related technologies that can only terminate recording when the cache space is full, the method of this application embodiment can automatically switch to an externally connected second non-volatile storage medium when the first non-volatile storage medium is full, thereby further improving the video recording time.

[0047] In some embodiments, the process of interpolating the original video data at a second frame rate to determine the intermediate video data may include: The entire segment of the original video data is interpolated at the second frame rate to determine the intermediate video data.

[0048] Here, the entire original video data can be used as the object of frame interpolation. Frame interpolation is performed on the entire segment of the original video data at the second frame rate to determine the intermediate video data.

[0049] In some embodiments, frame interpolation can be performed on the entire original video data to obtain intermediate video data when motion detection is not enabled during video recording. For example, when motion detection is not enabled, the video recording preview interface will display a user interaction message such as... Figure 2 The message displayed is "Motion detection: Off".

[0050] This application provides an exemplary process diagram of a video processing method, such as... Figure 3 As shown, it includes: S301, the camera application layer 10 starts video recording and sends preview and recording requests to the camera hardware abstraction layer 11.

[0051] S302, the camera hardware abstraction layer 11 calls the image sensor 12 to perform video acquisition based on the preview request and the recording request.

[0052] S303, the image sensor 12 outputs the raw video stream at a frame rate of 240fps (equivalent to the first frame rate) and outputs the preview video stream at a relatively low frame rate, and returns the raw video stream and the preview video stream to the camera hardware abstraction layer 11.

[0053] S304, the camera hardware abstraction layer 11 returns the raw video stream and the preview video stream to the camera application layer 10.

[0054] It should be noted that the preview video stream can be used to display on the preview interface of the video capture.

[0055] S305, the camera application layer 10 directly stores the 240fps raw video stream into the non-volatile storage medium 13 through the target transmission channel.

[0056] S306. After the acquisition is completed, when the user is viewing or the processor is idle, the algorithm processing module 14 reads the raw video data at 240fps from the non-volatile storage medium 13.

[0057] S307, Algorithm processing module 14 uses a frame rate conversion algorithm to perform frame interpolation on the original video data at 240fps to obtain high frame rate data at 960fps (equivalent to intermediate video data).

[0058] S308. The high frame rate data of 960fps is encoded and output at the target encoding frame rate of 60fps through the encoding module 15 to obtain the target video.

[0059] Understandably, by asynchronously processing video stream storage and frame interpolation, the peak load on the processor and system power consumption are reduced, supporting the capture of long-term continuous high-speed motion, such as the entire race or bird flight paths. Compared to related technologies that can only capture instantaneous fragments, this greatly improves the freedom of shooting and the user experience.

[0060] Currently, for situations where motion detection is enabled during video recording, the process of generating slow-motion video typically involves: detecting the presence of moving objects during recording; initiating frame interpolation when motion is detected; and stopping interpolation when the motion stops. This method of initiating frame interpolation only after motion is detected introduces significant latency (typically around 50ms), easily leading to failures in capturing high-speed targets. Real-world testing in tennis serve capture scenarios showed a failure rate as high as 30%, thus reducing the success rate of slow-motion video generation and the overall video quality.

[0061] In some embodiments, the process of interpolating the original video data at a second frame rate to determine the intermediate video data can be as follows: Figure 4 As shown, it includes: S401. Obtain the timestamps corresponding to the original video data from the target storage space.

[0062] In this embodiment, the video segment corresponding to the motion process of a moving object in the original video data can be determined by comparing two or more consecutive video frames in time sequence. For example, by analyzing the changes in pixels between consecutive frames, if a large number of pixels move regularly, it can be determined that a moving object has appeared or started moving; if pixels are almost stationary, it can be determined that the moving object has disappeared or stopped moving. This allows the determination of the video segment corresponding to the motion process of the moving object in the original video data, and then the generation of a time stamp corresponding to that video segment. In this way, the time interval of the moving object's motion in the original video data can be characterized by the time stamp.

[0063] In some embodiments, when motion detection is enabled, the motion detection function can be used to determine the time markers in the raw video data. Here, the motion detection function is a function configured in the video acquisition module of the terminal for detecting moving objects or moving scenes in the video frame. When the motion detection function is enabled during video acquisition, it can determine whether it is a moving scene and / or identify moving objects in the video frame by comparing two or more frames of video frames in the preceding and following time sequence.

[0064] S402. Using the second frame rate, perform frame interpolation on the first original video segment corresponding to the time marker in the original video data to determine the intermediate video data.

[0065] In this embodiment, given a time marker, the motion time interval requiring frame interpolation and the corresponding video segment can be determined based on the time marker, serving as the first original video segment. Then, according to the second frame rate, frame interpolation is performed on the first original video segment corresponding to the time marker in the original video data. Other video segments outside the first original video segment, i.e., video segments corresponding to time periods outside the motion time interval, are not interpolated, maintaining the original first frame rate, thereby obtaining intermediate video data.

[0066] In other words, the intermediate video data obtained by interpolating the first original video segment corresponding to the time marker in the original video data includes: a first video segment obtained by interpolating the first original video segment and at least one second original video segment with a first frame rate; the at least one second original video segment is a video segment outside the motion time interval represented by the time marker in the original video data.

[0067] For example, the original video data includes a sequence of video frames from T1 to T10, and the motion time interval corresponding to the time markers is from T3 to T6, indicating that the time interval from T3 to T6 is the time period of motion of the moving object. Therefore, T3 to T6 can be used as the motion time interval, and the video segments within this time interval can be used as the first original video segments and interpolated to the second frame rate. In the resulting intermediate video data, the frame rate from T1 to T2 is the first frame rate, the frame rate from T3 to T6 is the second frame rate, and the frame rate from T7 to T10 is the first frame rate.

[0068] As can be seen, the intermediate video obtained by time-stamped frame interpolation contains video segments with two frame rates: a first frame rate and a second frame rate. The first video segment with the second frame rate has more frames than both the first and second original video segments. Therefore, after encoding the intermediate video data at the target encoding frame rate, the first video segment has a longer playback duration due to the increased number of frames. This allows the target video to achieve the effect of playing at normal speed before the moving object moves, playing in slow motion during the moving object's movement, and resuming normal speed playback after the moving object stops. This results in the target video exhibiting a speed variation, for example, presenting a segmented speed variation effect of "normal speed-slow motion-normal speed".

[0069] It is understood that the embodiments of this application decouple motion detection from frame interpolation. When motion is detected during video capture, frame interpolation is not performed; instead, a time stamp is generated and recorded first. After video capture ends, the original video data and the recorded time stamp are read, and frame interpolation is performed according to the motion time interval of the time stamp. This resolves the contradiction between real-time video stream storage and motion capture, reducing motion capture latency. Relying on high-precision system clock synchronization, the error of the motion time interval corresponding to the time stamp in actual testing is less than 1ms, improving the success rate of high-speed target capture. It can achieve instantaneous capture and slow-motion generation in scenarios such as water droplet impact and ball entering the net. In actual testing, the capture success rate reaches 100% even with a tennis serve speed of 200km / h. Furthermore, it can generate target videos with segmented speed variations, further enhancing the performance of slow-motion videos, and eliminating the need for manual trimming of slow-motion segments by the user, thus improving the efficiency of slow-motion video generation and user experience.

[0070] In some embodiments, optical flow motion detection can be performed on the preview video stream corresponding to the original video stream to determine the first time marker and the second time marker; the first time marker and the second time marker are then stored in the target storage space.

[0071] In this embodiment, the preview video stream is time-synchronized with the original video stream, but its frame rate is lower. Therefore, during video acquisition, optical flow motion detection can be performed on the preview video stream while simultaneously acquiring the video. This reduces the processing load on the algorithm and further lowers latency and system power consumption. The first time marker is used to characterize the start time of motion; the second time marker is used to characterize the end time of motion. In some embodiments, if motion detection is enabled during video acquisition, optical flow motion detection can be performed on the preview video stream to determine the first and second time markers. The first and second time markers are then synchronously stored in the target storage space along with the acquired original video stream.

[0072] For example, in response to a video recording command, the image signal processor can enable the optical flow analysis engine, calculate the motion vector of pixels between consecutive video frames using an optical flow motion analysis algorithm, determine the start time of the moving object's motion based on the motion vector of pixels between consecutive video frames, generate a first time marker, determine the end time of the moving object's motion, and generate a second time marker.

[0073] In some embodiments, the process of performing optical flow motion analysis on the preview video stream to determine the first time marker and the second time marker includes: Identify the target region in the preview video stream; perform optical flow motion detection on the target region; if a moving pixel with a displacement exceeding a preset displacement threshold is detected in the target region, determine a first time marker; if the position of the moving pixel exceeds the target region, determine a second time marker.

[0074] In this embodiment of the application, a target area in the preview video stream can be determined by receiving a user's region specification operation, such as a user's click, drag, or draw operation on a screen location. For example, it can be done through... Figure 5 The video recording preview interface shown indicates to the user the enabled status of the motion detection function (e.g., Figure 5The image shows "Motion Detection: On" and a target region 501. The target region 501 is designated as the Region of Interest (ROI), and optical flow motion detection is performed on it. If a portion of moving pixels appear within the target region 501, and the displacement of these moving pixels exceeds a preset displacement threshold, it indicates that a moving object has begun to move. At this point, a first time marker is generated; exemplarily, a motion start timestamp is generated as the first time marker. If the detected position of a moving pixel exceeds the target region 501, it indicates that the moving object has moved beyond the image area expected to be represented with a slow-motion effect, or that there is no longer any motion event in the ROI. At this point, a second time marker is generated; exemplarily, a motion end timestamp is generated as the second time marker.

[0075] For example, the preset displacement threshold can be 5% of the screen width or length. Using an optical flow analysis algorithm, the time when the moving object begins to move and its direction of movement can be determined. When the displacement of the moving pixels representing the moving object in the screen width or length direction exceeds 5%, a first time marker is generated.

[0076] Understandably, generating first and second time markers by performing optical flow motion detection on target areas in the video frame further reduces the processing pressure on the optical flow analysis algorithm. Furthermore, this approach marks the start of motion time when motion enters the region of interest and the end of motion time when motion leaves the region of interest. Combined with the aforementioned frame interpolation strategy, video segments outside the motion markers can be left uninterpolated, while motion within the region of interest can be interpolated and played in slow motion. This allows for more refined slow-motion playback of motion within the user's region of interest, increasing the diversity and flexibility of slow-motion video presentation and ultimately enhancing the user experience.

[0077] In some embodiments, during video capture, if the sensor data of the terminal detects that the degree of shaking of the terminal does not meet the stability conditions, a prompt is displayed on the video capture interface of the terminal.

[0078] In this embodiment, to prevent terminal jitter from affecting motion detection and the captured image during video acquisition, the degree of jitter or shaking during video acquisition can be detected using sensor data from the terminal. If the degree of jitter or shaking does not meet the stability requirements, a prompt is displayed on the terminal's video acquisition interface to remind the user to take timely measures to improve the device stability during video acquisition and avoid false detections or poor video quality due to jitter.

[0079] For example, the degree of phone shaking can be determined using the phone's gyroscope data. If the angular velocity of the gyroscope data is greater than 2 degrees per second, it is determined that the stability condition is not met, and this can be reflected in the video recording preview interface. Figure 6 The image stabilization prompt is shown.

[0080] Understandably, by promptly detecting and alerting users to stabilize their images, the original video stream was able to capture moving objects effectively, further improving the quality of slow-motion videos created from the acquired original video stream.

[0081] This application also provides an exemplary process diagram of a video processing method, such as... Figure 7 As shown, it includes: S701, the camera application layer 10 starts video recording and sends preview and recording requests to the camera hardware abstraction layer 11.

[0082] Here, the camera application layer 10 can enable motion detection during video recording.

[0083] S702, the camera hardware abstraction layer 11 calls the image sensor 12 to perform video acquisition based on the preview request and the recording request.

[0084] S703, image sensor 12 outputs raw video stream at a frame rate of 240fps (equivalent to the first frame rate), and outputs preview video stream at a relatively low frame rate, returning the raw video stream and preview video stream to camera hardware abstraction layer 11.

[0085] S704, the camera hardware abstraction layer 11 returns the raw video stream and the preview video stream to the camera application layer 10.

[0086] It should be noted that the preview video stream can be used to display on the preview interface of the video capture.

[0087] S705, the camera application layer 10 directly stores the original video stream at 240fps frame rate into the non-volatile storage medium 13 through the target transmission channel.

[0088] S706, the algorithm processing module 14 acquires the preview video stream and performs anti-shake detection on the preview video stream using a gyroscope and accelerometer.

[0089] S707, Algorithm processing module 14 determines whether the degree of phone shaking meets the stability condition. If the stability condition is not met, S708 is executed; otherwise, S709 is executed.

[0090] S708, Algorithm Processing Module 14 pops up a message: "Please remain stable, we are detecting movement of objects within the frame."

[0091] In the S708, if the degree of phone shaking does not meet the stability requirements, a pop-up message will appear on the video capture interface saying "Please keep stable, detecting movement of objects within the frame".

[0092] S709, Algorithm processing module 14 determines whether there is moving objects in the target area.

[0093] In S709, if the phone's shaking level meets the stability requirements, the motion detection function is used to detect motion in the target area of ​​the preview video stream to determine if there is movement in the target area. If some moving pixels appear in the target area, and the displacement of the moving pixels exceeds a preset displacement threshold, it is determined that there is movement in the target area, and S710 is executed. Otherwise, if no movement is detected in the target area, the process jumps to S708, maintaining the anti-shake prompt displayed on the interface to remind the user to keep the phone stable at all times to capture moving objects.

[0094] S710, Algorithm Processing Module 14 determines the time stamp.

[0095] In S710, when motion of an object in the target area is detected, a time marker for the start time of the motion and a time marker for the end time of the motion (equivalent to a first time marker and a second time marker) are determined and used as time markers.

[0096] S711, the algorithm processing module 14 writes the time stamp to the non-volatile storage medium 13.

[0097] S712. After the acquisition is completed, when the user is viewing or the processor is idle, the algorithm processing module 14 reads the first time marker, the second time marker and the original video data at 240fps frame rate from the non-volatile storage medium 13 and performs frame interpolation processing.

[0098] S713 and the algorithm processing module 14 use a frame rate conversion algorithm to interpolate the video segment between the first time mark and the second time mark to 960fps, while the video segment outside the interval corresponding to the first time mark and the second time mark retains the original frame rate, thus obtaining high frame rate data (equivalent to intermediate video data).

[0099] S714. The high frame rate data is encoded and output at a target encoding frame rate of 60fps through the encoding module 15, resulting in a three-segment target video of "normal speed-slow motion-normal speed".

[0100] For example, in some scenarios, the video processing method with motion detection enabled in this application embodiment was compared with related technologies, and the results are as follows: In tests conducted on a tennis serve (200km / h) capture scenario, the missed capture rate using related technologies exceeded 30%, while the embodiment of this application achieved 100% successful capture, significantly improving the reliability of high-speed motion capture. In tests conducted on a water droplet impact (0.1s) capture scenario, the missed capture rate using related technologies exceeded 40%, while the embodiment of this application achieved 100% successful capture, realizing millisecond-level response. In tests conducted on a vehicle capture scenario (80km / h), the missed capture rate using related technologies exceeded 25%, while the embodiment of this application achieved 100% successful capture, and the ample system resources during frame interpolation processing allow for the use of high-precision frame interpolation algorithms, effectively eliminating motion blur.

[0101] Understandably, by detecting motion in the preview video stream, time stamping can be performed on the frame in which a moving object enters the target area. Relying on a high-precision system clock, the error can be controlled within a very small range (e.g., 1ms), thus solving the pain point in related technologies where "the target has moved out of the frame when motion detection is completed." This ensures that slow-motion effects such as bullet time accurately lock onto key moments (e.g., a basketball hitting the net, glass breaking). At the same time, by linking gyroscope stabilization with ROI area detection, invalid recordings caused by hand shake are avoided, improving the success rate of slow-motion video production.

[0102] The following further explains the frame interpolation process described above, including the frame interpolation process when motion detection is not enabled and the frame interpolation process when motion detection is enabled, including: Determine the current original video frame sequence; if the motion complexity of the current original video frame sequence is less than the motion complexity threshold, perform frame interpolation on the current original video frame sequence by frame duplication; if the motion complexity of the current original video frame sequence is greater than or equal to the motion complexity threshold, perform frame interpolation on the current original video frame sequence by optical flow interpolation; perform frame interpolation on the next original video frame sequence until the frame interpolation of the original video data or the first original video segment is completed.

[0103] In this embodiment, for the entire segment of original video data requiring frame interpolation, or the first original video segment, at least two consecutive original video frames that need interpolation are first determined as the current original video frame sequence. Motion complexity analysis is performed on the current original video frame sequence; for example, the image difference is compared, and the motion complexity is determined based on the comparison results. If the motion complexity is less than the motion complexity threshold, it indicates that the image differences between the original video frames in the current original video frame sequence are small, and frame interpolation can be performed by frame duplication, consuming less computational power. If the motion complexity is greater than or equal to the motion complexity threshold, it indicates that the image differences between the original video frames in the current original video frame sequence are large, usually due to the movement of moving objects. In this case, one or more intermediate frames are generated using optical flow frame interpolation, and these intermediate frames are inserted between the original video frames in the current original video frame sequence to achieve smoother and more fluid image transitions. Then, the same process is used to perform frame interpolation on the next original video frame sequence in the entire segment of original video data or the first original video segment until the frame interpolation of the original video data or the first original video segment is completed.

[0104] For example, when the displacement between consecutive original video frames does not exceed 5% of the screen width or length, and the motion complexity is determined to be less than the motion complexity threshold, frame interpolation using frame duplication takes approximately 1ms per copy. When rapid rotation, distortion, or other image differences occur between consecutive original video frames, and the motion complexity is determined to be greater than the motion complexity threshold, optical flow interpolation using the Recurrent All-Pairs Field Transforms (RAFT) model takes approximately 8ms per frame. This avoids excessive processor usage and balances video quality with power consumption.

[0105] Understandably, by using different frame interpolation methods based on varying motion complexity, the efficiency of the frame interpolation algorithm and the smoothness of the slow-motion video after interpolation are balanced, reducing system power consumption and improving the playback effect of slow-motion video.

[0106] In some embodiments, this application also provides a video processing method, including: Based on user actions, export the target video; combine the target video with preset effect templates to generate the target effect video.

[0107] For example, when a user clicks "One-Click Export" on the terminal interface, the target video is exported to the target path, significantly improving video generation efficiency. Furthermore, the target video can be intelligently bound to preset dynamic effects templates, such as motion trajectory light effects, water splash particle effects, etc., automatically synthesizing a video playback effect combining slow motion and special effects, further enhancing video production quality and user experience.

[0108] In some embodiments, the system can also receive editing instructions from the user via voice or text, and combine the editing instructions with time markers to trim the target video and determine the target edited video.

[0109] For example, for a target video captured and generated using motion detection mode, the user can verbally give instructions (such as "keep the moment of the tennis serve"), and the built-in artificial intelligence neural network can combine the time stamp of the target video with the user's instructions to accurately cut out the video segment required by the user, achieving zero manual operation and further improving the user experience.

[0110] Based on the above embodiments, this application also provides a video processing system, including a terminal and a server, wherein: When video capture is initiated, the terminal uses an image sensor to store the raw video stream at the first frame rate into the target storage space via the target transmission channel; after capture is completed, the terminal sends the raw video data in the target storage space to the server. After the server completes the acquisition, it performs frame interpolation on the original video data at a second frame rate to determine the intermediate video data. The second frame rate is greater than the first frame rate. The intermediate video data is then compared with the target encoding frame rate to determine the target video. The target encoding frame rate is less than the first frame rate. The target video is configured to be played with a duration greater than that of the original video data, based on the number of frames corresponding to the second frame rate and the target encoding frame rate.

[0111] For example, raw video data and timestamps can be uploaded to a server via a mobile phone or other terminal. The server then uses an optical flow interpolation algorithm (such as the RAFT model) to generate high-frame-rate intermediate video data. This reduces the heat generation and frequency reduction caused by executing the interpolation algorithm on the terminal, and leverages the server's stronger computing power to improve the processing time for complex scenes such as optical flow interpolation, thereby increasing the speed and efficiency of slow-motion video generation.

[0112] In some embodiments, based on the architecture of the video processing system described above, the raw video streams can include multiple video streams captured by multiple cameras on the terminal. Thus, by synchronously uploading multiple video streams, such as dual video streams with a wide-angle and a telephoto lens, the server can synthesize surround-view effects (such as a 360° display of a basketball going into the net) in the cloud, rendering multi-view slow-motion videos and further improving video production effects.

[0113] In some embodiments, to address the issue of single-camera phones being unable to achieve cinematic-level "bullet time" multi-view effects, multiple terminals can be networked to construct a surround shooting array. For example, three or more phones can be synchronized via Wi-Fi P2P networking (latency < 5ms); the networked terminals are then connected to nearby edge nodes (such as the built-in computing power of a 5G base station), and the frame interpolation task is offloaded to the edge server, reducing the host load to generate multi-camera slow motion. Furthermore, during video capture, the main terminal among the multiple terminals dynamically allocates computing tasks (e.g., terminal A is responsible for optical flow motion detection and analysis, and terminal B is responsible for gyroscope stabilization), and synchronization between terminals can be achieved using a clock synchronization protocol (such as PTPv2). In this way, multi-view "bullet time" can be achieved at low cost, improving creative freedom and user experience.

[0114] In some embodiments, the terminal can access at least one edge computing node nearby and use the at least one edge computing node to perform slow-motion video generation and multi-view rendering. GPU resources can also be dynamically allocated among the at least one edge computing node to achieve a balance between video quality and energy consumption.

[0115] In some embodiments, a server can perform real-scene analysis on the target video uploaded by the first user to determine the real physical scene of the target video. Combined with spatial anchoring technology, a virtual target can be superimposed on the real physical scene, such as superimposing a ball landing point prompt box on a real sports field. When the second user shoots the sports field, the ball landing point prompt box is displayed on the preview interface, guiding the second user to use the ball landing point prompt box to generate a new slow-motion effect video, replicating the slow-motion video effect shot by the first user.

[0116] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a video processing apparatus, such as... Figure 8 As shown, the video processing device 1 includes: The recording module 11 is used to store the raw video stream of the first frame rate into the target storage space through the target transmission channel via the image sensor when video acquisition is started. The frame interpolation module 12 is used to perform frame interpolation processing on the original video data at a second frame rate after the acquisition is completed, to determine intermediate video data; the second frame rate is greater than the first frame rate. The encoding module 13 is used to encode the intermediate video data at a target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

[0117] In some embodiments, the frame interpolation module 12 is further configured to obtain a time stamp corresponding to the original video data from the target storage space; the time stamp is used to characterize the motion time interval of the moving object in the original video data; and to perform frame interpolation processing on the first original video segment corresponding to the time stamp in the original video data at the second frame rate to determine the intermediate video data; the intermediate video data includes a first video segment obtained by interpolating the first original video segment and at least one second original video segment at the first frame rate; the at least one second original video segment is a video segment in the original video data that is outside the motion time interval characterized by the time stamp.

[0118] In some embodiments, the target transmission channel includes: a storage device controller and a non-volatile storage bus; the target storage space includes a non-volatile storage medium; the recording module 11 is further configured to output the original video stream to the storage device controller via the image sensor, and use the storage device controller to store the original video stream into the non-volatile storage medium via the non-volatile storage bus.

[0119] In some embodiments, the recording module 11 is further configured to output the original video stream through the image sensor; divide the original video stream according to a preset time interval to determine the original video blocks; and transmit the original video blocks to the storage device controller so that the storage device controller can store the original video blocks into the non-volatile storage medium through the non-volatile storage bus.

[0120] In some embodiments, the time stamp includes a first time stamp and a second time stamp; the first time stamp is used to characterize the start time of motion; the second time stamp is used to characterize the end time of motion; the video processing device 1 further includes a marking module, which is used to perform optical flow motion detection on the preview video stream corresponding to the original video stream, determine the first time stamp and the second time stamp, and store the first time stamp and the second time stamp in the target storage space.

[0121] In some embodiments, the marking module is further configured to determine a target region in the preview video stream; perform optical flow motion detection on the target region; determine a first time marker when a moving pixel with a displacement exceeding a preset displacement threshold is detected in the target region; and determine a second time marker when the position of the moving pixel exceeds the target region.

[0122] In some embodiments, the video processing device 1 further includes a stabilization module, which is used to provide a prompt on the video acquisition interface of the terminal when the sensor data of the terminal detects that the degree of shaking of the terminal does not meet the stability conditions.

[0123] In some embodiments, the frame interpolation module 12 is further configured to perform frame interpolation processing on the entire segment of the original video data at the second frame rate to determine the intermediate video data.

[0124] In some embodiments, the frame interpolation module 12 is further configured to: determine the current original video frame sequence; if the motion complexity of the current original video frame sequence is less than a motion complexity threshold, perform frame interpolation processing on the current original video frame sequence by frame duplication; if the motion complexity of the current original video frame sequence is greater than or equal to the motion complexity threshold, perform frame interpolation processing on the current original video frame sequence by optical flow interpolation; and perform frame interpolation processing on the next original video frame sequence until the frame interpolation processing of the original video data or the first original video segment is completed.

[0125] In some embodiments, the non-volatile storage medium includes a first non-volatile storage medium built into the terminal and a second non-volatile storage medium connected externally to the terminal; the recording module 11 is further configured to store the original video stream at a first frame rate into the first non-volatile storage medium through a target transmission channel via an image sensor; and store the original video stream into the second non-volatile storage medium when the available storage space of the first non-volatile storage medium is less than or equal to a preset available threshold.

[0126] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0127] In practical applications, the aforementioned video processing apparatus can be an electronic device that implements video processing functions, or a chip applied to an electronic device. In this application, the apparatus can implement the functions of multiple units through software, hardware, or a combination of both, enabling the apparatus to execute the video processing method provided in any of the above embodiments. Furthermore, the technical effects of each technical solution of the apparatus can be referenced to the technical effects of the corresponding technical solutions in the video processing method; therefore, this application will not elaborate further on these effects.

[0128] Based on the hardware implementation of each unit in the aforementioned video processing device, embodiments of this application also provide an electronic device, such as... Figure 9As shown, the electronic device 90 includes: a processor 901 and a memory 902 configured to store computer programs capable of running on the processor; The processor 901 is configured to execute the method steps in the foregoing embodiments when running a computer program.

[0129] Of course, in practical applications, such as Figure 9 As shown, the various components in the electronic device 90 are coupled together via a bus system 903. It is understood that the bus system 903 is used to enable communication between these components. In addition to a data bus, the bus system 903 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 903 in the figure.

[0130] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0131] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0132] Optionally, the electronic device 90 can be a chip, which may further include an input interface. The processor can control the input interface to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0133] Optionally, the chip may also include an output interface. The processor can control this output interface to communicate with other devices or chips; specifically, it can output information or data to other devices or chips.

[0134] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to perform the steps of the aforementioned method.

[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the embodiments of this application.

[0136] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0137] This application also provides a computer program. Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0138] It should be understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0139] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.

[0140] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0141] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0143] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0144] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video processing method, characterized in that, include: When video capture is started, the raw video stream of the first frame rate is stored in the target storage space through the target transmission channel via the image sensor. After the acquisition is completed, the original video data is interpolated at the second frame rate to determine the intermediate video data; The second frame rate is greater than the first frame rate; The intermediate video data is encoded at the target encoding frame rate to determine the target video; The target encoded frame rate is less than the second frame rate.

2. The method according to claim 1, characterized in that, The step of interpolating the original video data at a second frame rate to determine intermediate video data includes: Obtain the time stamp corresponding to the original video data from the target storage space; the time stamp is used to characterize the motion time interval of the moving object in the original video data; At the second frame rate, the first original video segment corresponding to the time marker in the original video data is subjected to frame interpolation to determine the intermediate video data; the intermediate video data includes a first video segment obtained by interpolating the first original video segment and at least one second original video segment at the first frame rate; the at least one second original video segment is a video segment in the original video data that is outside the motion time interval represented by the time marker.

3. The method according to claim 1 or 2, characterized in that, The target transmission channel includes: a storage device controller and a non-volatile storage bus; the target storage space includes a non-volatile storage medium; the step of storing the original video stream at the first frame rate into the target storage space through the target transmission channel via the image sensor includes: The image sensor outputs the raw video stream to the storage device controller, and the storage device controller stores the raw video stream into the non-volatile storage medium via the non-volatile storage bus.

4. The method according to claim 3, characterized in that, The step of outputting the raw video stream to the storage device controller via the image sensor includes: The raw video stream is output through the image sensor; The original video stream is divided into segments according to a preset time interval to determine the original video blocks. The original video chunks are transmitted to the storage device controller, so that the storage device controller can store the original video chunks into the non-volatile storage medium via the non-volatile storage bus.

5. The method according to claim 2, characterized in that, The time stamp includes a first time stamp and a second time stamp; The first time marker is used to characterize the start time of the motion; The second time stamp is used to characterize the end time of the motion, and the method further includes: Optical flow motion detection is performed on the preview video stream corresponding to the original video stream to determine the first time marker and the second time marker; Store the first time stamp and the second time stamp into the target storage space.

6. The method according to claim 5, characterized in that, The step of performing optical flow motion detection on the preview video stream corresponding to the original video stream to determine the first time marker and the second time marker includes: Determine the target region in the preview video stream; Optical flow motion detection is performed on the target area, and if a moving pixel with a displacement exceeding a preset displacement threshold is detected in the target area, the first time marker is determined. If the position of the moving pixel is detected to be outside the target area, the second time marker is determined.

7. The method according to claim 5 or 6, characterized in that, The method further includes: If the sensor data of the terminal detects that the degree of shaking of the terminal does not meet the stability threshold, a prompt will be displayed on the video acquisition interface of the terminal.

8. The method according to claim 1, characterized in that, The step of interpolating the original video data at a second frame rate to determine intermediate video data includes: The entire segment of the original video data is interpolated at the second frame rate to determine the intermediate video data.

9. The method according to claim 2, or any one of claims 4-6, or claim 8, characterized in that, Frame interpolation processing includes: Determine the current sequence of original video frames; If the motion complexity of the current original video frame sequence is less than the motion complexity threshold, the current original video frame sequence is interpolated by frame duplication. If the motion complexity of the current original video frame sequence is greater than or equal to the motion complexity threshold, the current original video frame sequence is interpolated by optical flow frame interpolation. The next original video frame sequence is subjected to frame interpolation processing until the frame interpolation processing of the original video data or the first original video segment is completed.

10. The method according to claim 1 or 2, or any one of claims 4-6, or claim 8, characterized in that, The target storage space includes a first non-volatile storage medium built into the terminal and a second non-volatile storage medium connected externally to the terminal. The step of storing the original video stream at the first frame rate into the target storage space via the target transmission channel through the image sensor includes: The original video stream at the first frame rate is stored in the first non-volatile storage medium through the target transmission channel via the image sensor. If the available storage space of the first non-volatile storage medium is less than or equal to a preset available threshold, the original video stream is stored in the second non-volatile storage medium.

11. A video processing system, characterized in that, Includes terminals and servers, among which: The terminal is used to, when video acquisition is started, store the raw video stream of the first frame rate into the target storage space through the target transmission channel via an image sensor; after acquisition is completed, send the raw video data in the target storage space to the server. The server is configured to, after acquisition, perform frame interpolation on the original video data at a second frame rate to determine intermediate video data; the second frame rate is greater than the first frame rate; encode the intermediate video data at a target encoding frame rate to determine the target video; the target encoding frame rate is less than the second frame rate.

12. A video processing apparatus, characterized in that, include: The recording module is used to store the raw video stream of the first frame rate into the target storage space through the target transmission channel when video capture is started, via an image sensor. The frame interpolation module is used to perform frame interpolation processing on the original video data at a second frame rate after the acquisition is completed, to determine intermediate video data; the second frame rate is greater than the first frame rate. The encoding module is used to encode the intermediate video data at a target encoding frame rate to determine the target video; The target encoded frame rate is less than the second frame rate.

13. An electronic device, characterized in that, include: Memory and processor; among which, The memory is used to store executable instructions. The processor is configured to implement the method as described in any one of claims 1-10 when executing executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, It stores executable instructions for causing the processor to execute, thereby implementing the method as described in any one of claims 1-10.

15. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-10.

Images