Video frame processing method, device, apparatus and storage medium

Abstract

This application discloses a video frame processing method, device, apparatus, and storage medium, relating to the field of smart glasses technology. The disclosed video frame processing method includes: receiving a first video frame corresponding to each buffered single frame of a smart device and its streaming transmission; performing corresponding first processing on the first video frame to obtain a first processing result; wherein, during the corresponding first processing of the first video frame, corresponding second processing is also performed simultaneously on a second video frame, the second video frame being distinct from the first video frame. That is, by reducing unnecessary processing steps in the smart device, unnecessary processing time in the smart device is saved, and by using parallel processing on the mobile terminal, the time for the mobile terminal to process video frames is saved, thereby reducing data transmission latency.

Description

Technical Field

[0001] This application relates to the field of smart glasses technology, and in particular to methods, devices, apparatuses, and storage media for processing video frames. Background Technology

[0002] With the development of AI glasses technology, there are more and more scenarios for processing camera data, and the requirements are getting higher and higher.

[0003] Currently, after the camera in AI glasses collects data, it is usually saved locally until a complete video file is obtained locally. Then, the complete video file is sent to the mobile device, where it decodes the complete video file frame by frame, performs image stabilization, and finally generates the final video displayed to the user. However, this data processing takes a long time, thus increasing the latency of data transmission.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a video frame processing method that aims to solve the technical problem of long data processing time, which increases the latency of data transmission.

[0006] To achieve the above objectives, this application proposes a video frame processing method applied to a mobile terminal, the method comprising:

[0007] Receive the first video frame that is streamed after each buffered single frame from the smart device.

[0008] The first video frame is subjected to a first processing to obtain a first processing result. During the first processing of the first video frame, the second video frame is also subjected to a second processing simultaneously. The second video frame is distinct from the first video frame.

[0009] In one embodiment, when the first process is to perform image stabilization on the first video frame, the second process is to receive the second video frame and decode the second video frame; when the first process is to decode the first video frame, the second process is to perform image stabilization on the second video frame.

[0010] The step of performing corresponding first processing on the first video frame to obtain a first processing result, wherein during the corresponding first processing on the first video frame, the corresponding second processing on the second video frame is also performed simultaneously, includes:

[0011] The first video frame is stabilized to obtain a first processing result. During the stabilization process of the first video frame, the second video received afterward is also decoded simultaneously to obtain a second processing result.

[0012] Alternatively, the first video frame can be decoded to obtain a first processing result, wherein during the decoding process of the first video frame, the second video can also be stabilized simultaneously to obtain a second processing result.

[0013] In one embodiment, the first video frame includes a stabilized video frame, and the step of stabilizing the first video frame to obtain a first processing result includes:

[0014] Acquire the shooting data when the smart device captures the first video frame;

[0015] Based on the captured data, the first video frame is stabilized to obtain a stabilized video frame.

[0016] In one embodiment, the first video frame includes the decoded original video frame, and the step of decoding the first video frame to obtain a first processing result includes:

[0017] Obtain the encoding rules used by the smart device to encode the first video frame;

[0018] Based on the encoding rules, the first video frame is asynchronously decoded to obtain the decoded original video frame.

[0019] In one embodiment, before the step of performing corresponding first processing on the first video frame to obtain a first processing result, the method further includes:

[0020] Determine whether the first video frame is a valid frame;

[0021] If the first video frame is not a valid frame, then the first video frame is discarded;

[0022] The step of performing corresponding first processing on the first video frame to obtain a first processing result further includes:

[0023] If the first video frame is a valid frame, then the first video frame is subjected to corresponding first processing to obtain a first processing result.

[0024] In one embodiment, after the step of performing corresponding first processing on the first video frame to obtain a first processing result, the method further includes:

[0025] Determine whether the first video frame is the last frame;

[0026] If the first video frame is the last frame, then after stabilizing the first video frame, the first video is stitched into the video segment obtained based on the previous video frames to obtain complete video data.

[0027] Furthermore, to achieve the above objectives, this application also proposes a video frame processing method applied to a smart device, the method comprising:

[0028] Encode the currently cached single frame to obtain a video frame;

[0029] Based on a preset streaming transmission, the video frames are sent to the mobile terminal in real time.

[0030] Furthermore, to achieve the above objectives, this application also proposes a video frame processing apparatus, which includes:

[0031] The receiving module is used to receive the first video frame that is streamed after each buffered single frame by the smart device.

[0032] The processing module is used to perform corresponding first processing on the first video frame to obtain a first processing result. During the first processing of the first video frame, corresponding second processing is also performed on the second video frame simultaneously. The second video frame is distinct from the first video frame.

[0033] In addition, to achieve the above objectives, this application also proposes a video frame processing apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video frame processing method described above.

[0034] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the video frame processing method described above.

[0035] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the video frame processing method described above.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] After a smart device captures and caches a video frame, it transmits that single video frame (the first video frame) to the mobile terminal in real time via streaming. Upon receiving the first video frame, the mobile terminal performs corresponding first processing on it. During this processing, it also simultaneously performs corresponding second processing on the second video frame, achieving parallel processing of video frames. This saves the mobile terminal's time in processing video frames. Furthermore, the smart device sends each cached video frame to the mobile terminal immediately, avoiding the need to save the complete video data on the smart device before sending it frame by frame. This reduces the processing steps for video frames on the smart device, thereby saving unnecessary processing time. Therefore, it reduces the data processing time during the transmission of video from the smart glasses to the mobile terminal, thus reducing data transmission latency. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating the video frame processing method of this application (Example 1).

[0041] Figure 2 This is a flowchart illustrating a second embodiment of the video frame processing method of this application.

[0042] Figure 3 This is a flowchart illustrating the video frame processing method of Embodiment 3 of this application;

[0043] Figure 4 This is a flowchart illustrating an embodiment of the video frame processing method of this application applied to a smart device;

[0044] Figure 5 A simplified flowchart illustrating the video frame processing method provided in this application embodiment;

[0045] Figure 6 This is a schematic diagram of the module structure of the video frame processing device according to an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the video frame processing method in the embodiments of this application.

[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0049] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of this application embodiment is: the mobile terminal receives the first video frame corresponding to each cached single frame of the smart device and is streamed; the first video frame is subjected to corresponding first processing to obtain a first processing result, wherein, during the corresponding first processing of the first video frame, the second video frame is also subjected to corresponding second processing simultaneously, and the second video frame is different from the first video frame.

[0051] In this embodiment, for ease of description, the following description uses a mobile terminal as the execution subject.

[0052] Because the camera in AI glasses typically saves the collected data locally until a complete video file is obtained locally, it then sends the complete video file to the mobile device. The mobile device decodes the complete video file frame by frame, performs image stabilization, and finally generates the final video displayed to the user. However, this data processing takes a long time, thus increasing the data transmission latency.

[0053] This application provides a solution whereby a smart device captures and caches a single video frame, then transmits that single video frame (the first video frame) to a mobile terminal in real time via streaming. Upon receiving the first video frame, the mobile terminal performs corresponding first processing on it, and simultaneously performs corresponding second processing on a second video frame, achieving parallel processing of video frames. This saves the mobile terminal's time in processing video frames. Furthermore, the smart device sends each cached video frame to the mobile terminal immediately, avoiding the need to save the complete video data on the smart device before sending it frame by frame to the mobile terminal. This reduces the processing steps for video frames on the smart device, thereby saving unnecessary processing time. Therefore, it reduces the data processing time during the transmission of video from smart glasses to the mobile terminal, thus reducing data transmission latency.

[0054] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or mobile terminal capable of performing the above functions. The following description uses a mobile terminal as an example to illustrate this embodiment and the subsequent embodiments.

[0055] Based on this, embodiments of this application provide a method for processing video frames, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the video frame processing method of this application.

[0056] In this embodiment, the video frame processing method includes steps S10 to S20:

[0057] Step S10: Receive the first video frame corresponding to each buffered single frame from the smart device and its streaming transmission. It should be noted that the smart device is an electronic device with data processing, storage, and communication functions, such as a smartphone, tablet, smart TV, or smart glasses. In this embodiment, smart glasses are used as an example. A single frame is a video frame currently captured by the smart device. Streaming transmission is a method of transmitting video data to the receiving end in a continuous stream; that is, the smart glasses transmit one video frame to the mobile terminal each time it buffers a single frame. The first video frame is the latest buffered single video frame from the smart glasses.

[0058] Understandably, by reducing the processing steps of each cached single frame by the smart device, and by reducing these processing steps without negatively impacting the video frames obtained by the mobile terminal, the processing time of the smart device for cached single frames is saved, and the mobile terminal is guaranteed to obtain complete video frames. This allows the mobile terminal to process and display video frames faster, thereby reducing video latency.

[0059] Step S20: Perform corresponding first processing on the first video frame to obtain a first processing result. During the first processing of the first video frame, perform corresponding second processing on the second video frame simultaneously. The second video frame is distinct from the first video frame.

[0060] It should be noted that the first processing can be either image stabilization or decoding. The second processing can also be either decoding or image stabilization. Specifically, when the first processing is image stabilization, the second processing is decoding; conversely, when the first processing is decoding, the second processing is image stabilization. The second video frame can be a video frame received by the mobile terminal before or after receiving the first video frame.

[0061] Understandably, the mobile terminal processes the second video frame simultaneously while processing the first video frame, making full use of the device's processing power and the idle time of the mobile terminal when processing a single video frame. This improves the utilization rate of device resources, reduces the processing time of the mobile terminal for video frames, and improves the overall processing efficiency, thereby reducing unnecessary waiting and delays.

[0062] In a specific implementation, the second video frame can be received and decoded simultaneously during the image stabilization process of the first video frame. In this case, the first video frame is the video frame before the value of the second video frame. Alternatively, the second video frame can be stabilized simultaneously during the decoding process of the first video frame. In this case, the first video frame is the video frame after the value of the second video frame.

[0063] It should be noted that when the first process is to perform image stabilization on the first video frame, the second process is to receive the second video frame and decode the second video frame; when the first process is to decode the first video frame, the second process is to perform image stabilization on the second video frame.

[0064] The process after step S20 also includes:

[0065] Determine whether the first video frame is the last frame;

[0066] If the first video frame is the last frame, then after stabilizing the first video frame, the first video is stitched into the video segment obtained based on the previous video frames to obtain complete video data.

[0067] It's important to note that the last frame is the last video frame in the video sequence, that is, the last video frame of the relevant video sent by the smart device to the mobile terminal. A video segment is composed of multiple processed, consecutive video frames. Complete video data is the final video content obtained by stitching all video frames together after processing by the mobile terminal.

[0068] Understandably, since the smart device sends the single video frame to the mobile terminal after caching it, the smart device does not need to determine the integrity of the video. In order for the mobile terminal to obtain complete video data based on the received video frame, the integrity of the video data is ensured by determining whether the first video frame is the last frame and then stitching it into the video segment after image stabilization of the last frame. This avoids the problem of video incompleteness caused by the loss of the last frame in the video segment.

[0069] Understandably, image stabilization can reduce jitter and blur in video, making the video image more stable and clearer. Stabilizing the last frame ensures that every frame of the video data has been optimized, thus improving the overall quality of the video.

[0070] Understandably, ensuring video integrity and improving video quality enhances the user's viewing experience. This allows users to watch a complete and high-quality video, reducing the negative experience caused by missing frames or poor video quality.

[0071] In its implementation, the mobile terminal can determine whether the first video frame being processed is the last frame in the video sequence by checking the end marker, frame number, or other relevant mechanisms. If the first video frame is not the last frame, the mobile terminal will continue to receive and process subsequent video frames. If it is the last frame, the mobile terminal uses algorithms to detect and compensate for motion changes in the video frame to stabilize the first video frame, reducing jitter or blur and making the video image more stable and clear. This stabilized first video frame is then stitched together with the previously processed video segments to ensure the continuity of video frames in time and space, forming a complete video sequence and thus obtaining complete video data. The previously processed video segments contain all the previously processed video frames.

[0072] This embodiment provides a video frame processing method. After a smart device captures and caches a video frame, it transmits the single video frame (first video frame) to a mobile terminal in real time via streaming. Upon receiving the first video frame, the mobile terminal performs corresponding first processing on the first video frame. During the processing, it also performs corresponding second processing on the second video frame simultaneously, achieving parallel processing of video frames to save the mobile terminal's video frame processing time. Furthermore, the smart device sends each cached video frame to the mobile terminal immediately, avoiding the need to save the complete video data on the smart device before sending it frame by frame to the mobile terminal. This reduces the processing steps of the smart device on the video frames, thereby saving unnecessary processing time in the smart device. Therefore, it reduces the data processing time during the transmission of video from the smart glasses to the mobile terminal, thus reducing the data transmission latency.

[0073] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S20, the video frame processing method further includes steps S21 to S22:

[0074] Step S21: Stabilize the first video frame to obtain a first processing result. During the stabilization process of the first video frame, the second video received afterward is also decoded simultaneously to obtain a second processing result.

[0075] Step S22, or, the first video frame is decoded to obtain a first processing result, wherein during the decoding process of the first video frame, the second video is simultaneously stabilized to obtain a second processing result.

[0076] It's important to note that image stabilization is an image processing technique that reduces jitter or blur in video frames, resulting in a more stable and clearer video image. Specifically, image stabilization can be achieved by using algorithms to detect and compensate for motion changes in video frames. Additionally, decoding is the process of converting encoded video data into raw video frames that can be further processed. Specific decoding processes may involve decompressing and converting the video data into different formats.

[0077] In this embodiment, the first processing result is obtained after stabilizing or decoding the first video frame; it can be a clear video frame that has undergone image stabilization or a decoded video frame. The second processing result is obtained after decoding or stabilizing the second video; it can be decoded video frame data or a clear video frame that has undergone image stabilization.

[0078] In this implementation, the mobile terminal performs image stabilization processing on the first video frame. This involves using algorithms to detect and compensate for motion changes in the first video frame, reducing jitter or blurring and making the first video frame more stable and clear. After processing, a first processing result is obtained, namely the stabilized video frame. Simultaneously, during the image stabilization of the first video frame, the mobile terminal receives and decodes the second video frame, converting the encoded video data into a raw video frame suitable for further processing. After this decoding, a second processing result is obtained, namely the decoded video frame data. It should be noted that in this implementation, the first video frame is received before the second video frame.

[0079] In a specific implementation, when the first video frame is received after the second video frame, the mobile terminal can also perform image stabilization processing on the second video frame. This involves using algorithms to detect and compensate for motion changes in the second video frame, reducing jitter or blur, and making the second video frame more stable and clear. After processing, a second processing result is obtained. Simultaneously, during the image stabilization of the second video frame, the mobile terminal decodes the first video frame to convert the encoded video data into raw video frames suitable for further processing. After this process, a first processing result is obtained, namely the decoded video frame data.

[0080] Furthermore, step S21 also includes:

[0081] The first video frame includes a stabilized video frame, and the step of stabilizing the first video frame to obtain a first processing result includes:

[0082] Acquire the shooting data when the smart device captures the first video frame;

[0083] Based on the captured data, the first video frame is stabilized to obtain a stabilized video frame.

[0084] In this embodiment, the first video frame includes the stabilized video frame after image stabilization.

[0085] It should be noted that shooting data refers to additional information generated by the smart device when shooting video frames, such as device motion data (gyroscope data and / or accelerometer data) and shooting parameters (exposure time, aperture size, and / or shutter speed). Stabilized video frames are those that have undergone image stabilization processing, resulting in reduced jitter or blur, and a more stable and clearer image.

[0086] Understandably, since the shooting data provides detailed motion information of the device during the shooting process, the image stabilization process can more accurately compensate for motion changes in the video frame. Therefore, the mobile terminal can perform image stabilization on the first video frame based on the shooting data to effectively reduce jitter or blur in the first video frame, making the video picture more stable and clear.

[0087] Understandably, mobile terminals effectively reduce jitter or blur in video frames, making the video image more stable and clearer. This not only improves video quality but also optimizes the user experience and increases processing efficiency.

[0088] In a specific implementation, during the process of the smart device capturing the first video frame, the smart device can also record shooting data such as gyroscope data, accelerometer data, exposure time, aperture size and / or shutter speed, and then send the shooting data along with the first video frame to the mobile terminal. When the mobile terminal performs image stabilization processing on the first video frame, it can analyze the motion information in the shooting data based on the shooting data, detect jitter or blur in the video frame, and perform corresponding compensation, thereby achieving image stabilization processing of the first video frame and obtaining a stable video frame.

[0089] Further, the step of decoding the first video frame to obtain the first processing result includes:

[0090] Obtain the encoding rules used by the smart device to encode the first video frame;

[0091] Based on the encoding rules, the first video frame is asynchronously decoded to obtain the decoded original video frame.

[0092] In this implementation example, the first video frame includes the decoded original video frame.

[0093] It should be noted that encoding rules are the rules and parameters followed by smart devices when encoding video frames. Asynchronous decoding is the process where the smart device encodes the first video frame, and then the mobile terminal decodes the first video frame.

[0094] It is understandable that by using asynchronous decoding, the decoder in the mobile terminal can decode the first video frame encoded in the smart device, thus avoiding the need to decode and send it to the mobile terminal after the smart device has encoded and buffered the video. Therefore, the processing steps for the first video frame are reduced, thereby improving the efficiency of the mobile terminal in obtaining video data and reducing the latency of video data during transmission.

[0095] Understandably, mobile terminals can accurately decode the first encoded video frame using encoding rules, ensuring that the decoded video frame is consistent with the original video frame, thereby guaranteeing that the video data obtained by the mobile terminal is the video data captured by the smart device.

[0096] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S20, the video frame processing method further includes steps S01 to S03:

[0097] Step S01: Determine whether the first video frame is a valid frame;

[0098] Step S02: If the first video frame is not a valid frame, then discard the first video frame.

[0099] It should be noted that a valid frame is a video frame related to the desired video, and that the video frame is not corrupted and meets the video processing requirements. Discarding means removing video frames that do not meet the requirements from the processing flow and no longer processing them.

[0100] Understandably, since the initial video frames sent by the smart device may not be aligned with the target being filmed, or the video frames may contain afterimages, it is necessary to determine whether the first video frame is valid in order to process the video frames efficiently and reduce the latency of video data transmission. Invalid frames are discarded to reduce the processing of invalid data by the system, save computing resources and time, thereby improving the overall processing efficiency and filtering out the valid frames needed by the user.

[0101] Understandably, by discarding invalid frames, mobile terminals can ensure that all video frames entering subsequent processing are valid and compliant, thereby improving the accuracy of video processing.

[0102] In practical implementation, the mobile terminal can check whether the video frame contains all the necessary data, without any loss or corruption, and / or verify whether the video frame conforms to the expected video format and encoding rules, and / or check whether the timestamps of the video frame are continuous and logical, ensuring the correct order of the video frames, and / or evaluate whether the quality of the video frame meets the predetermined standards, such as resolution and clarity, to determine whether the first video frame is a valid frame. If the first video frame is determined to be an invalid frame, the mobile terminal removes the frame from the processing flow and stops further processing to prevent invalid frames from interfering with subsequent processing, ensuring the accuracy and efficiency of video processing.

[0103] Furthermore, step S20 also includes:

[0104] Step S03: If the first video frame is a valid frame, then perform corresponding first processing on the first video frame to obtain a first processing result.

[0105] Understandably, performing predetermined processing operations on the first video frame after confirming it as a valid frame ensures the accuracy and efficiency of the video processing workflow. This not only improves the accuracy of video processing but also enhances processing efficiency, thereby increasing the stability and reliability of the system.

[0106] Furthermore, embodiments of this application also provide a method for processing video frames in smart devices, referencing... Figure 4 The method includes steps S1 to S2:

[0107] Step S1: Encode the currently buffered single frame to obtain a video frame;

[0108] Step S2: Based on the preset streaming transmission, the video frame is sent to the mobile terminal in real time.

[0109] Understandably, sending the first video to the mobile terminal immediately after the smart device caches a single frame reduces the time the video frame stays in the smart device's buffer, allowing the video frames captured by the smart device to be processed quickly by the mobile terminal, thus improving the real-time performance of the final video data.

[0110] For example, to help understand the implementation flow of the video frame processing method obtained by combining this embodiment with the above embodiments, please refer to... Figure 5 , Figure 5 A simplified flowchart of a video frame processing method is provided, specifically:

[0111] When the camera in the smart glasses captures the first video frame, it caches that frame in local storage. The smart glasses then stream the first video frame via Wi-Fi to the mobile terminal, where it is received by the mobile receiver. The mobile receiver then transmits the first video frame to the decoder, which asynchronously decodes it and sends the decoded first video frame (the current frame) to the EIS (Enhanced Image Stabilizer). The EIS performs real-time image stabilization on the first video frame. In other words, the camera outputs images frame by frame, sending each frame to the mobile terminal. After receiving 2-3 frames, the mobile terminal begins processing them using an algorithm. Within the processing loop, serial processing is replaced with parallel processing to reduce processing time.

[0112] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the processing method of video frames in this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0113] This application also provides a video frame processing apparatus applied to a mobile terminal, please refer to... Figure 6 The video frame processing device includes:

[0114] The receiving module 10 is used to receive the first video frame that is streamed after each buffered single frame by the smart device;

[0115] The processing module 20 is used to perform corresponding first processing on the first video frame to obtain a first processing result. During the first processing of the first video frame, corresponding second processing is also performed on the second video frame simultaneously. The second video frame is distinct from the first video frame.

[0116] Optionally, when the first process is to perform image stabilization on the first video frame, the second process is to receive the second video frame and decode the second video frame; when the first process is to decode the first video frame, the second process is to perform image stabilization on the second video frame.

[0117] The processing module 20 is further configured to perform image stabilization processing on the first video frame to obtain a first processing result, wherein during the image stabilization processing of the first video frame, the second video received subsequently is also decoded simultaneously to obtain a second processing result; or, the first video frame is decoded to obtain a first processing result, wherein during the decoding processing of the first video frame, the second video is also simultaneously stabilized to obtain a second processing result.

[0118] Optionally, the first video frame includes a stabilized video frame after image stabilization;

[0119] The processing module 20 is further configured to acquire the shooting data when the smart device captures the first video frame; and based on the shooting data, perform image stabilization processing on the first video frame to obtain a stabilized video frame.

[0120] Optionally, the first video frame includes the decoded original video frame;

[0121] The processing module 20 is further configured to acquire the encoding rules when the smart device encodes the first video frame; and based on the encoding rules, perform asynchronous decoding processing on the first video frame to obtain the decoded original video frame.

[0122] Optionally, the video frame processing apparatus further includes:

[0123] The judgment module 30 is used to determine whether the first video frame is a valid frame; if the first video frame is not a valid frame, the first video frame is discarded.

[0124] Optionally, the processing module 20 is further configured to perform corresponding first processing on the first video frame if the first video frame is a valid frame, and obtain a first processing result.

[0125] Optionally, the judgment module 30 is further configured to determine whether the first video frame is the last frame; if the first video frame is the last frame, then after stabilizing the first video frame, the first video is spliced ​​into the video segment obtained based on the previous video frames to obtain complete video data.

[0126] Optionally, the video frame processing apparatus can also be applied to smart devices, and the video frame processing apparatus further includes:

[0127] Encoding module 40 is used to encode the currently buffered single frame to obtain a video frame;

[0128] The transmission module 50 is used to send the video frames to the mobile terminal in real time based on a preset streaming transmission.

[0129] The video frame processing apparatus provided in this application, employing the video frame processing method described in the above embodiments, can solve the technical problem of long data processing time, thereby increasing data transmission latency. Compared with the prior art, the beneficial effects of the video frame processing apparatus provided in this application are the same as those of the video frame processing method described in the above embodiments, and other technical features in the video frame processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0130] This application provides a video frame processing apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video frame processing method in the above embodiment 1.

[0131] The following is for reference. Figure 7 This document illustrates a schematic diagram of a processing device suitable for implementing video frames in the embodiments of this application. The video frame processing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The video frame processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] like Figure 7As shown, the video frame processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the video frame processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the video frame processing device to communicate wirelessly or wiredly with other devices to exchange data. Although video frame processing devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0133] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0134] The video frame processing device provided in this application, employing the video frame processing method described in the above embodiments, can solve the technical problem of long data processing time, thereby increasing data transmission latency. Compared with the prior art, the beneficial effects of the video frame processing device provided in this application are the same as those of the video frame processing method provided in the above embodiments, and other technical features in this video frame processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0135] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0136] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0137] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the video frame processing method described in the above embodiments.

[0138] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0139] The aforementioned computer-readable storage medium may be included in a video frame processing device; or it may exist independently and not assembled into a video frame processing device.

[0140] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a video frame processing device, cause the video frame processing device to: receive a first video frame corresponding to each buffered single frame of the smart device and transmit it in a streaming manner; perform corresponding first processing on the first video frame to obtain a first processing result, wherein, during the corresponding first processing of the first video frame, corresponding second processing is also performed simultaneously on a second video frame, the second video frame being distinct from the first video frame.

[0141] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0143] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0144] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video frame processing method. This solves the technical problem of long data processing times, which increases data transmission latency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video frame processing method provided in the above embodiments, and will not be repeated here.

[0145] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video frame processing method described above.

[0146] The computer program product provided in this application can solve the technical problem that the long data processing time increases the latency of data transmission. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the video frame processing method provided in the above embodiments, and will not be repeated here.

[0147] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for processing video frames, characterized in that, Applied to mobile terminals, the method includes: Receive the first video frame that is streamed after each buffered single frame from the smart device. The first video frame is subjected to a first processing to obtain a first processing result. During the first processing of the first video frame, the second video frame is also subjected to a second processing simultaneously. The second video frame is distinct from the first video frame.

2. The method as described in claim 1, characterized in that, When the first processing is to perform image stabilization processing on the first video frame, the second processing is to receive the second video frame and decode the second video frame; when the first processing is to decode the first video frame, the second processing is to perform image stabilization processing on the second video frame. The step of performing corresponding first processing on the first video frame to obtain a first processing result, wherein during the corresponding first processing on the first video frame, the corresponding second processing on the second video frame is also performed simultaneously, includes: The first video frame is stabilized to obtain a first processing result. During the stabilization process of the first video frame, the second video received afterward is also decoded simultaneously to obtain a second processing result. Alternatively, the first video frame can be decoded to obtain a first processing result, wherein during the decoding process of the first video frame, the second video can also be stabilized simultaneously to obtain a second processing result.

3. The method as described in claim 2, characterized in that, The first video frame includes a stabilized video frame, and the step of stabilizing the first video frame to obtain a first processing result includes: Acquire the shooting data when the smart device captures the first video frame; Based on the captured data, the first video frame is stabilized to obtain a stabilized video frame.

4. The method as described in claim 2, characterized in that, The first video frame includes the decoded original video frame, and the step of decoding the first video frame to obtain the first processing result includes: Obtain the encoding rules used by the smart device to encode the first video frame; Based on the encoding rules, the first video frame is asynchronously decoded to obtain the decoded original video frame.

5. The method as described in claim 1, characterized in that, Before the step of performing corresponding first processing on the first video frame to obtain the first processing result, the method further includes: Determine whether the first video frame is a valid frame; If the first video frame is not a valid frame, then the first video frame is discarded; The step of performing corresponding first processing on the first video frame to obtain a first processing result further includes: If the first video frame is a valid frame, then the first video frame is subjected to corresponding first processing to obtain a first processing result.

6. The method as described in claim 1, characterized in that, After the step of performing corresponding first processing on the first video frame to obtain the first processing result, the method further includes: Determine whether the first video frame is the last frame; If the first video frame is the last frame, then after stabilizing the first video frame, the first video is stitched into the video segment obtained based on the previous video frames to obtain complete video data.

7. A method for processing video frames, characterized in that, Applied to smart devices, the method includes: Encode the currently cached single frame to obtain a video frame; Based on a preset streaming transmission, the video frames are sent to the mobile terminal in real time.

8. A video frame processing apparatus, characterized in that, The device includes: The receiving module is used to receive the first video frame that is streamed after each buffered single frame by the smart device. The processing module is used to perform corresponding first processing on the first video frame to obtain a first processing result. During the first processing of the first video frame, corresponding second processing is also performed on the second video frame simultaneously. The second video frame is distinct from the first video frame.

9. A video frame processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for processing video frames as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video frame processing method as described in any one of claims 1 to 7.

Images