Video processing method, device, storage medium and program product

Abstract

The embodiment of the present disclosure provides a video processing method, device, storage medium and program product, which receives an image frame to be rendered output by an image acquisition device; performs a first special effect rendering operation on the image frame to be rendered by a first thread to obtain a first rendered image frame; outputs the first rendered image frame to a second thread and a third thread, and requests the image acquisition device to output a next image frame to be rendered; the third thread performs video encoding on the first rendered image frame; the second thread performs a second special effect rendering operation on the first rendered image frame to obtain a second rendered image frame, and performs video display based on the second rendered image frame. The embodiment of the present disclosure performs special effect rendering in two threads, requests the image acquisition device to output a next image frame to be rendered after the first thread is rendered, simultaneously performs special effect rendering, screen image frame drawing of the second thread and video encoding of the third thread, improves the frequency of output image frames to be rendered, and improves the video frame rate of encoding recording.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to a video processing method, apparatus, storage medium, and program product. Background Technology

[0002] With the development and widespread adoption of mobile devices, more and more users are using them to shoot videos and record their daily lives. During the video recording process, users may add various effects, such as beauty filters, augmented reality (AR) effects, 3D effects, etc., to render the original video, helping users publish more exquisite works and enhancing the user experience.

[0003] In the existing technology, the preview of image acquisition devices is based on a single thread. After the image acquisition device outputs an image frame, it performs special effects rendering and finally screen drawing, all of which occur within a single thread. After the screen drawing is completed, the image acquisition device continues to output the next image frame for the next special effects rendering and screen drawing.

[0004] In existing technologies, if the special effects rendering process takes a long time, it will affect the timing of the image acquisition device outputting the next image frame, resulting in a reduction in the frame rate of the encoded and recorded image. Summary of the Invention

[0005] This disclosure provides a video processing method, device, storage medium, and program product to reduce the time consumption of special effects rendering and increase the frame rate of encoded and recorded images.

[0006] In a first aspect, embodiments of this disclosure provide a video processing method, including:

[0007] Receive the image frame to be rendered output by the image acquisition device;

[0008] The first rendering image frame is obtained by performing a first special effect rendering operation on the image frame to be rendered through the first thread.

[0009] The first rendered image frame is output to the second and third threads, and the image acquisition device is requested to output the next image frame to be rendered.

[0010] The first rendered image frame is video encoded using the third thread;

[0011] The second thread performs a second effect rendering operation on the first rendered image frame to obtain a second rendered image frame, and then displays the video based on the second rendered image frame.

[0012] In a second aspect, embodiments of this disclosure provide a video processing apparatus, including:

[0013] The receiving unit is used to receive the image frames to be rendered output by the image acquisition device;

[0014] The first processing unit is configured to perform a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame; output the first rendered image frame to the second thread and the third thread, and request the image acquisition device to output the next image frame to be rendered;

[0015] An encoding unit is used to perform video encoding on the first rendered image frame through the third thread;

[0016] The second processing unit is used to perform a second special effect rendering operation on the first rendered image frame through the second thread to obtain a second rendered image frame, and to display video based on the second rendered image frame.

[0017] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0018] The memory stores computer-executed instructions;

[0019] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the video processing method as described in the first aspect and various possible designs of the first aspect.

[0020] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video processing method described in the first aspect and various possible designs of the first aspect.

[0021] Fifthly, embodiments of this disclosure provide a computer program product including computer instructions that, when executed by a processor, implement the video processing method described in the first aspect and various possible designs of the first aspect.

[0022] The video processing method, apparatus, storage medium, and program product provided in this disclosure include: receiving an image frame to be rendered from an image acquisition device; performing a first special effect rendering operation on the image frame to be rendered using a first thread to obtain a first rendered image frame; outputting the first rendered image frame to a second thread and a third thread, and requesting the image acquisition device to output the next image frame to be rendered; performing video encoding on the first rendered image frame using the third thread; performing a second special effect rendering operation on the first rendered image frame using the second thread to obtain a second rendered image frame, and displaying video based on the second rendered image frame. This disclosure performs special effect rendering in two threads. After the first thread finishes rendering, it requests the image acquisition device to output the next image frame to be rendered while simultaneously performing special effect rendering and screen image frame drawing in the second thread, and video encoding in the third thread. This eliminates the need to wait for the second thread to finish rendering and the screen image frame to finish drawing before requesting the image acquisition device to output the next image frame to be rendered, increasing the frequency at which the image acquisition device outputs the image frame to be rendered. This allows the encoder to perform video encoding in a timely manner, increasing the frame rate of the encoded image. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an example diagram of a video processing method in the prior art;

[0025] Figure 2 A schematic diagram illustrating a video processing method provided in an embodiment of this disclosure;

[0026] Figure 3 A flowchart illustrating a video processing method provided in an embodiment of this disclosure;

[0027] Figure 4 A flowchart illustrating a video processing method provided in another embodiment of this disclosure;

[0028] Figure 5 A flowchart illustrating a video processing method provided in another embodiment of this disclosure;

[0029] Figure 6 This is a structural block diagram of a video processing device provided in one embodiment of the present disclosure;

[0030] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] When users are shooting videos using terminal devices, they may add various special effects, such as beauty filters, augmented reality (AR) effects, 3D effects, etc., to render the original video and help users publish more exquisite works, thus improving the user experience. Terminal devices include, but are not limited to, mobile phones, tablets, cameras, camcorders, wearable devices, etc.

[0033] In existing technologies, the preview of image acquisition devices on terminal equipment is based on a single thread, such as... Figure 1 As shown, after the image acquisition device outputs an image frame, it needs to pass through the InputBin image processing node to process and encapsulate the image output by the image acquisition device into a Pipeline structure, which is an encapsulation structure for the rendered image and frame. Then, the SourceBin image processing node performs operations such as rotation and cropping on the Pipeline. After passing through the EffectBin special effects rendering node, effect processing such as beauty and filter effects is performed, and the rendering result is output to the EncodeBin encoding node and the PostProcessBin special effects rendering node. The EncodeBin encoding node can encode the rendering result of the effect processing into video. The PostProcessBin special effects rendering node can perform Amazing effects processing on the rendering result of the effect processing, such as AR and 3D effects. Finally, the RenderBin screen drawing node draws the rendered image frame onto the screen for display. After the screen drawing is completed, the image acquisition device continues to output the next image frame for the next effect rendering and screen drawing. The above processing chain all occurs within a single thread. The EncodeBin encoding node must also wait for the entire processing chain of the current image frame to be completed before it can process the next image frame.

[0034] In existing technologies, if only the video rendered by the EffectBin special effects rendering node is desired, the time consumed by the PostProcessBin special effects rendering node leads to a decrease in the frame rate of the encoded and recorded image. For example, if the time from the InputBin image processing node to the end of the EffectBin special effects rendering node's effect processing is 25ms, while the PostProcessBin effect rendering node takes 20ms and the RenderBin screen drawing node takes 5ms, then the entire processing chain takes 50ms. In this case, the preview frame rate is only 20fps, and correspondingly, the frame rate of the encoded and recorded image is also only 20fps, resulting in a decrease in the frame rate of the encoded and recorded image.

[0035] To address the aforementioned technical problems, this disclosure implements, for example... Figure 2 As shown, two threads are used for special effects rendering. The first thread performs special effects rendering at the EffectBin special effects rendering node, while the second thread performs special effects rendering at the PostProcessBin special effects rendering node and image frame drawing at the RenderBin screen drawing node. After the first thread finishes rendering, it requests the image acquisition device to output the next image frame to be rendered. At the same time, the third thread performs video encoding through the EncodeBin encoding node, as well as special effects rendering and screen image frame drawing of the second thread. It does not need to wait for the second thread to finish rendering and the screen image frame to finish drawing before requesting the image acquisition device to output the next image frame to be rendered. This can increase the frequency of the image acquisition device outputting the image frames to be rendered, thereby enabling the encoder to perform video encoding in a timely manner and increasing the frame rate of the encoded and recorded image.

[0036] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0037] refer to Figure 3 , Figure 3 This is a schematic flowchart of a video processing method provided in an embodiment of this disclosure. The method of this embodiment can be applied to a terminal device or a server, and the video processing method includes:

[0038] S201, Receive the image frame to be rendered output by the image acquisition device.

[0039] In this embodiment, the image acquisition device can output image frames to be rendered, i.e., camera frames. Assuming the image acquisition device can output these frames at a frame rate of 30fps, it typically requests the next frame approximately every 33ms. In this embodiment, the image acquisition device only outputs the next image frame upon receiving a request to do so. Therefore, increasing the frequency at which the image acquisition device receives requests to output the next image frame will increase the frequency at which it outputs the next image frame, thus achieving or approaching the typical output frame rate of the image acquisition device.

[0040] S202. The first rendering image frame is obtained by performing a first special effect rendering operation on the image frame to be rendered through the first thread.

[0041] In this embodiment, since beauty effects and / or filter effects have relatively short rendering times, while AR effects and / or 3D effects have relatively long rendering times, effects with shorter rendering times, such as beauty effects and / or filter effects, can be designated as the first effect, and effects with longer rendering times, such as AR effects and / or 3D effects, can be designated as the second effect. In another embodiment, the first effect can be an effect required for video recording, while the second effect can be an effect not required for video recording.

[0042] In this embodiment, the first effect can be rendered in the first thread, while the second effect can be rendered in the second thread.

[0043] In the first thread, the rendering operation of the first special effect of the image frame to be rendered can be performed to obtain the first rendered image frame. Specifically, the processing of the InputBin image processing node, SourceBin image processing node, and EffectBin special effect rendering node in the existing technology can be executed in the first thread, such as... Figure 4 As shown, the process can be described as follows:

[0044] S2021. Through the first thread, perform a first image processing and encapsulation operation on the image frame to be rendered to obtain a rendering pipeline encapsulation structure, and perform a second image processing operation on the rendering pipeline encapsulation structure.

[0045] S2022. Perform the first special effect rendering operation on the result of the second image processing operation to obtain the first rendered image frame.

[0046] In this embodiment, in the first thread, the image output by the image acquisition device can be processed and encapsulated into a Pipeline structure, which is an encapsulation structure for the rendered image and frame; then, the Pipeline structure is subjected to second image processing operations such as rotation and cropping; then, the rendering operation of the first special effect is performed to obtain the first rendered image frame.

[0047] S203. Output the first rendered image frame to the second thread and the third thread, and request the image acquisition device to output the next image frame to be rendered.

[0048] In this embodiment, after obtaining the first rendered image frame in the first thread, the first rendered image frame can be output to the second and third threads, and the image acquisition device can be requested to output the next image frame to be rendered. That is, after the first thread completes the rendering of the first special effect, the image acquisition device can output the next image frame to be rendered to the first thread and re-execute S201. At the same time, the second and third threads can continue to perform subsequent operations S204 and S205 simultaneously, without affecting the frame output of the image acquisition device or the video encoding of the third thread, ensuring that the encoder performs video encoding in a timely manner and improving the video frame rate of the encoded recording.

[0049] S204. The first rendered image frame is video encoded by the third thread.

[0050] In this embodiment, the third thread is a video encoding thread. The first thread outputs the first rendered image frame to the third thread, and the third thread can encode the first rendered image frame into video using the EncodeBin encoding node.

[0051] S205. The second thread performs a second special effect rendering operation on the first rendered image frame to obtain a second rendered image frame, and performs video based on the second rendered image frame.

[0052] In this embodiment, the first thread outputs the first rendered image frame to the second thread. The second thread can perform a second special effect rendering operation on the first rendered image frame, including but not limited to AR, 3D special effects, etc. After the second special effect rendering operation is completed, the rendered second rendered image frame can be drawn onto the screen for display, thereby realizing the preview of the image frame.

[0053] The video processing method provided in this embodiment receives an image frame to be rendered from an image acquisition device; performs a first effect rendering operation on the image frame to be rendered using a first thread to obtain a first rendered image frame; outputs the first rendered image frame to a second thread and a third thread, and requests the image acquisition device to output the next image frame to be rendered; performs video encoding on the first rendered image frame using the third thread; performs a second effect rendering operation on the first rendered image frame using the second thread to obtain a second rendered image frame, and displays the video based on the second rendered image frame. This embodiment performs effect rendering in two threads. After the first thread finishes rendering, it requests the image acquisition device to output the next image frame to be rendered while simultaneously performing effect rendering and screen image frame drawing in the second thread, and video encoding in the third thread. This eliminates the need to wait for the second thread to finish rendering and the screen image frame to finish drawing before requesting the image acquisition device to output the next image frame to be rendered, increasing the frequency at which the image acquisition device outputs image frames to be rendered. This allows the encoder to perform video encoding in a timely manner, increasing the frame rate of the encoded video.

[0054] Based on any of the above embodiments, such as Figure 5 As shown, the rendering operation of the second effect on the first rendered image frame through the second thread in step S205 may specifically include:

[0055] S301. Add the first rendered image frame to the execution queue of the second thread for caching;

[0056] S302, The second thread sequentially performs the second effect rendering operation on the first rendered image frame cached in the execution queue.

[0057] In this embodiment, since the rendering time of the first special effects, such as beauty effects and / or filter effects, is relatively short, while the rendering time of the second special effects, such as AR effects and / or 3D effects, is relatively long, it is possible that while the first thread has completed the rendering of the first special effect to obtain a new first rendered image frame, the second thread has not yet completed the rendering of the second special effect or the drawing of the screen image frame for the old first rendered image frame. Therefore, in this embodiment, the second thread can adopt an execution queue approach, in which the first rendered image frame can be added to the execution queue of the second thread. The second thread can sequentially perform the second special effect rendering operation on the first rendered image frames cached in the execution queue, sequentially obtain the corresponding second rendered image frames, and sequentially display the video based on each second rendered image frame to achieve video preview.

[0058] Optionally, if the execution queue of the second thread is empty, the second effect rendering operation is directly performed on the first rendered image frame through the second thread. That is, if the execution queue of the second thread is empty, it means that the second thread has completed the second effect rendering operation and screen image frame drawing of the previous first rendered image frame. At this time, it is in an idle state and can directly perform the second effect rendering operation on the current first rendered image frame to obtain the current second rendered image frame. Then, based on the current second rendered image frame, the screen image frame is drawn to complete the video preview of the current frame.

[0059] If the execution queue of the second thread is not empty, the first rendered image frame is added to the execution queue, and the second thread sequentially performs the second effect rendering operation on the first rendered image frames cached in the execution queue. That is, if the execution queue of the second thread is empty, it means that the second thread is currently executing the second effect rendering operation of the previous first rendered image frame and drawing the screen image frame. At this time, it is in a busy state, and the current first rendered image frame can be cached in the execution queue of the second thread. After the other first rendered image frames before it have completed the second effect rendering operation, the second effect rendering operation of the current first rendered image frame can continue.

[0060] Based on any of the above embodiments, in order to avoid delays in the video preview, the number of first rendered image frames cached in the execution queue of the second thread should not be too large, for example, it can not exceed a preset number.

[0061] Furthermore, if the number of first rendered image frames cached in the execution queue of the second thread exceeds a preset number, the output of the first rendered image frames to the second thread can be paused. Optionally, the first thread can control the delay in requesting the image acquisition device to output the next image frame to be rendered, thereby delaying the output of the first rendered image frame. However, this also results in the delay in outputting the first rendered image frame to the video encoding of the third thread, which reduces the video frame rate of the encoded video to some extent compared to before the delay. Once the number of first rendered image frames cached in the execution queue of the second thread does not exceed the preset number, the request to the image acquisition device to output the next image frame to be rendered is resumed. Alternatively, other methods can be used to control the pause in outputting the first rendered image frame to the second thread; these will not be listed here.

[0062] Corresponding to the video processing method in the above embodiments, Figure 6 This is a structural block diagram of a video processing apparatus provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 6 The video processing device 60 includes: a receiving unit 601, a first processing unit 602, an encoding unit 603, and a second processing unit 604.

[0063] The receiving unit 601 is used to receive the image frame to be rendered output by the image acquisition device;

[0064] The first processing unit 602 is configured to perform a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame; output the first rendered image frame to the second thread and the third thread, and request the image acquisition device to output the next image frame to be rendered;

[0065] Encoding unit 603 is used to perform video encoding on the first rendered image frame through the third thread;

[0066] The second processing unit 604 is used to perform a second special effect rendering operation on the first rendered image frame through the second thread to obtain a second rendered image frame, and to display video based on the second rendered image frame.

[0067] According to one or more embodiments of this disclosure, when the first processing unit 602 performs a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame, it is configured to:

[0068] The first thread performs a first image processing and encapsulation operation on the image frame to be rendered to obtain a rendering pipeline encapsulation structure, and then performs a second image processing operation on the rendering pipeline encapsulation structure.

[0069] The result of the second image processing operation is used to perform the first special effect rendering operation to obtain the first rendered image frame.

[0070] According to one or more embodiments of this disclosure, when the second processing unit 604 performs a second effect rendering operation on the first rendered image frame through the second thread, it is configured to:

[0071] The first rendered image frame is added to the execution queue of the second thread for caching;

[0072] The second thread sequentially renders the first rendered image frame cached in the execution queue with the second effect.

[0073] According to one or more embodiments of this disclosure, when the second processing unit 604 performs a second effect rendering operation on the first rendered image frame through the second thread, it is configured to:

[0074] If the execution queue of the second thread is empty, then the second thread directly performs the rendering operation of the second effect on the first rendered image frame; or

[0075] If the execution queue of the second thread is not empty, the first rendered image frame is added to the execution queue, and the second thread sequentially performs the second effect rendering operation on the first rendered image frame cached in the execution queue.

[0076] According to one or more embodiments of this disclosure, the first processing unit 602 is further configured to:

[0077] If the number of first rendered image frames cached in the execution queue of the second thread exceeds a preset number, then control pauses the output of the first rendered image frames to the second thread.

[0078] According to one or more embodiments of this disclosure, when the second processing unit 604 performs video display based on the second rendered image frame, it is configured to:

[0079] The second rendered image frame is drawn onto the screen for display.

[0080] According to one or more embodiments of this disclosure, the rendering operation of the second special effect takes longer than the rendering operation of the first special effect.

[0081] According to one or more embodiments of this disclosure, the first special effect includes beauty effects and / or filter effects; the second special effect includes augmented reality effects and / or three-dimensional effects.

[0082] The video processing device provided in this embodiment can be used to execute the technical solutions of the above-described video processing method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0083] The video processing device provided in this embodiment receives an image frame to be rendered from an image acquisition device; performs a first effect rendering operation on the image frame to be rendered using a first thread to obtain a first rendered image frame; outputs the first rendered image frame to a second thread and a third thread, and requests the image acquisition device to output the next image frame to be rendered; performs video encoding on the first rendered image frame using the third thread; performs a second effect rendering operation on the first rendered image frame using the second thread to obtain a second rendered image frame, and displays the video based on the second rendered image frame. This embodiment performs effect rendering in two threads. After the first thread finishes rendering, it requests the image acquisition device to output the next image frame to be rendered while simultaneously performing effect rendering and screen image frame drawing in the second thread, and video encoding in the third thread. This eliminates the need to wait for the second thread to finish rendering and the screen image frame to finish drawing before requesting the image acquisition device to output the next image frame to be rendered, increasing the frequency at which the image acquisition device outputs the image frame to be rendered. This allows the encoder to perform video encoding in a timely manner, increasing the frame rate of the encoded video.

[0084] refer to Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0085] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0086] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0087] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0088] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0089] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0090] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0091] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0094] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0095] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0096] In a first aspect, according to one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0097] Receive the image frame to be rendered output by the image acquisition device;

[0098] The first rendering image frame is obtained by performing a first special effect rendering operation on the image frame to be rendered through the first thread.

[0099] The first rendered image frame is output to the second and third threads, and the image acquisition device is requested to output the next image frame to be rendered.

[0100] The first rendered image frame is video encoded using the third thread;

[0101] The second thread performs a second effect rendering operation on the first rendered image frame to obtain a second rendered image frame, and then displays the video based on the second rendered image frame.

[0102] According to one or more embodiments of this disclosure, the step of performing a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame includes:

[0103] The first thread performs a first image processing and encapsulation operation on the image frame to be rendered to obtain a rendering pipeline encapsulation structure, and then performs a second image processing operation on the rendering pipeline encapsulation structure.

[0104] The result of the second image processing operation is used to perform the first special effect rendering operation to obtain the first rendered image frame.

[0105] According to one or more embodiments of this disclosure, the rendering operation of the second effect on the first rendered image frame via the second thread includes:

[0106] The first rendered image frame is added to the execution queue of the second thread for caching;

[0107] The second thread sequentially renders the first rendered image frame cached in the execution queue with the second effect.

[0108] According to one or more embodiments of this disclosure, the rendering operation of the second effect on the first rendered image frame via the second thread includes:

[0109] If the execution queue of the second thread is empty, then the second thread directly performs the rendering operation of the second effect on the first rendered image frame; or

[0110] If the execution queue of the second thread is not empty, the first rendered image frame is added to the execution queue, and the second thread sequentially performs the second effect rendering operation on the first rendered image frame cached in the execution queue.

[0111] According to one or more embodiments of this disclosure, the method further includes:

[0112] If the number of first rendered image frames cached in the execution queue of the second thread exceeds a preset number, then control pauses the output of the first rendered image frames to the second thread.

[0113] According to one or more embodiments of this disclosure, the video display based on the second rendered image frame includes:

[0114] The second rendered image frame is drawn onto the screen for display.

[0115] According to one or more embodiments of this disclosure, the rendering operation of the second special effect takes longer than the rendering operation of the first special effect.

[0116] According to one or more embodiments of this disclosure, the first special effect includes beauty effects and / or filter effects;

[0117] The second effect includes augmented reality effects and / or 3D effects.

[0118] Secondly, according to one or more embodiments of this disclosure, a video processing apparatus is provided, comprising:

[0119] The receiving unit is used to receive the image frames to be rendered output by the image acquisition device;

[0120] The first processing unit is configured to perform a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame; output the first rendered image frame to the second thread and the third thread, and request the image acquisition device to output the next image frame to be rendered;

[0121] An encoding unit is used to perform video encoding on the first rendered image frame through the third thread;

[0122] The second processing unit is used to perform a second special effect rendering operation on the first rendered image frame through the second thread to obtain a second rendered image frame, and to display video based on the second rendered image frame.

[0123] According to one or more embodiments of this disclosure, when the first processing unit performs a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame, it is configured to:

[0124] The first thread performs a first image processing and encapsulation operation on the image frame to be rendered to obtain a rendering pipeline encapsulation structure, and then performs a second image processing operation on the rendering pipeline encapsulation structure.

[0125] The result of the second image processing operation is used to perform the first special effect rendering operation to obtain the first rendered image frame.

[0126] According to one or more embodiments of this disclosure, when the second processing unit performs a second effect rendering operation on the first rendered image frame through the second thread, it is configured to:

[0127] The first rendered image frame is added to the execution queue of the second thread for caching;

[0128] The second thread sequentially renders the first rendered image frame cached in the execution queue with the second effect.

[0129] According to one or more embodiments of this disclosure, when the second processing unit performs a second effect rendering operation on the first rendered image frame through the second thread, it is configured to:

[0130] If the execution queue of the second thread is empty, then the second thread directly performs the rendering operation of the second effect on the first rendered image frame; or

[0131] If the execution queue of the second thread is not empty, the first rendered image frame is added to the execution queue, and the second thread sequentially performs the second effect rendering operation on the first rendered image frame cached in the execution queue.

[0132] According to one or more embodiments of this disclosure, the first processing unit is further configured to:

[0133] If the number of first rendered image frames cached in the execution queue of the second thread exceeds a preset number, then control pauses the output of the first rendered image frames to the second thread.

[0134] According to one or more embodiments of this disclosure, when the second processing unit performs video display based on the second rendered image frame, it is configured to:

[0135] The second rendered image frame is drawn onto the screen for display.

[0136] According to one or more embodiments of this disclosure, the rendering operation of the second special effect takes longer than the rendering operation of the first special effect.

[0137] According to one or more embodiments of this disclosure, the first special effect includes beauty effects and / or filter effects; the second special effect includes augmented reality effects and / or three-dimensional effects.

[0138] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0139] The memory stores computer-executed instructions;

[0140] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the video processing method as described in the first aspect and various possible designs of the first aspect.

[0141] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0142] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the video processing method as described in the first aspect and various possible designs of the first aspect.

[0143] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0144] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0145] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, characterized in that, include: Receive the image frame to be rendered output by the image acquisition device; The first rendering image frame is obtained by performing a first special effect rendering operation on the image frame to be rendered through the first thread. The first rendered image frame is output to the second and third threads, and the image acquisition device is requested to output the next image frame to be rendered. The first rendered image frame is video encoded using the third thread; The second thread performs a second effect rendering operation on the first rendered image frame to obtain a second rendered image frame, and displays the video based on the second rendered image frame; The rendering time for the second special effect is greater than the rendering time for the first special effect; the first special effect includes beauty effects and / or filter effects; the second special effect includes augmented reality effects and / or 3D effects.

2. The method according to claim 1, characterized in that, The step of performing a first effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame includes: The first thread performs a first image processing and encapsulation operation on the image frame to be rendered to obtain a rendering pipeline encapsulation structure, and then performs a second image processing operation on the rendering pipeline encapsulation structure. The result of the second image processing operation is used to perform the first special effect rendering operation to obtain the first rendered image frame.

3. The method according to claim 1, characterized in that, The rendering operation of the second effect on the first rendered image frame through the second thread includes: The first rendered image frame is added to the execution queue of the second thread for caching; The second thread sequentially renders the first rendered image frame cached in the execution queue with the second effect.

4. The method according to claim 3, characterized in that, The rendering operation of the second effect on the first rendered image frame through the second thread includes: If the execution queue of the second thread is empty, then the second thread directly performs the rendering operation of the second effect on the first rendered image frame; or If the execution queue of the second thread is not empty, the first rendered image frame is added to the execution queue, and the second thread sequentially performs the second effect rendering operation on the first rendered image frame cached in the execution queue.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the number of first rendered image frames cached in the execution queue of the second thread exceeds a preset number, then control pauses the output of the first rendered image frames to the second thread.

6. The method according to any one of claims 1-4, characterized in that, The video display based on the second rendered image frame includes: The second rendered image frame is drawn onto the screen for display.

7. A video processing device, characterized in that, include: The receiving unit is used to receive the image frames to be rendered output by the image acquisition device; The first processing unit is used to perform a first special effect rendering operation on the image frame to be rendered through a first thread to obtain a first rendered image frame. The first rendered image frame is output to the second and third threads, and the image acquisition device is requested to output the next image frame to be rendered. An encoding unit is used to perform video encoding on the first rendered image frame through the third thread; The second processing unit is used to perform a second special effect rendering operation on the first rendered image frame through the second thread to obtain a second rendered image frame, and to display video based on the second rendered image frame; The rendering time for the second special effect is greater than the rendering time for the first special effect; the first special effect includes beauty effects and / or filter effects; the second special effect includes augmented reality effects and / or 3D effects.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the method described in any one of claims 1-6.

Images