Video processing method and video processing system

Abstract

The invention provides a video processing method and a video processing system, and the method comprises the steps: carrying out a video processing step through employing a GPU, and the step comprises the steps: carrying out the image preprocessing of a video image of a to-be-inputted algorithm model; reasoning the video image after image preprocessing by using an algorithm model, wherein the viewpoint of the video image after reasoning is different from the viewpoint of the video image to be input into the algorithm model; image post-processing is carried out on the video image after reasoning, the image post-processing comprises image splicing of the video image after reasoning and the video image of the algorithm model to be input which belong to the same frame, and a multi-view image is obtained. In the video processing process, the GPU is used for executing algorithm model reasoning on the video images, the GPU is also used for executing image preprocessing and image post-processing on the video images, so that the performance pressure of the CPU is reduced, meanwhile, frequent copying of data between the CPU and the GPU is reduced, the video processing time is shortened, and efficient and real-time video processing is achieved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method and a video processing system. Background Technology

[0002] Currently, there are several technical bottlenecks in the processing of naked-eye 4K video, mainly concentrated in the following aspects:

[0003] 1. Performance bottlenecks of the Central Processing Unit (CPU) and Graphics Processing Unit (GPU): In the video processing workflow of related technologies, the CPU needs to participate in multiple stages such as image preprocessing and image postprocessing. However, the high resolution and large data volume of 4K video put enormous performance pressure on the CPU during processing, often resulting in the processing speed not being able to keep up with the computing speed of the GPU.

[0004] 2. Data Copying Issues: In the video processing workflow of related technologies, image preprocessing and post-processing are performed on the CPU using the OpenCV library, and then algorithm model inference is performed on the GPU. Data needs to be copied frequently between the CPU and GPU, which not only increases video processing time but also increases system power consumption and complexity. Especially at 4K resolution, the large amount of image data copying becomes a bottleneck for real-time processing. Summary of the Invention

[0005] This invention provides a video processing method and system to address performance bottlenecks and data copying issues in the video processing workflow of related technologies.

[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0007] In a first aspect, embodiments of the present invention provide a video processing method, including:

[0008] The video processing steps are performed using a GPU, and the video processing steps include:

[0009] Perform image preprocessing on the video images to be used as input algorithm models;

[0010] The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model.

[0011] The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

[0012] Optionally, the image preprocessing includes at least one of the following: image scaling, pixel value normalization, channel order conversion, and data type conversion;

[0013] The image post-processing also includes at least one of the following: image scaling, pixel value denormalization, channel order conversion, and data type conversion.

[0014] Optionally, the algorithm model includes a depth estimation model and a viewpoint generation model;

[0015] The image preprocessing of the video image to be input to the algorithm model includes: performing a first image preprocessing on the video image to be input to the depth estimation model to obtain a preprocessed first video image;

[0016] The step of using an algorithm model to infer the video image after image preprocessing to obtain the inferred video image includes: using a depth estimation model to infer the first video image to obtain a depth estimation image;

[0017] The image preprocessing of the video image to be input to the algorithm model further includes: performing a second image preprocessing on the depth estimation image to be input to the viewpoint generation module to obtain a preprocessed second video image;

[0018] The step of using an algorithm model to infer the preprocessed video image to obtain the inferred video image further includes: using a viewpoint generation model to infer the second video image to obtain the inferred video image.

[0019] Optionally, the video processing steps further include:

[0020] The first video stream is decoded to obtain a decoded video image, which is used as the video image to be input into the algorithm model.

[0021] Optionally, the step of decoding the first video stream to obtain the decoded video image further includes:

[0022] The CPU is used to split the video file into a first video stream and an audio stream, and the first video stream is stored in a first queue.

[0023] Copy the first video stream from the first queue to the GPU.

[0024] Optionally, the video processing steps further include:

[0025] The multi-view image is encoded to obtain a second video stream.

[0026] Optionally, after encoding the multi-view images to obtain a second video stream, the process further includes:

[0027] Copy the second video stream to the CPU;

[0028] The CPU is used to synchronize and merge the second video stream with the audio stream to obtain a video file.

[0029] Optionally, the video processing steps further include:

[0030] The multi-view image is rendered.

[0031] Optionally, the step of using a GPU to perform video processing includes:

[0032] The video images of the algorithm model to be input are distributed to multiple GPUs, and the video processing steps are performed using multiple GPUs.

[0033] Optionally, distributing the video images of the input algorithm model to multiple GPUs includes:

[0034] Save the video image of the algorithm model to be input to the second queue;

[0035] The video images of the algorithm model to be input, stored in the second queue, are distributed to the multiple GPUs.

[0036] Optionally, the video processing steps further include:

[0037] The multi-viewpoint images are saved to the third queue;

[0038] The multi-view image is obtained from the third queue and rendered.

[0039] Optionally, the step of obtaining the multi-view image from the third queue for rendering includes:

[0040] The multi-view image is rendered using a designated GPU from among the plurality of GPUs.

[0041] Secondly, embodiments of the present invention provide a video processing system, including:

[0042] An execution module is configured to perform video processing steps using a GPU, the video processing steps including:

[0043] Perform image preprocessing on the video images to be used as input algorithm models;

[0044] The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model.

[0045] The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

[0046] Thirdly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the video processing method as described in the first aspect above.

[0047] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video processing method described in the first aspect above.

[0048] Fifthly, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the video processing method as described in the first aspect.

[0049] In this embodiment of the invention, during video processing, in addition to using the GPU to perform algorithm model inference on video images, the GPU is also used to perform image preprocessing and image postprocessing of video images to reduce the performance pressure on the CPU. At the same time, it also reduces the frequent copying of data between the CPU and the GPU, thereby reducing video processing time and achieving efficient and real-time video processing. Attached Figure Description

[0050] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0051] Figure 1 This is a schematic flowchart of a video processing method according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic flowchart of a video processing method according to another embodiment of the present invention;

[0053] Figure 3 This is a comparative diagram of the video processing method of this invention and video processing methods in related technologies;

[0054] Figure 4 This is a flowchart illustrating a video processing method according to another embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure of the video processing device according to an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The technical content of this invention will be briefly described below.

[0059] Depth estimation model: capable of estimating the depth information of the content in a given image.

[0060] Viewpoint generation model: It can generate an image that should be seen at a certain depth based on an image and its depth information. Generally, different viewing angles of the generated image can be achieved by adjusting the value of the depth information.

[0061] Viewpoint: The relative position of the observer with respect to the observed object. In photography, it generally refers to the relative position of the camera and the subject; more strictly speaking, it is the relative position of the front nodal point of the camera lens and the plane of the subject.

[0062] CUDA is a parallel computing platform and programming model. It significantly improves computational performance by utilizing the processing power of graphics processing units (GPUs). Its operations operate on data content within NVIDIA GPUs. In this invention, CUDA operators can be used to perform operations such as image preprocessing and image post-processing.

[0063] CUDA kernel functions are a crucial part of performing parallel computing tasks on GPUs. By writing and optimizing kernel functions, the powerful computing capabilities of GPUs can be fully utilized to accelerate various tasks such as scientific computing, image processing, and deep learning.

[0064] SDL (Simple DirectMedia Layer) is a cross-platform multimedia development kit that provides 2D, audio, and event-driven capabilities. In this invention, SDL can be used as a tool for playing audio data.

[0065] NVIDIA Media SDK: This is a development tool provided by NVIDIA for decoding video streams and encoding images into video streams. It can be used to decode and encode video, and it uses NVIDIA's GPU graphics cards, which not only have high processing efficiency but also reduce CPU usage.

[0066] Tensorrt: Tensorrt is a high-performance deep learning inference engine provided by NVIDIA, designed for deployment in production environments. It aims to improve the inference speed and efficiency of deep learning models running on NVIDIA GPUs, primarily to accelerate the running speed of AI algorithm models on NVIDIA GPUs.

[0067] FFmpeg: An open-source audio and video processing library. In this invention, FFmpeg can be used to decode the audio and video streams of a video file. The audio stream can be played using SDL, and the video stream can be decoded using NVIDIA video decoding.

[0068] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a video processing method according to an embodiment of the present invention. The video processing method includes:

[0069] Step 11: Perform video processing steps using the GPU, the video processing steps including:

[0070] Perform image preprocessing on the video images to be used as input algorithm models;

[0071] The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model.

[0072] The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

[0073] The multi-viewpoint image in this embodiment of the invention can be a dual-viewpoint image with two viewpoints or a multi-viewpoint image with more than two viewpoints. The dual-viewpoint image can also be referred to as a dual-viewpoint image with left and right eye perspectives.

[0074] In this embodiment of the invention, during video processing, in addition to using the GPU to perform algorithm model inference on video images, the GPU is also used to perform image preprocessing and image postprocessing of video images to reduce the performance pressure on the CPU. At the same time, it also reduces the frequent copying of data between the CPU and the GPU, thereby reducing video processing time and achieving efficient and real-time video processing.

[0075] In some embodiments, optionally, the image preprocessing includes at least one of the following: image scaling, pixel value normalization, channel order conversion, and data type conversion.

[0076] In some embodiments, optionally, the image post-processing includes, in addition to image stitching of the inferred video image and the video image of the input algorithm model, at least one of the following: image scaling, pixel value denormalization, channel order conversion, and data type conversion.

[0077] Image scaling refers to the process of changing the size of an image by adding or removing pixels.

[0078] Pixel value normalization refers to converting the pixel values ​​of an image to a specific range, such as [0,1] or [0,255], in order to facilitate the comparison and analysis of pixel values ​​between different images.

[0079] Channel order conversion refers to changing the order of color channels in an image.

[0080] Data type conversion refers to converting an image from one data format to another. For example, converting a YUV format image to an RGB format image.

[0081] The algorithm model in this embodiment of the invention can also be called an artificial intelligence (AI) model.

[0082] In some embodiments, optionally, the algorithm model includes a depth estimation model and a viewpoint generation model; the image preprocessing includes a first image preprocessing of the video image to be input to the depth estimation model and a second image preprocessing of the video image to be input to the viewpoint generation model; the inference includes: inference using the depth estimation model and inference using the viewpoint generation model.

[0083] The image preprocessing of the video image to be input to the algorithm model includes: performing a first image preprocessing on the video image to be input to the depth estimation model to obtain a preprocessed first video image;

[0084] The step of using an algorithm model to infer the video image after image preprocessing to obtain the inferred video image includes: using a depth estimation model to infer the first video image to obtain a depth estimation image;

[0085] The image preprocessing of the video image to be input to the algorithm model further includes: performing a second image preprocessing on the depth estimation image to be input to the viewpoint generation module to obtain a preprocessed second video image;

[0086] The step of using an algorithm model to infer the preprocessed video image to obtain the inferred video image further includes: using a viewpoint generation model to infer the second video image to obtain the inferred video image.

[0087] It should be noted that the first image preprocessing and the second image preprocessing may be the same or different. The content of the first image preprocessing is determined according to the requirements of the depth estimation model for the image, while the content of the second image preprocessing is determined according to the requirements of the viewpoint generation model for the image.

[0088] In this embodiment of the invention, optionally, a depth estimation model is used to infer the first video image to obtain a depth estimation image. After obtaining the depth estimation image, the depth information of the depth estimation image can be adjusted as needed to adjust the out-of-screen and in-screen effects.

[0089] In this embodiment of the invention, image preprocessing and image postprocessing operations can be performed by writing CUDA operators (also known as CUDA kernel function code).

[0090] In this embodiment of the invention, the preprocessed video image can be sent to the algorithm model in the GPU via the Tensorrt interface for further processing, avoiding frequent data copying between the CPU and GPU and improving overall processing efficiency. In contrast, related technologies require copying the preprocessed video image from the local OpenCV server to the GPU for algorithm model inference, which is a performance bottleneck. This embodiment of the invention completely avoids this issue.

[0091] In this embodiment of the invention, the algorithm model runs on the GPU to obtain the processed video image. The processed video image is also stored on the GPU for subsequent image post-processing. However, in related technologies, the video image processed by the GPU needs to be copied to the CPU for image post-processing, which is also a performance bottleneck. This embodiment of the invention can avoid this problem.

[0092] In some embodiments, optionally, the video processing step further includes: decoding the first video stream to obtain a decoded video image, which is then used as the video image to be input into the algorithm model. That is, in these embodiments of the invention, a GPU can be used to decode the video stream, further reducing the performance burden on the CPU.

[0093] In this embodiment of the invention, the NVIDIA Media SDK can be used to decode the video stream in the GPU, generate video images, and store them on the GPU, avoiding data copying between the GPU and the CPU.

[0094] In this embodiment of the invention, the decoded video image can be an nv12 format video image. nv12 is a video encoding format, belonging to a sampling method of the YUV color space. Generally, the input image of the algorithm model is required to be an RGB format image. Therefore, in this embodiment of the invention, before performing image preprocessing on the decoded video image, a format conversion of the decoded video image is also included.

[0095] In some embodiments, optionally, the step of decoding the first video stream to obtain the decoded video image may further include: splitting the video file using a CPU to obtain the first video stream and the audio stream.

[0096] In this embodiment of the invention, ffmpeg can be used to split the video file. This process can be performed using the CPU, but the CPU consumption is extremely low, and the impact on system performance is minimal.

[0097] In some embodiments, alternatively, please refer to Figure 4 The CPU is used to split the video file, resulting in a first video stream and an audio stream, followed by:

[0098] Store the first video stream in the first queue;

[0099] Copy the first video stream from the first queue to the GPU.

[0100] In this embodiment of the invention, the split video streams are stored in a first queue to ensure the real-time nature and continuity of the video streams, guaranteeing that subsequent processing can proceed sequentially. Furthermore, copying the video streams directly from the first queue to the GPU memory avoids subsequent data copying between the CPU and GPU, reducing memory access latency.

[0101] In this embodiment of the invention, in order to synchronize audio and video during the final rendering, it is also necessary to record the actual playback time of each frame in the first video stream.

[0102] In some embodiments, optionally, the video processing step further includes: encoding the multi-view image to obtain a second video stream. That is, in this embodiment of the invention, a GPU can be used to encode the multi-view image, reducing the reliance on the CPU during encoding and further reducing the performance pressure on the CPU. In this embodiment of the invention, the NVIDIA Video Encoder (NVIDIA Media SDK) can be used to encode the multi-view image.

[0103] In some embodiments, optionally, the process of encoding the multi-view images to obtain a second video stream further includes:

[0104] Copy the second video stream to the CPU;

[0105] The CPU is used to synchronize and merge the second video stream with the audio stream to obtain a video file.

[0106] In this embodiment of the invention, the corresponding audio stream data can be found based on the ID of the video image in the second video stream.

[0107] In this embodiment of the invention, the ffmpeg interface can be used to synchronize and merge the encoded video stream and audio stream, generate a standard MP4 video file, and save it locally, thus achieving efficient video saving.

[0108] In some embodiments, optionally, the video processing step further includes rendering the multi-viewpoint image. That is, in this embodiment of the invention, a GPU can be used to render the viewpoint image. Because GPU rendering is used, video can be processed in real time (e.g., achieving a video processing speed of >25fps), while further reducing the performance burden on the CPU.

[0109] In this embodiment of the invention, the Direct3D interface can be used to directly drive video images for rendering.

[0110] In some cases, electronic devices have multiple GPUs, but in related technologies, video processing usually only uses one GPU, which does not fully utilize the advantages of multi-GPU parallel processing, resulting in ineffective use of hardware resources.

[0111] To address the aforementioned issue of low hardware utilization, in some embodiments, optionally, the video processing steps using GPUs include: distributing the video image of the input algorithm model to multiple GPUs, and using the multiple GPUs to execute the video processing steps. That is, multiple GPUs are used simultaneously to process the allocated video images in parallel, thereby effectively improving video processing time.

[0112] In this embodiment of the invention, the method for distributing video images to be used as input to the algorithm model can be as follows: assuming there are n CPUs, the first frame can be distributed to the first GPU, the second frame to the second GPU, ..., the nth frame to the nth GPU, and then the (n+1)th frame to the first GPU, and so on. Alternatively, frames 1-10 can be distributed to the first GPU, frames 11-20 to the second GPU, and so on. This embodiment of the invention does not limit the distribution method.

[0113] In some embodiments, alternatively, please refer to Figure 4 The step of distributing the video images of the algorithm model to be input to multiple GPUs includes:

[0114] Save the video image of the algorithm model to be input to the second queue;

[0115] The video images of the algorithm model to be input, stored in the second queue, are distributed to the multiple GPUs.

[0116] In this embodiment of the invention, the second queue can also be called the input queue, or the decoded frame queue. The decoded frame queue refers to saving the decoded video images to the second queue. By setting up the second queue, the ability to process data in parallel in a multi-GPU environment can be supported.

[0117] In some embodiments, alternatively, please refer to Figure 4 The video processing steps further include:

[0118] The multi-viewpoint images are saved to the third queue;

[0119] The multi-view image is obtained from the third queue and rendered.

[0120] In this embodiment of the invention, the third queue can also be called the output queue, which is used to respond to the problem of image result temporary storage when the multi-GPU processing results are too fast.

[0121] In some embodiments, optionally, obtaining the multi-view image from the third queue for rendering includes: rendering the multi-view image using a designated GPU among the plurality of GPUs.

[0122] The video file in this embodiment of the invention can be a 4K video file, but it is not limited to this.

[0123] Please refer to Figure 2 and Figure 3 , Figure 2 This is a schematic flowchart of a video processing method according to another embodiment of the present invention. Figure 3 This is a comparative diagram of the video processing method of this invention and video processing methods in related technologies. The video processing method of this invention includes:

[0124] Step 21: Split the video stream and audio stream.

[0125] In this embodiment of the invention, the ffmpeg tool can be used to split the video source, separating the video stream (i.e., the first video stream in the above embodiment) and the audio stream. This process mainly utilizes the CPU, but the power consumption is extremely low, and the impact on system performance is minimal.

[0126] Step 22: Store the video stream in the queue.

[0127] In this embodiment of the invention, a queue (i.e., the first queue in the above embodiment) can be designed to store the split video streams. The design of this queue takes into account the real-time nature and continuity of the video streams, ensuring that subsequent processing can proceed smoothly.

[0128] In addition, to achieve audio and video synchronization during the final rendering, this step can record the actual playback time of each frame of video image in the video stream.

[0129] Step 23: Video stream decoding and GPU memory transfer.

[0130] In this step, the video stream data stored in the queue is copied to the GPU memory.

[0131] In this embodiment of the invention, for a video image in a video stream, only the change information of that video image relative to the previous video images can be stored, thereby reducing the amount of video stream data. Since the amount of video stream data is small, this process also consumes less CPU resources.

[0132] In this embodiment of the invention, the NVIDIA Media SDK is used to decode the video stream in the GPU memory, generate a video image, and store it on the GPU, thus avoiding data copying between the CPU and the GPU.

[0133] In this embodiment of the invention, the video image obtained after decoding can be an nv12 format video image.

[0134] Step 24: Image preprocessing.

[0135] In this embodiment of the invention, CUDA kernel function code can be written to perform preprocessing operations such as image scaling, pixel value normalization, and / or channel order conversion on the decoded video image. These operations are all performed on the GPU, without needing to copy the data back to the CPU.

[0136] The image preprocessing in this embodiment of the invention includes: preprocessing the video image to be used as the input depth estimation model (also known as the first preprocessing), and preprocessing the video image to be used as the input viewpoint generation model (also known as the second preprocessing).

[0137] It should be noted that since the decoded video image can be in nv12 format, while depth estimation models and viewpoint generation models typically require the input image to be in RGB format, the nv12 format video image can be converted to RGB format before the first preprocessing step. Figure 2 Frame preprocessing in the process.

[0138] Step 25: Algorithm model processing.

[0139] In this embodiment of the invention, the preprocessed video image is directly fed into the algorithm model on the GPU via the TensorRT interface for further processing. In related technologies, this requires copying the preprocessed image from the local OpenCV repository to the GPU, which is a performance bottleneck. This embodiment of the invention completely avoids this issue.

[0140] In this embodiment of the invention, the algorithm model runs on the GPU to obtain the processed video image, which is also stored on the GPU for subsequent post-processing. In related technologies, the processed image data needs to be copied to the CPU for image post-processing, which is also a performance bottleneck. This embodiment of the invention can avoid this problem.

[0141] In this embodiment of the invention, after obtaining the depth estimation image through the depth estimation model, the depth information of the depth estimation image can be adjusted to adjust the out-of-screen and in-screen effects.

[0142] Step 26: Image post-processing.

[0143] In this embodiment of the invention, CUDA kernel function code can be written to perform operations such as pixel inverse normalization, channel order conversion, and data type conversion on the video image (single viewpoint image) processed by the algorithm model, and then stitch it with the original video image (RGB image before image preprocessing) to generate a dual viewpoint video image. These post-processing steps are also completed on the GPU, which can avoid data copying between the GPU and the CPU. Due to the high bandwidth of the GPU, image copying operations in the GPU are much more efficient than copying operations in the CPU.

[0144] In this embodiment of the invention, when the number of GPUs on the local machine is greater than 1, a decoding frame queue (i.e., the second queue in the above embodiment) and a multi-view image queue (i.e., the third queue in the above embodiment) can be designed to achieve multi-GPU parallel processing.

[0145] That is, one of multiple GPUs (as shown in the attached image) Figure 2 and Figure 4 GPU0 performs video decoding, stores the decoded video images in the decoded frame queue (i.e., the second queue), and distributes the video images from the decoded frame queue to multiple GPUs (such as GPU0, GPU1, and GPU2). GPU0, GPU1, and GPU2 all perform image preprocessing, algorithm processing (inference), and image post-processing operations on the allocated video images. The processed multi-view images can be stored in the multi-view image queue (i.e., the third queue).

[0146] It should be noted that, Figure 2 and Figure 4The embodiments shown are merely illustrative. The GPUs used for decoding and image rendering can be the same GPU or different GPUs. Furthermore, the CPU used for decoding and / or rendering may not perform inference, i.e., it may not distribute data to the GPU. Various implementation methods are possible, and this invention is not limited thereto.

[0147] Step 27: Image display and video encoding.

[0148] In this embodiment of the invention, the Direct3D interface can be used to directly drive video images for rendering. In contrast, in related technologies, the image needs to be copied from the CPU to the integrated graphics device for display, which is also a performance bottleneck. This embodiment of the invention can avoid this limitation.

[0149] Furthermore, in this embodiment of the invention, the NVIDIA Video Encoder (NVIDIA Media SDK) can also be used to encode the multi-view images on the GPU to generate a video stream. The encoded video stream is then copied back to the CPU for local storage. In related technologies, the OpenCV library is required in this case, and CPU encoding is generally used, which is less efficient.

[0150] Step 28: Synchronize and save the video and audio streams.

[0151] In this embodiment of the invention, the corresponding original audio stream data can be found and synchronized and merged based on the ID of the video image in the video stream.

[0152] In this embodiment of the invention, the ffmpeg interface can be used to synchronize and merge the encoded video stream and audio stream, generate a standard MP4 video file and save it locally, achieving the effect of real-time video rendering on a macro scale and synchronous saving of the rendered video.

[0153] By implementing the embodiments of the present invention, the following beneficial effects can be achieved:

[0154] Significantly improves processing speed: Video decoding, image preprocessing, algorithm model inference, image post-processing, and encoding are all transferred to the GPU. In other words, the image data stays on the GPU throughout the entire process, starting from video decoding. There is no data copying between the CPU and the GPU, making the entire process extremely efficient and thus significantly improving processing speed.

[0155] Reduced CPU load: Since most computing tasks are moved to the GPU, the CPU load is greatly reduced, allowing other applications on electronic devices to run more smoothly.

[0156] Improved hardware utilization: This embodiment of the invention fully utilizes the advantages of multi-GPU parallel processing, which makes effective use of hardware resources and improves overall processing efficiency.

[0157] Optimized user experience: Thanks to the increased processing speed and reduced CPU load, users can enjoy a smoother and more efficient video processing experience.

[0158] The following test experiment compares the performance of the video processing method of this invention with that of related technologies.

[0159] Please refer to Table 1. In this test experiment, it can be clearly seen that compared with the video processing methods in related technologies, the video processing method of this embodiment has improved processing speed and GPU utilization, while reducing CPU utilization, which can reduce the impact on other applications on electronic devices.

[0160]

[0161]

[0162] Table 1

[0163] Please refer to Figure 5 This invention also provides a video processing system 40, comprising:

[0164] Execution module 41 is configured to perform video processing steps using a GPU, the video processing steps including:

[0165] Perform image preprocessing on the video images to be used as input algorithm models;

[0166] The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model.

[0167] The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

[0168] In this embodiment of the invention, during video processing, in addition to using the GPU to perform algorithm model inference on video images, the GPU is also used to perform image preprocessing and image postprocessing of video images to reduce the performance pressure on the CPU. At the same time, it also reduces the frequent copying of data between the CPU and the GPU, thereby reducing video processing time and achieving efficient and real-time video processing.

[0169] Optionally, the image preprocessing includes at least one of the following: image scaling, pixel value normalization, channel order conversion, and data type conversion;

[0170] The image post-processing also includes at least one of the following: image scaling, pixel value denormalization, channel order conversion, and data type conversion.

[0171] Optionally, the algorithm model includes a depth estimation model and a viewpoint generation model;

[0172] The image preprocessing of the video image to be input to the algorithm model includes: performing a first image preprocessing on the video image to be input to the depth estimation model to obtain a preprocessed first video image;

[0173] The step of using an algorithm model to infer the video image after image preprocessing to obtain the inferred video image includes: using a depth estimation model to infer the first video image to obtain a depth estimation image;

[0174] The image preprocessing of the video image to be input to the algorithm model further includes: performing a second image preprocessing on the depth estimation image to be input to the viewpoint generation module to obtain a preprocessed second video image;

[0175] The step of using an algorithm model to infer the preprocessed video image to obtain the inferred video image further includes: using a viewpoint generation model to infer the second video image to obtain the inferred video image.

[0176] Optionally, the video processing steps further include:

[0177] The first video stream is decoded to obtain a decoded video image, which is used as the video image to be input into the algorithm model.

[0178] Optionally, the step of decoding the first video stream to obtain the decoded video image further includes:

[0179] The CPU is used to split the video file into a first video stream and an audio stream, and the first video stream is stored in a first queue.

[0180] Copy the first video stream from the first queue to the GPU.

[0181] Optionally, the video processing steps further include:

[0182] The multi-view image is encoded to obtain a second video stream.

[0183] Optionally, after encoding the multi-view images to obtain a second video stream, the process further includes:

[0184] Copy the second video stream to the CPU;

[0185] The CPU is used to synchronize and merge the second video stream with the audio stream to obtain a video file.

[0186] Optionally, the video processing steps further include:

[0187] The multi-view image is rendered.

[0188] Optionally, the step of using a GPU to perform video processing includes:

[0189] The video images of the algorithm model to be input are distributed to multiple GPUs, and the video processing steps are performed using multiple GPUs.

[0190] Optionally, distributing the video images of the input algorithm model to multiple GPUs includes:

[0191] Save the video image of the algorithm model to be input to the second queue;

[0192] The video images of the algorithm model to be input, stored in the second queue, are distributed to the multiple GPUs.

[0193] Optionally, the video processing steps further include:

[0194] The multi-viewpoint images are saved to the third queue;

[0195] The multi-view image is obtained from the third queue and rendered.

[0196] Optionally, the step of obtaining the multi-view image from the third queue for rendering includes:

[0197] The multi-view image is rendered using a designated GPU from among the plurality of GPUs.

[0198] Please refer to Figure 6 The present invention also provides an electronic device 50, including a processor 51, a memory 52, and a computer program stored in the memory 52 and executable on the processor 51. When the computer program is executed by the processor 51, it implements the various processes of the above-described video processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0199] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described video processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0200] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0203] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A video processing method, characterized in that, include: The video processing steps are performed using a GPU, and the video processing steps include: Perform image preprocessing on the video images to be used as input algorithm models; The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model. The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

2. The video processing method as described in claim 1, characterized in that, The image preprocessing includes at least one of the following: image scaling, pixel value normalization, channel order conversion, and data type conversion; The image post-processing also includes at least one of the following: image scaling, pixel value denormalization, channel order conversion, and data type conversion.

3. The video processing method as described in claim 1, characterized in that, The algorithm model includes a depth estimation model and a viewpoint generation model; The image preprocessing of the video image to be input to the algorithm model includes: performing a first image preprocessing on the video image to be input to the depth estimation model to obtain a preprocessed first video image; The step of using an algorithm model to infer the video image after image preprocessing to obtain the inferred video image includes: using a depth estimation model to infer the first video image to obtain a depth estimation image; The image preprocessing of the video image to be input to the algorithm model further includes: performing a second image preprocessing on the depth estimation image to be input to the viewpoint generation module to obtain a preprocessed second video image; The step of using an algorithm model to infer the preprocessed video image to obtain the inferred video image further includes: using a viewpoint generation model to infer the second video image to obtain the inferred video image.

4. The video processing method as described in claim 1, characterized in that, The video processing steps also include: The first video stream is decoded to obtain a decoded video image, which is used as the video image to be input into the algorithm model.

5. The video processing method as described in claim 4, characterized in that, The step of decoding the first video stream to obtain the decoded video image includes, prior to: The CPU is used to split the video file into a first video stream and an audio stream, and the first video stream is stored in a first queue. Copy the first video stream from the first queue to the GPU.

6. The video processing method as described in claim 5, characterized in that, The video processing steps also include: The multi-view image is encoded to obtain a second video stream.

7. The video processing method as described in claim 6, characterized in that, The process of encoding the multi-view images to obtain a second video stream further includes: Copy the second video stream to the CPU; The CPU is used to synchronize and merge the second video stream with the audio stream to obtain a video file.

8. The video processing method as described in claim 1, characterized in that, The video processing steps also include: The multi-view image is rendered.

9. The video processing method as described in claim 1, characterized in that, The steps of using a GPU to perform video processing include: The video images of the algorithm model to be input are distributed to multiple GPUs, and the video processing steps are performed using multiple GPUs.

10. The video processing method as described in claim 9, characterized in that, The step of distributing the video images of the algorithm model to be input to multiple GPUs includes: Save the video image of the algorithm model to be input to the second queue; The video images of the algorithm model to be input, stored in the second queue, are distributed to the multiple GPUs.

11. The video processing method as described in claim 9, characterized in that, The video processing steps also include: The multi-viewpoint images are saved to the third queue; The multi-view image is obtained from the third queue and rendered.

12. The video processing method as described in claim 11, characterized in that, The step of obtaining the multi-view image from the third queue and rendering it includes: The multi-view image is rendered using a designated GPU from among the plurality of GPUs.

13. A video processing system, characterized in that, include: An execution module is configured to perform video processing steps using a GPU, the video processing steps including: Perform image preprocessing on the video images to be used as input algorithm models; The algorithm model is used to infer the video image after image preprocessing to obtain the inferred video image. The viewpoint of the inferred video image is different from the viewpoint of the video image to be input into the algorithm model. The inferred video images are subjected to image post-processing, which includes image stitching of the inferred video images and the video images of the algorithm model to be input into the same frame to obtain multi-view images.

14. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the video processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video processing method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the video processing method as described in any one of claims 1 to 12.

Images