Panoramic video processing method, device, video processing equipment and storage medium
By using multi-threading of the central processor and graphics processor in video processing equipment for video pulling, decoding, splicing and encoding processing, the problem of high video push delay is solved, the video push efficiency is improved, and the user's high quality is met Timeliness requirements.
Patent Information
- Application Number
- CN202411256526.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-09
AI Technical Summary
In the prior art, video push delay is high, making it difficult to meet the users' delay needs.
The panoramic video processing method is adopted, and multiple threads of the central processor and the graphics processor are used to pull, decode, splice and encode videos, making full use of the parallel processing capabilities of the graphics processor, and improving the efficiency of decoding, splicing and encoding processing.
It improves the efficiency of video push, reduces latency, and meets the high-time efficiency needs of real-time push of panoramic videos and large-resolution videos.
Smart Images

Figure CN118764573B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a panoramic video processing method, apparatus, video processing equipment and storage medium. Background Art
[0002] The processing of panoramic videos generally includes processes such as pulling, decoding, encoding, and pushing. Based on some application requirements, multiple video streams may be pulled from multiple video sources. Then, after decoding, multiple video streams need to be spliced and fused to form a panoramic video stream with a higher resolution. Then, the spliced panoramic video stream is encoded and pushed. However, the inventors have found that in the prior art, there is a relatively high delay in video push, which makes it difficult to meet the user's delay requirements. Summary of the invention
[0003] In view of this, the purpose of the present application is to provide a panoramic video processing method, apparatus, video processing device and storage medium to improve the problem of relatively high video push delay existing in the prior art.
[0004] To achieve the above purpose, the embodiment of the present application adopts the following technical solution:
[0005] A panoramic video processing method is applied to a video processing device, wherein the video processing device includes a central processing unit and a graphics processing unit. The panoramic video processing method includes:
[0006] Using multiple stream pulling threads in the central processing unit, multiple video sources are processed to obtain multiple original video streams, wherein there is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources and the multiple original video streams;
[0007] Using a plurality of decoding threads in the graphics processor, the plurality of original video streams are decoded to obtain a plurality of decoded video frame data, wherein there is a one-to-one correspondence between the plurality of original video streams, the plurality of decoding threads and the plurality of decoded video frame data;
[0008] Using the panoramic stitching thread in the graphics processor, stitching the plurality of decoded video frame data to obtain panoramic video frame data;
[0009] Using the panoramic encoding thread in the graphics processor, encoding the panoramic video frame data to obtain encoded video frame data;
[0010] In response to a request from a video terminal device, the encoded video frame data is pushed to the video terminal device using a panoramic streaming thread in the central processor.
[0011] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of encoding the panoramic video frame data using the panoramic encoding thread in the graphics processor to obtain the encoded video frame data includes:
[0012] Using the panoramic encoding thread in the graphics processor, the panoramic video frame data is obtained from the panoramic video data queue, and the panoramic video frame data is encoded to obtain encoded video frame data corresponding to the panoramic video frame data, wherein the panoramic video data queue is located in the video memory corresponding to the graphics processor;
[0013] The encoded video frame data is stored in an encoded data queue, wherein the encoded data queue is located in a memory corresponding to the central processing unit.
[0014] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of using the panoramic encoding thread in the graphics processor to obtain the panoramic video frame data from the panoramic video data queue, and encoding the panoramic video frame data to obtain the encoded video frame data corresponding to the panoramic video frame data includes:
[0015] Using the panoramic encoding thread in the graphics processor, calling a data copy function, wherein the data copy function includes a first parameter, a second parameter, and a third parameter, the first parameter is used to represent a location where the copied data needs to be stored, the second parameter is used to represent a location where the copied data itself is stored, and the third parameter is used to represent the number of bytes of the copied data;
[0016] The panoramic video frame data is copied from the panoramic video data queue by using the data copy function, and the panoramic video frame data is encoded to obtain the encoded video frame data corresponding to the panoramic video frame data.
[0017] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of storing the encoded video frame data in the encoded data queue includes:
[0018] After the encoded video frame data is obtained based on the encoding process, determining whether the queue state of the encoded data queue corresponding to the central processor is in a not full state;
[0019] When the queue state of the coded data queue is not full, writing the coded video frame data to the position pointed to by the current write index in the coded data queue;
[0020] After writing the encoded video frame data into the encoded data queue, the write index of the encoded data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the encoded data queue, the position pointed to by the updated write index is the first position in the encoded data queue, so that the position pointed to by the write index circulates in the encoded data queue.
[0021] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of using the panoramic stitching thread in the graphics processor to stitch the multiple decoded video frame data to obtain the panoramic video frame data includes:
[0022] Using the panoramic stitching thread in the graphics processor, the multiple decoded video frame data are obtained from multiple decoded video frame queues, wherein there is a one-to-one correspondence between the multiple decoded video frame queues and the multiple decoding threads, so that the decoded video frame data corresponding to one decoding thread is stored in the corresponding decoded video frame queue, and the decoded video frame queue is located in the video memory corresponding to the graphics processor;
[0023] The multiple decoded video frame data are spliced to obtain panoramic video frame data, and the panoramic video frame data is stored in a panoramic video data queue, wherein the panoramic video data queue is located in a video memory corresponding to the graphics processor.
[0024] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of splicing the multiple decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue includes:
[0025] Performing stitching processing on the multiple decoded video frame data obtained synchronously to obtain panoramic video frame data, wherein the panoramic stitching thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues;
[0026] Determine whether the queue status of the panoramic video data queue is not full;
[0027] When the queue state of the panoramic video data queue is not full, writing the panoramic video frame data to the position pointed to by the current write index in the panoramic video data queue;
[0028] After writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue.
[0029] In a preferred embodiment of the present application, in the above-mentioned panoramic video processing method, the step of splicing the multiple decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue includes:
[0030] After the multiple decoded video frame data are synchronously acquired, the multiple decoded video frame data are polled, and the following operations are performed on the currently polled decoded video frame data:
[0031] Determining whether a queue state in a queue of video data to be stitched is not full, wherein the panoramic stitching thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues, and the queue of video data to be stitched is located in the video memory of the graphics processor;
[0032] When the video data to be spliced is not full, writing the currently polled decoded video frame data into the position pointed to by the current write index in the queue of video data to be spliced;
[0033] After writing the currently polled decoded video frame data into the queue of video data to be spliced, updating the write index of the queue of video data to be spliced so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the queue of video data to be spliced, the position pointed to by the updated write index is the first position in the queue of video data to be spliced, so that the position pointed to by the write index circulates in the queue of video data to be spliced;
[0034] Determine whether the queue state in the queue of video data to be spliced is an empty state;
[0035] When the queue state of the video data queue to be spliced is not an empty state, read the decoded video frame data of the position pointed to by the current read index in the video data queue to be spliced, and after reading the decoded video frame data, update the current read index in the video data queue to be spliced so that the updated read index points to the next position, wherein, when the position pointed to by the current read index is the last position in the video data queue to be spliced, the position pointed to by the updated read index is the first position in the video data queue to be spliced, so that the position pointed to by the read index circulates in the video data queue to be spliced;
[0036] After the plurality of decoded video frame data are sequentially read out from the queue of video data to be spliced, the plurality of decoded video frame data are spliced to obtain panoramic video frame data;
[0037] Determine whether the queue status of the panoramic video data queue is not full;
[0038] When the queue state of the panoramic video data queue is not full, writing the panoramic video frame data to the position pointed to by the current write index in the panoramic video data queue;
[0039] After writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue.
[0040] The embodiment of the present application further provides a panoramic video processing device, which is applied to a video processing device, wherein the video processing device includes a central processing unit and a graphics processor, and the panoramic video processing device includes:
[0041] A video stream pulling module, used to use multiple stream pulling threads in the central processor to perform video pulling processing on multiple video sources to obtain multiple original video streams, wherein there is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources and the multiple original video streams;
[0042] A video decoding module, configured to decode the multiple original video streams using multiple decoding threads in the graphics processor to obtain multiple decoded video frame data, wherein there is a one-to-one correspondence between the multiple original video streams, the multiple decoding threads and the multiple decoded video frame data;
[0043] A video stitching module, configured to stitch the plurality of decoded video frame data using a panoramic stitching thread in the graphics processor to obtain panoramic video frame data;
[0044] A video encoding module, configured to use a panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data;
[0045] The video push module is used to respond to the request of the video terminal device and push the encoded video frame data to the video terminal device by using the panoramic streaming thread in the central processor.
[0046] On the basis of the above, the embodiment of the present application further provides a video processing device, including:
[0047] Memory for storing computer programs;
[0048] The processor connected to the memory is used to execute the computer program stored in the memory to implement the above-mentioned panoramic video processing method.
[0049] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is run, each step of the above-mentioned panoramic video processing method is executed.
[0050] The panoramic video processing method, device, video processing equipment and storage medium provided by the present application can, firstly, utilize multiple stream pulling threads in a central processing unit to perform video pulling processing on multiple video sources to obtain multiple original video streams; secondly, utilize multiple decoding threads in a graphics processor to perform decoding processing on multiple original video streams to obtain multiple decoded video frame data; then, utilize the panoramic stitching thread in the graphics processor to perform stitching processing on multiple decoded video frame data to obtain panoramic video frame data; thereafter, utilize the panoramic encoding thread in the graphics processor to perform encoding processing on the panoramic video frame data to obtain encoded video frame data; finally, utilize the panoramic streaming pushing thread in the central processing unit to push the encoded video frame data to a video terminal device. Based on the foregoing content, since relatively more complex tasks such as decoding, stitching and encoding are concentrated on the graphics processor, the parallel processing capabilities of the graphics processor can be fully utilized, that is, multiple decoding threads, panoramic stitching threads and panoramic encoding threads are deployed on the graphics processor, and the large number of threads in the graphics processor can be fully utilized. Therefore, the efficiency of decoding, stitching and encoding can be improved, thereby ensuring the efficiency of video push, and further improving the problem of relatively high video push delay in the prior art, so that it has high practical application value in the application of real-time push of panoramic video, and also has high practical application value in the application of panoramic large-resolution video push, fully meeting the high timeliness requirements of users. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.
[0052] Figure 1 This is a structural block diagram of a video processing device provided in an embodiment of the present application.
[0053] Figure 2 A flowchart of a panoramic video processing method provided in an embodiment of the present application.
[0054] Figure 3 A framework diagram of panoramic video processing provided in an embodiment of the present application.
[0055] Figure 4 A block diagram of a panoramic video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0057] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0058] like Figure 1 As shown, an embodiment of the present application provides a video processing device, which may include a memory, a processor, and a panoramic video processing apparatus.
[0059] Among them, the memory and the processor can be electrically connected directly or indirectly to realize data transmission or interaction. For example, the memory and the processor can be electrically connected through one or more communication buses or signal lines. The panoramic video processing device includes at least one software function module that can be stored in the memory in the form of software or firmware. The processor is used to execute an executable computer program stored in the memory, for example, the software function module and computer program included in the panoramic video processing device, so as to implement the panoramic video processing method provided in the embodiment of the present application, and the specific content is described below.
[0060] Optionally, the memory may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. Furthermore, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0061] Understandably, Figure 1 The structure shown is for illustration only. The video processing device may also include Figure 1 More or fewer components as shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for exchanging information with other devices (such as terminal devices), and for another example, may also include a graphics processing unit (GPU).
[0062] Combination Figure 2 The present application also provides a panoramic video processing method applicable to the above-mentioned video processing device. The method steps defined in the process related to the panoramic video processing method can be implemented by the video processing device. The video processing device may include a central processing unit and a graphics processing unit. Figure 2 The specific process shown is explained in detail.
[0063] Step S110, using the multiple stream pulling threads in the central processing unit, perform video pulling processing on the multiple video sources to obtain multiple original video streams.
[0064] In an embodiment of the present application, the video processing device can utilize multiple stream pulling threads in the central processor to perform video pulling processing on multiple video sources to obtain multiple original video streams. There is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources, and the multiple original video streams. That is, for each of the multiple stream pulling threads, the stream pulling thread corresponds to a video source and an original video stream. Figure 3 As shown, original video stream 1 can be pulled from video source 1 through stream pulling thread 1, original video stream 2 can be pulled from video source 2 through stream pulling thread 2, original video stream 3 can be pulled from video source 3 through stream pulling thread 3, and original video stream 4 can be pulled from video source 4 through stream pulling thread 4.
[0065] Step S120: using the multiple decoding threads in the graphics processor to decode the multiple original video streams to obtain multiple decoded video frame data.
[0066] In an embodiment of the present application, after multiple original video streams are pulled based on step S110, the video processing device can use multiple decoding threads in the graphics processor to decode the multiple original video streams to obtain multiple decoded video frame data. There is a one-to-one correspondence between the multiple original video streams, the multiple decoding threads, and the multiple decoded video frame data. That is, for each decoding thread in the multiple decoding threads, the decoding thread corresponds to one original video stream and one decoded video frame data. Figure 3 As shown, the original video stream 1 can be decoded by decoding thread 1 to obtain decoded video frame data 1, the original video stream 2 can be decoded by decoding thread 2 to obtain decoded video frame data 2, the original video stream 3 can be decoded by decoding thread 3 to obtain decoded video frame data 3, and the original video stream 4 can be decoded by decoding thread 4 to obtain decoded video frame data 4.
[0067] Step S130 , using the panoramic stitching thread in the graphics processor, stitching the multiple decoded video frame data to obtain panoramic video frame data.
[0068] In the embodiment of the present application, after decoding in step S120 to form a plurality of decoded video frame data, the video processing device may use the panoramic stitching thread in the graphics processor to stitch the plurality of decoded video frame data to obtain panoramic video frame data. Figure 3 As shown, the panoramic stitching thread can stitch decoded video frame data 1, decoded video frame data 2, decoded video frame data 3 and decoded video frame data 4 to obtain panoramic video frame data.
[0069] Step S140: utilizing the panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data.
[0070] In the embodiment of the present application, after obtaining the panoramic video frame data based on step S130, the video processing device can utilize the panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data.
[0071] Step S150, in response to a request from a video terminal device, the encoded video frame data is pushed to the video terminal device using the panoramic streaming thread in the central processor.
[0072] In an embodiment of the present application, after receiving a request from a video terminal device, the video processing device may respond to the request of the video terminal device and utilize the panoramic streaming thread in the central processor to push the encoded video frame data to the video terminal device.
[0073] Based on the foregoing content, since relatively more complex tasks such as decoding, stitching and encoding are concentrated on the graphics processor, the parallel processing capabilities of the graphics processor can be fully utilized, that is, multiple decoding threads, panoramic stitching threads and panoramic encoding threads are deployed on the graphics processor, and the large number of threads in the graphics processor can be fully utilized. Therefore, the efficiency of decoding, stitching and encoding can be improved, thereby ensuring the efficiency of video push, and further improving the problem of relatively high video push delay in the prior art, so that it has high practical application value in the application of real-time push of panoramic video, and also has high practical application value in the application of panoramic large-resolution video push, fully meeting the high timeliness requirements of users.
[0074] First, it should be noted that the specific method of video pulling and processing is not limited and can be configured according to actual needs.
[0075] For example, in an alternative implementation, multiple original video streams can be obtained in one device, that is, the multiple video sources belong to the same device. For example, after multiple video capture devices capture multiple video streams, they can be concentrated in one device first, so that the multiple stream pulling threads can pull multiple video streams from this one device.
[0076] For another example, in another alternative implementation, multiple original video streams can be obtained in multiple devices. For example, multiple original video streams can be directly obtained from multiple network cameras or live broadcast devices (mobile phones, tablets), etc. The original video stream can be RTSP (Real Time Streaming Protocol, a network protocol for controlling real-time data transmission on a computer network, which is commonly used in streaming media systems and supports real-time transmission of audio and video. RTSP is an application layer protocol whose purpose is to manage the transmission of streaming media through control messages between the client and the server) stream output by multiple network cameras or live broadcast devices. In other words, after obtaining the RTSP streams output by multiple network cameras or live broadcast devices, the RTSP streams can be parsed to obtain 4K (3840*2160) resolution, H.264 (also known as AVC, Advanced Video Coding) or H.265 (also known as HEVC, High Efficiency Video Coding) ES (Elementary Stream, which refers to the basic code stream format in the video coding standard, only contains the encoded data of the video or audio, and does not include other additional information in the container format (such as MP4, MKV, etc.)) encoded video streams, and then the encoded video stream can be used as the original video stream and can be stored in the memory corresponding to the central processor. In addition, based on some requirements, such as for the convenience of storage and acquisition, the original video stream can be stored through the corresponding data queue. That is, the stream pulling thread can write the original video stream to the data queue in the memory, and then the subsequent decoding thread can read the original video stream from the data queue. Among them, different original video streams can be stored in different data queues, such as the above-mentioned 4 original video streams, which can be stored in 4 data queues respectively.
[0077] Secondly, it should be noted that the specific method of decoding processing is not limited and can be configured according to actual needs.
[0078] For example, in an alternative implementation, the decoding thread can be used to decode the original video stream, such as obtaining a decoded video stream in NV12 format (a planar YUV format, where Y represents the brightness component and UV represents the chrominance component), and then the decoded video stream can be directly used as the decoded video frame data.
[0079] For another example, in another alternative implementation, the decoding thread can be used to decode the original video stream, such as obtaining a decoded video stream in NV12 format (a YUV format), and then, based on the Nv12ToColor32 algorithm, the decoded video stream can be converted and processed to obtain corresponding decoded video frame data (the format is Color32 format, i.e., RGB32, a 32-bit pixel format, where R, G, and B represent the three color channels of red, green, and blue). The decoded video frame data can be stored in the video memory of the graphics processor. For example, the decoded video frame data can be stored in a data queue in the video memory, such as a decoded video frame queue. For example, different decoded video frame data can be stored in different data queues, such as the above-mentioned 4 decoded video frame data, which can be stored in 4 decoded video frame queues respectively.
[0080] Thirdly, it should be noted that the specific method of splicing processing is not limited and can be configured according to actual needs.
[0081] For example, in an alternative implementation, in order to alleviate the storage pressure of the video memory corresponding to the graphics processor, the panoramic stitching thread in the image processor can be used to obtain multiple decoded video frame data, and then, after stitching is performed based on the multiple decoded video frame data to obtain panoramic video frame data, the panoramic video frame data can be stored in the memory corresponding to the central processing unit.
[0082] For another example, in another alternative implementation, in order to make the video processing more efficient, step S130 may further include step S131 and step S132, the specific contents of which are as follows.
[0083] Step S131, using the panoramic stitching thread in the graphics processor, obtaining the multiple decoded video frame data from multiple decoded video frame queues.
[0084] In an embodiment of the present application, after obtaining a plurality of decoded video frame data based on step S120, the panoramic stitching thread in the graphics processor can be used to obtain the plurality of decoded video frame data from the plurality of decoded video frame queues storing the plurality of decoded video frame data. There is a one-to-one correspondence between the plurality of decoded video frame queues and the plurality of decoding threads, so that the decoded video frame data corresponding to one decoding thread is stored in the corresponding decoded video frame queue, and the decoded video frame queue is located in the video memory corresponding to the graphics processor. For example, in combination with the examples in the foregoing text, decoded video frame data 1 can be stored in decoded video frame queue 1, decoded video frame data 2 can be stored in decoded video frame queue 2, decoded video frame data 3 can be stored in decoded video frame queue 3, and decoded video frame data 4 can be stored in decoded video frame queue 4. In this way, the panoramic stitching thread can read decoded video frame data 1 from decoded video frame queue 1, the panoramic stitching thread can read decoded video frame data 2 from decoded video frame queue 2, the panoramic stitching thread can read decoded video frame data 3 from decoded video frame queue 3, and the panoramic stitching thread can read decoded video frame data 4 from decoded video frame queue 4.
[0085] Step S132: splicing the multiple decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue.
[0086] In an embodiment of the present application, after obtaining the multiple decoded video frame data, the multiple decoded video frame data can be spliced to obtain panoramic video frame data, and the panoramic video frame data is stored in a panoramic video data queue. Wherein, the panoramic video data queue is located in the video memory corresponding to the graphics processor, so that for subsequent encoding processing, the panoramic encoding thread can directly read the panoramic video frame data from the video memory, which can make the time efficiency higher. Wherein, the specific splicing processing method is not limited. For example, the above-mentioned decoded video frame data 1, decoded video frame data 2, decoded video frame data 3 and decoded video frame data 4 can be directly combined (there may be no overlapping parts between the 4 decoded video frame data) to achieve splicing. For another example, the overlapping parts between the above-mentioned decoded video frame data 1, decoded video frame data 2, decoded video frame data 3 and decoded video frame data 4 can be calculated first, and then splicing is performed based on the overlapping parts. Exemplarily, the splicing process is as follows:
[0087] Import splicing script pto file: You need to import a splicing script file, which contains relevant information for video splicing, such as camera position, shooting angle, etc.
[0088] Calculate the overlapped bands using linear weighting or multi-band fusion algorithms: During the stitching process, there are overlapping areas between video frames, and it is necessary to determine how to merge the image information of these overlapping areas. For example, linear weighting or multi-band fusion algorithms can be used to calculate the overlapped bands to ensure smooth transition and consistency of the images in the overlapping areas.
[0089] Video image stitching and rendering parameter calculation: After determining the image information of the overlapping area, it is necessary to calculate the video image stitching and rendering parameters to ensure that the images from different cameras or image sources can achieve consistent appearance and color after stitching;
[0090] Image exposure and white balance self-calculation algorithm: During the video stitching process, there may be differences in exposure and white balance between different cameras or image sources, resulting in obvious color difference and inconsistent brightness in the stitched image. In this step, a self-calculation algorithm is used to calculate the color correction parameters of the image with color difference, so that the final output stitched video has consistent color and brightness;
[0091] Stitched image generation: The videos are stitched together according to the calculated stitching, rendering parameters, and color correction parameters to generate the final stitched video, ensuring that the stitched video has no obvious color difference and stitching seams, so that it presents a natural and coherent appearance visually.
[0092] It is understandable that the specific contents involved in the above-mentioned splicing process can refer to the relevant existing technologies and are not specifically limited or described here.
[0093] It can be understood that, in the above step S132, the specific method of performing the splicing process is not limited and can be configured according to actual needs.
[0094] For example, in an alternative implementation, in order to make the splicing process more efficient and to make the splicing process occupy fewer resources, step S132 may further include the following contents:
[0095] In the first step, the multiple decoded video frame data obtained synchronously can be spliced to obtain panoramic video frame data, wherein the panoramic splicing thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues, that is, after the panoramic splicing thread obtains the multiple decoded video frame data from the multiple decoded video frame queues, it can directly splice the obtained multiple decoded video frame data;
[0096] In the second step, it is possible to determine whether the queue state of the panoramic video data queue is not full, that is, it is possible to determine whether there is an empty position in the panoramic video data queue;
[0097] The third step is to write the panoramic video frame data to the position pointed to by the current write index in the panoramic video data queue when the queue state of the panoramic video data queue is not full, wherein, based on the pre-configuration, in the panoramic video data queue, when there is an empty position, its write index will point to the first empty position, so the panoramic video frame data can be directly written to the position pointed to by the current write index;
[0098] In the fourth step, after writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue can be updated so that the updated write index points to the next position (i.e., the current first empty position), wherein, when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue. It can be understood that when updating the position pointed to by the write index from the last position to the first position, the first position is required to be an empty position, i.e., it has no data. For example, subsequent threads, such as the panoramic encoding thread, have read the data at the first position (panoramic video frame data).
[0099] For another example, in another alternative implementation, in order to make the control of the splicing process more reliable, step S132 may further include the following contents:
[0100] After the multiple decoded video frame data are synchronously acquired, the multiple decoded video frame data may be polled (the specific polling method is not limited and may be arbitrary or implemented according to a certain strategy, such as polling in the order of decoded video frame data 1, decoded video frame data 2, decoded video frame data 3, and decoded video frame data 4), and the following operations may be performed on the currently polled decoded video frame data:
[0101] In the first step, it is possible to determine whether the queue status in the queue of video data to be stitched is not full, wherein the panoramic stitching thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues, and the queue of video data to be stitched is located in the video memory of the graphics processor and is used to store the decoded video frame data obtained from the decoded video frame queue;
[0102] In the second step, when the queue of video data to be spliced is not full, the currently polled decoded video frame data may be written into the position pointed to by the current write index in the queue of video data to be spliced, as described above;
[0103] In the third step, after writing the currently polled decoded video frame data into the queue of video data to be spliced, the write index of the queue of video data to be spliced can be updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the queue of video data to be spliced, the position pointed to by the updated write index is the first position in the queue of video data to be spliced, so that the position pointed to by the write index circulates in the queue of video data to be spliced, as described above;
[0104] In the fourth step, it can be determined whether the queue state in the queue of video data to be spliced is an empty state. For example, the empty state means that there is no data in the data queue;
[0105] The fifth step is to read the decoded video frame data of the position pointed to by the current read index in the video data queue to be spliced when the queue state of the video data queue to be spliced is not an empty state, and after reading the decoded video frame data, update the current read index in the video data queue to be spliced so that the updated read index points to the next position, wherein, when the position pointed to by the current read index is the last position in the video data queue to be spliced, the position pointed to by the updated read index is the first position in the video data queue to be spliced, so that the position pointed to by the read index circulates in the video data queue to be spliced, and it can be understood that when updating the position pointed to by the read index from the last position to the first position, the first position needs to have data, for example, data that has been written to the first position by the panoramic stitching thread, such as newly acquired decoded video frame data;
[0106] In the sixth step, after the multiple decoded video frame data are read out from the queue of video data to be spliced in sequence, the multiple decoded video frame data may be spliced to obtain panoramic video frame data. For example, decoded video frame data 1, decoded video frame data 2, decoded video frame data 3 and decoded video frame data 4 may be read from the queue of video data to be spliced, and then, decoded video frame data 1, decoded video frame data 2, decoded video frame data 3 and decoded video frame data 4 may be spliced to obtain panoramic video frame data. After that, new decoded video frame data 1, new decoded video frame data 2, new decoded video frame data 3 and new decoded video frame data 4 may be read from the queue of video data to be spliced, and then, new decoded video frame data 1, new decoded video frame data 2, new decoded video frame data 3 and new decoded video frame data 4 may be spliced to obtain new panoramic video frame data.
[0107] In the seventh step, it can be determined whether the queue status of the panoramic video data queue is not full, as described above;
[0108] In step 8, when the queue state of the panoramic video data queue is not full, the panoramic video frame data may be written into the position pointed to by the current write index in the panoramic video data queue, as described above;
[0109] In the ninth step, after writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue can be updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue, as described above.
[0110] Based on this, the panoramic stitching thread can cooperate with two data queues (i.e., the stitching video data queue and the panoramic video data queue) to make the panoramic stitching thread reliably cooperate in reading data, stitching data and storing data, and through the cooperation of the two data queues, data processing buffering can be achieved to a certain extent, which can improve the stitching errors caused by data conflicts and other problems to a certain extent.
[0111] Fourthly, it should be noted that the specific method of performing the encoding process is not limited and can be configured according to actual needs.
[0112] For example, in an alternative implementation, in order to reduce the consumption of storage space, the panoramic encoding thread can be used to encode the panoramic video frame data, and then the obtained encoded video frame data can be streamed. In this way, on the one hand, the efficiency of video streaming can be improved, and on the other hand, the consumption of storage space can be reduced.
[0113] For example, in another alternative implementation, encoding and subsequent processing (such as streaming, etc.) can be operated independently, so that the probability of data conflict can be reduced, and step S140 can further include step S141 and step S142, the specific contents of which are described as follows.
[0114] Step S141, using the panoramic encoding thread in the graphics processor, obtains the panoramic video frame data from the panoramic video data queue, and encodes the panoramic video frame data to obtain encoded video frame data corresponding to the panoramic video frame data.
[0115] In the embodiment of the present application, after the panoramic video frame data is obtained based on step S130, the panoramic video frame data can be obtained from the panoramic video data queue by using the panoramic encoding thread in the graphics processor, and the panoramic video frame data is encoded to obtain the encoded video frame data corresponding to the panoramic video frame data. The panoramic video data queue is located in the video memory corresponding to the graphics processor.
[0116] Step S142: storing the encoded video frame data in an encoded data queue.
[0117] In the embodiment of the present application, after the encoded video frame data is obtained based on step S141, the encoded video frame data can be stored in an encoded data queue. The encoded data queue is located in the memory corresponding to the central processing unit, so that when the central processing unit needs to process the encoded data queue, the encoded video frame data can be read from the encoded data queue, thereby achieving decoupling from the encoding process.
[0118] It can be understood that, in the above-mentioned step S141, the specific method of obtaining data from the panoramic video data queue and performing encoding processing is not limited and can be configured according to actual needs.
[0119] For example, in an alternative implementation, in order to enable the panoramic video frame data to be acquired from the panoramic video data queue at a high speed to improve the efficiency of encoding processing, the above step S141 may further include the following steps:
[0120] First, the panoramic encoding thread in the graphics processor can be used to call a data copy function, wherein the data copy function includes a first parameter, a second parameter and a third parameter, wherein the first parameter is used to characterize the location where the copied data needs to be stored, that is, the target location, the second parameter is used to characterize the location where the copied data itself is stored, that is, the current location, and the third parameter is used to characterize the number of bytes of the copied data, that is, the amount of data; illustratively, the data copy function can be a cudaMemcpyDeviceToDevice function, which has an efficient data transmission function, and the cudaMemcpyDeviceToDevice function can be used to achieve asynchronous transmission by setting a suitable stream, so that data transmission and computing tasks can be executed in parallel, making full use of the parallel computing capabilities of the graphics processor. In addition, the function has a similar interface to other memory copy functions, making it easy to use and integrate into existing CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model) code;
[0121] Secondly, the data copy function can be used to copy the panoramic video frame data from the panoramic video data queue, and the panoramic video frame data can be encoded to obtain the encoded video frame data corresponding to the panoramic video frame data; exemplarily, after copying the panoramic video frame data based on the data copy function, the panoramic video frame data in RGB32 color format can be converted into NV12 color format by Color32ToNv12 conversion algorithm, and then, the panoramic video frame data in NV12 color format can be encoded, such as encoded into H.264 or H.265 format, to obtain the corresponding encoded video frame data.
[0122] It can be understood that, in the above-mentioned step S142, the specific method of storing the encoded video frame data in the encoded data queue is not limited and can be configured according to actual needs.
[0123] For example, in an alternative implementation, in order to ensure the orderliness and reliability of the storage of the encoded video frame data, the above step S142 may further include the following steps:
[0124] First, after obtaining the encoded video frame data based on the encoding process, it can be determined whether the queue state of the encoding data queue corresponding to the central processor is in a not full state;
[0125] Secondly, when the queue state of the coded data queue is not full, writing the coded video frame data to the position pointed to by the current write index in the coded data queue;
[0126] Then, after writing the encoded video frame data into the encoded data queue, the write index of the encoded data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the encoded data queue, the position pointed to by the updated write index is the first position in the encoded data queue, so that the position pointed to by the write index circulates in the encoded data queue.
[0127] For the above steps of storing the encoded video frame data into the encoded data queue, the specific contents can refer to the relevant description in the previous text, which will not be repeated here.
[0128] Fifthly, it should be noted that for step S150, the specific method of pushing the panoramic video frame data is not limited and can also be configured according to actual needs.
[0129] For example, in an alternative implementation, after receiving a request from a video terminal device, the panoramic streaming thread in the central processor can be used to push the encoded video frame data to the video terminal device through the RTSP (Real-Time Streaming Protocol) protocol or the RTP (Real-Time Transport Protocol) protocol. The video terminal device can be a client for communicating with a streaming media server, used to send RTSP requests and receive responses from the server, thereby realizing control and playback of real-time streaming media, and the above-mentioned video processing device can be a streaming media server) or a general streaming media player (such as VLC, VideoLANClient, a software application for playing network streaming media, its main function is to obtain streaming media data from the network, decode and render it, so that users can watch videos or listen to audio content in real time) to obtain the encoded video frame data, and then the encoded video frame data can be played and processed.
[0130] Combination Figure 4 The embodiment of the present application also provides a panoramic video processing device applicable to the above-mentioned video processing device. The panoramic video processing device may include a video streaming module, a video decoding module, a video splicing module, a video encoding module and a video pushing module.
[0131] The video stream pulling module is used to use the multiple stream pulling threads in the central processor to perform video pulling processing on multiple video sources to obtain multiple original video streams, wherein there is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources and the multiple original video streams. In this embodiment, the video stream pulling module can be used to execute Figure 2 As shown in step S110, for the relevant content of the video streaming module, reference can be made to the above description of step S110.
[0132] The video decoding module is used to use the multiple decoding threads in the graphics processor to decode the multiple original video streams to obtain multiple decoded video frame data, wherein there is a one-to-one correspondence between the multiple original video streams, the multiple decoding threads and the multiple decoded video frame data. In this embodiment, the video decoding module can be used to perform Figure 2 As shown in step S120, the relevant contents of the video decoding module can refer to the above description of step S120.
[0133] The video stitching module is used to use the panoramic stitching thread in the graphics processor to stitch the multiple decoded video frame data to obtain panoramic video frame data. In this embodiment, the video stitching module can be used to execute Figure 2 As shown in step S130, the relevant contents of the video stitching module can refer to the above description of step S130.
[0134] The video encoding module is used to use the panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data. In this embodiment, the video encoding module can be used to perform Figure 2 As shown in step S140, the relevant contents of the video encoding module can refer to the above description of step S140.
[0135] The video push module is used to respond to the request of the video terminal device and use the panoramic push thread in the central processor to push the encoded video frame data to the video terminal device. In this embodiment, the video push module can be used to execute Figure 2 As shown in step S150, the relevant contents of the video push module can refer to the above description of step S150.
[0136] In the embodiment of the present application, corresponding to the panoramic video processing method applied to the video processing device, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is run, each step of the panoramic video processing method is executed. Among them, each step executed when the aforementioned computer program is run is not described one by one here, and reference can be made to the explanation of the panoramic video processing method in the previous text.
[0137] To summarize, the panoramic video processing method, apparatus, video processing device and storage medium provided by the present application can, firstly, utilize multiple stream pulling threads in the central processing unit to perform video pulling processing on multiple video sources to obtain multiple original video streams; secondly, utilize multiple decoding threads in the graphics processor to decode multiple original video streams to obtain multiple decoded video frame data; then, utilize the panoramic stitching thread in the graphics processor to stitch multiple decoded video frame data to obtain panoramic video frame data; thereafter, utilize the panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data; finally, utilize the panoramic streaming push thread in the central processing unit to push the encoded video frame data to the video terminal device. Based on the foregoing content, since relatively more complex tasks such as decoding, stitching and encoding are concentrated on the graphics processor, the parallel processing capabilities of the graphics processor can be fully utilized, that is, multiple decoding threads, panoramic stitching threads and panoramic encoding threads are deployed on the graphics processor, and the large number of threads in the graphics processor can be fully utilized. Therefore, the efficiency of decoding, stitching and encoding can be improved, thereby ensuring the efficiency of video push, and further improving the problem of relatively high video push delay in the prior art, so that it has high practical application value in the application of real-time push of panoramic video, and also has high practical application value in the application of panoramic large-resolution video push, fully meeting the high timeliness requirements of users.
[0138] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device and method embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0139] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0140] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.
[0141] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A panoramic video processing method, characterized in that: Applied to a video processing device, the video processing device includes a central processing unit and a graphics processing unit, and the panoramic video processing method includes: Using multiple stream pulling threads in the central processing unit, multiple video sources are processed to obtain multiple original video streams, wherein there is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources and the multiple original video streams; Using a plurality of decoding threads in the graphics processor, the plurality of original video streams are decoded to obtain a plurality of decoded video frame data, wherein there is a one-to-one correspondence between the plurality of original video streams, the plurality of decoding threads and the plurality of decoded video frame data; Using the panoramic stitching thread in the graphics processor, stitching the multiple decoded video frame data to obtain panoramic video frame data; Using the panoramic encoding thread in the graphics processor, encoding the panoramic video frame data to obtain encoded video frame data; In response to a request from a video terminal device, the encoded video frame data is pushed to the video terminal device using a panoramic streaming thread in the central processor; The step of using the panoramic stitching thread in the graphics processor to stitch the multiple decoded video frame data to obtain panoramic video frame data includes: Using the panoramic stitching thread in the graphics processor, the multiple decoded video frame data are obtained from multiple decoded video frame queues, wherein there is a one-to-one correspondence between the multiple decoded video frame queues and the multiple decoding threads, so that the decoded video frame data corresponding to one decoding thread is stored in the corresponding decoded video frame queue, and the decoded video frame queue is located in the video memory corresponding to the graphics processor; Performing splicing processing on the multiple decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue, wherein the panoramic video data queue is located in a video memory corresponding to the graphics processor; The step of encoding the panoramic video frame data using the panoramic encoding thread in the graphics processor to obtain the encoded video frame data includes: Using the panoramic encoding thread in the graphics processor, the panoramic video frame data is obtained from the panoramic video data queue, and the panoramic video frame data is encoded to obtain encoded video frame data corresponding to the panoramic video frame data, wherein the panoramic video data queue is located in the video memory corresponding to the graphics processor; Storing the encoded video frame data in an encoded data queue, wherein the encoded data queue is located in a memory corresponding to the central processing unit; The step of splicing the plurality of decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue includes: After the multiple decoded video frame data are synchronously acquired, the multiple decoded video frame data are polled, and the following operations are performed on the currently polled decoded video frame data: Determining whether a queue state in a queue of video data to be stitched is not full, wherein the panoramic stitching thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues, and the queue of video data to be stitched is located in the video memory of the graphics processor; When the video data to be spliced is not full, writing the currently polled decoded video frame data into the position pointed to by the current write index in the queue of video data to be spliced; After writing the currently polled decoded video frame data into the queue of video data to be spliced, updating the write index of the queue of video data to be spliced so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the queue of video data to be spliced, the position pointed to by the updated write index is the first position in the queue of video data to be spliced, so that the position pointed to by the write index circulates in the queue of video data to be spliced; Determine whether the queue state in the queue of video data to be spliced is an empty state; When the queue state of the video data queue to be spliced is not an empty state, read the decoded video frame data of the position pointed to by the current read index in the video data queue to be spliced, and after reading the decoded video frame data, update the current read index in the video data queue to be spliced so that the updated read index points to the next position, wherein, when the position pointed to by the current read index is the last position in the video data queue to be spliced, the position pointed to by the updated read index is the first position in the video data queue to be spliced, so that the position pointed to by the read index circulates in the video data queue to be spliced; After the plurality of decoded video frame data are sequentially read out from the queue of video data to be spliced, the plurality of decoded video frame data are spliced to obtain panoramic video frame data; Determine whether the queue status of the panoramic video data queue is not full; When the queue state of the panoramic video data queue is not full, writing the panoramic video frame data to the position pointed to by the current write index in the panoramic video data queue; After writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue.
2. The panoramic video processing method according to claim 1, characterized in that: The step of obtaining the panoramic video frame data from the panoramic video data queue by using the panoramic encoding thread in the graphics processor, and encoding the panoramic video frame data to obtain the encoded video frame data corresponding to the panoramic video frame data includes: Using the panoramic encoding thread in the graphics processor, calling a data copy function, wherein the data copy function includes a first parameter, a second parameter, and a third parameter, the first parameter is used to represent a location where the copied data needs to be stored, the second parameter is used to represent a location where the copied data itself is stored, and the third parameter is used to represent the number of bytes of the copied data; The panoramic video frame data is copied from the panoramic video data queue by using the data copy function, and the panoramic video frame data is encoded to obtain the encoded video frame data corresponding to the panoramic video frame data.
3. The panoramic video processing method according to claim 1, characterized in that: The step of storing the encoded video frame data in an encoded data queue comprises: After the encoded video frame data is obtained based on the encoding process, determining whether the queue state of the encoded data queue corresponding to the central processor is in a not full state; When the queue state of the coded data queue is not full, writing the coded video frame data to the position pointed to by the current write index in the coded data queue; After writing the encoded video frame data into the encoded data queue, the write index of the encoded data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the encoded data queue, the position pointed to by the updated write index is the first position in the encoded data queue, so that the position pointed to by the write index circulates in the encoded data queue.
4. A panoramic video processing device, characterized in that: Applied to a video processing device, the video processing device includes a central processing unit and a graphics processing unit, and the panoramic video processing device includes: A video stream pulling module, used to use multiple stream pulling threads in the central processor to perform video pulling processing on multiple video sources to obtain multiple original video streams, wherein there is a one-to-one correspondence between the multiple stream pulling threads, the multiple video sources and the multiple original video streams; A video decoding module, configured to decode the multiple original video streams using multiple decoding threads in the graphics processor to obtain multiple decoded video frame data, wherein there is a one-to-one correspondence between the multiple original video streams, the multiple decoding threads and the multiple decoded video frame data; A video stitching module, configured to stitch the plurality of decoded video frame data using a panoramic stitching thread in the graphics processor to obtain panoramic video frame data; A video encoding module, configured to use a panoramic encoding thread in the graphics processor to encode the panoramic video frame data to obtain encoded video frame data; A video push module, configured to respond to a request from a video terminal device and push the encoded video frame data to the video terminal device using a panoramic streaming thread in the central processor; The method of using the panoramic stitching thread in the graphics processor to stitch the plurality of decoded video frame data to obtain panoramic video frame data includes: Using the panoramic stitching thread in the graphics processor, the multiple decoded video frame data are obtained from multiple decoded video frame queues, wherein there is a one-to-one correspondence between the multiple decoded video frame queues and the multiple decoding threads, so that the decoded video frame data corresponding to one decoding thread is stored in the corresponding decoded video frame queue, and the decoded video frame queue is located in the video memory corresponding to the graphics processor; Performing splicing processing on the multiple decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue, wherein the panoramic video data queue is located in a video memory corresponding to the graphics processor; The step of encoding the panoramic video frame data using the panoramic encoding thread in the graphics processor to obtain the encoded video frame data includes: Using the panoramic encoding thread in the graphics processor, the panoramic video frame data is obtained from the panoramic video data queue, and the panoramic video frame data is encoded to obtain encoded video frame data corresponding to the panoramic video frame data, wherein the panoramic video data queue is located in the video memory corresponding to the graphics processor; Storing the encoded video frame data in an encoded data queue, wherein the encoded data queue is located in a memory corresponding to the central processing unit; The step of performing splicing processing on the plurality of decoded video frame data to obtain panoramic video frame data, and storing the panoramic video frame data in a panoramic video data queue includes: After the multiple decoded video frame data are synchronously acquired, the multiple decoded video frame data are polled, and the following operations are performed on the currently polled decoded video frame data: Determining whether a queue state in a queue of video data to be stitched is not full, wherein the panoramic stitching thread is used to synchronously obtain the multiple decoded video frame data from the multiple decoded video frame queues, and the queue of video data to be stitched is located in the video memory of the graphics processor; When the video data to be spliced is not full, writing the currently polled decoded video frame data into the position pointed to by the current write index in the queue of video data to be spliced; After writing the currently polled decoded video frame data into the queue of video data to be spliced, updating the write index of the queue of video data to be spliced so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the queue of video data to be spliced, the position pointed to by the updated write index is the first position in the queue of video data to be spliced, so that the position pointed to by the write index circulates in the queue of video data to be spliced; Determine whether the queue state in the queue of video data to be spliced is an empty state; When the queue state of the video data queue to be spliced is not an empty state, read the decoded video frame data of the position pointed to by the current read index in the video data queue to be spliced, and after reading the decoded video frame data, update the current read index in the video data queue to be spliced so that the updated read index points to the next position, wherein, when the position pointed to by the current read index is the last position in the video data queue to be spliced, the position pointed to by the updated read index is the first position in the video data queue to be spliced, so that the position pointed to by the read index circulates in the video data queue to be spliced; After the plurality of decoded video frame data are sequentially read out from the queue of video data to be spliced, the plurality of decoded video frame data are spliced to obtain panoramic video frame data; Determine whether the queue status of the panoramic video data queue is not full; When the queue state of the panoramic video data queue is not full, writing the panoramic video frame data to the position pointed to by the current write index in the panoramic video data queue; After writing the panoramic video frame data into the panoramic video data queue, the write index of the panoramic video data queue is updated so that the updated write index points to the next position, wherein when the position pointed to by the current write index is the last position in the panoramic video data queue, the position pointed to by the updated write index is the first position in the panoramic video data queue, so that the position pointed to by the write index circulates in the panoramic video data queue.
5. A video processing device, characterized in that: include: Memory for storing computer programs; A processor connected to the memory is used to execute a computer program stored in the memory to implement the panoramic video processing method described in any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is run, each step of the panoramic video processing method according to any one of claims 1 to 3 is executed.
