Plug flow methods, apparatus, systems, computing devices, storage media, and program product

Abstract

The application provides a push streaming method, device, system, computing equipment, storage medium and program product, and relates to the technical field of edge computing. The application separates two processes by executing a model inference process and a process of converting a processed target video frame into a to-be-merged video for disk writing by a first thread and a second thread, and realizes asynchronous communication between the threads through a frame buffer queue. Then, the first thread performs frame rate correction before transcoding, and performs delayed push streaming after transcoding by locally buffering video slices, so that the delay of model inference does not cause frame rate out of control or playback lag on the playback end.

Description

Technical Field

[0001] This invention relates to the field of edge computing, and more specifically, to a streaming method, apparatus, system, computing device, storage medium, and program product. Background Technology

[0002] With the rapid application of edge computing and artificial intelligence (AI) technologies in video analytics, real-time video streaming processing has become an indispensable key technology in various industries such as industrial production, traffic safety, education monitoring, and healthcare. Real-time video surveillance can not only significantly reduce the burden of on-site human supervision, but also achieve automated event identification and rapid response, effectively improving on-site safety and operational efficiency.

[0003] In existing technologies, if a deep learning model is deployed to perform inference tasks such as object detection, semantic segmentation, and behavior recognition on a video stream in real time, the time consumed by inference can easily lead to problems such as frame rate loss, playback stuttering, or audio-visual asynchrony during playback. Summary of the Invention

[0004] The purpose of this invention is to provide a streaming method, apparatus, system, computing device, storage medium, and program product to improve the problems existing in the prior art.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides a streaming method, the method comprising:

[0007] The first thread acquires the captured video stream in real time.

[0008] The first thread calls a deep learning model to process each original video frame in the video stream and adds each processed target video frame to the frame buffer queue.

[0009] The second thread reads the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and stores the multiple videos to be merged obtained in each write cycle into the same folder.

[0010] The first thread sequentially performs frame rate correction and dynamic transcoding on multiple videos to be merged in each folder to obtain a playlist file and multiple video slices in target formats.

[0011] The first thread caches the playlist file and multiple video slices locally each time it receives them, and then pushes them to the playback device with a delay.

[0012] In an optional implementation, the second thread reads the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and stores the multiple videos to be merged obtained in each write cycle into the same folder, including:

[0013] The second thread creates the current folder corresponding to the current write cycle in the preset root directory every preset write cycle, and adds the folder path corresponding to the current folder to the first queue.

[0014] The second thread reads all target video frames from the frame buffer queue at preset intervals during the current write cycle.

[0015] The second thread combines all the target video frames read each time into a video to be merged, names the video to be merged with the read timestamp, stores it in the current folder, and adds the storage path of the video to be merged to the second queue.

[0016] In an optional implementation, the first thread sequentially performs frame rate correction and dynamic transcoding on multiple videos to be merged in each folder to obtain multiple video slices of the target format, including:

[0017] The first thread periodically reads a folder path from the first queue and reads multiple target storage paths that match the folder path from the second queue;

[0018] The first thread reads the target videos to be merged from each target storage path in the preset root directory;

[0019] The first thread calculates the target output frame rate based on all target videos to be merged;

[0020] The first thread merges all target videos to be merged into a single combined video in sequence;

[0021] The first thread performs frame rate correction on the combined videos to be merged based on the target output frame rate, and then performs transcoding to obtain an output video file in the target format.

[0022] The first thread slices the output video file to obtain multiple video slices of the target format, and outputs the playlist file corresponding to the multiple video slices.

[0023] In an optional implementation, the step of the first thread calculating the target output frame rate based on all target videos to be merged includes:

[0024] The first thread counts the number of target video frames in each target video to be merged and then adds them together to get the total number of frames.

[0025] The first thread calculates the ratio of the total number of frames to the write cycle to obtain the target output frame rate.

[0026] In an optional implementation, the name of each folder is a timestamp corresponding to its creation time, and the method further includes:

[0027] The second thread periodically cleans up the data in the preset root directory, and each time it cleans up, it directly deletes each folder whose creation time is before a fixed time period.

[0028] In a second aspect, the present invention provides a streaming system, the streaming system comprising a computing device and an image acquisition device and a playback terminal device communicatively connected to the computing device;

[0029] The image acquisition device is used to acquire video streams in real time;

[0030] The computing device is used to process the video stream in real time according to the streaming method described in the first aspect above, and then stream it to the playback device with a delay.

[0031] Thirdly, the present invention provides a streaming device, the device comprising a first thread and a second thread for asynchronous communication;

[0032] The first thread is used to acquire the captured video stream in real time;

[0033] The first thread is also used to call a deep learning model to process each original video frame in the video stream, and add each processed target video frame to the frame buffer queue;

[0034] The second thread is used to read the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and to store the multiple videos to be merged obtained in each write cycle into the same folder;

[0035] The first thread is also used to sequentially perform frame rate correction and dynamic transcoding on multiple videos to be merged in each folder to obtain a playlist file and multiple video slices in target formats;

[0036] The first thread is also used to perform delayed streaming to the playback device based on the playlist file and multiple video slices obtained each time.

[0037] Fourthly, the present invention provides a computing device, comprising: a memory and a processor, wherein the memory stores a software program, and when the computing device is running, the processor executes the software program to implement the streaming method as described in the first aspect above.

[0038] Fifthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the streaming method described in the first aspect.

[0039] In a sixth aspect, the present invention provides a program product that, when executed by a processor, implements the streaming method described in the foregoing embodiments.

[0040] Compared with the prior art, the embodiments of the present invention provide a streaming method, apparatus, system, computing device, storage medium, and program product. The method is as follows: a first thread acquires the captured video stream in real time and calls a deep learning model to process each original video frame in the video stream, and adds each processed target video frame to a frame buffer queue; a second thread reads the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and stores the multiple videos to be merged obtained in each write cycle into the same folder; simultaneously, the first thread performs frame rate correction and dynamic transcoding processing on the multiple videos to be merged in each folder in sequence to obtain a playlist file and multiple video slices of the target format; the first thread caches the playlist file and multiple video slices obtained each time locally and then pushes them to the playback device with a delay, and the playback device can play the multiple video slices according to the playback order indicated by the playlist file. This invention separates the model inference process from the process of converting the processed target video frames into videos to be merged and stored. Furthermore, the first thread performs frame rate correction before transcoding, and after transcoding, the video slices are locally cached and then pushed with a delay, so that the delay of model inference will not cause frame rate loss or playback stuttering at the playback end. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a streaming method provided in an embodiment of the present invention.

[0043] Figure 2 This invention provides a storage architecture under a preset root directory.

[0044] Figure 3 This is a schematic diagram of the storage contents of the first queue and the second queue provided in an embodiment of the present invention.

[0045] Figure 4This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0049] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0050] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0051] With the rapid application of edge computing and artificial intelligence (AI) technologies in video analytics, real-time video stream processing has become an indispensable key technology in various industries such as industrial production, traffic safety, education monitoring, and healthcare. Real-time video surveillance not only significantly reduces the burden of on-site human supervision but also enables automated event recognition and rapid response, effectively improving on-site safety and operational efficiency. Currently, mainstream intelligent video systems typically employ the following three methods for video acquisition and streaming media forwarding:

[0052] (1) Video acquisition and writing mechanism based on FFmpeg (an open-source, cross-platform, powerful multimedia processing toolkit) or OpenCV (an open-source, cross-platform computer vision and machine learning software library): directly acquire image frames from video streams or cameras, and synchronously write the images or encoded videos to disk;

[0053] (2) Timed batch processing to generate streaming media format: Multiple video clips are merged in batches by script, uniformly transcoded into .ts files, and .m3u8 index files are generated to support HLS (HTTP Live Streaming, an adaptive streaming media transmission protocol) protocol, so as to realize real-time video playback on the Web end;

[0054] (3) Integrate RTMP or media server: The collected video stream is forwarded in RTMP or HLS protocol through streaming servers such as NGINX-RTMP, Wowza, FFserver to achieve “near real-time” remote access and playback.

[0055] While these technologies operate stably in relatively static, non-AI scenarios and can meet general monitoring needs, once combined with real-time computing tasks such as deep learning model inference, they expose several significant technical bottlenecks in industrial applications deployed on edge devices. These bottlenecks are mainly reflected in the following aspects:

[0056] While these technologies operate stably in relatively static, non-AI scenarios and can meet general monitoring needs, they encounter the following problems in scenarios requiring the deployment of deep learning models for real-time inference tasks such as object detection, semantic segmentation, and behavior recognition:

[0057] (i) Inference time consumption leads to uncontrolled video frame rate, causing playback stuttering or audio-visual desynchronization:

[0058] In AI video analytics scenarios, each frame of a video needs to undergo inference operations such as object detection, semantic segmentation, and behavior recognition using deep neural network models (e.g., YOLO, DETR, SegmentAnything). This process is often computationally intensive, and even on GPU devices, it can take as long as 100-300ms per frame. In actual operation, the inference time is dynamic, affected by factors such as video resolution, model complexity, and device load, which prevents the system from outputting images at a fixed frame rate.

[0059] For example, if an edge server deploys multiple camera streams for parallel analysis, and the frame rate of a certain channel is set to 25 FPS, while the average inference time of the AI ​​model is 150ms, the theoretical maximum output frame rate is approximately 6.7 FPS. In this case, if the original frames or the analyzed video are still pushed to the front end for playback using the traditional method, it will cause audio-visual synchronization errors, frequent page buffering, and a very poor user experience. More importantly, the traditional .mp4→.ts→.m3u8 video merging and transcoding process uses a fixed frame rate (e.g., 25 FPS) as the encoding reference time base (PTS) by default. However, the actual output frame rate changes non-linearly during the analysis process. Without frame rate adaptation, this will lead to disordered timestamps in the .ts slices, resulting in problems such as "frame skipping," "still images," and "audio-visual misalignment" on the playback end.

[0060] (ii) Directly pushing MP4 videos causes buffer buildup, making it difficult to cope with network instability or latency fluctuations:

[0061] In traditional video systems, .mp4 files already written to disk are often forwarded directly via HLS or RTMP during video transmission. However, this approach is highly sensitive to network conditions, especially in low-bandwidth, high-packet-loss industrial environments (such as mines, ports, and subways). Directly transmitting high-bitrate MP4 files can cause the following problems:

[0062] 1. Playback stuttering: Because MP4 files are not pre-segmented (chunks), the front-end player needs to read the complete GOP (Group of Pictures) data or a long buffer to decode and play them. When the network fluctuates, noticeable stuttering will occur.

[0063] 2. Streaming Blocking: If a video file is read simultaneously by the transcoding process while it is still being written, it will cause IO conflicts, write failures, or incomplete content.

[0064] 3. Unfriendly to concurrency: Multiple clients simultaneously requesting video segments for a certain period of time will cause disk contention and bandwidth overload;

[0065] In addition, the conventional FFmpeg transcoding process requires manually specifying parameters such as frame rate, bit rate, and resolution, making it difficult to adaptively adjust according to the actual inference output rate of the current model.

[0066] (iii) The separation of multi-threaded write and merging logic is prone to errors and lacks a complete transcoding scheduling strategy:

[0067] Since model inference and video writing are performed asynchronously, without a reasonable thread scheduling and caching mechanism, the following problems can easily occur:

[0068] 1. Video frames are lost due to I / O bottlenecks in the write thread;

[0069] 2. The merging thread started reading before the video segment was completely written, causing transcoding failure;

[0070] 3. Accumulated temporary video files consume a large amount of disk space and may even cause system crashes;

[0071] 4. After the program is interrupted, some .mp4 files are missing or incomplete, and the transcoding process will report an error.

[0072] In actual deployments, these issues can easily trigger system instability, especially in unattended edge device scenarios, which greatly affects the system's robustness and engineering practicality.

[0073] In view of this, the present invention provides a streaming method in which the inference task and the writing task are performed asynchronously by the first thread and the second thread, respectively. Subsequently, the first thread performs frame rate correction and dynamic transcoding processing before delaying the streaming to the playback device, so that the playback device will not experience frame rate loss or stuttering during real-time playback.

[0074] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a streaming method provided in an embodiment of the present invention. This method can be applied to computing devices, such as edge servers, dedicated video servers or video processing devices, personal computing devices, and personal terminal devices. The computing device runs a first thread and a second thread for asynchronous communication, and simultaneously deploys a deep learning model. Taking the connection between the computing device and an image acquisition device as an example, the method includes the following steps:

[0075] S101, The first thread acquires the captured video stream in real time.

[0076] In this embodiment, the image acquisition device acquires the video stream in real time, while the computing device acquires the acquired video stream in real time. A `cap.isOpened()` monitoring mechanism can be set in the first thread to reacquire the stream source if an interruption is detected.

[0077] S102. The first thread calls the deep learning model to process each original video frame in the video stream and adds each processed target video frame to the frame buffer queue.

[0078] In this embodiment, the frame buffer queue is used to store the target video frames after inference. The deep learning model can be used for object detection, semantic segmentation, behavior recognition, etc. For example, assuming that the image acquisition equipment is installed in scenarios such as mines, ports, and construction sites, the deep learning model can be used to detect whether workers are wearing safety equipment (such as helmets and protective vests) correctly. This example is only illustrative, and the present invention does not limit the actual role of the deep learning model.

[0079] S103. The second thread reads the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and stores the multiple videos to be merged obtained in each write cycle into the same folder.

[0080] In this embodiment, each time the second thread reads the cache queue, it reads all the target video frames in the queue and combines them into a single video to be merged. That is, the second thread performs a full read of the cache queue each time, and the multiple videos to be merged obtained in each write cycle are stored sequentially in the same folder on the disk. For example, the write cycle can be 1 minute or 2 minutes, and the format of the videos to be merged can be MP4 (i.e., filenames with the .mp4 extension) or other video formats.

[0081] S104. The first thread sequentially performs frame rate correction and dynamic transcoding on multiple videos to be merged in each folder to obtain a playlist file and multiple video slices in target formats.

[0082] Before performing dynamic transcoding, the first thread of this invention performs frame rate correction to ensure that the generated video slices remain continuous on the timeline at the playback end.

[0083] S105. The first thread caches the playlist file and multiple video slices locally each time and then pushes them to the playback device with a delay.

[0084] In this embodiment, delayed streaming means that the obtained video slices and playlist files are first cached locally, and then the streaming is performed after a certain period of time or when specific conditions are met (such as the playback end actively pulling them).

[0085] The target format can be a .ts file conforming to the HLS protocol, and the playlist file can be a .m3u8 file, used to guide the playback device to play multiple video slices in a specified order. Therefore, during the streaming process, the first thread provides the .m3u8 playlist file and multiple .ts video slices to the playback device via the HTTP protocol. After receiving the playlist file, the playback device requests and plays the multiple video slices sequentially according to the playback order indicated therein, thereby achieving continuous playback of the video content.

[0086] In this invention, the first thread does not process and push the original video stream in real time during streaming. Instead, it performs offline transcoding and slicing based on locally cached videos to be merged, forming video slices and playlist files that can be retrieved on demand by the playback device. This processing method effectively alleviates playback problems caused by unstable frame rates during deep learning model inference, while improving the system's stability and fault tolerance under network fluctuations.

[0087] Optionally, the implementation of step S103 above may include:

[0088] S1031. Every preset write cycle, the second thread creates the current folder corresponding to the current write cycle in the preset root directory and adds the folder path corresponding to the current folder to the first queue.

[0089] In this embodiment, the first queue is used to cache the folder path of each folder created periodically by the second thread. Taking a write cycle of 1 minute as an example, the second thread will create a folder in the preset root directory of the disk every minute, name the folder with the creation time, and add the folder path to the first queue.

[0090] S1032. During the current write cycle, the second thread reads all target video frames from the frame buffer queue at preset intervals.

[0091] S1033. The second thread combines all the target video frames read each time into a video to be merged, names the video to be merged with the read timestamp, stores it in the current folder, and adds the storage path of the video to be merged to the second queue.

[0092] In this embodiment, the second queue is used to cache the storage path of each video to be merged that the second thread writes to disk. Taking a write cycle of 1 minute and a preset duration of 1 second as an example, within the 1-minute event, the second thread will read the frame cache queue once per second. After each read, it will combine all the target video frames read into a video to be merged, name the video to be merged with the read timestamp, store it in the current folder, and add the storage path of the video to be merged to the second queue.

[0093] For example, assuming a write cycle of 1 minute, a preset duration of 1 second, and a preset root directory of / cache / device01 / (device01 represents the device number of the image acquisition device), such as... Figure 2 The folder created by the second thread at 14:00:00 on August 5, 2025, will be named "202508051400". If the first thread's video stream processing continues uninterrupted within that minute, this folder can contain up to 60 videos to be merged, stored by the second thread. Even if the first thread experiences a brief interruption in pulling the video stream, it can resume quickly. Figure 2 Correspondingly, the first queue and the second queue are as follows: Figure 3 As shown.

[0094] It should be noted that, Figure 2 and Figure 3 The examples shown are for illustrative purposes only and do not represent a limitation on the size of the first and second queues.

[0095] This thread-isolated architecture ensures smooth operation of the main inference process while preventing video frame drops or blocking caused by fluctuations in model processing time (such as inference times exceeding 50ms for some complex frames). Furthermore, the second thread has breakpoint recovery capabilities; for example, it writes one video to be merged per second and creates a new folder for writing every minute, ensuring that even if the system crashes, existing videos to be merged remain intact and usable, greatly improving system robustness and disaster recovery capabilities.

[0096] The second thread manages the folder path and the storage path of the video to be merged through the first queue and the second queue respectively, ensuring that the first thread can efficiently locate the storage location of the video to be merged from the preset root directory without relying on the database index, which greatly improves the transcoding efficiency.

[0097] In AI scenarios, due to the significant differences in complexity between different frames for inference models such as YOLO and Transformer, the model processing time fluctuates significantly, resulting in the actual output frame rate of the inference model being much lower than the standard frame rate. However, traditional video inference systems generally use a fixed standard frame rate (such as 25fps or 15fps) as the transcoding parameter for direct transcoding during the transcoding stage, which may lead to uneven timestamps in HLS slices, causing stuttering or accelerated / decelerated playback on the playback end.

[0098] Therefore, this invention performs frame rate correction calculation for each folder before transcoding, and performs transcoding based on this target output frame rate to avoid screen jumps or playback stuttering caused by traditional fixed frame rate direct transcoding. That is, the sub-steps of step S104 above may include:

[0099] S1041. The first thread periodically reads a folder path from the first queue and reads multiple target storage paths that match the folder path from the second queue.

[0100] In this embodiment, the first thread can read a folder path from the first queue every write cycle (e.g., 1 minute). For each folder path read, multiple target storage paths matching the folder path are then read from the second queue.

[0101] S1042. The first thread reads the target videos to be merged from the preset root directory corresponding to each target storage path.

[0102] For example, taking the video to be merged as an MP4 format, combined with Figure 3 If the first thread starts from Figure 3 The first thread reads the folder path " / cache / device01 / 202508051400 / " from the first queue. Then the first thread will... Figure 3 The second queue reads multiple target storage paths: / cache / device01 / 202508051400 / 1722866400.mp4 to / cache / device01 / 202508051400 / 1722866459.mp4. Then, it reads all target MP4 videos from the preset root directory, which are 1722866400.mp4 to 1722866459.mp4. It should be noted that this example is for illustrative purposes only and is not intended to be limiting.

[0103] S1043, The first thread calculates the target output frame rate based on all target videos to be merged.

[0104] In this embodiment, the target output frame rate represents the average frame rate of each target video frame in all target videos to be merged in the inference output of the first thread. The target output frame rate can be calculated as follows: (1) the first thread counts the number of target video frames in each target video to be merged and adds them together to obtain the total number of frames; (2) the first thread calculates the ratio of the total number of frames to the writing cycle to obtain the target output frame rate.

[0105] S1044. The first thread merges all target videos to be merged into a single combined video in sequence.

[0106] S1045. The first thread performs frame rate correction on the combined videos based on the target output frame rate and then performs transcoding to obtain the output video file in the target format.

[0107] S1046. The first thread slices the output video file to obtain multiple video slices of the target format, and outputs the playlist file corresponding to the multiple video slices.

[0108] To address the issues of unstable frame rates and conflicts with real-time network transmission during inference, this invention employs an offline merging + batch transcoding strategy: For each read of all target videos to be merged, a three-stage process of "merging (concat) → transcoding → HLS slicing" can be executed using ffmpeg, where the slicing parameters can be configured as follows:

[0109] (1) hls_time 10: Each .ts segment is 10 seconds long;

[0110] (2) hls_list_size 6: retains a maximum of 6 segments to achieve soft real-time;

[0111] (3) hls_flags delete_segments+append_list+program_date_time: ensures that the playlist is dynamically updated and deletes expired segments to reduce transmission pressure;

[0112] (4) setpts: dynamically set to the currently calculated target output frame rate.

[0113] As can be seen from step S105, the playlist file (.m3u8 file) and multiple video slices (.ts file) generated by the first thread each time can be cached locally in the hls / subdirectory for use by the web player to achieve a stable and smooth playback effect.

[0114] Compared to traditional direct RTMP push solutions, the streaming method of this invention has stronger network adaptability. Through local caching and delayed streaming mechanism, the playback device has the following advantages when playing video slices:

[0115] Even when the network is unstable, the local .ts cache segment of the most recent 10 to 60 seconds can still be accessed to avoid "black screen" and "screen lag" phenomena;

[0116] The system supports delayed retransmission: Since the video segments are all local independent fragments, even if there is a brief network outage between the computing device and the playback device, transcoding and slicing can still be performed. When the network is restored, the playback device can automatically pull back the missing segments to achieve complete frame replacement.

[0117] Optionally, to avoid accumulating a large number of historical cached videos to be merged in the preset root directory, the second thread can periodically (e.g., every 1 or 2 minutes) clean up the data in the preset root directory, and each time it cleans up, it will directly delete each folder whose creation time is before a fixed duration (e.g., 5 or 10 minutes).

[0118] Optionally, the first thread can include exception handling and error logging for all transcoding processes, facilitating quick troubleshooting and service restart.

[0119] This invention also provides a streaming system, which includes a computing device and an image acquisition device and a playback device communicatively connected to the computing device. The image acquisition device is used to acquire video streams in real time; the computing device is used to process the video streams in real time according to the above-described streaming method and then push the streams to the playback device with a delay; the playback device pulls multiple corresponding video slices based on the playlist file pushed by the computing device for playback.

[0120] In order to perform the corresponding steps in the above method embodiments and various possible implementations, the present invention also provides a streaming device, which includes a first thread and a second thread for asynchronous communication.

[0121] The first thread is used to acquire the captured video stream in real time;

[0122] The first thread is also used to call the deep learning model to process each original video frame in the video stream and add each processed target video frame to the frame buffer queue.

[0123] The second thread is used to read the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and to store the multiple videos to be merged obtained in each write cycle into the same folder.

[0124] The first thread is also used to sequentially perform frame rate correction and dynamic transcoding on multiple videos to be merged in each folder to obtain a playlist file and multiple video slices in target formats;

[0125] The first thread is also used to perform delayed streaming to the playback device based on the playlist file and multiple video slices obtained each time.

[0126] Optionally, the second thread can be used to: create a current folder corresponding to the current writing cycle in a preset root directory every preset writing cycle, and add the folder path corresponding to the current folder to the first queue; within the current writing cycle, read all target video frames in the frame buffer queue every preset duration; combine all the target video frames read each time into a video to be merged, name the video to be merged with the reading timestamp, store it in the current folder, and add the storage path of the video to be merged to the second queue.

[0127] Optionally, the first thread can be used to: periodically read a folder path from a first queue and read multiple target storage paths matching that folder path from a second queue; read the target videos to be merged corresponding to each target storage path from a preset root directory; calculate the target output frame rate based on all target videos to be merged; merge all target videos to be merged sequentially into a combined video to be merged; perform frame rate correction on the combined video to be merged based on the target output frame rate and then perform transcoding to obtain an output video file in the target format; slice the output video file to obtain multiple video slices in the target format, and output a playlist file corresponding to the multiple video slices.

[0128] Optionally, the first thread can also be used to: count the number of target video frames in each target video to be merged and add them together to get the total number of frames; calculate the ratio of the total number of frames to the write cycle to get the target output frame rate.

[0129] Optionally, the name of each folder is a timestamp corresponding to its creation time. The second thread can also be used to: periodically clean up data in a preset root directory, and delete each folder whose creation time is before a fixed time period during each cleanup.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the first and second threads in the above-described streaming device can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0131] Please see Figure 4 , Figure 4 This is a schematic diagram of a computing device according to an embodiment of the present invention. The computing device 300 includes a processor 310, a memory 320, and a bus 330, with the processor 310 connected to the memory 320 via the bus 330.

[0132] The memory 320 can be used to store software programs or firmware, for example, the software program or firmware corresponding to the aforementioned streaming device. The processor 310 executes various functional applications and data processing by running the software program stored in the memory 320 to implement the streaming method provided in the embodiments of the present invention.

[0133] The memory 320 may be, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0134] The processor 310 can be an integrated circuit chip with signal processing capabilities, capable of executing software programs, such as the software program corresponding to the aforementioned streaming device. The processor 310 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), SoC (System on Chip), etc.; it can also be: DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0135] Understandable. Figure 4 The structure shown is for illustrative purposes only; the computing device 300 may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0136] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the streaming method disclosed in the above embodiments. The computer-readable storage medium can be, but is not limited to, various media capable of storing program code, such as a USB flash drive, external hard drive, ROM, RAM, PROM, EPROM, EEPROM, FLASH disk, or optical disk.

[0137] This invention also provides a program product that, when executed by a processor, implements the above-described streaming method.

[0138] In summary, the embodiments of the present invention provide a streaming method, apparatus, system, computing device, storage medium, and program product. The method comprises: a first thread acquiring the captured video stream in real time and calling a deep learning model to process each original video frame in the video stream, and adding each processed target video frame to a frame buffer queue; a second thread reading the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and storing the multiple videos to be merged obtained in each write cycle in the same folder; simultaneously, the first thread sequentially performs frame rate correction and dynamic transcoding processing on the multiple videos to be merged in each folder to obtain a playlist file and multiple video slices of the target format; the first thread locally caches the obtained playlist file and multiple video slices each time and then pushes them to the playback device with a delay, so that the playback device plays the multiple video slices according to the playback order indicated by the playlist file. This invention separates the model inference process from the process of converting the processed target video frames into videos to be merged and stored. Furthermore, the first thread performs frame rate correction before transcoding, and after transcoding, the video slices are locally cached and then pushed with a delay, so that the delay of model inference will not cause frame rate loss or playback stuttering at the playback end.

[0139] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A plug flow method characterized by, The method comprises: The first thread acquires the collected video stream in real time; The first thread calls a deep learning model to process each raw video frame in the video stream, and adds each processed target video frame to a frame buffer queue; The second thread reads the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and stores the multiple videos to be merged obtained in each write cycle to the same folder under a preset root directory; The first thread sequentially performs frame rate correction and dynamic transcoding processing on the multiple videos to be merged in each folder to obtain a playlist file and multiple video slices in a target format; The first thread locally caches the playlist file and the multiple video slices obtained each time and then pushes them to a playback terminal device with a delay; The name of each folder under the preset root directory is a timestamp corresponding to its creation time, and the method further comprises that the second thread periodically cleans up data in the preset root directory, and each time the data is cleaned up, each folder whose creation time is located before a fixed time length is directly deleted.

2. The plug flow method of claim 1, wherein, The step of the second thread reading the frame buffer queue multiple times in each write cycle to obtain multiple videos to be merged, and storing the multiple videos to be merged obtained in each write cycle to the same folder comprises: The second thread creates a current folder corresponding to the current write cycle under the preset root directory every preset write cycle, and adds a folder path corresponding to the current folder to a first queue; The second thread reads all target video frames in the frame buffer queue every preset time length within the current write cycle; The second thread combines all target video frames read each time into a video to be merged, names the video to be merged with a read timestamp, and stores the video to be merged in the current folder, and simultaneously adds a storage path of the video to be merged to a second queue.

3. The plug flow method of claim 2, wherein, The step of the first thread sequentially performing frame rate correction and dynamic transcoding processing on the multiple videos to be merged in each folder to obtain multiple video slices in a target format comprises: The first thread reads a folder path from the first queue and multiple target storage paths matched with the folder path from the second queue; The first thread reads target videos to be merged corresponding to each target storage path from the preset root directory; The first thread calculates a target output frame rate based on all target videos to be merged; The first thread merges all target videos to be merged into a combined video to be merged; The first thread performs transcoding processing on the combined video to be merged after frame rate correction based on the target output frame rate to obtain an output video file in a target format; The first thread slices the output video file to obtain multiple video slices in a target format, and outputs a playlist file corresponding to the multiple video slices.

4. The plug flow method of claim 3, wherein, The step of the first thread calculating a target output frame rate based on all target videos to be merged comprises: The first thread adds the number of target video frames in each target video to be merged to obtain a total frame number; The first thread calculates a ratio of the total frame number and the write period to obtain the target output frame rate.

5. A plug flow system characterized by, The push streaming system comprises a computing device, and an image acquisition device and a playback terminal device connected in communication with the computing device. The image acquisition device is configured to acquire a video stream in real time. The computing device is configured to perform real-time processing on the video stream according to the push streaming method of any one of claims 1-4 and perform delayed push streaming to the playback terminal device.

6. A push flow device characterized by, The device comprises a first thread and a second thread of asynchronous communication; The first thread is configured to acquire the acquired video stream in real time; The first thread is further configured to call a deep learning model to process each raw video frame in the video stream and add each target video frame processed to a frame buffer queue; The second thread is configured to read the frame buffer queue multiple times in each write period to obtain multiple videos to be merged, and store the multiple videos to be merged obtained in each write period to a same folder under a preset root directory; The first thread is further configured to perform frame rate correction and dynamic transcoding processing on the multiple videos to be merged in each folder in sequence to obtain a playlist file and multiple video slices of a target format; The first thread is further configured to perform delayed push streaming to a playback terminal device based on the playlist file and the multiple video slices obtained each time; The name of each folder under the preset root directory is a timestamp corresponding to a creation time of the folder, and the second thread is further configured to clean data of the preset root directory at a fixed time interval, and directly delete each folder whose creation time is located before a fixed time length each time the data is cleaned.

7. A computing device, comprising: Comprise: a memory and a processor, the memory stores a software program, when the computing device runs, the processor executes the software program to realize the push streaming method of any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the push streaming method of any one of claims 1-4.

9. A program product, characterized by The program product is executed by the processor to realize the push streaming method of any one of claims 1-4.

Images