Method and system for simultaneous multi-stream video playback on same screen

Abstract

The present application relates to a method and system for simultaneous multi-stream video playback on a same screen and belongs to the technical field of video playback. The method comprises: establishing a stream pulling library for pulling data streams according to calling, the stream pulling library realizing a thread pool, and the thread pool maintaining a plurality of threads for multi-stream pulling; when the stream pulling library is called for stream pulling and the stream pulling library receives a corresponding unused stream identifier, the stream pulling library using an idle thread in the thread pool to pull the stream, and, according to the received stream identifier, marking the thread; when the stream pulling library has retrieved the data stream, providing a corresponding data stream by means of callback; and, according to the stream identifier, decoding and playing a corresponding video, wherein when being transmitted into the stream pulling library, the unused stream identifier is updated to "in use". The present application uses stream identifiers to establish a one-to-one correspondence relationship of threads, stream pulling, callback, and decoding and playback, so that the number of streams of videos played at a same moment on a same screen and the position of the videos displayed on the same screen can be flexibly adjusted. The present application substantively relates to multi-stream pulling and multi-stream playback, and requires no merging of multiple video streams.

Description

Method and system for simultaneous same-screen playback of multiple videos

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410150342.5 filed with the Chinese Patent Office on February 2, 2024, entitled “A method and system for simultaneous playback of multiple videos on the same screen”. The entire contents of the above application are incorporated by reference into this application. Technical Field

[0003] The present application relates to a method and system for simultaneously playing multiple videos on the same screen, belonging to the field of video playback technology. Background Art

[0004] Currently, smart cameras for home use on the market display videos to users in a single channel, whether in an APP (Application, application or mobile software) or in smart hardware devices. That is, users can only view the video status of one camera at a time, which can also be understood as only one video can be played on the same display screen at the same time.

[0005] Although the hardware system of smart screen devices, as intelligent hardware devices, can connect to multiple cameras, they still only display one video channel, as is common on the market, so the device's large screen is not fully utilized. Displaying only one camera video channel cannot meet the demand for simultaneous playback of camera videos from multiple rooms in a home, nor can it better utilize the large screen. Therefore, a solution is needed to enable the simultaneous display of multiple camera channels on the app or device.

[0006] Patent document CN116248950A discloses an interactive video multi-screen presentation method and device, storage medium, and terminal. The method includes: obtaining multiple videos to be merged and decoding the videos to be merged into corresponding video data streams to be merged; merging the video data streams to be merged into a composite video data stream according to a conventional layout method; obtaining screen layout information of the composite video data stream; generating a composite screen video based on the composite video data stream and the screen layout information; and streaming the composite screen video to a cloud server platform. After the viewer obtains the composite screen video from the cloud server platform, it can adjust the screen display layout of the composite video data stream based on the screen layout information. The screen layout information includes the position information of each video screen in the screen display of the composite video data stream. This method is essentially a single-channel stream pull. Although it can display multiple videos on the same screen simultaneously, the merging method it adopts first increases the corresponding cost. Second, the screen layout of the multiple videos is fixed when merging and then displaying. When the current display screen needs to be rearranged, it must be re-arranged, re-merged, and then displayed, which is a cumbersome step.

[0007] Summary of the Invention

[0008] The purpose of this application includes, for example, providing a method and system for simultaneous playback of multiple videos on the same screen, so as to solve the problem of increased costs caused by the existing method of merging multiple videos for simultaneous playback of multiple videos on the same screen.

[0009] The present application provides a method for simultaneously playing multiple videos on the same screen, including: establishing a stream pulling library for pulling data streams according to calls, the stream pulling library implementing a thread pool, and the thread pool maintaining multiple threads for multi-channel stream pulling; when the stream pulling library is called to pull a stream and the stream pulling library receives a corresponding unused channel identifier, the stream pulling library takes an idle thread from the thread pool to pull the stream and marks the thread according to the received channel identifier; after the stream pulling library pulls the data stream, it provides the corresponding data stream through a callback; the corresponding video is decoded and played according to the channel identifier; the unused channel identifier is updated to used when it is passed to the stream pulling library.

[0010] Optionally, a multi-channel playback manager is established to manage the usage status of the channel identifiers of the identification threads, so that unused channel identifiers are transmitted to the stream pulling library through the multi-channel playback manager, and the unused channel identifiers transmitted to the stream pulling library are updated to used ones.

[0011] Optionally, a data manager is established to receive the data stream provided by the stream pulling library through a callback data management pool in the data manager, and decode and play the corresponding video through a player in the data manager according to the channel identifier.

[0012] Optionally, the unused route identifiers are obtained by searching based on the total route identifiers and the used route identifiers.

[0013] Optionally, the unused road identifiers are obtained by cyclically searching the total road identifiers until the first unused road identifier is found and returned, so as to use the first unused road identifier as the currently unused road identifier.

[0014] Optionally, a way number array is further established to store the way number identifier and an information field for indicating whether the way number identifier is used.

[0015] Optionally, the total number of pull-down streams at that moment is also limited in the following manner: when the total number of used channel identifiers reaches a set value, the callback cannot continue.

[0016] Optionally, when the total number of used path identifiers reaches a set value, the used path identifiers find a corresponding release channel through a release interface, so that the callback can continue.

[0017] Optionally, an interface for external calls is also provided.

[0018] Optionally, operations such as screenshots or screen recording are performed when multiple videos are played simultaneously on the same screen through the externally called interface.

[0019] Optionally, the stream pulling library is directly connected to the camera to obtain the data stream by pulling the stream.

[0020] Optionally, the stream pulling library obtains the data stream through a stream pushing service of an Internet cloud stream pulling address.

[0021] Optionally, the multi-channel playback manager provides a stream pulling interface externally and maintains a globally unique channel number array internally.

[0022] Optionally, the multi-channel playback manager limits the number of channels to a maximum of 16.

[0023] The present application also provides a system for simultaneous playback of multiple videos on the same screen, including a processor that executes instructions to implement the above-mentioned method for simultaneous playback of multiple videos on the same screen.

[0024] The beneficial effects of this application include, for example:

[0025] Different from the existing method of combining multiple videos to display multiple videos on the same screen at the same time, the present application provides a new method and system for simultaneous playback of multiple videos on the same screen. The thread pool is realized by establishing a stream pulling library. When the stream pulling library is called to pull the stream and the stream pulling library receives the corresponding unused channel identifier, the stream pulling library takes an idle thread from the thread pool to pull the stream and marks the thread according to the received channel identifier; after the stream pulling library pulls the data stream, it provides the corresponding data stream through a callback; decodes and plays the corresponding video according to the channel identifier; the unused channel identifier is updated to used when it is passed to the stream pulling library; wherein, the stream pulling library pulls the data stream according to the call; the thread pool maintains multiple threads for multi-channel stream pulling. The present application establishes a one-to-one correspondence between threads, stream pulling, callbacks, and decoding and playback through channel identifiers, which can flexibly adjust the channels of videos played on the same screen at the same time and the positions displayed on the same screen. The essence of this application is multi-channel streaming and multi-channel playback. There is no need to merge multiple videos, that is, there is no need to merge multiple data streams. Compared with the existing simultaneous display of multiple videos on the same screen, there is no need to merge them, which saves the relevant costs of merging and effectively avoids the cost increase caused by merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flowchart of a process for simultaneously playing multiple videos on the same screen according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0028] In order to solve the problems in the background technology, different from the existing method of merging first and then displaying to achieve the simultaneous display of multiple videos on the same screen, the present application provides a new method and system for the simultaneous playback of multiple videos on the same screen, specifically by establishing a stream pulling library, which implements a thread pool; when the stream pulling library is called to pull the stream and the stream pulling library receives the corresponding unused channel identifier, the stream pulling library takes an idle thread from the thread pool to pull the stream, and marks the thread according to the received channel identifier; after the stream pulling library pulls the data stream, it provides the corresponding data stream through a callback; decodes and plays the corresponding video according to the channel identifier; the unused channel identifier is updated to used when it is passed to the stream pulling library. Among them, the stream pulling library pulls the data stream according to the call; the thread pool maintains multiple threads for multi-channel stream pulling. The present application establishes a one-to-one correspondence between threads, stream pulling, callbacks, and decoding and playback through channel identifiers, which can flexibly adjust the channels of videos played on the same screen at the same time and the positions displayed on the same screen. The essence of this application is multi-channel streaming and multi-channel playback. There is no need to merge multiple videos, that is, there is no need to merge multiple data streams. Compared with the existing simultaneous display of multiple videos on the same screen, there is no need for merging, which saves the related costs of merging and effectively avoids the cost increase caused by merging.

[0029] This application provides an embodiment of a method for simultaneously playing multiple videos on the same screen:

[0030] A method for simultaneously playing multiple channels of video on the same screen includes establishing a stream pull library that implements a thread pool. When the library is called to pull a stream and receives a corresponding unused channel identifier, the library selects an idle thread from the thread pool to pull the stream and identifies the thread based on the received channel identifier. After the library pulls the data stream, it provides the corresponding data stream via a callback. The corresponding video is decoded and played based on the channel identifier. The library pulls the data stream based on the call; the thread pool maintains multiple threads for pulling multiple channels; and unused channel identifiers are updated to used when they are passed to the library.

[0031] In one embodiment, a multi-channel playback manager is established to manage the usage status of channel identifiers. The multi-channel playback manager transmits unused channel identifiers to the stream pull library and updates the unused channel identifiers transmitted to the stream pull library to be used. The channel identifiers are used to identify threads, establishing a one-to-one correspondence between threads and stream pulls.

[0032] In one embodiment, a data manager is established to receive the data stream provided by the stream pulling library through a callback data management pool in the data manager, and decode and play the corresponding video through a player in the data manager according to the channel identifier.

[0033] A callback refers to a callback function, a bidirectional calling pattern whereby the callee's interface also calls the other party's interface when it is called. A callback function is a function or procedure implemented by the caller for the callee's use. In this embodiment, the stream pull library acts as the callee, and the data manager acts as the caller. The data manager invokes the stream pull library's multi-channel data streams through the callback function, which then decodes and plays the data streams provided by the stream pull library.

[0034] In one embodiment, the unused path identifiers are obtained by searching based on the total path identifiers and the used path identifiers.

[0035] In one embodiment, the unused path identifiers are obtained by cyclically searching the total path identifiers until the first unused path identifier is found and returned, so that the first unused path identifier is used as the current unused path identifier.

[0036] In one embodiment, a way number array is further established to store the way number identifier and an information field for indicating whether the way number identifier is used.

[0037] In one embodiment, the total number of pull-down streams at that moment is further limited in the following manner: when the total number of used channel identifiers reaches a set value, the callback cannot continue.

[0038] In one embodiment, when the total number of used path identifiers reaches a set value, the used path identifiers find a corresponding release channel through a release interface, so that the callback can continue.

[0039] In one embodiment, an interface for external calls is also provided, that is, an interface for external calls is provided through a streaming playback library formed by a streaming library, a multi-channel playback manager and a data manager, so as to add additional functions through the external call interface, such as screenshots or screen recording when multiple videos are played simultaneously on the same screen.

[0040] In an embodiment of the present application, as shown in FIG1 , a stream playback library is composed of a stream pull library, a multi-channel playback manager (also called a multi-channel stream pull playback manager), and a data manager. The stream pull library obtains data streams by pulling or pushing streams. The data stream here refers to the audio and video data streams acquired by the camera. When the stream pull library is directly connected to the camera, the stream pull library obtains the data stream by pulling streams; of course, the stream pull library can also obtain the data stream through the push stream service of the Internet cloud stream pull address.

[0041] The streaming library implements a thread pool that maintains multiple threads for streaming. It uses identifiers to establish a one-to-one correspondence between threads and channels: thread 1 for streaming corresponds to channel 1, thread 2 for streaming corresponds to channel 2, and so on. Thread N for streaming corresponds to channel N. Given that streaming is asynchronous, identifiers are used to establish a one-to-one correspondence between multiple threads, assigning callbacks to the corresponding streaming channels.

[0042] Among them, there is a bridging process between the stream pulling library as a bottom-level library using the bottom-level language and the multi-channel playback manager and data manager using the upper-level language, that is, the bridging between the stream pulling library and the multi-channel playback manager is realized through the bridging interface, and the bridging between the stream pulling library and the data manager is realized through the bridging interface. The bottom-level language is the bottom-level programming language, including assembly language, C language, C++ language, etc.; the upper-level language is the high-level language, including JAVA, Python, etc. The stream pulling library of this application adds a multi-channel stream pulling thread manager on the basis of the existing single-channel stream pulling library to support multi-channel stream pulling.

[0043] The multi-channel playback manager manages the identifier library, which is a globally unique channel array. Identifiers are divided into used and unused based on whether they are passed to the stream pull library. Only unused channel identifiers are passed to the stream pull library, establishing a one-to-one correspondence between threads and stream pulls. That is, an idle thread and the channel to be pulled are established with an unused channel identifier that is different from other ones. When an unused channel identifier is passed to the stream pull library, it is synchronously updated to used. The globally unique channel array is managed by the channel management pool in the multi-channel playback manager. Specifically, unused channel identifiers are automatically searched and allocated based on the total channel identifiers and the currently used channel identifiers. That is, the globally unique channel array stores used channel identifiers. When an unused channel identifier is obtained, it will cyclically search to see if the channel identifier has been used until the first unused channel identifier is found. The first unused channel identifier here is used as the unused channel identifier to be passed to the stream pull library in the next step. The channels in Figure 1 are channel identifiers, and the total channel identifier is the total number of channels in Figure 1. By adding a multi-channel playback manager (multi-channel player or multi-channel manager), the usage and unused status of multiple channels can be managed, and unused channels can be provided to the outside world for streaming. The multi-channel player provides a streaming interface to the outside world, and internally maintains a globally unique channel array, storing key information such as channel identifiers and callback objects. It automatically loops to retrieve unused channels and calls the streaming library streaming interface.

[0044] The multi-channel streaming playback manager, which originally only supported single-channel streaming playback, now supports multi-channel identification management. A new globally unique channel number array is added to store channel number identification and information fields such as whether the channel is in use.

[0045] When the multi-channel manager calls the stream pull interface externally, it loops through all channels, finds an unused channel ID, and then instantiates the stream pull interface, passing in the corresponding ID. It also updates the channel ID to a used ID and stores the corresponding callback object (CallBack) for subsequent data callbacks.

[0046] The multi-channel manager limits the maximum number of channels to 16, keeping the stream library's memory usage and performance low and reducing risks. When the multi-channel identifier reaches 16, the callback to the upper layer cannot continue, and an external release interface must be called. The external release interface does not need to worry about the corresponding channel number. The multi-channel manager can find the corresponding channel to be released based on the channel number identifier.

[0047] The data manager finds the corresponding channel according to the channel identifier callback, obtains the corresponding data stream through the callback, and performs subsequent decoding and playback. The callback data stream is pulled by the callback data management pool in the data manager, and the decoding and playback can be performed according to the decoding player inside the data manager, or according to the external calling interface provided by the stream playback library. The multi-channel audio decoders are multi-channel and parallel without interfering with each other, and the multi-channel video decoders are multi-channel and parallel without interfering with each other. The callback data management pool calls back the corresponding audio and video data according to the channel identifier, that is, the globally unique channel array finds the corresponding channel according to the channel identifier callback, obtains the corresponding data, and distributes it one by one. The globally unique channel array matches the channel identifier stored in the channel management pool, that is, the unique channel identifier that is different from other channel identifiers, with the channel identifier called back by the stream playback library to the callback data management pool, and distributes the data stream. The data manager, based on the data stream and identifier received from the stream pull library callback, finds the corresponding callback object based on the globally unique channel array, calls back the corresponding data stream, and stores it in the decoding queue. The decoder reads the queue, decodes the audio and video, and plays them simultaneously. The data stream manager of this application adds multi-channel data callbacks, which can correctly handle the decoding of the corresponding channels of video and audio, and throw the data to the system for decoding and playback; the decoding player can support the simultaneous playback of multiple channels of audio and video.

[0048] The data manager, when the stream library gets the data stream from the streaming service, calls back to the upper layer according to the thread identifier to the corresponding data stream and identifier. The data manager uses the globally unique path array and then finds the corresponding callback object (CallBack) provided to the outside according to the callback identifier. The callback object then calls back this data stream to the outside and calls back to the audio and video decoder for decoding and playback.

[0049] The decoder instantiates audio and video decoding threads. When the data manager calls back to the data stream, it stores it in the data stream queue. The decoder continuously reads the data stream from this queue, parses the audio and video according to the data stream, and inserts the parsed audio into the system player for playback. The parsed video stream is inserted into the system video player for playback. The system automatically handles this internally, so audio playback can still be performed simultaneously without any processing.

[0050] As shown in Figure 1, the smiling face in Figure 1 represents the user, and the user can implement other functions based on the interface of the streaming playback library for external calls, such as screenshots or screen recording. Among them, the interface of the streaming playback library for external calls refers to the multi-channel playback data manager's start playback interface and the data manager's data callback interface.

[0051] This application adopts the idea of ​​unified encapsulation, that is, when the encapsulated streaming playback library provides the streaming playback interface to the outside world, the outside world does not need to consider which channel it is, and directly uses the streaming playback API (Application Programming Interface, application programming interface, interface or channel). The streaming playback library automatically manages the number of channels. When the maximum number of channels is reached (in principle, there can be many channels, no upper limit, but in order to meet the performance requirements of the streaming library memory and the network, it is limited to 16 channels in this embodiment. Of course, the maximum number of channels here can also be set according to actual conditions), if the maximum number of channels is reached, a callback is made to the upper layer to prevent it from continuing to play, and the useless channels need to be released externally. Among them, the API is essentially a pre-defined function that is responsible for the communication between a program and other software.

[0052] The streaming library implements a thread pool that maintains multiple threads for streaming and uses identifiers to match threads and routes one to one. Because streaming is asynchronous, different streaming data must be called back to the corresponding routes under multiple threads, so an identifier is needed for one-to-one matching.

[0053] When an external call is made to the streaming library to pull a stream and the corresponding channel identifier is passed in from the multi-channel playback manager, the library retrieves a thread from the thread pool and identifies this thread with the passed channel identifier. After pulling the data stream from the streaming server, the library then uses this thread identifier to make a callback to provide the corresponding data stream, that is, to provide the corresponding data stream to the data manager. The streaming library maintains a thread pool, which is a thread pool that maintains multiple threads, each of which acts as a channel for pulling streams. Streaming media: multimedia data is continuously sent from a streaming provider to a client, and the client can start playing the multimedia without downloading the entire multimedia data to the server. Pulling stream: the process of pulling live content from a server using a specified address; the process of obtaining a video stream from a server and playing it locally is also called playback. Pushing stream: the process of pushing live content to a server.

[0054] This application provides an embodiment of a system for simultaneously playing multiple videos on the same screen:

[0055] A system for simultaneously playing multiple videos on the same screen includes a processor that executes instructions to implement the aforementioned method for simultaneously playing multiple videos on the same screen. The method for simultaneously playing multiple videos on the same screen has been described in detail in an embodiment of a method for simultaneously playing multiple videos on the same screen and will not be repeated here. Industrial Applicability

[0056] The present application provides a method and system for simultaneous playback of multiple videos on the same screen, which can realize multi-channel streaming and multi-channel playback without the need to merge multiple videos, greatly reducing the cost of playing multiple videos on the same screen at the same time.

[0057] In addition, it can be understood that the method and system for simultaneous playback of multiple videos on the same screen of the present application are reproducible and can be widely used in the field of video playback technology.

Claims

1. A method for simultaneously playing multiple videos on the same screen, characterized in that: A stream pulling library is established for pulling data streams according to calls, and the stream pulling library implements a thread pool, which maintains multiple threads for multi-channel stream pulling. When the stream pulling library is called to pull a stream and receives a corresponding unused channel identifier, the stream pulling library takes an idle thread from the thread pool and marks the thread according to the received channel identifier. After the stream pulling library pulls the data stream, it provides the corresponding data stream through a callback. Decode and play the corresponding video according to the channel number identifier; The unused channel identifiers are updated to used when they are passed to the stream pulling library.

2. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: A multi-channel playback manager is established for managing the usage status of the channel identifiers of the identification threads, so that unused channel identifiers are transmitted to the stream pulling library through the multi-channel playback manager, and the unused channel identifiers transmitted to the stream pulling library are updated to used ones.

3. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: A data manager is established to receive the data stream provided by the stream pulling library through a callback data management pool in the data manager, and decode and play the corresponding video through a player in the data manager according to the path identifier.

4. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: The unused route identifiers are obtained by searching based on the total route identifiers and the used route identifiers.

5. The method for simultaneously playing multiple videos on the same screen according to claim 4, characterized in that: The unused route identifiers are obtained by cyclically searching the total route identifiers until the first unused route identifier is found and returned, so as to use the first unused route identifier as the currently unused route identifier.

6. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: A way number array is also established to store the way number identifier and an information field for indicating whether the way number identifier is used.

7. The method for simultaneously playing multiple videos on the same screen according to claim 1, characterized in that: The total number of pull-down streams at that moment is also limited in the following way: when the total number of used stream identifiers reaches the set value, the callback cannot continue.

8. The method for simultaneously playing multiple videos on the same screen according to claim 7, characterized in that: When the total number of used path identifiers reaches the set value, the used path identifiers find the corresponding release channel through the release interface so that the callback can continue.

9. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: It also provides an interface for external calls.

10. The method for simultaneously playing multiple videos on the same screen according to claim 9, characterized in that: Through the externally called interface, operations such as screenshots or screen recording can be performed when multiple videos are played simultaneously on the same screen.

11. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: The stream pulling library is directly connected to the camera to obtain the data stream by pulling the stream.

12. The method for simultaneously playing multiple videos on the same screen according to claim 1, wherein: The stream pulling library obtains the data stream through the push stream service of the stream pulling address on the Internet cloud.

13. The method for simultaneously playing multiple videos on the same screen according to claim 2, wherein: The multi-channel playback manager provides a stream pulling interface externally and maintains a globally unique channel number array internally.

14. The method for simultaneously playing multiple videos on the same screen according to claim 2, characterized in that: The multi-channel playback manager is limited to a maximum of 16 channels.

15. A system for simultaneously playing multiple videos on the same screen, comprising a processor, characterized in that: The processor executes instructions to implement the method for simultaneous playback of multiple videos on the same screen as described in any one of claims 1 to 14.

Images