Audio and video stream pushing method and device, server and storage medium
By dynamic detection and data integration of media devices in the streaming media server, and pushing synthetic audio and video streams, the problem that the teaching process in classroom scenarios is difficult to be clearly understood, and the user experience of teaching supervisors is improved.
Patent Information
- Application Number
- CN202310501565.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2024-11-05
AI Technical Summary
In classroom scenarios, it is difficult for the prior art to clearly understand the teaching process through the player, especially in the interaction between teachers and students, where the students’ voices are difficult to capture and the focus is difficult to understand by the teaching supervisor.
By dynamically detecting media devices in the streaming media server, synthesized audio and video are generated, integrated audio and video stream data are pushed, and the online status and type of media devices are monitored to meet the needs of the multimedia course patrol system.
It improves the teaching supervisor's understanding of the teaching process, improves the user experience of the multimedia course patrol system, and ensures the integrity and clarity of audio and video data during the teaching process.
Smart Images

Figure CN118900338A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of audio and video editing technology, and in particular to a method, device, server and storage medium for pushing audio and video streams. Background Art
[0002] In the classroom scene, there are usually multiple video devices (such as cameras), audio devices (such as microphones) and other media devices to collect media data of the classroom, and then display multiple videos, audios or audio and video in the player, so that teaching supervisors can patrol the class through the player and understand the teaching process of each classroom.
[0003] However, in actual application, the inventors found that the supervision and class inspection program in the relevant technology was restricted by the functions of the media equipment, making it difficult to achieve good results in some scenarios, and the teaching supervisors could not clearly understand the teaching process through the player.
[0004] For example, some video devices do not have the function of collecting audio. When displaying the picture corresponding to the video device, the player does not play any sound. Or some video devices collect the sound of a single object, and only the sound of some objects in the classroom is played in the player.
[0005] Therefore, in the teacher-student interaction scenario where the teacher asks questions to the students and the students answer, the currently displayed audio and video is only a display of the teacher's actions or actions and sounds, while the students' voices are small or even completely inaudible, making it difficult for the teaching supervisor to clearly understand the teaching process through the player. Moreover, in some teaching scenarios, there are focuses in the teaching process, and the player in the related technology is difficult to inform the teaching supervisor of the corresponding focus, making it difficult for the teaching supervisor to clearly understand the teaching process through the player. Summary of the invention
[0006] The embodiments of the present application provide a method, device, server and storage medium for pushing audio and video streams. The solution can meet the needs of supervision and inspection in classroom scenarios, thereby providing integrated audio and video stream data, which helps to improve the user experience of the multimedia inspection system.
[0007] In a first aspect, an embodiment of the present application provides an audio and video stream push method, which is applied to a streaming media server in a multimedia class tour system. The multimedia class tour system includes a player, a streaming media server, and multiple media devices. The multiple media devices are all registered in the streaming media server, and the player and the media devices are all connected to the streaming media server. The method includes:
[0008] Dynamically detect media devices to determine the online status, media type, and streaming address of each media device;
[0009] According to the pull stream address, obtain the media stream of the media device and generate a media stream set;
[0010] Classifying and merging each media stream in the media stream set to generate a synthesized audio and a synthesized video;
[0011] Based on the synthesized audio and video, the preview audio and video with the director screen is output to the player;
[0012] When receiving a play request transmitted by a player, determining a target media device associated with the play request from the media devices, and determining streaming data according to the media type of the target media device, the streaming data being audio and video stream data including at least two of the media stream, synthesized audio, and synthesized video of the target media device;
[0013] Output the streaming data to the player and monitor the online status and media type of the target media device to determine whether to restart the dynamic detection of each media device.
[0014] In a second aspect, an embodiment of the present application further provides an audio and video stream pushing device, the device comprising:
[0015] The device detection module is configured to dynamically detect the media devices to determine the online status, media type and streaming address corresponding to each media device;
[0016] The data acquisition module is configured to acquire the media stream of the media device according to the pull stream address and generate a media stream set;
[0017] A data synthesis module, configured to classify and merge the media streams in the media stream set to generate synthesized audio and synthesized video;
[0018] A pre-push module is configured to output a preview audio and video with a director screen to a player based on the synthesized audio and the synthesized video;
[0019] The data selection module is configured to, upon receiving a play request transmitted by a player, determine a target media device associated with the play request from the media devices, and determine the push stream data according to the media type of the target media device, wherein the push stream data is audio and video stream data including at least two of the media stream, synthesized audio, and synthesized video of the target media device;
[0020] The data push module is configured to output push stream data to the player and monitor the online status and media type of the target media device to determine whether to restart the dynamic detection of each media device.
[0021] In a third aspect, an embodiment of the present application further provides a server, the server comprising:
[0022] one or more processors;
[0023] A storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the audio and video stream pushing method in any of the above embodiments.
[0024] In a fourth aspect, an embodiment of the present application further provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to execute the audio and video stream pushing method as in any of the above embodiments.
[0025] The embodiment of the present application integrates the audio data, video data, etc. of each media device in the classroom to meet the needs of playing multiple-channel classroom patrols on a single screen and single-screen single-screen classroom patrols during the classroom patrol process, and then pushes the corresponding data to the player. In addition, the present application solution also monitors the online status of the media device and the media type to determine whether the audio and video stream data needs to be reintegrated, so that the teaching supervisor can better understand the teaching process, which helps to improve the teaching supervisor's experience in using the multimedia classroom patrol system. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A schematic diagram of the architecture of a multimedia class tour system is provided for an embodiment of the present application;
[0027] Figure 2 A flowchart of the steps of the audio and video stream push method provided in one embodiment of the present application;
[0028] Figure 3 A flowchart of the steps for generating synthesized audio provided in an embodiment of the present application;
[0029] Figure 4 A flowchart of the steps for generating a composite video provided in an embodiment of the present application;
[0030] Figure 5 A schematic diagram of a player display interface provided by an embodiment of the present application;
[0031] Figure 6 A flowchart of the steps for selecting a director screen provided in an embodiment of the present application;
[0032] Figure 7 A flowchart of steps for monitoring a target media device provided in an embodiment of the present application;
[0033] Figure 8 A schematic diagram of the structure of an audio and video stream push device provided in an embodiment of the present application;
[0034] Fig. 9 A schematic diagram of the structure of a server provided in one embodiment of the present application. DETAILED DESCRIPTION
[0035] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only the parts related to the embodiments of the present application rather than all structures are shown in the accompanying drawings, and those skilled in the art should be able to think of it after reading the specification of this application that as long as the technical features do not contradict each other, any combination of the technical features can constitute an optional implementation method.
[0036] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship. In the description of the present application, "multiple" means two or more, and "several" means one or more.
[0037] In a classroom scenario, multiple video devices and audio devices are usually set up in the classroom to present multiple images and sounds in the classroom in corresponding players, so that teaching supervisors can understand the current teaching process.
[0038] Figure 1 An architectural schematic diagram of a multimedia class tour system is provided for one embodiment of the present application. As shown in the figure, the multimedia class tour system includes a player, a streaming media server and multiple media devices. The streaming media server establishes communication connections with the player and multiple media devices respectively to transmit and receive data.
[0039] Multiple media devices in the classroom, such as video devices, audio devices, etc., all output video data or audio data to the streaming media server, and the streaming media server outputs corresponding audio and video stream data to the player. It is understandable that multiple media devices are registered on the streaming media server so that they can transmit media streams to the streaming media server. It should be conceivable that the above-mentioned media streams include video data and audio data, such as video devices can push video data or video data and audio data to the streaming media server, and audio devices can push audio data to the streaming media server.
[0040] Figure 2This is a flowchart of the steps of the audio and video stream push method provided in an embodiment of the present application. The method is applied to a streaming media server. As shown in the figure, the audio and video stream push method of the present application includes the following steps:
[0041] Step S210: Dynamically detect the media devices to determine the online status, media type and streaming address corresponding to each media device.
[0042] Each media device has completed registration in the streaming media server, but the streaming media server still needs to dynamically detect it and record the corresponding data. During the dynamic detection process, the streaming media server obtains the online status, media type and streaming address of the media device.
[0043] Among them, the online status is used to determine whether the media device can push data normally; the media type is used to determine whether the media stream output by the media device is video data or audio data; and the pull stream address is used to obtain the media stream of the media device.
[0044] Step S220: Acquire the media stream of the media device according to the stream pull address, and generate a media stream set.
[0045] It is conceivable that the media stream set is a set of media streams corresponding to each media device, which includes video data and audio data. The streaming media server will also record the mapping relationship between video data and corresponding media devices and audio data and corresponding media devices to facilitate the extraction of media streams.
[0046] Step S230: classify and merge the media streams in the media stream set to generate synthesized audio and synthesized video.
[0047] After the streaming media server obtains the media streams of each media device, it can classify and merge the obtained data according to the media type of the media stream, such as grouping all the media streams according to the media type to obtain a group corresponding to the audio data and another group corresponding to the video data. In addition, the audio data is merged and the video data is merged to obtain the synthesized audio and the synthesized video.
[0048] It should be noted that the media stream set includes not only video data and audio data, but also synthesized audio and synthesized video, so that the streaming media server can select data to be pushed to the player.
[0049] Figure 3 The flowchart of the steps for generating synthesized audio provided in an embodiment of the present application, for generating synthesized audio, as shown in the figure, the audio and video stream push method of the present application also includes the following steps:
[0050] Step S310: traverse each media stream in the media stream set, and extract audio data in each media stream into the audio set.
[0051] Step S320: read the audio data in the audio set, eliminate duplicate audio data, and synthesize the remaining audio data to generate synthesized audio.
[0052] It is understandable that by traversing the data in the media stream set, the streaming media server can determine the data type of each media stream. When the streaming media server determines that the media type of the current media stream is an audio type, the streaming media server extracts the audio data into the audio set. It should be noted that the audio set is used to temporarily store audio data.
[0053] The streaming media server reads the audio data in the audio collection to perform a deduplication operation on the audio data, that is, to eliminate duplicate audio data. For example, for two identical audio data, the streaming media server can select the audio data with better sound quality to merge, and eliminate the other audio data. Then, the remaining audio data is merged to generate a synthetic audio.
[0054] Therefore, by deduplicating and merging the audio data, the streaming media server can obtain better audio data, so as to push audio data that can reflect the sounds in the current classroom scene to the player, which helps the teaching supervisor to clearly understand the teaching process through the player.
[0055] Figure 4 The flowchart of the steps for generating a composite video provided in an embodiment of the present application, as shown in the figure, for generating a composite video, the audio and video stream push method of the present application also includes the following steps:
[0056] Step S410: traverse each media stream in the media stream set, and extract video data in each media stream into the video set.
[0057] Step S420: read the video data in the video set, and splice the video frames corresponding to the respective video data to generate a composite video.
[0058] It is understandable that by traversing the data in the media stream set, the streaming media server can determine the data type of each media stream. When the streaming media server determines that the media type of the current media stream is a video type, the streaming media server extracts the corresponding video data into the video set. It should be noted that the video set is used to temporarily store video data.
[0059] The streaming media server reads the video data in the video set, and merges the video data. The streaming media server splices the video pictures corresponding to the video data into one picture, which is used as the picture of the composite video, that is, the composite video is used as one channel of video data, and the video picture of the composite video includes multiple video pictures corresponding to different media devices.
[0060] Therefore, by merging the video data, the streaming media server can splice the video images corresponding to multiple video devices into one video image, so that the teaching supervisor can watch the images shot by multiple video devices in the player.
[0061] It should be noted that when a media stream sent by a media device includes video data and audio data, the streaming media server can extract the audio data therefrom and store it in an audio collection. In addition, the streaming media server can extract the video data therefrom and store it in a video collection.
[0062] Step S240: Based on the synthesized audio and the synthesized video, the preview audio and video carrying the director's screen is output to the player.
[0063] The streaming media server can provide preview audio and video to the player, and the preview audio and video also provides a guide screen, so as to provide the user with a video screen that can represent the current classroom situation. For example, the streaming media server outputs synthesized audio and synthesized video to the player, that is, the synthesized audio and synthesized video are used as the preview audio and video.
[0064] Figure 5 A schematic diagram of a player display interface provided for an embodiment of the present application, as shown in the figure, a classroom list and a playback interface are displayed in the display interface of the player. Multiple classrooms are recorded in the classroom list, and each classroom corresponds to a preview audio and video. The user can select any classroom in the classroom list to make the player display the corresponding video screen. Multiple video screens are displayed in the playback interface to facilitate the user to observe the current situation in the classroom. Of course, the user can select a video screen in the player so that only the video screen is played in the playback interface.
[0065] Step S250: When a play request transmitted by the player is received, a target media device associated with the play request is determined from the media devices, and streaming data is determined according to the media type of the target media device.
[0066] It is conceivable that the teaching supervisor can interact with the streaming media server through the player. For example, the teaching supervisor can select the video screen that needs to be played separately on the player, and then the player generates a corresponding playback request and sends the playback request to the streaming media server.
[0067] After receiving the play request, the streaming media server selects a target media device from the media devices, and the target media device is the media device associated with the play request. Exemplarily, the play request is associated with the teacher's close-up picture, and the video device corresponding to the teacher's close-up picture is camera 1, and correspondingly, camera 1 is the target media device.
[0068] The streaming media server also determines the streaming data according to the media type of the target media device, wherein the streaming data is audio and video streaming data including at least two of the media stream, synthesized audio, and synthesized video of the target media device. Exemplarily, the media stream corresponding to the camera 1 as the target media device is video type data, and correspondingly, the streaming data includes not only the media stream corresponding to the camera 1, but also the synthesized audio, so that the teaching supervisor can hear the synthesized audio while watching the selected video screen.
[0069] Step S260: output the streaming data to the player, and monitor the online status and media type of the target media device to determine whether to restart the dynamic detection of each media device.
[0070] It is conceivable that when the online status and / or media type of the target media device changes, for example, Camera II is a media device that provides video data and audio data. When its media type is changed to provide only audio data, if the current streaming data is still maintained, no video screen will be displayed in the player's playback interface.
[0071] Therefore, while outputting streaming data to the player, the streaming media server also needs to continuously monitor the online status and media type of the target media device to avoid changes such as media device disconnection or media type changes that affect the streaming data, and then determine whether it is necessary to restart the dynamic detection of each media device.
[0072] It should be borne in mind that when the online status and media type of the target media device change, the streaming media server will restart the dynamic detection of each media device accordingly, thereby re-determining the relevant information of each media device, and then re-acquiring the synthesized audio and synthesized video, that is, executing the audio and video stream push method again.
[0073] It can be seen from the above scheme that in the classroom application scenario, the scheme integrates the audio data, video data, etc. of various media devices in the classroom to meet the needs of single-screen multi-channel picture tours and single-screen single-picture tours during the class tour. In addition, this application scheme is also universal and can be compatible with cameras, recorders, audio devices and other equipment from different manufacturers and with different media configurations in the classroom. Moreover, the streaming media server in the scheme can dynamically integrate audio and video stream data based on the monitoring of the online status of media devices and media types, so that teaching supervisors can better understand the teaching process in the classroom, which helps to improve the teaching supervisors' experience in using the multimedia class tour system.
[0074] In some embodiments, the preview audio and video output by the player carries a director screen, which is provided by the target device. Figure 6 A flowchart of the steps for selecting a director screen is provided in an embodiment of the present application, such as Figure 6 As shown, the audio and video stream push method of the present application includes the following steps:
[0075] Step S610: Determine the current teaching subject in the classroom according to the preset timetable.
[0076] Step S620: Determine the media device as the target device according to the teaching subject, and use the video screen of the target device as the director screen.
[0077] It is understandable that each classroom has a preset timetable, in which the teaching subjects conducted at each class time are recorded. The streaming media server can determine the teaching subject currently being conducted in the classroom based on the timetable. In addition, for each teaching subject, the streaming media server will select a media device as the target device, and then use the video screen corresponding to the target device as the guide screen.
[0078] In one embodiment, the teaching subjects have multiple types, such as a first subject corresponding to a self-study type, a second subject corresponding to a multimedia type, a third subject corresponding to an experiment type, and a fourth subject corresponding to a lecture type. Therefore, in the process of selecting the director screen, the type of the teaching subject can be further combined to select the corresponding video screen as the director screen.
[0079] If the teaching subject is the first subject, the target device is determined to be the video device corresponding to the student panoramic screen, and the student panoramic screen is used as the guide screen. It can be understood that for self-study subjects such as self-study classes and painting classes, the guide screen should focus more on the student screen. Therefore, the streaming media server selects the video device corresponding to the student panoramic screen as the target device, and then uses the corresponding video screen as the guide screen.
[0080] If the teaching subject is the second subject, the target device is determined to be a video device corresponding to the multimedia screen, and the multimedia screen is used as the guide screen. For the multimedia screen, for example, in a subject that uses computers as a teaching medium, such as a computer class, the display screen on the computer can be used as the guide screen to show the focus of the subject to the teaching supervisor.
[0081] If the teaching subject is the third subject, the target device is determined to be the video device corresponding to the booth screen, and the booth screen is used as the director screen. It can be understood that for subjects that mainly display objects, such as experimental classes, the director screen should focus more on the booth screen, and accordingly, the target device is the video device corresponding to the booth screen.
[0082] If the teaching subject is the fourth subject, the target device is determined to be a video device corresponding to the teacher's panoramic screen, and the teacher's panoramic screen is used as the guide screen. In subjects where the teacher's lecture is the main subject, the guide screen is selected to be the teacher's panoramic screen, so as to highlight the teacher's teaching process.
[0083] Therefore, in different teaching subjects, the streaming media server selects the corresponding video screen as the director screen, so that the player plays the video screen that better reflects the current classroom situation, realizes the flexible setting of the director strategy for teaching subjects, and effectively improves the accuracy of automatic director.
[0084] In one embodiment, the streaming media server needs to determine the push data according to the media type of the target media device when determining the push data, that is, the streaming media server pushes different data corresponding to media streams of different media types.
[0085] When the media type of the target media device is a video type, the media stream of the target media device is extracted from the media stream set, and the media stream is mixed with the synthesized audio into audio and video stream data. That is, when the target media device outputs video data, the push data output by the streaming media server to the player is the audio and video stream data synthesized by the video data and the synthesized audio. When the media type of the target media device is an audio type, the synthesized video and the synthesized audio are mixed into audio and video stream data.
[0086] In addition, when the media type of the target media device is an audio and video type, the video data in the media stream of the target media device is obtained, and the video data and the synthesized audio are mixed into audio and video stream data. It can be understood that some media devices can provide video data and audio data, and correspondingly, the media type of the media device is an audio and video type. Therefore, the push data output by the streaming media server to the player is the audio and video stream data synthesized by the video data of the target media device and the synthesized audio.
[0087] Therefore, this solution selects different streaming data for target media devices of different media types, so as to output the integrated audio and video stream data to the player to meet the needs of supervision and inspection in classroom scenarios.
[0088] Figure 7 A flowchart of steps for monitoring a target media device provided in an embodiment of the present application. In an embodiment, the streaming media server monitors the online status of the target media device and whether the media type has changed while outputting push data, such as Figure 7 As shown, the audio and video stream data of this application also includes the following steps:
[0089] Step S710: Detect the IP address and port of the target media device to determine whether the online status has changed.
[0090] Step S720: Detect the media type of the target media device to determine whether the media type has changed.
[0091] Step S730: When the online status and / or media type changes, determine to restart the dynamic detection of each media device.
[0092] It is understandable that, for the monitoring of online status, the streaming media server monitors the IP (Internet Protocol) address and port of the target media device. When the IP address changes, it can be determined that the online status of the target media device changes. In addition, the port includes a virtual port and a physical port, which is used to transmit data with the streaming media server. Therefore, when the port changes, it can also be determined that the online status of the target media device changes.
[0093] Specifically, the streaming media server can determine whether the online status of the target media device has changed by detecting whether the IP address and the port are reachable. If both the IP address and the port are reachable, the online status of the target media device has not changed.
[0094] Likewise, the media type of the target media device needs to be detected to determine whether the media type has changed. It is worth noting that the monitoring of the online status and the media type can be performed synchronously or asynchronously.
[0095] After the detection is completed, it is necessary to determine whether to restart the dynamic detection to re-push the data. The streaming media server is set with corresponding conditions. When the corresponding conditions are met, the streaming media server restarts the dynamic detection, such as when any one of the online status and the media type changes, that is, when the online status and / or the media type changes, it is determined to restart the dynamic detection of each media device. Therefore, the streaming media server monitors the online status and the media type to avoid the situation where the pushed data cannot meet the user's requirements.
[0096] Figure 8 This is a structural schematic diagram of an audio and video stream pushing device provided in one embodiment of the present application. The device is used to execute the audio and video stream pushing method as described in the above embodiment, and has functional modules and beneficial effects corresponding to the execution of the above method. As shown in the figure, the audio and video stream pushing device includes a device detection module 801, a data acquisition module 802, a data synthesis module 803, a pre-push module 804, a data selection module 805 and a data push module 806.
[0097] Among them, the device detection module 801 is configured to dynamically detect the media devices to determine the online status, media type and streaming address corresponding to each media device; the data acquisition module 802 is configured to obtain the media stream of the media device according to the streaming address, and generate a media stream set; the data synthesis module 803 is configured to classify and merge the media streams in the media stream set to generate synthetic audio and synthetic video; the pre-push module 804 is configured to output the audio and video with the director screen preview to the player based on the synthetic audio and synthetic video; the data selection module 805 is configured to determine the target media device associated with the playback request from the media device when receiving the playback request transmitted by the player, and determine the streaming data according to the media type of the target media device, and the streaming data is the audio and video stream data including at least two of the media stream, synthetic audio and synthetic video of the target media device; the data push module 806 is configured to output the streaming data to the player, and monitor the online status and media type of the target media device to determine whether to restart the dynamic detection of each media device.
[0098] Based on the above embodiment, the data synthesis module 803 is further configured as follows:
[0099] Traverse each media stream in the media stream set, and extract the audio data in each media stream into the audio set;
[0100] The audio data in the audio collection is read, duplicate audio data is eliminated, and the remaining audio data is synthesized to generate synthesized audio.
[0101] Based on the above embodiment, the data synthesis module 803 is further configured as follows:
[0102] Traverse each media stream in the media stream set, and extract video data in each media stream into the video set;
[0103] The video data in the video set are read, and the video frames corresponding to the respective video data are spliced to generate a composite video.
[0104] Based on the above embodiment, the pre-push module 804 is further configured as follows:
[0105] According to the preset timetable, determine the current teaching subject in the classroom;
[0106] According to the teaching subject, the media device as the target device is determined, and the video screen of the target device is used as the director screen.
[0107] Based on the above embodiment, the pre-push module 804 is further configured as follows:
[0108] If the teaching subject is the first subject, the target device is determined to be a video device corresponding to the student's panoramic picture, and the student's panoramic picture is used as the guide picture;
[0109] If the teaching subject is the second subject, the target device is determined to be a video device corresponding to the multimedia screen, and the multimedia screen is used as the guide screen;
[0110] If the teaching subject is the third subject, the target device is determined to be the video device corresponding to the booth screen, and the booth screen is used as the director screen;
[0111] If the teaching subject is the fourth subject, the target device is determined to be the video device corresponding to the teacher's panoramic picture, and the teacher's panoramic picture is used as the director picture.
[0112] Based on the above embodiment, the data selection module 805 is further configured as follows:
[0113] When the media type of the target media device is a video type, extracting the media stream of the target media device from the media stream set, and mixing the media stream with the synthesized audio into audio and video stream data;
[0114] When the media type of the target media device is an audio type, the synthesized video and the synthesized audio are mixed into audio and video stream data;
[0115] When the media type of the target media device is an audio and video type, video data in a media stream of the target media device is obtained, and the video data is mixed with the synthesized audio into audio and video stream data.
[0116] Based on the above embodiment, the data push module 806 is further configured as follows:
[0117] Detect the IP address and port of the target media device to determine whether the online status has changed;
[0118] Detecting the media type of the target media device to determine whether the media type has changed;
[0119] When the online status and / or media type changes, it is determined that dynamic detection of each media device is restarted.
[0120] It is worth noting that in the embodiment of the above-mentioned audio and video stream pushing device, each functional module is only divided according to functional logic, but is not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific name of each functional module is only for the convenience of distinguishing each other, and is not used to limit the scope of protection of this application.
[0121] Fig. 9 A structural diagram of a server provided in an embodiment of the present application, the server is used to execute the audio and video stream push method provided in the above embodiment, and has functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the server includes a processor 901, a memory 902, an input device 903 and an output device 904. The number of processors 901 in the server can be one or more, and one processor 901 is taken as an example in the figure; the processor 901, the memory 902, the input device 903 and the output device 904 can be connected by a bus or other means, and the figure takes the connection through a bus as an example. The memory 902, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio and video stream push method in the embodiment of the present application. The processor 901 executes various corresponding functional applications and data processing by running the software programs, instructions and modules stored in the memory 902, that is, the above-mentioned audio and video stream push method is realized.
[0122] The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data recorded or created during use, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 902 may further include a memory remotely arranged relative to the processor 901, and the remotely arranged memory may be connected to a server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0123] The input device 903 can be used to input corresponding digital or character information to the processor 901, and to generate key signal input related to the user settings and function control of the device; the output device 904 can be used to send or display key signal output related to the user settings and function control of the device.
[0124] An embodiment of the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to perform relevant operations in the audio and video stream pushing method provided in any embodiment of the present application.
[0125] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0126] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0127] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims
1. A method for pushing audio and video streams, characterized in that: A streaming media server applied to a multimedia class tour system, the multimedia class tour system comprising a player, a streaming media server and a plurality of media devices, the plurality of media devices are all registered in the streaming media server, and the player and the media devices are all in communication connection with the streaming media server, the method comprising: Dynamically detect the media devices to determine the online status, media type and streaming address of each media device; According to the stream pull address, obtain the media stream of the media device and generate a media stream set; Classifying and merging the media streams in the media stream set to generate a synthesized audio and a synthesized video; Based on the synthesized audio and the synthesized video, outputting the preview audio and video carrying the director screen to the player; When receiving a play request transmitted by the player, determining a target media device associated with the play request from the media devices, and determining streaming data according to a media type of the target media device, the streaming data being audio and video stream data including a media stream of the target media device, the synthesized audio, and at least two of the synthesized video; The streaming data is output to the player, and the online status and the media type of the target media device are monitored to determine whether to restart the dynamic detection of each of the media devices.
2. The method for pushing audio and video streams according to claim 1, characterized in that: Classifying and merging each media stream in the media stream set to generate a synthesized audio and a synthesized video, including: Traversing each media stream in the media stream set, and extracting audio data in each media stream into an audio set; The audio data in the audio set is read, duplicate audio data is eliminated, and the remaining audio data is synthesized to generate the synthesized audio.
3. The method for pushing audio and video streams according to claim 1 or 2, characterized in that: The classifying and merging the media streams in the media stream set to generate a synthesized audio and a synthesized video includes: Traversing each media stream in the media stream set, and extracting video data in each media stream into the video set; The video data in the video set are read, and the video frames corresponding to the respective video data are spliced to generate the composite video.
4. The method for pushing audio and video streams according to claim 1, characterized in that: Before outputting the preview audio and video carrying the director screen to the player based on the synthesized audio and the synthesized video, the method further includes: According to the preset timetable, determine the current teaching subject in the classroom; According to the teaching subject, a media device is determined as a target device, and a video screen of the target device is used as the director screen.
5. The method for pushing audio and video streams according to claim 4, characterized in that: The teaching subjects include a first subject corresponding to a self-study type, a second subject corresponding to a multimedia type, a third subject corresponding to an experimental type, and a fourth subject corresponding to a lecture type; The step of determining a media device as a target device according to the teaching subject, and using a video screen of the target device as the directing screen, includes: If the teaching subject is the first subject, determining that the target device is a video device corresponding to the student panoramic picture, and using the student panoramic picture as the director picture; If the teaching subject is the second subject, determining that the target device is a video device corresponding to a multimedia screen, and using the multimedia screen as the directing screen; If the teaching subject is the third subject, determining that the target device is a video device corresponding to the booth screen, and using the booth screen as the director screen; If the teaching subject is the fourth subject, the target device is determined to be a video device corresponding to the teacher's panoramic picture, and the teacher's panoramic picture is used as the director picture.
6. The method for pushing audio and video streams according to claim 1, characterized in that: When receiving a play request transmitted by the player, determining a target media device associated with the play request from the media device, and determining streaming data according to the media type of the target media device, the streaming data being audio and video stream data including at least two of the media stream of the target media device, the synthesized audio, and the synthesized video, including: When the media type of the target media device is a video type, extracting the media stream of the target media device from the media stream set, and mixing the media stream with the synthesized audio into the audio and video stream data; When the media type of the target media device is an audio type, mixing the synthesized video and the synthesized audio into the audio and video stream data; When the media type of the target media device is an audio and video type, video data in the media stream of the target media device is obtained, and the video data is mixed with the synthesized audio to form the audio and video stream data.
7. The method for pushing audio and video streams according to claim 1, characterized in that: Outputting the streaming data to the player and monitoring the online status and the media type of the target media device to determine whether to restart the dynamic detection of each media device includes: Detecting the IP address and port of the target media device to determine whether the online status has changed; Detecting the media type of the target media device to determine whether the media type has changed; When the online status and / or the media type changes, it is determined to restart the dynamic detection of each of the media devices.
8. An audio and video stream pushing device, characterized in that: A streaming media server used in a multimedia class tour system, the multimedia class tour system comprising a player, a streaming media server and a plurality of media devices, the plurality of media devices are all registered in the streaming media server, and the player and the media devices are all connected to the streaming media server in communication, the audio and video stream push device comprising: A device detection module is configured to dynamically detect the media devices to determine the online status, media type and streaming address corresponding to each media device; A data acquisition module, configured to acquire the media stream of the media device according to the stream pull address, and generate a media stream set; A data synthesis module, configured to classify and merge the media streams in the media stream set to generate synthesized audio and synthesized video; A pre-push module is configured to output the audio and video with the director screen preview to the player based on the synthesized audio and the synthesized video; a data selection module configured to, upon receiving a play request transmitted by the player, determine a target media device associated with the play request from the media devices, and determine streaming data according to a media type of the target media device, wherein the streaming data is audio and video stream data including at least two of the media stream of the target media device, the synthesized audio, and the synthesized video; The data push module is configured to output the push stream data to the player and monitor the online status and the media type of the target media device to determine whether to restart the dynamic detection of each media device.
9. A server, characterized in that: The server comprises: one or more processors; A storage device for storing one or more programs. When one or more of the programs are executed by one or more of the processors, the one or more processors implement the audio and video stream pushing method as described in any one of claims 1 to 7.
10. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, they are used to execute the audio and video stream pushing method according to any one of claims 1 to 7.
Citation Information
Cited By
Multi-device plug-flow live broadcast method and related device
CN119865633A
