Interactive classroom live video processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an interactive classroom live video processing method and device, equipment and a storage medium. According to the technical scheme provided by the embodiment of the invention, the rendering view of the whiteboard view control is acquired, and the stream data of the teaching whiteboard display picture is generated based on the acquired rendering view, so that the teaching whiteboard display picture can be pushed to the cloud server together with the stream data of classroom live broadcast in the form of video stream, and the teaching whiteboard display picture can be pushed to the cloud server. The teaching whiteboard display picture and the classroom live broadcast video are shared in the interactive classroom live broadcast scene, the teaching whiteboard display picture and the classroom live broadcast video stream are guaranteed to be synchronous, the interactive classroom live broadcast effect is optimized, and the interactive classroom live broadcast experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video live broadcast technology, and in particular to a method, device, equipment and storage medium for processing interactive classroom live broadcast video. Background Art

[0002] At present, with the development and progress of video live broadcast technology, video live broadcast is often used for remote classroom teaching and interaction in classroom teaching scenarios. When conducting interactive classroom live broadcast, the lecturer's teaching video screen will be pushed to the listening end, so that the listening end can remotely watch the lecturer's teaching video screen in real time, so as to improve the interactive effect of classroom live broadcast.

[0003] However, since the live broadcast scene requires the video streams of each client in the room to be mixed and then pushed through the content distribution network, each client enters the live broadcast room through the corresponding live broadcast browser and receives the mixed stream pushed by the content distribution network to watch the classroom live broadcast. As for the display screen of the classroom teaching whiteboard, it is generally necessary to install a device with a whiteboard plug-in to share it, but the live broadcast browser usually does not have this whiteboard plug-in, resulting in the inability to share the teaching whiteboard display screen with the audience in the live broadcast scene, which will affect the interactive classroom live broadcast effect and cause the user's classroom experience to deviate. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment and storage medium for processing live video of an interactive classroom, which can synchronously push the teaching whiteboard display screen and the live video stream, thereby solving the technical problem that the teaching whiteboard display screen cannot be shared in the interactive classroom live broadcast scene.

[0005] In a first aspect, an embodiment of the present application provides a method for processing an interactive classroom live video, comprising:

[0006] Collect classroom live video and microphone audio, generate first-class data and push it to the cloud server;

[0007] When the teaching whiteboard view control is detected to be running, the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, and the rendering view is compressed, format converted and encoded to generate the second stream data in the specified format;

[0008] The first stream data and the second stream data are pushed to the cloud server in real time based on the acquisition timestamp, so as to be mixed and distributed through the cloud server.

[0009] It can be seen that the interactive classroom live video processing method of the present application collects the rendering view of the whiteboard view control, and generates the streaming data of the teaching whiteboard display screen based on the collected rendering view, so that the teaching whiteboard display screen can be pushed to the cloud server together with the streaming data of the classroom live broadcast in the form of a live video stream. In this way, the teaching whiteboard display screen and the classroom live video can be shared in the interactive classroom live broadcast scene, ensuring the synchronization of the teaching whiteboard display screen and the classroom live video stream, optimizing the interactive classroom live broadcast effect, and improving the interactive classroom live broadcast experience.

[0010] In addition, this solution collects the rendering view of the teaching whiteboard view control through the screen acquisition thread, which can realize the accurate acquisition of the whiteboard display screen, filter out irrelevant display information, and improve the acquisition efficiency and accuracy of the whiteboard display screen.

[0011] Furthermore, based on the picture acquisition thread, the rendering view of the teaching whiteboard view control is collected, and further includes:

[0012] The collection time interval of the rendering view is determined based on the frame rate of the second stream data, and the rendering view is collected according to the collection time interval and the collection time consumption of the rendering view.

[0013] Further, collecting the rendering view according to the collection time interval and the collection time consumption of the rendering view includes:

[0014] When the acquisition time interval is greater than the acquisition time consumption, the sleep time duration is determined based on the acquisition time interval and the acquisition time consumption. After each screen acquisition thread completes acquiring a rendering view, the system sleeps based on the sleep time duration.

[0015] By putting the screen acquisition thread to sleep, you can reduce resource usage and lower system performance costs.

[0016] Furthermore, when the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, it also includes:

[0017] The multimedia audio data inside the teaching whiteboard is synchronously collected, an audio stream is generated based on the multimedia audio data, and the audio stream is placed into the second stream data.

[0018] Furthermore, synchronously collecting multimedia audio data inside the teaching whiteboard and generating an audio stream based on the multimedia audio data include:

[0019] Obtain screen recording permission, enable multimedia sound collection service based on screen recording permission, and set multimedia sound collection configuration instance according to multimedia sound collection service;

[0020] Create an audio recording object based on the multimedia sound acquisition configuration instance, synchronously collect the multimedia audio data inside the teaching whiteboard through the audio recording object, and transcode the multimedia audio data into an audio stream in a specified format.

[0021] By calling the multimedia sound collection service to collect multimedia audio data inside the whiteboard, external noise interference can be avoided and the collection of teaching whiteboard stream data can be optimized.

[0022] Furthermore, the resolution of the rendered view is compressed, including:

[0023] Comparing a preset resolution list according to the resolution of the rendered view, where the resolution list includes a plurality of preset target resolutions;

[0024] Compresses the resolution of the render view down to the closest target resolution.

[0025] In a second aspect, an embodiment of the present application provides an interactive classroom live video processing system, including a lecture terminal, a lecture terminal and a cloud server;

[0026] The lecturer terminal is used to execute the interactive classroom live video processing method of the first aspect;

[0027] The cloud server is used to receive the first stream data and the second stream data transmitted by the main lecture end, perform stream data processing on the first stream data and the second stream data, and distribute the mixed streams to the listening end.

[0028] Furthermore, the listening end also includes a stage end, which is used to collect third-stream data during the live broadcast of the class and push the third-stream data to the cloud server for mixing and distribution.

[0029] In a third aspect, an embodiment of the present application provides an interactive classroom live video processing device, comprising:

[0030] The first acquisition module is used to collect classroom live video and microphone audio, generate first-stream data and push it to the cloud server;

[0031] The second acquisition module is used to acquire the rendering view of the teaching whiteboard view control based on the screen acquisition thread when detecting that the teaching whiteboard view control is running, perform resolution compression, format conversion and encoding on the rendering view, and generate a second stream data in a specified format;

[0032] The streaming module is used to push the first stream data and the second stream data to the cloud server in real time based on the acquisition timestamp, so as to mix and distribute the data through the cloud server.

[0033] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0034] memory and one or more processors;

[0035] A memory for storing one or more programs;

[0036] When one or more programs are executed by one or more processors, the one or more processors implement the interactive classroom live video processing method as in the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the interactive classroom live video processing method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a method for processing interactive classroom live video provided in Example 1 of the present application;

[0039] Figure 2 It is a structural diagram of the interactive classroom live video processing system in Example 1 of the present application;

[0040] Figure 3 This is a schematic diagram of collecting audio streams in Embodiment 1 of the present application;

[0041] Figure 4 It is a structural diagram of an interactive classroom live video processing device provided in Example 2 of the present application;

[0042] Figure 5 It is a structural schematic diagram of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0044] The interactive classroom live video processing method provided in this application aims to achieve the synchronous push of the teaching whiteboard display screen and the classroom live video stream, reduce the display delay between the two, and improve the interactive classroom live broadcast effect.

[0045] In the classroom live broadcast scenario, the cloud server will obtain the video streams pushed by the lecturer and the listener, and then mix the video streams and push them to each client of the classroom live broadcast (i.e. the lecturer and the listener). The lecturer and the listener use the classroom live broadcast software installed on their respective clients to interact with the classroom live broadcast. As for the lecturer, if a teaching whiteboard is used, the display screen of the teaching whiteboard will be synchronized to the teaching software of other clients through the teaching software for display. Generally speaking, due to application restrictions, the lecturer cannot open the teaching software and the live broadcast software at the same time to share the teaching whiteboard display screen of the classroom live video. Even if the teaching software and the live broadcast software can be opened separately, since the processing of the two is independent of each other, there may be a delay in the display of the classroom live video and the teaching whiteboard display screen when they are transmitted to other clients, which will affect the interactive classroom live broadcast effect and result in a poor classroom live broadcast experience for users.

[0046] Based on this, an interactive classroom live video processing method of an embodiment of the present application is provided to solve the technical problem of push delay between the teaching whiteboard display screen and the classroom live video stream. By collecting the rendering view of the whiteboard view control, the stream data of the teaching whiteboard display screen is generated based on the collected rendering view, so that the teaching whiteboard display screen can be pushed to the cloud server together with the stream data of the classroom live broadcast in the form of a video stream, so as to realize the sharing of the teaching whiteboard display screen and the classroom live video in the interactive classroom live broadcast scene, optimize the interactive classroom live broadcast effect, and enhance the interactive classroom live broadcast experience.

[0047] In addition, this solution collects the rendering view of the teaching whiteboard view control through the screen acquisition thread, which can realize the accurate acquisition of the whiteboard display screen, filter out irrelevant display information, and improve the acquisition efficiency and accuracy of the whiteboard display screen.

[0048] Embodiment 1:

[0049] Figure 1 A flowchart of an interactive classroom live video processing method provided in Example 1 of the present application is given. The interactive classroom live video processing method provided in this embodiment can be executed by an interactive classroom live video processing device, which can be implemented by software and / or hardware. The interactive classroom live video processing device can be composed of two or more physical entities, or it can be composed of one physical entity. Generally speaking, the interactive classroom live video processing device can be a lecture terminal for classroom live broadcast, specifically a terminal device such as a teaching whiteboard (smart whiteboard).

[0050] The following description takes the lecturer end as an example to execute the interactive classroom live video processing method.

[0051] Reference Figure 1 , the interactive classroom live video processing method specifically includes:

[0052] S110 collects classroom live video and microphone audio, generates first-rate data and pushes it to the cloud server.

[0053] In the classroom live broadcast scenario, after the live broadcast software is turned on at the lecturer's end, the lecturer's classroom live broadcast video and microphone audio will be collected in real time through the camera and microphone set at the lecturer's end, and stream data will be generated and pushed to the cloud server, and the stream data will be defined as the first-stream data. If the lecturer's end does not turn on the teaching whiteboard at this time, the lecturer's end will only push the first-stream data to the cloud server, and then the cloud server will process and mix the stream data. When the lecturer turns on the teaching whiteboard, in order to facilitate the sharing of the teaching whiteboard display screen during the classroom live broadcast and improve the classroom live broadcast effect, it is necessary to push a separate auxiliary stream for the teaching whiteboard display screen, and send it to the cloud server together with the first-stream data for live broadcast mixing, so as to achieve the synchronous push of the teaching whiteboard display screen and the classroom live broadcast stream data.

[0054] S120. When it is detected that the teaching whiteboard view control is running, the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, and the rendering view is compressed, format converted and encoded to generate a second stream data in a specified format.

[0055] The embodiment of the present application detects whether the teaching whiteboard is turned on in the live broadcast scenario, and then determines whether to share the teaching whiteboard display screen, so as to generate a stream data corresponding to the teaching whiteboard display screen, which is defined as the second stream data. It should be noted that only when the classroom live broadcast and the teaching whiteboard are turned on at the same time, the lecture end will push the classroom live video stream and the teaching whiteboard display screen synchronously. If only the classroom live broadcast or the teaching whiteboard is turned on, the classroom live video stream and the teaching whiteboard display screen will not be pushed synchronously.

[0056] Reference Figure 2 , provides an interactive classroom live video processing system according to an embodiment of the present application, the interactive classroom live video processing system includes a lecture terminal 11, a listening terminal 13 and a cloud server 12; the lecture terminal 11 is used to execute the interactive classroom live video processing method as in the first aspect; the cloud server 12 is used to receive the first stream data and the second stream data transmitted by the lecture terminal 11, perform stream data processing on the first stream data and the second stream data, and distribute the mixed streams to the listening terminal 13.

[0057] The lecture terminal 11 pushes the first stream data and the second stream data to the cloud server 12 for stream data processing and then mixes and distributes them to each listening terminal 13. Since the first stream data and the second stream data are pushed to the cloud server synchronously, after the mixed stream is distributed to each listening terminal 13, the listening terminal 13 can synchronously display the lecturer's teaching whiteboard display screen and the classroom live video stream without a large delay, thereby achieving a better interactive classroom live experience.

[0058] Optionally, the listening end also includes a stage end, which is used to collect third-stream data during the live broadcast of the class and push the third-stream data to the cloud server for mixing and distribution.

[0059] For the listening end, some of them can use the "classroom stage function" to share the classroom live streaming data of the listening end, and define this part of the streaming data as the third stream data. The third stream data contains the classroom live video stream and microphone audio of the students attending the class. The third stream data will also be pushed to the cloud server for mixed streaming. For the stage end, it will receive a mixed stream of the first stream data, the second stream data, and the third stream data of other stages. For ordinary listening ends (non-stage ends), they will receive a mixed stream of the first stream data, the second stream data, and all the third stream data. For the main lecture end, in addition to pushing the first stream data and the second stream data, it will also receive the third stream data of each stage end. In this way, different mixed stream data can be pushed synchronously according to the roles of different clients in the intercommunication classroom live broadcast scenario, thereby improving the user's classroom live broadcast experience.

[0060] For example, after the lecturer uses the lecture terminal to start the classroom live broadcast, the first-stream data is pushed to the cloud server by default. When the lecturer opens the teaching whiteboard, the collection and push of the second-stream data will be triggered.

[0061] Specifically, the lecture end collects the display screen of the teaching whiteboard through a screen collection thread, and then generates the second stream data. In addition, in order to avoid putting unnecessary display content (such as the operation bar) on the teaching whiteboard into the second stream data, the embodiment of the present application collects the rendering view of the teaching whiteboard view control, and the rendering view does not contain part of the operation bar content, so that the display screen of the teaching whiteboard can be simplified and the live display effect can be optimized. When the lecture end detects that the teaching whiteboard view control is running, it means that the teaching whiteboard is turned on. At this time, the collection and push of the second stream data can be triggered to share the display screen of the teaching whiteboard of the lecture end.

[0062] When collecting the second stream data, the main speaker starts a screen acquisition thread. The screen acquisition thread continuously obtains the rendered image from the view control View of the teaching whiteboard through the View.drawToBitmap method, that is, the rendered view, which is in ARGB8888 format and has an aspect ratio of 16:9.

[0063] The resolution of the rendering view is then compressed, wherein the resolution of the rendering view is compared with a preset resolution list, the resolution list including a plurality of preset target resolutions; and the resolution of the rendering view is compressed down to the closest target resolution.

[0064] Assuming that the target resolutions preset in the intrinsic resolution list are 1920*1080, 1280*720, 960*540, 640*360, and 480*270, then the above target resolutions are compared according to the resolution of the current rendering view, and the target resolution that is closest to the resolution of the current rendering view and smaller than the resolution of the current rendering view is selected, and the resolution of the rendering view is compressed to the target resolution. In this way, the resolution of the output second stream data can be fixed, which is convenient for subsequent stream data processing.

[0065] Then, the rendering view of the compressed resolution is converted into the YUV420 format and encoded into the H264 video stream, that is, the second stream data, so as to complete the stream data collection of the corresponding teaching whiteboard display screen.

[0066] Optionally, in the process of collecting the rendering view of the teaching whiteboard view control based on the screen collection thread, it also includes:

[0067] The collection time interval of the rendering view is determined based on the frame rate of the second stream data, and the rendering view is collected according to the collection time interval and the collection time consumption of the rendering view.

[0068] It is understandable that according to the frame rate required by the second-stream data, assuming the frame rate is 10fps, 10 rendering views need to be collected within 1 second, and one frame of rendering view needs to be completed within 100 milliseconds. Assuming that the overall time consumption of processing the rendering view into the second-stream data is n milliseconds, when n>=100, it is necessary to take the next frame of the picture immediately after sending the current frame to meet the frame rate requirements of the second-stream data as much as possible. On the contrary, when the acquisition time interval is greater than the acquisition time, the sleep time is determined based on the acquisition time interval and the acquisition time, and each time the screen acquisition thread completes the acquisition of a rendering view, it sleeps based on the sleep time. That is, if n<100, the screen acquisition thread can be put to sleep for 100-n milliseconds after each acquisition of a frame of rendering view. By sleeping the screen acquisition thread, resource usage can be reduced and system performance expenses can be reduced.

[0069] In this way, the rendering view of the teaching whiteboard view control View area is continuously collected and H264 video data is generated through the screen acquisition thread, so as to complete the conversion of the teaching whiteboard display screen to the live streaming data in the classroom live broadcast scene.

[0070] Furthermore, when the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, it also includes:

[0071] The multimedia audio data inside the teaching whiteboard is synchronously collected, an audio stream is generated based on the multimedia audio data, and the audio stream is placed into the second stream data.

[0072] In order to further optimize the sharing effect of the teaching whiteboard content, the embodiment of the present application also synchronously collects the multimedia audio data inside the teaching whiteboard when collecting the rendering view to generate the audio stream of the second stream data. In this way, when the second stream data is played on the listening end, not only can the teaching whiteboard display screen be seen, but also the audio content corresponding to the teaching whiteboard display screen can be heard, achieving a better classroom live teaching experience.

[0073] Among them, refer to Figure 3 , synchronously collect the multimedia audio data inside the teaching whiteboard, and generate audio streams based on the multimedia audio data, including:

[0074] S1201, obtaining screen recording permission, enabling multimedia sound collection service based on the screen recording permission, and setting a multimedia sound collection configuration instance according to the multimedia sound collection service;

[0075] S1202: Create an audio recording object based on the multimedia sound acquisition configuration instance, synchronously acquire the multimedia audio data inside the teaching whiteboard through the audio recording object, and transcode the multimedia audio data into an audio stream in a specified format.

[0076] The lecturer's classroom live broadcast software applies for screen recording permission from the user, and enables a multimedia sound collection service when the screen recording permission is enabled. Then, a MediaProjection instance is obtained based on the multimedia sound collection service, and the multimedia sound collection configuration instance AudioPlaybackCaptureConfiguration is used to set the multimedia sound collection configuration instance. Then, an audio recording object AudioRecord is created using the multimedia sound collection configuration instance AudioPlaybackCaptureConfiguration to record audio, obtain the original PCM audio data, and encode it into an AAC audio stream, and push the AAC audio stream in the second stream data. In this way, by calling the multimedia sound collection service to collect multimedia audio data inside the whiteboard, external noise interference can be avoided and the collection of teaching whiteboard stream data can be optimized.

[0077] Optionally, in practical applications, considering that the audio output by the teaching whiteboard may be collected by the main speaker, that is, the microphone audio of the main speaker may include the audio output by the teaching whiteboard, which results in both the first stream data and the second stream data including the audio output by the teaching whiteboard. Therefore, based on the audio stream collected in the second stream data, the microphone audio of the first stream data can be subjected to noise reduction processing, thereby achieving a better audio output effect when the first stream data and the second stream data are output synchronously.

[0078] In addition, when collecting audio, you can further judge the system version. For versions lower than Android 10, the main speaker outputs a prompt: the system is low and cannot collect internal media playback sound. Only versions higher than or equal to Android 10 will collect the internal media sound of the teaching whiteboard.

[0079] S130: Push the first stream data and the second stream data to the cloud server in real time based on the acquisition timestamp, so as to perform stream mixing and distribution through the cloud server.

[0080] Finally, based on the first and second streams of data collected in real time, they can be pushed to the cloud server in real time according to the collection timestamp. On the corresponding cloud server side, by receiving the two streams of data respectively, the two streams of data are mixed according to the receiving time and pushed to each listening end. The listening end can watch the teaching whiteboard display screen while watching the live classroom teaching video, achieving a better interactive classroom live broadcast experience.

[0081] In the above, by collecting the classroom live video and microphone audio, the first stream data is generated and pushed to the cloud server; when the teaching whiteboard view control is detected to be running, the rendering view of the teaching whiteboard view control is collected based on the screen acquisition thread, and the rendering view is compressed, converted and encoded to generate the second stream data in the specified format; the first stream data and the second stream data are pushed to the cloud server in real time based on the acquisition timestamp, so as to be mixed and distributed through the cloud server. By adopting the above technical means, by collecting the rendering view of the whiteboard view control, the stream data of the teaching whiteboard display screen is generated based on the collected rendering view, so that the teaching whiteboard display screen can be pushed to the cloud server together with the stream data of the classroom live broadcast in the form of a video stream, so as to realize the sharing of the teaching whiteboard display screen and the classroom live broadcast video in the interactive classroom live broadcast scene, optimize the interactive classroom live broadcast effect, and enhance the interactive classroom live broadcast experience. In addition, this solution collects the rendering view of the teaching whiteboard view control through the screen acquisition thread, which can realize the accurate acquisition of the whiteboard display screen, filter irrelevant display information, and improve the acquisition efficiency and acquisition accuracy of the whiteboard display screen.

[0082] Embodiment 2:

[0083] Based on the above embodiments, Figure 4This is a schematic diagram of the structure of an interactive classroom live video processing device provided in Example 2 of the present application. Figure 4 The interactive classroom live video processing device provided in this embodiment specifically includes: a first acquisition module 21, a second acquisition module 22 and a streaming module 23.

[0084] Among them, the first acquisition module 21 is used to collect classroom live video and microphone audio, generate first stream data and push it to the cloud server;

[0085] The second acquisition module 22 is used for acquiring the rendering view of the teaching whiteboard view control based on the screen acquisition thread when detecting that the teaching whiteboard view control is running, performing resolution compression, format conversion and encoding on the rendering view, and generating second stream data in a specified format;

[0086] The streaming module 23 is used to push the first stream data and the second stream data to the cloud server in real time based on the acquisition timestamp, so as to perform stream mixing and distribution through the cloud server.

[0087] Specifically, based on the picture acquisition thread, the rendering view of the teaching whiteboard view control is collected, and also includes:

[0088] The collection time interval of the rendering view is determined based on the frame rate of the second stream data, and the rendering view is collected according to the collection time interval and the collection time consumption of the rendering view.

[0089] The rendering view is collected according to the collection time interval and the collection time consumption of the rendering view, including:

[0090] When the acquisition time interval is greater than the acquisition time consumption, the sleep time duration is determined based on the acquisition time interval and the acquisition time consumption. After each screen acquisition thread completes acquiring a rendering view, the system sleeps based on the sleep time duration.

[0091] By putting the screen acquisition thread to sleep, you can reduce resource usage and lower system performance costs.

[0092] Specifically, when the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, it also includes:

[0093] The multimedia audio data inside the teaching whiteboard is synchronously collected, an audio stream is generated based on the multimedia audio data, and the audio stream is placed into the second stream data.

[0094] Among them, synchronously collecting multimedia audio data inside the teaching whiteboard and generating an audio stream based on the multimedia audio data include:

[0095] Obtain screen recording permission, enable multimedia sound collection service based on screen recording permission, and set multimedia sound collection configuration instance according to multimedia sound collection service;

[0096] Create an audio recording object based on the multimedia sound acquisition configuration instance, synchronously collect the multimedia audio data inside the teaching whiteboard through the audio recording object, and transcode the multimedia audio data into an audio stream in a specified format.

[0097] By calling the multimedia sound collection service to collect multimedia audio data inside the whiteboard, external noise interference can be avoided and the collection of teaching whiteboard stream data can be optimized.

[0098] Specifically, the resolution of the rendered view is compressed, including:

[0099] Comparing a preset resolution list according to the resolution of the rendered view, where the resolution list includes a plurality of preset target resolutions;

[0100] Compresses the resolution of the render view down to the closest target resolution.

[0101] In the above, by collecting the classroom live video and microphone audio, the first stream data is generated and pushed to the cloud server; when the teaching whiteboard view control is detected to be running, the rendering view of the teaching whiteboard view control is collected based on the screen acquisition thread, and the rendering view is compressed, converted and encoded to generate the second stream data in the specified format; the first stream data and the second stream data are pushed to the cloud server in real time based on the acquisition timestamp, so as to be mixed and distributed through the cloud server. By adopting the above technical means, by collecting the rendering view of the whiteboard view control, the stream data of the teaching whiteboard display screen is generated based on the collected rendering view, so that the teaching whiteboard display screen can be pushed to the cloud server together with the stream data of the classroom live broadcast in the form of a video stream, so as to realize the sharing of the teaching whiteboard display screen and the classroom live broadcast video in the interactive classroom live broadcast scene, optimize the interactive classroom live broadcast effect, and enhance the interactive classroom live broadcast experience. In addition, this solution collects the rendering view of the teaching whiteboard view control through the screen acquisition thread, which can realize the accurate acquisition of the whiteboard display screen, filter irrelevant display information, and improve the acquisition efficiency and acquisition accuracy of the whiteboard display screen.

[0102] The interactive classroom live video processing device provided in Example 2 of the present application can be used to execute the interactive classroom live video processing method provided in Example 1 above, and has corresponding functions and beneficial effects.

[0103] Embodiment three:

[0104] Embodiment 3 of the present application provides an electronic device, referring to Figure 5The electronic device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The number of processors in the electronic device can be one or more, and the number of memories in the electronic device can be one or more. The processor, memory, communication module, input device and output device of the electronic device can be connected via a bus or other means.

[0105] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the interactive classroom live video processing method described in any embodiment of the present application (for example, the first acquisition module, the second acquisition module and the streaming module in the interactive classroom live video processing device). The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0106] The communication module is used for data transmission.

[0107] The processor executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory, thereby realizing the above-mentioned interactive classroom live video processing method.

[0108] The input device can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device can include display devices such as display screens.

[0109] The electronic device provided above can be used to execute the interactive classroom live video processing method provided in the above embodiment 1, and has corresponding functions and beneficial effects.

[0110] Embodiment 4:

[0111] An embodiment of the present application also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute a method for processing interactive classroom live video. The method comprises: collecting classroom live video and microphone audio, generating a first stream of data and pushing it to a cloud server; when detecting that a teaching whiteboard view control is running, collecting a rendering view of the teaching whiteboard view control based on a screen acquisition thread, performing resolution compression, format conversion and encoding on the rendering view, and generating a second stream of data in a specified format; and pushing the first stream of data and the second stream of data to the cloud server in real time based on the acquisition timestamp, so as to mix and distribute the data through the cloud server.

[0112] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0113] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the interactive classroom live video processing method described above, and can also execute related operations in the interactive classroom live video processing method provided in any embodiment of the present application.

[0114] The interactive classroom live video processing device, storage medium and electronic device provided in the above embodiments can execute the interactive classroom live video processing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the interactive classroom live video processing method provided in any embodiment of the present application.

[0115] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for processing live interactive classroom videos. It is characterized in that include: Collect classroom live video and microphone audio, generate first-class data and push it to the cloud server; When the teaching whiteboard view control is detected to be running, the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, and the rendering view is compressed, format converted and encoded to generate second stream data in a specified format; The first stream data and the second stream data are pushed to the cloud server in real time based on the acquisition timestamp, so as to be mixed and distributed through the cloud server.

2. The interactive classroom live video processing method according to claim 1, It is characterized in that The rendering view of the teaching whiteboard view control is collected based on the screen collection thread, and further includes: The acquisition time interval of the rendering view is determined based on the frame rate of the second stream data, and the rendering view is acquired according to the acquisition time interval and the acquisition time consumption of the rendering view.

3. The interactive classroom live video processing method according to claim 2, It is characterized in that The collecting the rendering view according to the collection time interval and the collection time consumption of the rendering view includes: When the acquisition time interval is greater than the acquisition time consumption, the sleep time duration is determined based on the acquisition time interval and the acquisition time consumption, and each time the screen acquisition thread completes acquiring one of the rendering views, the screen acquisition thread sleeps based on the sleep time duration.

4. The interactive classroom live video processing method according to claim 1, It is characterized in that When the rendering view of the teaching whiteboard view control is collected based on the screen collection thread, it also includes: The multimedia audio data inside the teaching whiteboard is synchronously collected, an audio stream is generated based on the multimedia audio data, and the audio stream is placed into the second stream data.

5. The interactive classroom live video processing method according to claim 4, It is characterized in that The synchronously collecting multimedia audio data inside the teaching whiteboard and generating an audio stream based on the multimedia audio data include: Obtain screen recording permission, enable a multimedia sound collection service based on the screen recording permission, and set a multimedia sound collection configuration instance according to the multimedia sound collection service; An audio recording object is created based on the multimedia sound collection configuration instance, and multimedia audio data inside the teaching whiteboard is synchronously collected through the audio recording object, and the multimedia audio data is transcoded into the audio stream in a specified format.

6. The interactive classroom live video processing method according to claim 1, It is characterized in that Performing resolution compression on the rendered view, including: Comparing a preset resolution list according to the resolution of the rendered view, the resolution list including a plurality of preset target resolutions; The resolution of the rendered view is compressed down to the closest target resolution.

7. An interactive classroom live video processing system, including a lecture terminal, a lecture terminal and a cloud server; The main lecture terminal is used to execute the interactive classroom live video processing method as described in any one of claims 1-6; The cloud server is used to receive the first stream data and the second stream data transmitted by the lecture end, perform stream data processing on the first stream data and the second stream data, and distribute the mixed streams to the listening end.

8. The interactive classroom live video processing system according to claim 7, It is characterized in that The listening end also includes a stage end, which is used to collect third-stream data during the live broadcast of the class and push the third-stream data to the cloud server for mixing and distribution.

9. An interactive classroom live video processing device, It is characterized in that include: The first acquisition module is used to collect classroom live video and microphone audio, generate first-stream data and push it to the cloud server; A second acquisition module is used for acquiring a rendering view of the teaching whiteboard view control based on a screen acquisition thread when detecting that the teaching whiteboard view control is running, performing resolution compression, format conversion and encoding on the rendering view, and generating a second stream data in a specified format; The streaming module is used to push the first stream data and the second stream data to the cloud server in real time based on the acquisition timestamp, so as to mix and distribute them through the cloud server.

10. An electronic device, It is characterized in that include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the interactive classroom live video processing method as described in any one of claims 1-6.

11. A storage medium containing computer executable instructions, It is characterized in that The computer executable instructions, when executed by a computer processor, are used to execute the interactive classroom live video processing method as described in any one of claims 1-6.

Images