Content coding adjustment method for AI online education

Abstract

The invention discloses a content coding adjustment method for AI online education, which is applied among a server, a teacher end and all student ends, and comprises the following steps: coding captured video data and audio data by the teacher end and each student end to obtain original video streams of each end, and sending the original video streams to the server; and the server stores the related parameters of each original video stream to a mapping table. According to the AI online education content coding adjustment method, the pressure transmission is reduced, the data transmission efficiency is improved, the network bandwidth requirement is reduced by merging the multiple paths of original video streams of the student end, meanwhile, the teacher end or the student end can request video streams with different resolutions and formats according to the network condition and the processing capacity of the teacher end or the student end, and the user experience is improved. Fluency of watching experience is ensured; and meanwhile, by reducing the number of video streams, the processing burden of teacher-side and student-side equipment is relieved, and CPU and memory resources are saved.

Description

Technical Field

[0001] The present invention belongs to the field of online education technology, and specifically relates to a content coding adjustment method for AI online education. Background Art

[0002] In online education, teachers and students use a specific online education platform for distance learning. Teachers and students typically log in to the education platform through electronic devices such as computers or tablets equipped with cameras and microphones to enter a virtual classroom. A common video layout is for the teacher's video to take center stage, while the student's video window is smaller and arranged as a pane to the side or bottom of the screen. Teachers and students can communicate visually through the camera, seeing each other's expressions and body language; they can communicate in real time through a microphone and speaker; and in the chat window, students can type questions or teachers can post text messages. Teachers can also use the education platform for screen sharing, whiteboard interaction, and other functions.

[0003] From the student's perspective, the video screen at the student end will include a teacher video screen and multiple student video screens (pane format, for example, 2 2, 3 3, etc., and supports pane switching to support more student video screen display).

[0004] In the existing technology, to realize the video screen of the student end, the traditional solution is to receive at least one teacher end video stream and multiple student end video streams. The size of the video pane determines the number of video streams that the student end needs to receive. For example, if you configure one teacher end video and one 2 2 student panes need to receive 5 video streams (1 teacher + 4 students), and configure 1 teacher video and a 3 A pane of 3 needs to receive 10 video streams (1 teacher + 9 students). Although video compression (video streams are compressed before transmission, achieved through encoding technologies such as H.264 or H.265, to reduce the size of video files while minimizing quality loss) and adaptive streaming (automatically adjusting video quality according to the student's network status to ensure a smooth viewing experience, that is, if the student's network bandwidth is limited, the video stream will automatically reduce quality to maintain smooth playback) and other technologies will be used to reduce bandwidth requirements, in general, the student side needs to receive multiple video streams, which still has a large bandwidth and decoding pressure. Each video stream requires a certain amount of bandwidth to transmit, and the decoder on the student side needs to process these streams to display the video. Moreover, as the pane size increases (such as 3 4,4 4, etc.), the problem will become more prominent. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems raised in the background technology and propose a content coding adjustment method for AI online education.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The present invention proposes a content coding adjustment method for AI online education, which is applied between a server, a teacher terminal, and all student terminals, wherein:

[0008] The teacher and each student end encode the captured video data and audio data to obtain the original video stream of each end, and send the original video stream to the server. The server stores the relevant parameters of each original video stream in the mapping table;

[0009] The server sends each received original video stream according to the original video channel number requested by the teacher or student. The teacher and student terminals play and display each received original video stream in the first page of the preset area in the form of preset panes, and each window in the pane plays and displays an original video stream of the student terminal.

[0010] The teacher or student terminal monitors the frame rate of each original video stream of the current window pane played locally in real time. When the frame rate is less than a first preset value, the teacher or student terminal sends a request message for merging video streams to the server, requesting that the original video streams of multiple student terminals in the current window pane of the student terminal be merged into one.

[0011] The server calculates the ID number of the original video stream of the student terminal associated with the request information according to the parameters of the request information for merging video streams, merges the original video streams of the student terminal associated with the ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream in the mapping table;

[0012] The server then sends a request response message for merging the video streams to the teacher or student terminal. When the operation result of the request response message for merging the video streams is successful, the server sends a merged video stream with corresponding parameters to the teacher or student terminal.

[0013] After the teacher or student receives the combined video stream, it plays and displays it in the current pane;

[0014] The teacher or student side also switches the layout type, page number and video stream resolution of the pane that plays and displays the merged video stream.

[0015] Preferably, the ID number sequence of all student-side original video streams is ;

[0016] For the original video stream of the student end played and displayed in the pane by the teacher or student end, if there is no student speaking, the original video stream of the student end is displayed in each window one by one according to the sequence of ID numbers, and the order in which the original video stream of the student end is displayed in all windows is called the video stream order, and is expressed as ,in Indicates the first The video stream corresponding to each window, and ;

[0017] When there is a student speaking, the original video stream position of the student end corresponding to the speaking student is swapped with the window position of the first row and column in the pane of the first page, and the server regenerates the video stream of the pane where the two swapped window positions are located to achieve the position swap of the two windows. Every time the position swap occurs, the server records the latest video stream order.

[0018] Preferably, when the teacher or student terminal switches the layout type, page number or video stream resolution of the pane for playing and displaying the merged video stream, respectively, it sends a request message for switching the pane layout type, a request message for switching the pane page number or a request message for switching the video stream resolution to the server;

[0019] The server searches the mapping table for the corresponding merged video stream according to the pane layout type switching request information, the pane page number switching request information, or the video stream resolution switching request information.

[0020] If the request is not found in the mapping table, the server calculates the ID number of the original video stream of the associated student end according to the parameters of the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution, and according to the latest video stream sequence, and performs the latest merger of the original video streams of the student end with the associated ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream in the mapping table;

[0021] When the corresponding merged video stream is found or after the latest merge, the server sends a request response message for pane layout type switching, a request response message for pane page number switching, or a request response message for video stream resolution switching to the teacher or student end. When the operation result of the request response message for pane layout type switching, the request response message for pane page number switching, or the request response message for video stream resolution switching is successful, the server sends a merged video stream with corresponding parameters to the teacher or student end.

[0022] Preferably, the request information for merging video streams, the request information for switching pane layout types, the request information for switching pane page numbers, and the request information for switching video stream resolution all include parameters of request information ID, requester ID, course instance ID, operation type, pane layout type, pane page number, and video stream resolution;

[0023] The request response message for merging video streams, the request response message for switching pane layout types, the request response message for switching pane page numbers, and the request response message for switching video stream resolution all include a response message ID, a requester ID, a course instance ID, an operation result, a merged video stream ID, and a merged video address.

[0024] Preferably, the server calculates the ID number of the original video stream of the student end associated with the request information based on the parameters in the request information for merging video streams, the request information for switching pane layout types, the request information for switching pane page numbers, or the request information for switching video stream resolution, merges the original video streams of the student end with the associated ID number to obtain a corresponding merged video stream, and stores the relevant parameters of the merged video stream in a mapping table, including:

[0025] The server switches the video stream according to the request information for merging the video streams, the request information for switching the pane layout type, the request information for switching the pane page number, or the pane layout type in the request information for switching the video stream resolution. and pane page numbers , and according to the latest video stream order , the original video stream ID number sequence of the student terminal associated with the request information is calculated as , and merge the original video streams of the students with the associated ID numbers into a merged video;

[0026] Transcoding the obtained resolution of the merged video stream to a corresponding video resolution in the request information for the merged video stream, the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution;

[0027] Generate a merged video stream ID and a merged video stream address for the transcoded merged video stream;

[0028] A mapping relationship is established between the merged video stream and the corresponding pane layout type, pane page number, video stream resolution, merged video stream ID, merged video stream address and ID number of the associated student-end original video stream, and stored in a mapping table. For the original video stream, the mapping table stores multiple video resolutions, multiple original video stream IDs and multiple original video stream addresses corresponding to each original video stream.

[0029] Preferably, for the teacher side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the first area of ​​the playback interface, and the original video stream received from the teacher side is displayed in the second area of ​​the playback interface, and the playback interface is further provided with a third area for switching the pane layout type in the first area, a fourth area for switching the pane page number in the first area, and a fifth area for switching the resolution of each video stream in the first area;

[0030] For the student side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the sixth area of ​​the playback interface, and the original video stream received from the teacher side is displayed in the seventh area of ​​the playback interface. The playback interface also has an eighth area for switching the pane layout type in the sixth area, a ninth area for switching the pane page number in the sixth area, and a tenth area for switching the resolution of each video stream in the sixth area.

[0031] Preferably, the content encoding adjustment method for AI online education further includes the teacher or student end double-clicking the single-window video stream screen in the pane that plays and displays the merged video stream to zoom in or out:

[0032] When double-clicking to zoom in, the teacher or student first calculates the number of rows and columns in the pane where the double-click event is located, and the pane layout type is , the coordinates of the upper left corner of the pane are , the position coordinates of the lower right corner of the pane are , the position coordinates of the double-click event are , then the double-click event is in the number of rows in the pane , double-click the column number in the event pane ,in Indicates rounding down;

[0033] When the calculation is completed, the teacher or student sends a request message to the server for zooming in on the single-window video stream in the pane, and the request message includes a request message ID, a requester ID, a course instance ID, an operation type, a pane layout type, a pane page number, a zoomed-in video stream resolution, and the number of rows and columns in the pane where the double-click event is located.

[0034] After receiving the request message for zooming in on the single-window video stream in the pane, the server searches the mapping table for the original video stream ID sequence of the student end associated with the merged video stream based on the pane layout type, pane page number, and zoomed-in video stream resolution in the request message, and obtains the ID number of the original video stream of the student end corresponding to the double-click event based on the number of rows and columns in the pane where the double-click event is located and the pane layout type;

[0035] Then search the mapping table for the student end original video stream corresponding to the ID number and the original video stream ID and original video stream address of the corresponding video stream resolution;

[0036] The server sends a request response message to the teacher or student terminal to enlarge the single screen in the pane. The request response message includes the response message ID, the requester ID, the course instance ID, the operation result, the original video stream ID and the original video address corresponding to the enlarged single window;

[0037] If the operation result in the response message is successful, the server sends the original video stream of the student end with the corresponding ID number and video stream resolution to the requester of the corresponding ID;

[0038] After receiving the original video stream, the requester of the corresponding ID plays the original video stream in a form covering the entire first area or the sixth area.

[0039] Preferably, when the enlarged single-window video stream screen is double-clicked to reduce, the student terminal or the teacher terminal sends a request message for double-clicking to reduce the enlarged single-window video stream screen to the server, and the request message includes a request message ID, a requester ID, a course instance ID, an operation type, a pane layout type, a pane page number, and a video stream resolution;

[0040] After receiving the request message for double-clicking to zoom out the enlarged single-window video stream, the server searches the mapping table for the corresponding merged video stream according to the pane layout type, pane page number, and video stream resolution in the request message;

[0041] The server sends a request response message to the teacher or student end for the enlarged single-window video stream screen to be reduced by double-clicking. The request response message includes the response message ID, requester ID, course instance ID, operation result, merged video stream ID and merged video address. When the operation result is successful, the server sends the merged video stream with corresponding parameters to the teacher or student end.

[0042] Preferably, the content encoding adjustment method for AI online education further includes adaptive adjustment of video resolution:

[0043] When the size of the playback interface of the teacher's terminal or the student's terminal is smaller than a second preset value, the teacher's terminal or the student's terminal determines whether the resolution of each video stream displayed on the current playback interface is a preset minimum resolution. If the resolution of any video stream is not at the preset minimum resolution, a request message for switching the resolution of the corresponding video stream is sent to the server, requesting the resolution of the video stream to be adjusted, and the video stream resolution in the request message is the preset minimum resolution.

[0044] When the playback interface of the teacher's or student's end is greater than or equal to the second preset value, the teacher's or student's end determines whether the resolution of each video stream displayed on the current playback interface is the preset highest-level resolution. When the resolution of any video stream is not at the preset highest level, a request message for switching the video stream resolution requesting the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the preset highest level.

[0045] Preferably, when the video resolution is adaptively adjusted, the process also includes: when the teacher end or the student end is stuck in playing the merged video stream, the teacher end or the student end determines whether the resolution of each video stream displayed on the current playback interface is at a preset minimum gear; if the resolution of any video stream is not at the preset minimum gear, a request message for switching the resolution of the video stream to request the server to adjust the resolution of the corresponding video stream, and the video stream resolution in the request message is the video resolution of the first preset gear;

[0046] When adaptively adjusting the video resolution, it also includes when the teacher's or student's end is in the playback process of playing and displaying the merged video stream, and the CPU usage of the teacher's or student's end is lower than the third preset value, the teacher's or student's end determines whether the resolution of each video stream played and displayed on the current playback interface is at the preset highest level. When there is a video stream whose resolution is not at the preset highest level, a request message for switching the video stream resolution to request the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the video resolution of the second preset level.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This content encoding adjustment method for AI online education combines multiple original video streams on the student side, reduces pressure transmission, improves data transmission efficiency, and reduces network bandwidth requirements. At the same time, the teacher or student side can request video streams of different resolutions and formats according to their own network conditions and processing capabilities, ensuring a smooth viewing experience. At the same time, by reducing the number of video streams, the processing burden of the teacher and student devices is reduced, saving CPU and memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the playback interface display of the teacher terminal of the present invention;

[0050] Figure 2 This is the playback interface display of the student terminal of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] like Figure 1-Figure 2 As shown, a content encoding adjustment method for AI online education is provided, which is applied between the server, the teacher end, and all student ends, wherein:

[0053] The teacher and each student end encode the captured video data and audio data (specifically, the teacher and each student end capture the video data and audio data through their respective cameras and microphones and encode them), obtain the original video streams of each end (that is, obtain the original video stream of the teacher end and the original video streams of each student end), and send the original video streams to the server (the teacher and each student end send their own encoded original video streams to the server), and the server stores the relevant parameters of each original video stream in the mapping table;

[0054] The server sends the received original video streams according to the original video channels requested by the teacher or student. The teacher and student display the received original video streams in the first page of the preset area in the form of preset panes (such as 2 2 pane types) for playback and display, and each window in the pane plays and displays the original video stream of a student end.

[0055] Among them, for the teacher side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the first area of ​​the playback interface (each pane occupies one page), and the original video stream received from the teacher side is displayed in the second area of ​​the playback interface. The playback interface also has a third area for switching the pane layout type in the first area, a fourth area for switching the pane page number in the first area, and a fifth area for switching the resolution of each video stream in the first area (such as Figure 1 As shown, in Figure 1 In the video, the left pane is the first area, the original video stream of the teacher's terminal in the lower right corner is the second area, the pane switching is the third area, the page number switching is the fourth area, and the resolution switching is the fifth area);

[0056] For the student side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the sixth area of ​​the playback interface (each pane occupies one page), and the original video stream received from the teacher side is displayed in the seventh area of ​​the playback interface. The playback interface also has an eighth area for switching the pane layout type in the sixth area, a ninth area for switching the pane page number in the sixth area, and a tenth area for switching the resolution of each video stream in the sixth area (such as Figure 2 As shown, in Figure 2 In the diagram, the right pane is the sixth area, the teacher's original video stream on the left is the seventh area, pane switching is the eighth area, page switching is the ninth area, and resolution switching is the tenth area).

[0057] The teacher or student side monitors the frame rate of each original video stream of the current pane played locally in real time. When the frame rate is less than the first preset value (for example, the frame rate of one of the original video streams is less than the first preset value), the teacher or student side sends a request message for merging video streams to the server, requesting that the original video streams of multiple student sides in the student side pane be merged into one (for example, for 22 students, if the frame rate is 2 2 panes are split, which involves 6 merged video streams (22=4×5+2), that is, the first 5 merged video streams all contain 4 (2 2) student-end video screens, the sixth one includes the remaining 2 student-end video screens; if according to 3 If the video is split into 3 panes, it involves 3 merged video streams (22=9×2+4), that is, the first 2 merged video streams contain 9 (3 3) student-end video screens, and the third one contains the remaining 4 student-end video screens);

[0058] The server calculates the original video stream ID number of the student terminal associated with the request information based on the parameters of the request information for merging video streams (i.e., the request information ID, the requester ID (including the teacher terminal ID and the IDs of all student terminals), the course instance ID, the operation type, the pane layout type, the pane page number, and the video stream resolution), merges the original video streams of the student terminal with the associated ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream (i.e., the pane layout type, the pane page number, the video stream resolution, the merged video stream ID, the merged video stream address, and the ID number of the associated student terminal original video stream) in the mapping table;

[0059] The server then sends a request response message for merging the video streams to the teacher or student terminal. When the operation result of the request response message for merging the video streams is successful, the server sends a merged video stream with corresponding parameters to the teacher or student terminal.

[0060] After the teacher or student receives the merged video stream, it is played and displayed in the current pane (the merged video stream is played and displayed in the pane in the following way: each window corresponds to the original video stream before the merge).

[0061] In one embodiment, the teacher or student terminal further switches the layout type, page number, and video stream resolution of the pane for playing and displaying the merged video stream.

[0062] When the teacher or student terminal switches the layout type, page number or video stream resolution of the pane for playing and displaying the merged video stream, it sends a request message for switching the pane layout type, a request message for switching the pane page number or a request message for switching the video stream resolution to the server respectively;

[0063] The server searches the mapping table for the corresponding merged video stream according to the pane layout type switching request information, the pane page number switching request information, or the video stream resolution switching request information.

[0064] If the request is not found in the mapping table, the server calculates the ID number of the original video stream of the associated student end according to the parameters of the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution, and according to the latest video stream sequence, and performs the latest merger of the original video streams of the student end with the associated ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream in the mapping table;

[0065] When the corresponding merged video stream is found or after the latest merge, the server sends a request response message for pane layout type switching, a request response message for pane page number switching, or a request response message for video stream resolution switching to the teacher or student end. When the operation result of the request response message for pane layout type switching, the request response message for pane page number switching, or the request response message for video stream resolution switching is successful, the server sends a merged video stream with corresponding parameters to the teacher or student end.

[0066] It should be noted that the request information for merging video streams, the request information for switching pane layout types, the request information for switching pane pages, and the request information for switching video stream resolution all include the request information ID, the requester ID (including the teacher's ID and the IDs of all student terminals), the course instance ID, the operation type, the pane layout type, the pane page number, and the parameters of the video stream resolution. (The operation type in the request information for merging video streams is to merge the original video streams of the student terminal, and the operation type in the request information for switching pane layout types is to switch pane layout types. In this embodiment, the pane layout types have 1 1, 2 2 and 3 3, the operation type in the request information for pane page switching is pane page switching, and the operation type in the request information for video stream resolution switching is video stream resolution switching. In this embodiment, the types of video stream resolutions (including the original video stream of the teacher end, the original video stream of the student end, and the merged video stream) are 480P, 720P, and 1080P);

[0067] The request response message for merging video streams, the request response message for switching pane layout types, the request response message for switching pane page numbers and the request response message for switching video stream resolution all include the response message ID, requester ID, course instance ID, operation result, merged video stream ID and merged video address (the operation result in the request response message for merging video streams is the operation result of success or failure in merging video streams, the operation result in the request response message for switching pane layout types is the operation result of success or failure in switching pane layout types, the operation result in the request response message for switching pane page numbers is the operation result of success or failure in switching pane page numbers, and the operation result in the request response message for switching video stream resolution is the operation result of success or failure in switching video stream resolution).

[0068] In one embodiment, the ID number sequence of all student-side original video streams is ;

[0069] For the original video stream of the student side played and displayed in the pane by the teacher or student side, if there is no student speaking (and no student speaking has appeared yet), the original video stream of the student side will be displayed in each window one by one according to the sequence of ID numbers. The order in which the original video stream of the student side is displayed in all windows is called the video stream order and is expressed as ,in Indicates the first The video stream corresponding to each window (that is, after all windows in all panes are arranged in order, the first The video stream corresponding to each window, that is, no matter The video stream corresponding to each window is the original video stream of the student terminal in the ID number sequence. To express the window corresponding to the video stream), and (In which, the windows are sorted in the order of the pane page numbers and from left to right and then from top to bottom in a pane. For example, there are two pages, 8 students, and the pane layout type is 2 2, the four windows in the pane with page number 1 are the first window, the second window, the third window, and the fourth window from left to right and from top to bottom, and the four windows in the pane with page number 2 are the fifth window, the sixth window, the seventh window, and the eighth window from left to right and from top to bottom);

[0070] When there is a student speaking, the original video stream position of the student end corresponding to the speaking student is swapped with the window position of the first row and column in the pane of the first page, and the server regenerates the video stream of the pane where the two swapped window positions are located to achieve the position swap of the two windows. Every time the position swap occurs, the server records the latest video stream order.

[0071] In one embodiment, the server calculates the ID number of the original video stream of the student end associated with the request information based on the parameters in the request information for merging video streams, the request information for switching pane layout types, the request information for switching pane page numbers, or the request information for switching video stream resolutions, merges the original video streams of the student end associated with the ID number to obtain a corresponding merged video stream, and stores the relevant parameters of the merged video stream in a mapping table, including:

[0072] The server responds to the request for merging video streams, switching pane layout types, switching pane pages, or switching video stream resolution based on the pane layout type in the request. and pane page numbers , and according to the latest video stream order , the original video stream ID number sequence of the student terminal associated with the request information is calculated as (This sequence is obtained by sorting the windows in the pane from left to right and then from top to bottom), where, Indicates the first The video stream corresponding to each window, Indicates the first The video stream corresponding to each window is obtained, and the original video stream of the student end with the associated ID number is merged into a merged video;

[0073] Transcoding the obtained resolution of the merged video stream to a corresponding video resolution in the request information for the merged video stream, the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution;

[0074] Generate a merged video stream ID and a merged video stream address for the transcoded merged video stream;

[0075] A mapping relationship is established between the merged video stream and the corresponding pane layout type, pane page number, video stream resolution, merged video stream ID, merged video stream address and ID number of the associated student-side original video stream, and stored in a mapping table. For the original video stream, the mapping table stores multiple video resolutions (480P, 720P and 1080P) corresponding to each original video stream, multiple original video stream IDs and multiple original video stream addresses. The mapping table is shown in Table 1 (where N / A in Table 1 means empty):

[0076] Table 1

[0077] In one embodiment, the content encoding adjustment method for AI online education further includes the teacher or student end double-clicking a single-window video stream image in a pane that plays and displays the merged video stream to zoom in or out:

[0078] When double-clicking to zoom in, the teacher or student first calculates the number of rows and columns in the pane where the double-click event is located, and the pane layout type is , the coordinates of the upper left corner of the pane are , the position coordinates of the lower right corner of the pane are , the position coordinates of the double-click event are , then the double-click event is in the number of rows in the pane , double-click the column number in the event pane ,in Indicates rounding down;

[0079] When the calculation is completed, the teacher or student sends a request message to the server for zooming in on the video stream in the single window of the pane. The request message includes the request message ID, the requester ID, the course instance ID, the operation type, the pane layout type, the pane page number, the zoomed-in video stream resolution, and the number of rows and columns in the pane where the double-click event occurs (the operation type in the request message for zooming in on the single window in the pane is: zooming in on the single window in the pane).

[0080] After receiving the request information for zooming in on the single-window video stream in the pane, the server searches the mapping table for the original video stream ID sequence of the associated student terminal corresponding to the merged video stream according to the pane layout type, pane page number, and video stream resolution in the request message, and finds the original video stream ID sequence of the associated student terminal according to the number of rows and columns in the pane and the pane layout type of the double-click event. , using the formula The double-click event is calculated as the first ID in the original video stream ID sequence of the associated student terminal. The original video stream of the student end is obtained, and then the ID number of the original video stream of the student end corresponding to the double-click event is obtained (for example, the original video stream sequence of the student end with the associated ID number is , according to the double-click event, the number of rows and columns in the pane is the second row and the second column and the pane layout type is 2 2. Using formula The calculation shows that the double-click event is the 4th student-end original video stream in the original video stream ID number sequence of the associated student end, and the ID number of the student-end original video stream corresponding to the double-click event is 8);

[0081] Then, the original video stream ID and original video stream address of the student end corresponding to the ID number and the corresponding video stream resolution (i.e., the amplified video stream resolution) are searched in the mapping table;

[0082] The server sends a request response message to the teacher or student terminal to enlarge the single screen in the pane. The request response message includes the response message ID, the requester ID, the course instance ID, the operation result, the original video stream ID and the original video address corresponding to the enlarged single window;

[0083] If the operation result in the response message is successful, the server sends the original video stream of the student end with the corresponding ID number and video stream resolution (i.e., the magnified video stream resolution) to the requester with the corresponding ID;

[0084] After receiving the original video stream, the requester of the corresponding ID plays the original video stream in a form that covers the entire first area or the sixth area;

[0085] When the enlarged single-window video stream screen is double-clicked to reduce, the student or teacher terminal sends a request message to the server for double-clicking to reduce the enlarged single-window video stream screen, and the request message includes the request message ID, requester ID, course instance ID, operation type, pane layout type, pane page number, and video stream resolution (the operation type is double-clicking to reduce the enlarged single-window video stream screen);

[0086] After receiving the request message for double-clicking to zoom out the enlarged single-window video stream, the server searches the mapping table for the corresponding merged video stream according to the pane layout type, pane page number, and video stream resolution in the request message;

[0087] The server sends a request response message to the teacher or student end for the zoomed-in single-window video stream screen to be double-clicked to zoom out. The request response message includes the response message ID, requester ID, course instance ID, operation result, merged video stream ID and merged video address. When the operation result is successful, the server sends a merged video stream with corresponding parameters to the teacher or student end (to enable the server to switch from 1-channel single student-end original video stream to 1-channel virtual merged video stream before the original video stream is zoomed in).

[0088] In one embodiment, the content encoding adjustment method for AI online education further includes adaptive adjustment of video resolution:

[0089] When the size of the playback interface of the teacher's terminal or the student's terminal is smaller than a second preset value, the teacher's terminal or the student's terminal determines whether the resolution of each video stream displayed on the current playback interface is a preset minimum resolution. If the resolution of any video stream is not at the preset minimum resolution, a request message for switching the resolution of the corresponding video stream is sent to the server, requesting the resolution of the video stream to be adjusted, and the video stream resolution in the request message is the preset minimum resolution.

[0090] When the playback interface of the teacher's or student's end is greater than or equal to the second preset value, the teacher's or student's end determines whether the resolution of each video stream displayed on the current playback interface is the preset highest-level resolution. When the resolution of any video stream is not at the preset highest level, a request message for switching the video stream resolution requesting the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the preset highest level.

[0091] When the video resolution is adaptively adjusted, the teacher or student terminal also includes the following steps: when the teacher or student terminal is playing a merged video stream and the playback is stuck (obtained by detecting the frame rate), the teacher or student terminal determines whether the resolution of each video stream played and displayed on the current playback interface is at a preset minimum gear; if the resolution of any video stream is not at the preset minimum gear, a request message for switching the video stream resolution to request the server to adjust the resolution of the corresponding video stream, and the video stream resolution in the request message is the video resolution of the first preset gear;

[0092] When adaptively adjusting the video resolution, it also includes when the teacher's or student's end is in the playback process of playing and displaying the merged video stream, and the CPU usage of the teacher's or student's end is lower than the third preset value, the teacher's or student's end determines whether the resolution of each video stream played and displayed on the current playback interface is at the preset highest level. When there is a video stream whose resolution is not at the preset highest level, a request message for switching the video stream resolution to request the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the video resolution of the second preset level.

[0093] The present application also includes a content coding adjustment device for AI online education, comprising a processor and a memory storing a plurality of computer instructions. When the computer instructions are executed by the processor, a process of a content coding adjustment method for AI online education is implemented.

[0094] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A content coding adjustment method for AI online education, characterized by: The method is applied between the server, the teacher terminal and all the student terminals, wherein: The teacher and each student end encode the captured video data and audio data to obtain the original video stream of each end, and send the original video stream to the server. The server stores the relevant parameters of each original video stream in the mapping table; The server sends each received original video stream according to the original video channel number requested by the teacher or student. The teacher and student terminals play and display each received original video stream in the first page of the preset area in the form of preset panes, and each window in the pane plays and displays an original video stream of the student terminal. The teacher or student terminal monitors the frame rate of each original video stream of the current window pane played locally in real time. When the frame rate is less than a first preset value, the teacher or student terminal sends a request message for merging video streams to the server, requesting that the original video streams of multiple student terminals in the current window pane of the student terminal be merged into one. The server calculates the ID number of the original video stream of the student terminal associated with the request information according to the parameters of the request information for merging video streams, merges the original video streams of the student terminal associated with the ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream in the mapping table; The server then sends a request response message for merging the video streams to the teacher or student terminal. When the operation result of the request response message for merging the video streams is successful, the server sends a merged video stream with corresponding parameters to the teacher or student terminal. After the teacher or student receives the combined video stream, it plays and displays it in the current pane; The teacher or student side also switches the layout type, page number and video stream resolution of the pane that plays and displays the merged video stream.

2. The content encoding adjustment method for AI online education according to claim 1, characterized in that: The ID number sequence of all student-side original video streams is ; For the original video stream of the student end played and displayed in the pane by the teacher or student end, if there is no student speaking, the original video stream of the student end is displayed in each window one by one according to the sequence of ID numbers, and the order in which the original video stream of the student end is displayed in all windows is called the video stream order, and is expressed as ,in Indicates the first The video stream corresponding to each window, and ; When there is a student speaking, the original video stream position of the student end corresponding to the speaking student is swapped with the window position of the first row and column in the pane of the first page, and the server regenerates the video stream of the pane where the two swapped window positions are located to achieve the position swap of the two windows. Every time the position swap occurs, the server records the latest video stream order.

3. The content encoding adjustment method for AI online education according to claim 2, characterized in that: When the teacher or student terminal switches the layout type, page number or video stream resolution of the pane for playing and displaying the merged video stream, it sends a request message for switching the pane layout type, switching the pane page number or switching the video stream resolution to the server respectively; The server searches the mapping table for the corresponding merged video stream according to the pane layout type switching request information, the pane page number switching request information, or the video stream resolution switching request information. If the request is not found in the mapping table, the server calculates the ID number of the original video stream of the associated student end according to the parameters of the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution, and according to the latest video stream sequence, and performs the latest merger of the original video streams of the student end with the associated ID number to obtain the corresponding merged video stream, and stores the relevant parameters of the merged video stream in the mapping table; When the corresponding merged video stream is found or after the latest merge, the server sends a request response message for pane layout type switching, a request response message for pane page number switching, or a request response message for video stream resolution switching to the teacher or student end. When the operation result of the request response message for pane layout type switching, the request response message for pane page number switching, or the request response message for video stream resolution switching is successful, the server sends a merged video stream with corresponding parameters to the teacher or student end.

4. The content encoding adjustment method for AI online education according to claim 3, characterized in that: The request information for merging video streams, request information for switching pane layout types, request information for switching pane pages, and request information for switching video stream resolution all include parameters of request information ID, requester ID, course instance ID, operation type, pane layout type, pane page number, and video stream resolution; The request response message for merging video streams, the request response message for switching pane layout types, the request response message for switching pane page numbers, and the request response message for switching video stream resolution all include a response message ID, a requester ID, a course instance ID, an operation result, a merged video stream ID, and a merged video address.

5. The content encoding adjustment method for AI online education according to claim 4, characterized in that: The server calculates the ID number of the original video stream of the student end associated with the request information based on the parameters in the request information for merging video streams, the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution, merges the original video streams of the student end associated with the obtained ID number to obtain a corresponding merged video stream, and stores the relevant parameters of the merged video stream in a mapping table, including: The server switches the video stream according to the request information for merging the video streams, the request information for switching the pane layout type, the request information for switching the pane page number, or the pane layout type in the request information for switching the video stream resolution. and pane page numbers , and according to the latest video stream order , the original video stream ID number sequence of the student terminal associated with the request information is calculated as , and merge the original video streams of the students with the associated ID numbers into a merged video; Transcoding the obtained resolution of the merged video stream to a corresponding video resolution in the request information for the merged video stream, the request information for switching the pane layout type, the request information for switching the pane page number, or the request information for switching the video stream resolution; Generate a merged video stream ID and a merged video stream address for the transcoded merged video stream; A mapping relationship is established between the merged video stream and the corresponding pane layout type, pane page number, video stream resolution, merged video stream ID, merged video stream address and ID number of the associated student-end original video stream, and stored in a mapping table. For the original video stream, the mapping table stores multiple video resolutions, multiple original video stream IDs and multiple original video stream addresses corresponding to each original video stream.

6. The content encoding adjustment method for AI online education according to claim 1, characterized in that: For the teacher side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the first area of ​​the playback interface, and the original video stream received from the teacher side is displayed in the second area of ​​the playback interface. The playback interface is also provided with a third area for switching the pane layout type in the first area, a fourth area for switching the pane page number in the first area, and a fifth area for switching the resolution of each video stream in the first area; For the student side, the original video stream or merged video stream received from the student side is displayed in the form of a pane in the sixth area of ​​the playback interface, and the original video stream received from the teacher side is displayed in the seventh area of ​​the playback interface. The playback interface also has an eighth area for switching the pane layout type in the sixth area, a ninth area for switching the pane page number in the sixth area, and a tenth area for switching the resolution of each video stream in the sixth area.

7. The content encoding adjustment method for AI online education according to claim 1, characterized in that: The content encoding adjustment method for AI online education further includes the teacher or student end double-clicking a single window video stream image in a pane that plays and displays the merged video stream to zoom in or out: When double-clicking to zoom in, the teacher or student first calculates the number of rows and columns in the pane where the double-click event is located, and the pane layout type is , the coordinates of the upper left corner of the pane are , the position coordinates of the lower right corner of the pane are , the position coordinates of the double-click event are , then the double-click event is in the number of rows in the pane , double-click the column number in the event pane ,in Indicates rounding down; When the calculation is completed, the teacher or student sends a request message to the server for zooming in on the single-window video stream in the pane, and the request message includes a request message ID, a requester ID, a course instance ID, an operation type, a pane layout type, a pane page number, a zoomed-in video stream resolution, and the number of rows and columns in the pane where the double-click event is located. After receiving the request message for zooming in on the single-window video stream in the pane, the server searches the mapping table for the original video stream ID sequence of the student end associated with the merged video stream based on the pane layout type, pane page number, and zoomed-in video stream resolution in the request message, and obtains the ID number of the original video stream of the student end corresponding to the double-click event based on the number of rows and columns in the pane where the double-click event is located and the pane layout type; Then search the mapping table for the student end original video stream corresponding to the ID number and the original video stream ID and original video stream address of the corresponding video stream resolution; The server sends a request response message to the teacher or student terminal to enlarge the single screen in the pane. The request response message includes the response message ID, the requester ID, the course instance ID, the operation result, the original video stream ID and the original video address corresponding to the enlarged single window; If the operation result in the response message is successful, the server sends the original video stream of the student end with the corresponding ID number and video stream resolution to the requester of the corresponding ID; After receiving the original video stream, the requester of the corresponding ID plays the original video stream in a form covering the entire first area or the sixth area.

8. The content encoding adjustment method for AI online education according to claim 7, characterized in that: When the enlarged single-window video stream screen is double-clicked to shrink, the student terminal or the teacher terminal sends a request message to the server for double-clicking to shrink the enlarged single-window video stream screen, and the request message includes the request message ID, the requester ID, the course instance ID, the operation type, the pane layout type, the pane page number, and the video stream resolution; After receiving the request message for double-clicking to zoom out the enlarged single-window video stream, the server searches the mapping table for the corresponding merged video stream according to the pane layout type, pane page number, and video stream resolution in the request message; The server sends a request response message to the teacher or student end for the enlarged single-window video stream screen to be reduced by double-clicking. The request response message includes the response message ID, requester ID, course instance ID, operation result, merged video stream ID and merged video address. When the operation result is successful, the server sends the merged video stream with corresponding parameters to the teacher or student end.

9. The content encoding adjustment method for AI online education according to claim 1, characterized in that: The content encoding adjustment method of the AI ​​online education also includes adaptive adjustment of video resolution: When the size of the playback interface of the teacher's terminal or the student's terminal is smaller than a second preset value, the teacher's terminal or the student's terminal determines whether the resolution of each video stream displayed on the current playback interface is a preset minimum resolution. If the resolution of any video stream is not at the preset minimum resolution, a request message for switching the resolution of the corresponding video stream is sent to the server, requesting the resolution of the video stream to be adjusted, and the video stream resolution in the request message is the preset minimum resolution. When the playback interface of the teacher's or student's end is greater than or equal to the second preset value, the teacher's or student's end determines whether the resolution of each video stream displayed on the current playback interface is the preset highest-level resolution. When the resolution of any video stream is not at the preset highest level, a request message for switching the video stream resolution requesting the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the preset highest level.

10. The content encoding adjustment method for AI online education according to claim 9, characterized in that: When the video resolution is adaptively adjusted, when the teacher or student end is playing a merged video stream and the playback is stuck, the teacher or student end determines whether the resolution of each video stream displayed on the current playback interface is at a preset minimum gear. If the resolution of any video stream is not at the preset minimum gear, a request message for switching the resolution of the corresponding video stream is sent to the server, and the video stream resolution in the request message is the video resolution of the first preset gear. When adaptively adjusting the video resolution, it also includes when the teacher's or student's end is in the playback process of playing and displaying the merged video stream, and the CPU usage of the teacher's or student's end is lower than the third preset value, the teacher's or student's end determines whether the resolution of each video stream played and displayed on the current playback interface is at the preset highest level. When there is a video stream whose resolution is not at the preset highest level, a request message for switching the video stream resolution to request the corresponding video stream resolution adjustment is sent to the server, and the video stream resolution in the request message is the video resolution of the second preset level.

Images