A video communication method and related apparatus

Abstract

Embodiments of the present application disclose a video communication method and related equipment, which are used for saving bandwidth transmission resources in a video communication process. The method of the embodiments of the present application comprises: a terminal acquires real-time character video data, the terminal acquires character image model driving parameters according to the real-time character video data, the terminal sends the character image model driving parameters to a server, the server takes the character image model driving parameters as input, adopts a character image model corresponding to the terminal to obtain the output of a dynamic character image, and acquires first interface display data comprising the dynamic character image according to the dynamic character image.

Description

Technical Field

[0001] This application relates to the field of communications, and more particularly to a video communication method and a terminal device. Background Technology

[0002] With the development of modern communication technologies and services, people's communication needs have shifted from the initial single voice requirement to the need for video and audio communication. Video communication services that integrate the transmission of voice, data, and video have become a hot topic in the field of communication and are being used more and more widely in areas such as video conferencing, remote video medical care, and remote video education.

[0003] One method of video communication involves one terminal receiving real-time video recordings of people captured by another terminal via a camera, playing the video data from the real-time video recordings through a screen or monitor, and playing the audio data from the real-time video recordings through speakers or headphones.

[0004] However, real-time video recordings of people are large in volume. When there are many people participating in video communication, such as in large-scale online conferences or remote video education scenarios, a single terminal receiving real-time video recordings from multiple terminals simultaneously will consume a large amount of bandwidth, leading to unstable data transmission and high bandwidth resource costs. Summary of the Invention

[0005] This application provides a video communication method and related equipment to save bandwidth resources and reduce bandwidth usage in video communication.

[0006] The first aspect of this application provides a video communication method:

[0007] The terminal uses a front-facing camera to capture real-time video recordings of the user. These recordings can be of the user's upper body, full body, or face.

[0008] The terminal uses an AR engine to analyze and calculate real-time video recordings of people to obtain the driving parameters of the user's character model. These driving parameters represent all or part of the user's actions.

[0009] After the terminal analyzes and calculates the real-time human image video to obtain the human image model driving parameters, it sends the human image model driving parameters to the server. The server takes the human image model driving parameters as input, uses the human image model, and obtains the output of a dynamic human image. Based on the dynamic human image, it obtains the data for the first interface display. The data for the first interface display includes the dynamic human image, the human image model, and the corresponding terminal.

[0010] After the terminal sends the character model driving parameters to the server, it will receive the first interface display data from the server, which includes the dynamic character image driven by the character model driving parameters.

[0011] It is understandable that the terminal only sends the character image model driving parameters to the server, and the terminal receives the first interface display data, including the character image model, sent by the server. The bandwidth occupied during the transmission is much less than that occupied by the transmission of real-time character video recordings. Video communication through dynamic character image models saves bandwidth resources.

[0012] Based on the first aspect, the embodiments of this application provide a first implementation of the first aspect:

[0013] The terminal's display device displays data on the first interface. The terminal user can create a request based on the currently displayed data. For example, by clicking on a character model through a touchscreen or mouse, the user can indicate that they want to conduct a one-to-one video communication with the terminal user corresponding to that character model. The operation request based on the data displayed on the first interface can be defined by the user. After the terminal creates an operation request based on the data displayed on the first interface, it sends the operation request to the server, which then adjusts the arrangement and combination of the dynamic characters according to the operation request and obtains the data displayed on the second interface.

[0014] After the terminal sends an operation request, it will receive data from the server showing the second interface after adjusting the dynamic character image according to the operation request.

[0015] Based on the first aspect or the first embodiment of the first aspect, this application provides a second embodiment of the first aspect in a real-time example:

[0016] After the terminal sends the character image model driving parameters to the server, it can not only obtain the driven character image model, but also some text information about the terminal user's current state obtained by the server after analyzing the character image model driving parameters, such as sleeping, serious, etc.

[0017] Based on the first aspect to the second implementation of the first aspect, this application provides a third implementation of the first aspect:

[0018] The driving parameters for the character model include at least one of the following: human posture parameters and facial expression parameters.

[0019] Based on the first aspect to the second implementation of the first aspect, this application provides a fourth implementation of the first aspect:

[0020] Character models include at least one of virtual character models and 3D character scan models.

[0021] A second aspect of the embodiments of this application provides a video communication method:

[0022] The server receives human posture parameters and facial expression parameters obtained by analyzing and calculating real-time video recordings of people.

[0023] The server is pre-configured with various character models. Through pre-communication with the terminal, the server determines the character model corresponding to the terminal. After receiving the human posture parameters and facial expression parameters sent by the terminal, the server calls the corresponding character model of the terminal, takes the human posture parameters and facial expression parameters as input, and obtains the output of a dynamic character image. This dynamic character image can express all or part of the terminal user's actions expressed by the human posture parameters and facial expression parameters.

[0024] After obtaining the dynamic human figure driven by human posture parameters and facial image parameters, the server can create an immersive scene display screen based on the dynamic human figure and application requirements. The scene display screen can include one or more dynamic human figures, and the positions between the dynamic human figures can be set as needed.

[0025] After receiving the human posture parameters and facial expression parameters sent by the terminal, the server drives the character model to obtain the first interface display data of the scene with realism and immersion, and then synchronously sends the first interface display data to all terminals.

[0026] Based on the second aspect, the embodiments of this application provide a first implementation of the second aspect:

[0027] The server receives operation requests created by end users on the terminal based on the data displayed on the first interface. For example, clicking on a character model via a touch screen or mouse indicates a desire to conduct one-to-one video communication with the end user corresponding to that character model. The operation requests based on the data displayed on the first interface can be defined by the user.

[0028] After receiving the operation request from the terminal, the server adjusts the arrangement and combination of the dynamic characters set in the data displayed on the first interface according to the operation request. After rearranging or combining the dynamic characters, the data displayed on the second interface is obtained.

[0029] The server sends the adjusted second-interface display data to the terminal based on the operation request.

[0030] Based on the second aspect or the first implementation of the second aspect, this application provides a second implementation of the second aspect:

[0031] The server can analyze the driving parameters of the character image model to obtain some text information that indicates the current state of the end user, such as sleeping or being serious.

[0032] The server sends status text information to the terminal, obtained after analyzing the character image driving parameters.

[0033] Based on the second aspect and the second implementation method of the second aspect, this application provides a third implementation method of the second aspect:

[0034] The driving parameters for the character model include at least one of the following: human posture parameters and facial expression parameters.

[0035] Based on the second aspect and the second implementation method of the second aspect, this application provides a fourth implementation method of the second aspect:

[0036] Character models include at least one of virtual character models and 3D character scan models.

[0037] A third aspect of this application provides a terminal that implements the functions of the terminal described in the first aspect. These functions can be implemented in hardware or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the aforementioned functions.

[0038] A fourth aspect of this application provides a server that implements the functions of the server described in the second aspect above. These functions can be implemented in hardware or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the aforementioned functions.

[0039] A fifth aspect of this application provides a terminal, including a processor, a memory, an input / output device, and a bus;

[0040] The processor, memory, input / output devices are connected to the bus;

[0041] The processor is used to execute the method described in the first aspect above.

[0042] A sixth aspect of this application provides a server, including a processor, a memory, an input / output device, and a bus;

[0043] The processor, memory, input / output devices are connected to the bus;

[0044] The processor is used to execute the method described in the second aspect above.

[0045] A seventh aspect of this application provides a computer-readable storage medium storing a program that, when executed by a computer, performs the process of the method described in any one of the first to second aspects.

[0046] An eighth aspect of this application provides a computer program product that, when executed on a computer, allows the computer to perform the process of any one of the methods described in the first to second aspects.

[0047] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The terminal acquires real-time human video recording data, the terminal acquires human image model driving parameters based on the human video recording data, the terminal sends the human image model driving parameters to the server, the server takes the human image model driving parameters as input, uses the human image model to obtain a dynamic human image output, and acquires first interface display data based on the dynamic human image. The first interface display data includes the dynamic human image, the human image model corresponds to the terminal, and the terminal receives the first interface display data sent by the server. The terminal only sends the human image model driving parameters to the server and receives the first interface display data including the human image model sent by the server. The bandwidth occupied during transmission is much less than the bandwidth occupied by the transmission of real-time human video recordings. Video communication through dynamic human image models saves bandwidth resources. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the architecture of the video communication system in the embodiments of this application;

[0049] Figure 2 A flowchart illustrating a video communication method in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the character model in the embodiments of this application;

[0051] Figure 4a This is a schematic diagram of a remote video education scenario in an embodiment of this application;

[0052] Figure 4b This is another flowchart illustrating the video communication method in this application embodiment;

[0053] Figure 5 A schematic diagram illustrating the effect of displaying data in a live classroom for students.

[0054] Figure 6 A schematic diagram illustrating the effect of displaying data in a live classroom for teachers.

[0055] Figure 7 A schematic diagram illustrating the display effect of dynamic character images in classroom management;

[0056] Figure 8 A schematic diagram illustrating the effect of displaying data on the interface in a group discussion scenario;

[0057] Figure 9 This is a schematic diagram of the terminal structure in an embodiment of this application;

[0058] Figure 10 This is a schematic diagram of the server structure in an embodiment of this application;

[0059] Figure 11 This is a schematic diagram of the hardware structure of a student terminal in an embodiment of this application;

[0060] Figure 12 This is a schematic diagram of the software architecture of a student terminal in an embodiment of this application;

[0061] Figure 13 This is a schematic diagram of a hardware structure of a live streaming server in an embodiment of this application;

[0062] Figure 14 This is a schematic diagram of a software architecture for a live streaming server in an embodiment of this application;

[0063] Figure 15 This is a schematic diagram of the hardware structure of a teacher terminal in an embodiment of this application;

[0064] Figure 16 This is a schematic diagram of the software architecture of a teacher terminal in an embodiment of this application;

[0065] Figure 17 This is a schematic diagram of the terminal structure in an embodiment of this application;

[0066] Figure 18 This is a schematic diagram of the server structure in an embodiment of this application. Detailed Implementation

[0067] This application provides a video communication method for saving bandwidth transmission resources.

[0068] With the development of modern communication technologies and services, people's communication needs have shifted from the initial single voice requirement to the need for video and audio communication. Video communication services that integrate the transmission of voice, data, and video have become a hot topic in the field of communication and are being used more and more widely in areas such as video conferencing, remote video medical care, and remote video education.

[0069] In scenarios such as video conferencing and remote video education, there are usually many users involved in the video communication process. The current main method of video communication is to transmit real-time video recordings to multiple terminals simultaneously.

[0070] However, real-time video recordings consume significant bandwidth. Due to bandwidth limitations, achieving ideal video communication results is difficult, and only a limited number of video streams can be accessed simultaneously. Taking remote video education as an example, each student can only see the real-time videos of a few classmates at a time. With classmates' videos refreshing in turn, it's impossible to simultaneously and intuitively perceive the dynamics of all classmates, lacking the immersive and intuitive experience of a traditional classroom. This impacts learning efficiency, causing a significant drop in grades for many children during online learning during the pandemic.

[0071] Therefore, there is an urgent need for a new video communication method that can reduce bandwidth usage, save bandwidth resources, and enhance the immersive experience of video communication.

[0072] This application provides a video communication system architecture; please refer to [link / reference]. Figure 1 :

[0073] Multiple terminals are remotely connected to a remote server. There is no direct connection between the first terminal and a second or other terminals. All data processing activities are handled centrally by the remote server. Each terminal can transmit data to the remote server, which can process the data transmitted by each terminal using cloud services and simultaneously send the processed data back to each terminal, thus achieving remote video communication.

[0074] based on Figure 1 The system architecture for remote video communication shown is illustrated below. The video communication method in the embodiments of this application is described below:

[0075] Please see Figure 2 In this embodiment, the terminal analyzes and calculates real-time human video recording data to obtain human posture parameters and facial expression parameters. After sending the human posture parameters and facial expression parameters to the server, the server drives the human image model to obtain a dynamic human image and further obtains the first interface display data. The terminal can obtain an operation request based on the first interface display data and send the operation request to the server so that the server can further obtain the adjusted second interface display data according to the operation request and the dynamic human image. After the server sends it to the terminal, the terminal displays the second interface display data.

[0076] 201. The terminal acquires real-time video recordings of people.

[0077] The terminal uses a front-facing camera to capture real-time video recordings of the user. These recordings can be of the user's upper body, full body, or face.

[0078] 202. The terminal obtains human posture parameters and facial expression parameters based on real-time video recordings of people.

[0079] The terminal uses an AR engine to analyze and calculate real-time video recordings of people, obtaining the user's body posture parameters and facial expression parameters. These parameters are used to represent all or part of the user's actions.

[0080] Human posture parameters can include the data shown in Table 1 below:

[0081] Table 1

[0082]

[0083] Facial expression parameters can include the data shown in Table 2 below:

[0084] Table 2

[0085]

[0086] 203. The terminal sends human posture parameters and facial expression parameters to the server.

[0087] After the terminal analyzes and calculates the human posture parameters and facial expression parameters by the real-time human video recording, it sends the human posture parameters and facial expression parameters to the server.

[0088] 204. The server takes human posture parameters and facial expression parameters as input, uses a human image model, and outputs a dynamic human image.

[0089] The server is pre-configured with various character models. Through pre-communication with the terminal, the server determines the character model corresponding to the terminal. After receiving the human posture parameters and facial expression parameters sent by the terminal, the server calls the corresponding character model of the terminal, takes the human posture parameters and facial expression parameters as input, and obtains the output of a dynamic character image. This dynamic character image can express all or part of the terminal user's actions expressed by the human posture parameters and facial expression parameters.

[0090] The server can analyze human posture parameters and facial expression parameters to obtain text information that indicates the current state of the end user, such as whether they are sleeping or focused.

[0091] The name of the terminal user corresponding to the character model can be displayed around the character model.

[0092] It should be noted that the character model can be a virtual character or other character models, such as a 3D character scan model; there are no specific limitations here.

[0093] Character model such as Figure 3 As shown, Figure 3Image (A) shows a virtual character model, which can be customized or uses existing virtual character models. Figure 3 Figure (B) shows a 3D human figure scan model. A model corresponding to a real person can be created by scanning a real person with a laser. Then, the human posture parameters and facial expression parameters of the real person can be used to drive the real person model, which can be more vivid, lifelike and expressive.

[0094] 205. The server obtains the data displayed on the first screen based on the dynamic character image.

[0095] After obtaining the dynamic human figure driven by human posture parameters and facial image parameters, the server can create an immersive scene display screen based on the dynamic human figure and application requirements. The scene display screen can include one or more dynamic human figures, and the positions between the dynamic human figures can be set as needed.

[0096] 206. The server sends the data for the first screen display to the terminal.

[0097] After receiving the human posture parameters and facial expression parameters sent by the terminal, the server drives the character model to obtain the first interface display data of the scene with realism and immersion, and then synchronously sends the first interface display data to all terminals.

[0098] 207. The terminal displays the data from the first interface on the terminal screen.

[0099] After receiving the first interface display data sent by the server, the terminal displays the first interface display data on the terminal's display device for the terminal user to view.

[0100] 208. The terminal displays a data acquisition operation request based on the first interface.

[0101] The terminal's display device displays data on the first interface. The terminal user can create requests based on the currently displayed data. For example, by clicking on a character model through a touch screen or mouse, the user can indicate that they want to conduct a one-to-one video communication with the terminal user corresponding to that character model. The operation requests based on the data displayed on the first interface can be defined by the user.

[0102] 209. The terminal sends an operation request to the server.

[0103] After creating an operation request based on the data displayed on the first interface, the terminal sends the operation request to the server.

[0104] 210. The server obtains the data for displaying the second interface based on the operation request and the dynamic character image.

[0105] After receiving the operation request from the terminal, the server adjusts the arrangement and combination of the dynamic characters set in the data displayed on the first interface according to the operation request. After rearranging or combining the dynamic characters, the data displayed on the second interface is obtained.

[0106] The second interface can also display data that includes status text information indicating operation requests. For example, after issuing a "raise hand" operation request, the corresponding dynamic character model will display the "raise hand" text information.

[0107] 211. The server sends the data for the second interface display to the terminal.

[0108] 212. The terminal displays the data from the second interface on the terminal screen.

[0109] Steps 211 to 212 in this embodiment are similar to steps 206 to 207 in this embodiment, and will not be described in detail here.

[0110] Understandably, the terminal only sends human posture parameters and facial expression parameters to the server, and receives the first interface display data, including the human image model, sent by the server. The bandwidth occupied during the transmission is much less than that occupied by the transmission of real-time human video recordings. Video communication through dynamic human image models saves bandwidth resources; and the display effect can be further adjusted according to operation requests to obtain the second interface display data, making the display effect more vivid and lifelike.

[0111] Based on the above description, the following describes an application scenario of the video communication method in this application:

[0112] Please see Figure 4a The video communication method in this embodiment can be applied to remote video education scenarios. Remote video education scenarios include student terminals and teacher terminals. Student terminals communicate with other student terminals or teacher terminals via a remote server, while teacher terminals communicate with student terminals via a remote server or perform classroom management.

[0113] based on Figure 4a The following describes the video communication method in the remote video education scenario illustrated in this application embodiment:

[0114] The video communication method in the remote video education scenario in this application embodiment is divided into two parts: the first part is the synthesis of live classroom display data, and the other part is classroom management based on the synthesized live classroom.

[0115] First, the method for synthesizing live classrooms in the embodiments of this application will be introduced. Please refer to [link / reference needed]. Figure 4b :

[0116] In this embodiment, the live streaming server drives the corresponding character model through the human posture parameters and facial expression parameters of students or teachers, and receives the teacher's real-time lecture video and the teacher's terminal demonstration video to synthesize a live classroom scene display interface that meets the learning needs.

[0117] 401. Student terminals acquire real-time student video recording data.

[0118] 402. The student terminal acquires the student's body posture parameters and the student's facial expression parameters.

[0119] 403. The student terminal sends the student's body posture parameters and the student's facial expression parameters to the live streaming server.

[0120] It should be noted that in this embodiment, steps 401 to 403 are... Figure 2 Steps 201 to 203 in the illustrated embodiment are similar and will not be described in detail here.

[0121] 404. Teacher terminal obtains real-time teacher video recording data.

[0122] It should be noted that step 404 in this embodiment is similar to step 401 in this embodiment, and will not be described in detail here.

[0123] 405. The teacher's terminal sends real-time teacher recording data to the live streaming server.

[0124] After acquiring real-time teacher recording data, the teacher terminal can directly send the real-time teacher recording data to the live streaming server to provide students with a direct experience of the teacher's lecture during the live class.

[0125] 406. The teacher terminal acquires the teacher's body posture parameters and the teacher's facial expression parameters.

[0126] 407. The teacher's terminal sends the teacher's body posture parameters and the teacher's facial expression parameters to the live streaming server.

[0127] It should be noted that steps 406 to 407 in this embodiment are similar to steps 402 to 403 in this embodiment, and will not be described in detail here.

[0128] It should be noted that when steps 406 and 407 are executed, step 405 may not be executed in this embodiment; when step 405 is executed, steps 406 and 407 may not be executed in this embodiment; or steps 405, 406 and 407 may all be executed.

[0129] 408. Teacher terminal obtains demonstration video data.

[0130] The teacher's terminal acquires demonstration screens used for detailed explanations during the teacher's presentation. These screens can be recorded data, playing videos, or slideshow presentations.

[0131] 409. The teacher's terminal sends demonstration video data to the live streaming server.

[0132] The teacher's terminal sends the demonstration video data to the live streaming server.

[0133] It should be noted that steps 404 to 409 can be executed before step 401, after step 403, or at any time between steps 401 and 403; there are no specific limitations here.

[0134] 410. The live streaming server synthesizes dynamic character images.

[0135] It should be noted that in this embodiment, step 410 and... Figure 2 Step 204 in the illustrated embodiment is similar and will not be described in detail here.

[0136] It should be noted that in this embodiment, step 410 can be executed after step 409, or after step 403 or step 407, and the specific execution is not limited here.

[0137] 411. The live streaming server synthesizes the data displayed in the live classroom.

[0138] The live classroom data displayed by the live streaming server includes both student-side and teacher-side live classroom data.

[0139] The content framework of a live online class for teachers or students is as follows: Figure 5 As shown in Figure (A), the live classroom display data synthesized by the live server for teachers or students includes not only dynamic human figures, but also teacher demonstration screens and real-time recordings of teacher lectures. In addition, some control menus for classroom management can be set up, which contain commonly used functions in the classroom.

[0140] The display effect of the live classroom on the student's end is as follows: Figure 5 As shown in Figure (B), the student's body posture parameters and facial expression parameters are used to drive the cartoon virtual character model to form a character image set; the terminal user, namely "the boss (self)," is highlighted in the character image set; the upper left corner displays the geometric problem displayed on the teacher's terminal, and the upper right corner displays the real-time recording of the teacher's live lecture. Since the real-time recording of the teacher's lecture has already appeared, the teacher's character image can be omitted from the dynamic character image set, and only the dynamic character images of the students can be added. Of course, the teacher's dynamic character image can also be added when needed to make the live classroom scene more vivid.

[0141] The display effect of the live classroom on the teacher's end is as follows: Figure 6 As shown, a particular student may not be highlighted in the dynamic character set; otherwise, the display effect is similar to that of a live classroom on the student's end.

[0142] Based on the above introduction, the classroom management scenario in this application embodiment will be described below:

[0143] The control menu includes the following function options in the data displayed in the live classroom on the student's end:

[0144] Table 3

[0145]

[0146] The control menu in the teacher's live classroom display data includes the following functional options, as shown in Table 4:

[0147] Table 4

[0148]

[0149] When a student terminal sends a hand-raising request to the live streaming server, the live streaming server can display a "hand raised" status around the dynamic avatar of the student terminal that initiated the hand-raising request, and forward the hand-raising request to the teacher terminal. When the teacher terminal initiates a "agree to raise hands" request, the live streaming server forwards the student terminal's real-time audio data to all terminals participating in the live class.

[0150] When a teacher initiates a question request, the teacher can directly select the student's corresponding dynamic avatar and click the question button. After the teacher's question request is sent to the live streaming server, the server will directly initiate a connection request for the student's device to connect to the audio. The student's device system automatically captures the audio through the microphone (the microphone is already turned on when joining the virtual classroom) and transmits the student's audio to the live streaming server, where it is combined with the teacher's audio and broadcast to all students and teachers' devices. This eliminates the need for students to connect to the microphone, simplifying student classroom operations.

[0151] For easier classroom management, some student information needs to be displayed in text format. Please refer to [link / reference]. Figure 7 :

[0152] Analyzing the student's posture and facial expression parameters reveals that the student's status is "attentive in class". When a student sends a question request, the word "Question" can be displayed around the student's dynamic avatar, and the student's name is displayed below the corresponding dynamic avatar.

[0153] When a teacher initiates a request to enable group discussion restrictions, students can request to have one-on-one conversations with other students. A student or teacher can only appear in one discussion group at a time. The live streaming server groups and arranges the dynamic avatars of students and teachers according to their groups. Voice data is synthesized and sent based on the discussion groups selected by the students and teachers, achieving the effect of group discussions without interference between groups. The display effect of group discussions is as follows: Figure 8 As shown.

[0154] Based on the above description, the video communication method in this application embodiment can also be applied to large-scale online conference scenarios. The conference host terminal is similar to the teacher terminal, and the participants are similar to the student terminal. The display effect of the display interface data and the function options of the control menu can be adjusted according to actual needs.

[0155] The video communication method in the embodiments of this application has been described above. The terminal and server in the embodiments of this application are described below.

[0156] Please see Figure 9 In one embodiment of this application, the terminal structure includes an acquisition unit 901, a sending unit 902, and a receiving unit 903.

[0157] Acquisition unit 901 is used to acquire real-time video recordings of people;

[0158] Acquisition unit 901 is used to acquire the driving parameters of the character image model based on the character video recording data;

[0159] The sending unit 902 is used to send character image model driving parameters to the server, so that the server takes the character image model driving parameters as input, adopts the character image model, obtains the output of dynamic character image, and obtains the first interface display data based on the dynamic character image. The first interface display data includes the dynamic character image, the character image model and the terminal correspondence.

[0160] The receiving unit 903 is used to receive the first interface display data sent by the server.

[0161] The sending unit 902 is used to send an operation request to the server based on the data displayed on the first interface, so that the server can obtain the data displayed on the second interface according to the operation request and the dynamic character image.

[0162] The receiving unit 903 is used to receive the second interface display data sent by the server.

[0163] The receiving unit 903 is used to receive status text information sent by the server, which indicates the changing status of the dynamic character image.

[0164] Please see Figure 10In one embodiment of this application, the server structure includes a receiving unit 1001, an acquiring unit 1002, a processing unit 1003, and a sending unit 1004.

[0165] The receiving unit 1001 is used to receive the character image model driving parameters sent by the terminal;

[0166] Processing unit 1003 is used to take the character image model driving parameters as input, use the character image model, and obtain the output of dynamic character image. The character image model corresponds to the terminal.

[0167] The acquisition unit 1002 is used to acquire the first interface display data based on the dynamic character image, and the first interface display data includes the dynamic character image.

[0168] The sending unit 1004 is used to send the first interface display data to the terminal.

[0169] The receiving unit 1001 is used to receive an operation request sent by the terminal based on the data displayed on the first interface;

[0170] The acquisition unit 1002 is used to acquire data displayed on the second interface according to the operation request;

[0171] The sending unit 1004 is used to send the second interface display data to the terminal.

[0172] The acquisition unit 1002 is used to acquire status text information based on the character image model driving parameters;

[0173] The sending unit 1004 is used to send status text information to the terminal.

[0174] Based on the above description, the following section describes a hardware structure of the student terminal in an embodiment of this application. Please refer to [link / reference]. Figure 11 :

[0175] The video capture module in the student terminal is responsible for capturing video of students in class, which can be upper body or full-body video. The human posture and facial expression calculation module analyzes the captured video to determine the current human body state and facial expression data of the students. This data can include the following information: whether the student is in the current camera, the position and posture data of the student's body, hands, and head, and the student's facial expression parameters. Facial expression parameters can express various states of the eyes, mouth, nose, and face, such as the degree of opening and closing of the left eye and the direction of the right eye's gaze. Various facial expressions are provided in the form of multiple parameters. Huawei's AR Engine uses 64 parameters to express facial expressions, and Apple's AR... Kit uses 51 parameters to represent facial expressions, then packages the student's body posture and facial expression parameters and sends them to the live streaming server via a network connection. The sound acquisition module is responsible for collecting the student's voice. When a student needs to speak, the teacher can use the student's virtual avatar to turn on the microphone and activate the sound acquisition module. Students can also actively turn on their microphones to collect sound when communicating with other students before, during, and after class. All collected sound needs to be sent to the live streaming server via a network connection. The network connection module is responsible for establishing a connection with the live streaming server and transmitting data between the student's end and the live streaming server. The network connection module sends the student's body posture and facial expression data, as well as the student's actions, from the student's device. The network connection module receives data from the live streaming server, including the teacher's device display, lecture video, and the physical states, facial expressions, and operation data of all students and teachers. The user input module receives UI input from students, such as raising their hands, initiating conversations with other students, and adjusting the virtual classroom synthesis format via their mobile phones or tablets. The virtual classroom synthesis module is a key module on the student's device. It combines the teacher's device display and lecture video received from the live streaming server with the physical postures and facial expressions of all students and teachers into a complete virtual classroom interactive interface for the student. The video and audio are then output through the monitor and speakers, respectively.

[0176] The following describes a software architecture of the student terminal in an embodiment of this application. Please refer to [link / reference]. Figure 12 :

[0177] The virtual classroom student terminal system software architecture includes video acquisition module, sound acquisition module, human posture and facial expression calculation module, network connection module, user input module, and virtual classroom synthesis module. All of these modules are developed based on the system adaptation layer, which can be adapted to Windows systems on PCs, Android systems on mobile phones and tablets, or iOS systems.

[0178] The following describes a hardware structure of the live streaming server in an embodiment of this application. Please refer to [link / reference]. Figure 13 :

[0179] The live streaming server includes a virtual classroom management service responsible for creating the virtual classroom (i.e., creating a virtual classroom service instance, a student operation request service instance, and a teacher operation request service instance), and responsible for teachers and all students finding and logging into the virtual classroom (i.e., connecting to the correct student operation request service and teacher operation request service); a virtual classroom synthesis service responsible for collecting the body postures and facial expressions of all students in the entire classroom, the teacher's device interface video, the teacher's lecture video, and the voice data of each student and teacher, and sending this data as needed according to the different situations of each student and teacher (because some students only watch the teacher's device interface video or the teacher's lecture video), and managing the voice call relationships between students; a student operation request service responsible for handling the operation requests of each student, such as raising their hand, talking to other students individually, etc.; a teacher operation request service responsible for handling the teacher's operation requests, such as calling roll, inviting students to speak, and managing classroom discipline, etc.; and a network connection module responsible for establishing network connection relationships with the devices of teachers and all students, and sending and receiving various network data.

[0180] The following describes a software architecture of the live streaming server in an embodiment of this application. Please refer to [link / reference]. Figure 14 :

[0181] The software architecture of the virtual classroom live streaming server system includes a virtual classroom management service module, a virtual classroom synthesis service module, a network connection module, a student operation request service module, and a teacher operation request service module. All of these modules are developed based on the system adaptation layer, which can be adapted to Windows server, Linux server, and UNIX server systems, respectively.

[0182] The following describes a hardware structure of the teacher terminal in an embodiment of this application. Please refer to [link / reference]. Figure 15 :

[0183] The teacher's device system includes a video capture module responsible for capturing the teacher's lecture video; a human posture and facial expression calculation module responsible for calculating the teacher's human posture and facial expression, outputting the human posture and facial expression parameters, packaging them, and sending them to the live streaming server via the network connection module. The teacher's lecture video is also sent to the live streaming server simultaneously and then to the virtual classroom synthesis module to synthesize the virtual classroom on the teacher's device; an audio capture module captures the teacher's lecture audio and sends it directly to the live streaming server via the network connection module; a user input module receives input from the teacher, such as taking attendance, inviting students to speak, and managing classroom discipline; and a virtual classroom synthesis module responsible for combining the human posture, facial expression, and operation data of all students sent from the live streaming server with the teacher's current lecture content and lecture video to synthesize the virtual classroom on the teacher's device, and outputting the display from the monitor and the sound from the speaker.

[0184] The following describes a software architecture of the teacher terminal in an embodiment of this application. Please refer to [link / reference]. Figure 16 :

[0185] The software architecture of the virtual classroom teacher's side system is similar to that of the student's side system software architecture. The application's video acquisition module, sound acquisition module, human posture and facial expression calculation module, network connection module, user input module, and virtual classroom synthesis module all develop corresponding functions based on the system adaptation layer. The system adaptation layer can be adapted to the Windows system of PCs, and the Android or iOS systems of mobile phones and tablets, respectively.

[0186] For example, Figure 17 A schematic diagram of a terminal structure is shown.

[0187] The terminal 1700 may include a tagging module 1701, a packet-splitting module 1702, a sensor module 1703, an internal memory 1704, an external memory interface 1705, a processor 1706, an audio module 1707, a display screen 1708, an indicator 1709, a USB interface 1710, a charging management module 1711, a power management module 17111, a battery 17112, a mobile communication module 1712, a wireless communication module 1713, an antenna 1, an antenna 2, etc.

[0188] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on terminal 1700. In other embodiments, terminal 1700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components can be implemented entirely in hardware, entirely in software, or a combination of software and hardware.

[0189] The wireless communication function of terminal 1700 can be implemented through antenna 1, antenna 2, mobile communication module 1712, wireless communication module 1713, modem processor and baseband processor.

[0190] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 1700 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0191] Mobile communication module 1712 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal 1700. Mobile communication module 1713 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. Mobile communication module 1712 can receive electromagnetic waves through antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to modem processor for demodulation. Mobile communication module 1712 can also amplify the signal modulated by modem processor and convert it into electromagnetic waves for radiation through antenna 1. In some embodiments, some functional modules of mobile communication module 1712 may be located in processor 1706.

[0192] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device or displays an image or video through the display screen 1708. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 1706 and may be housed in the same device as the mobile communication module 1712 or other functional modules.

[0193] The wireless communication module 1713 can provide solutions for wireless communication applications on the terminal 1700, including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, and infrared (IR) technologies. The wireless communication module 1713 can be one or more devices integrating at least one communication processing module. The wireless communication module 1713 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 1706. The wireless communication module 1713 can also receive signals to be transmitted from processor 1706, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0194] In some embodiments, antenna 1 of terminal 1700 is coupled to mobile communication module 1712, and antenna 2 is coupled to wireless communication module 1713, enabling terminal 1700 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0195] In some embodiments, the tagging module 1701 is used to tag the data.

[0196] In some embodiments, the sub-packet module 1702 is used to determine the data transmission channel and send different types of data to their respective transmission channels for transmission.

[0197] The sensor module 1703 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, a distance sensor, a temperature sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc. The touch sensor may be located on the display screen 1708; the touch sensor and the display screen 1708 together form a touchscreen, also known as a "touchscreen".

[0198] Internal memory 1704 can be used to store computer executable program code, which includes instructions. Processor 1706 causes terminal 1700 to execute the data transmission method, as well as various functional applications and data processing, provided in the embodiments of this application by executing the instructions stored in internal memory 1704.

[0199] The external memory interface 1705 can be used to connect an external memory card, expanding the storage capacity of the terminal 1700. The external memory card communicates with the processor 1706 through the external memory interface 1705 to realize data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0200] The processor 1706 may include one or more processing units, which may be application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs), etc. The different processing units may be independent devices or integrated into one or more processors.

[0201] Audio module 1707 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 1707 can also be used for encoding and decoding audio signals. Audio module 1707 includes at least one functional module, which may be a speaker, receiver, or microphone, and is not specifically limited herein. In some embodiments, audio module 1707 may be located in processor 1706, or some functional modules of audio module 1707 may be located in processor 1706.

[0202] Indicator 1709 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0203] The charging management module 1711 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1711 receives charging input from the wired charger via a USB interface 1710. In some wireless charging embodiments, the charging management module 1711 receives wireless charging input via the wireless charging coil of the terminal 1700. While charging the battery 17112, the charging management module 17111 can also supply power to the electronic device via the power management module 17111.

[0204] The power management module 17111 connects the battery 17112, the charging management module 1711, and the processor 1706. The power management module 17111 receives input from the battery 17112 and / or the charging management module 1711, supplying power to the processor 1706, internal memory 1704, display screen 1708, and wireless communication module 1713, etc. The power management module 17111 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 17111 may also be located within the processor 1706. In other embodiments, the power management module 17111 and the charging management module 1711 may be located in the same device.

[0205] In addition, the terminal 1700 may also include one or more components such as buttons, motor, indicator, camera, and subscriber identity module (SIM) card interface, and this application embodiment does not impose any restrictions on this.

[0206] Terminal 1700 can execute Figure 2 The operations performed by the terminal in the illustrated embodiment or Figure 4b The specific operations performed by the student terminal or teacher terminal in the illustrated embodiment will not be described in detail here.

[0207] Please see Figure 18 This application also provides a server 1800.

[0208] Server 1800 may include processor 1802 and transceiver 1803, wherein transceiver 1803 may be connected to processor 1802, such as Figure 18As shown. Transceiver 1803 may include a receiver and a transmitter, and can be used to receive or send messages or data. Transceiver 1803 may be a network interface card (NIC). Server 1800 may also include an acceleration component (which may be called an accelerator). When the acceleration component is a network acceleration component, the acceleration component may be a NIC. Processor 1802 may be the control center of server 1800, connecting various parts of server 1800, such as transceiver 1803, through various interfaces and lines. In this invention, processor 1802 may be a central processing unit (CPU). Optionally, processor 1802 may include one or more processing units. Processor 1802 may also be a digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), GPU, or other programmable logic devices. Server 1800 may also include a memory 1801, which can be used to store software programs and modules. Processor 1802 executes various functional applications and data processing of server 1800 by reading the software code and modules stored in memory 1801.

[0209] Server 1800 can execute Figure 2 The operations performed by the server in the illustrated embodiment or Figure 4b The specific operations performed by the live streaming server in the illustrated embodiment will not be described in detail here.

[0210] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0212] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0214] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of video communication, characterized by, The method comprises the following steps: a terminal acquires real-time character video data; the terminal acquires character image model driving parameters according to the real-time character video data; the terminal sends the character image model driving parameters to a server, so that the server takes the character image model driving parameters as input, adopts a character image model to obtain an output of a dynamic character image, and acquires first interface display data according to the dynamic character image, wherein the first interface display data comprises the dynamic character image, and the character image model corresponds to the terminal; the terminal receives the first interface display data sent by the server; based on the first interface display data, the terminal sends an operation request to the server, so that the server adjusts arrangement and combination modes of the dynamic character image set in the first interface display data according to the operation request to obtain second interface display data, wherein the second interface display data comprises state text information of the operation request; the terminal receives the second interface display data sent by the server.

2. The video communication method according to claim 1, characterized in that, The method further comprises the following steps: the terminal receives state text information sent by the server, and the state text information represents a change state of the dynamic character image.

3. The video communication method according to claim 2, characterized in that, The character image model driving parameters comprise at least one of body posture parameters and facial expression parameters.

4. The video communication method according to claim 2, characterized by, The character image model comprises at least one of a virtual image model and a three-dimensional character scanning model.

5. A method of video communication, characterized by The method comprises the following steps: a server receives character image model driving parameters sent by a terminal; the server takes the character image model driving parameters as input, adopts a character image model to obtain an output of a dynamic character image, and the character image model corresponds to the terminal; the server acquires first interface display data according to the dynamic character image, and the first interface display data comprises the dynamic character image; the server sends the first interface display data to the terminal; the server receives an operation request sent by the terminal based on the first interface display data; the server adjusts arrangement and combination modes of the dynamic character image set in the first interface display data according to the operation request to obtain second interface display data, and the second interface display data comprises state text information of the operation request; the server sends the second interface display data to the terminal.

6. The video communication method according to claim 5, characterized in that, The method further comprises the following steps: the server acquires state text information according to the character image model driving parameters; the server sends the state text information to the terminal.

7. The video communication method of any of claims 5-6, The character image model driving parameters comprise at least one of body posture parameters and facial expression parameters.

8. The video communication method of any of claims 5-6, The character image model comprises at least one of a virtual image model and a three-dimensional character scanning model.

9. A terminal, characterized by comprising: The method comprises the following steps: an acquisition unit is configured to acquire real-time character video data; the acquisition unit is configured to acquire character image model driving parameters according to the character video data; The sending unit is configured to send the character model driving parameter to the server, and make the server take the character model driving parameter as input, adopt a character model to obtain an output of a dynamic character, and acquire first interface display data according to the dynamic character, wherein the first interface display data comprises the dynamic character, the character model, and the terminal. The receiving unit is configured to receive the first interface display data sent by the server. The sending unit is configured to send an operation request to the server based on the first interface display data, and make the server adjust arrangement and combination of the dynamic character set in the first interface display data according to the operation request to obtain second interface display data, wherein the second interface display data comprises state text information of the operation request. The receiving unit is configured to receive the second interface display data sent by the server.

10. The terminal of claim 9, wherein The receiving unit is configured to receive state text information sent by the server, wherein the state text information indicates a change state of the dynamic character.

11. A server, characterized by The server comprises: The receiving unit is configured to receive character model driving parameter sent by the terminal. The processing unit is configured to take the character model driving parameter as input, adopt a character model to obtain an output of a dynamic character, wherein the character model corresponds to the terminal. The acquiring unit is configured to acquire first interface display data according to the dynamic character, wherein the first interface display data comprises the dynamic character. The sending unit is configured to send the first interface display data to the terminal. The receiving unit is configured to receive an operation request sent by the terminal based on the first interface display data. The acquiring unit is configured to adjust arrangement and combination of the dynamic character set in the first interface display data according to the operation request to obtain second interface display data, wherein the second interface display data comprises state text information of the operation request. The sending unit is configured to send the second interface display data to the terminal.

12. The server of claim 11, wherein The acquiring unit is configured to acquire state text information according to the character model driving parameter. The sending unit is configured to send the state text information to the terminal.

13. A terminal, characterized by The server comprises: A processor, a memory, an input / output device, and a bus; The processor, the memory, the input / output device, and the bus are connected; The processor is configured to execute the method in any one of claims 1 to 4.

14. A server, characterized by The terminal comprises: A processor, a memory, an input / output device, and a bus; The processor, the memory, the input / output device, and the bus are connected; The processor is configured to execute the method in any one of claims 5 to 8.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and when the computer executes the program, the method in any one of claims 1 to 8 is executed.

16. A computer program product, characterised in that, When the computer program product is executed on a computer, the computer performs the method as claimed in any one of claims 1 to 8.

Images