Video communication method and device

Abstract

The present application provides a method and device for video communication, which includes, after detecting that a user has selected a virtual image model, obtaining the user's portrait motion information and audio data based on the data collected by the collector, fitting the portrait motion information with the virtual image model, generating the user's virtual image data, and fusing the virtual image data with the audio data to generate the user's first audio and video stream data, and then sending the first audio and video stream data to a second terminal device participating in the same video communication with the user through a video cloud platform. In this way, since the solution enables users to use virtual image data for communication during video communication, it can avoid the leakage of the user's real personal image data, and the user does not need to care about his or her actual clothing or facial image, thereby effectively ensuring the security and privacy of the user's real personal image data, and greatly facilitating the user's use.

Description

Technical Field

[0001] The present application relates to the field of video call privacy and security technology, and in particular to a method and device for video communication. Background Art

[0002] With the rapid development of video calling technology, more and more video conferencing and video calling-related applications are becoming increasingly popular. However, when using these applications, user image data may be leaked. Therefore, effectively protecting user image data has become an urgent issue.

[0003] Existing solutions primarily distinguish between the user's portrait and background during video, retaining the portrait while blurring or replacing the background to protect the user's environmental information. However, this solution still poses the risk of leaking user portrait data and fails to effectively protect user image data.

[0004] In summary, there is an urgent need for a video communication method to effectively ensure the security and privacy of users' real personal image data. Summary of the Invention

[0005] In an exemplary embodiment of the present application, a method and apparatus for video communication are provided to effectively ensure the security and privacy of a user's real personal image data.

[0006] In a first aspect, an exemplary embodiment of the present application provides a method for video communication, comprising:

[0007] After detecting that a user has selected a virtual image model, obtaining the user's portrait motion information and audio data based on the data collected by the collector; the virtual image model is obtained by three-dimensional modeling;

[0008] Fitting the portrait motion information with the virtual image model to generate virtual image data of the user;

[0009] fusing the virtual image data with the audio data to generate first audio and video stream data of the user;

[0010] The first audio and video stream data is sent to a second terminal device participating in the same video communication with the user through a video cloud platform.

[0011] In the above technical solution, taking a video communication application installed on a terminal device as an example, after detecting that a user has logged into the video communication application and selected an avatar model, the video communication application obtains the user's portrait motion information and audio data based on the data collected by the collector, and uses this to construct virtual image data for the user. Specifically, the portrait motion information is fitted with the avatar model to generate the user's virtual image data, and the virtual image data is then merged with the audio data to generate a high-definition first audio and video stream data. This high-definition first audio and video stream data is then transmitted via the video cloud platform to a second terminal device participating in the same video communication as the user, thereby allowing the user's real-time portrait motion to be displayed to other users participating in the same video communication in a timely and effective manner through the avatar model. In this way, since this solution allows users to communicate using virtual image data during video communication, it can prevent the leakage of the user's real personal image data, and allows users to not worry about their actual clothing or facial appearance, effectively ensuring the security and privacy of the user's real personal image data, and greatly facilitating user use.

[0012] In some exemplary embodiments, the virtual image model is obtained by three-dimensional human body modeling based on the user's real image;

[0013] Before the user selects an avatar model, it also includes:

[0014] Acquire a first portrait image captured by the collector;

[0015] Determine whether the first portrait image matches a pre-stored verification image; the pre-stored verification image is obtained when constructing the virtual image model.

[0016] In the above technical solution, when a user logs in to a video communication application, the video communication application will verify the user's identity information, that is, determine whether the user's first portrait image matches the pre-stored verification image. This can prevent illegal users from impersonating others to log in and steal video communication content, and can also prevent illegal users from impersonating others to log in and steal information data stored by users in the video communication application, thereby ensuring the user's privacy and security.

[0017] In some exemplary embodiments, the method further comprises:

[0018] When it is detected that the user has not selected the avatar model, obtaining the second portrait image of the user and the audio data collected by the collector;

[0019] generating second audio and video stream data of the user based on the second portrait image and the audio data;

[0020] The second audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform.

[0021] In the above technical solution, if a user does not select an avatar model after logging into the video communication application, the video communication application will directly generate a high-definition second audio and video stream based on the user's real image data (i.e., the second portrait image) and audio data, and display this high-definition second audio and video stream data to other users participating in the same video communication. In this way, the solution can generate different audio and video data based on the different needs of users. In other words, the displayed audio and video data can be flexibly adjusted according to the user's needs, making the solution's audio and video data display more flexible.

[0022] In some exemplary embodiments, the portrait motion information includes at least one of the following: facial expression motion, eye movement, head movement, and limb torso movement.

[0023] In the above technical solution, during the video communication process, the data collected by the collector can obtain the user's real-time portrait movement details, such as facial expressions, eye movements, head movements, and limb and torso movements, and the virtual portrait model can be adjusted in real time based on the real-time portrait movement details, so that the user's real-time portrait movement details can be displayed to other users participating in the same video communication in a timely and effective manner.

[0024] In some exemplary embodiments, after sending the first audio and video stream data to a second terminal device participating in the same video communication with the user via the video cloud platform, the method further includes:

[0025] During the video communication process, the user is authenticated at least once to determine whether the user is allowed to continue using the virtual image model.

[0026] In this technical solution, to ensure the reliability of the user's portrait image data during video calls, the user's portrait is periodically verified, for example, every few minutes, to determine whether to allow the user to continue using the avatar model. This solution prevents the leakage of video call content caused by a user changing to someone else (i.e., someone other than the registered account holder), thereby effectively and timely ensuring the security and privacy of the user's portrait image data.

[0027] In some exemplary embodiments, performing at least one verification on the user during the video communication process includes:

[0028] During the video communication, obtaining a third person portrait image of the user collected by a collector;

[0029] Determine whether the third portrait image matches a pre-stored verification image.

[0030] In the above technical solution, during the video communication process, the collector will periodically collect the user's portrait information and compare the portrait information with the pre-stored verification image to verify whether the user has changed during the video communication process (for example, whether the user has become a person who is not the registered account), thereby ensuring the reliability of the user's portrait image data.

[0031] In some exemplary embodiments, determining whether the third portrait image matches a pre-stored verification image includes:

[0032] Based on a feature comparison method or an image similarity method, it is determined whether the third portrait image matches the pre-stored verification image.

[0033] In the above technical solution, by adopting the feature comparison method or the image similarity method, it is possible to promptly and accurately determine whether the third portrait image matches the pre-stored verification image, thereby providing support for ensuring the reliability of the user's portrait image data.

[0034] In a second aspect, an exemplary embodiment of the present application provides a video communication device, including:

[0035] an acquisition unit, configured to acquire, upon detecting that a user has selected a virtual image model, portrait motion information and audio data of the user based on data collected by a collector; the virtual image model is obtained by three-dimensional modeling;

[0036] A processing unit is used to fit the portrait motion information with the virtual image model to generate virtual image data of the user; fuse the virtual image data with the audio data to generate first audio and video stream data of the user; and send the first audio and video stream data to a second terminal device participating in the same video communication with the user through a video cloud platform.

[0037] In some exemplary embodiments, the virtual image model is obtained by three-dimensional human body modeling based on the user's real image;

[0038] The processing unit is further configured to:

[0039] Before detecting that the user has selected the avatar model, obtaining a first portrait image captured by the collector;

[0040] Determine whether the first portrait image matches a pre-stored verification image; the pre-stored verification image is obtained when constructing the virtual image model.

[0041] In some exemplary embodiments, the processing unit is further configured to:

[0042] When it is detected that the user has not selected the avatar model, obtaining the second portrait image of the user and the audio data collected by the collector;

[0043] generating second audio and video stream data of the user based on the second portrait image and the audio data;

[0044] The second audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform.

[0045] In some exemplary embodiments, the portrait motion information includes at least one of the following: facial expression motion, eye movement, head movement, and limb torso movement.

[0046] In some exemplary embodiments, the processing unit is further configured to:

[0047] After the first audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform, the user is verified at least once during the video communication to determine whether the user is allowed to continue using the virtual image model.

[0048] In some exemplary embodiments, the processing unit is specifically configured to:

[0049] During the video communication, obtaining a third person portrait image of the user collected by a collector;

[0050] Determine whether the third portrait image matches a pre-stored verification image.

[0051] In some exemplary embodiments, the processing unit is specifically configured to:

[0052] Based on a feature comparison method or an image similarity method, it is determined whether the third portrait image matches the pre-stored verification image.

[0053] In a third aspect, an embodiment of the present application provides a computing device comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes any of the video communication methods described in the first aspect above.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computing device. When the program runs on the computing device, the computing device executes any of the video communication methods described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 A schematic diagram of a video communication system architecture provided in some embodiments of the present application;

[0057] Figure 2 A flowchart of a video communication method provided in some embodiments of the present application;

[0058] Figure 3 A communication diagram of a video conference provided in some embodiments of the present application;

[0059] Figure 4 A schematic structural diagram of a video communication device provided in some embodiments of the present application;

[0060] Figure 5 A schematic diagram of the structure of a computing device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0062] In order to facilitate understanding of the embodiments of the present application, first Figure 1 The system structure shown in the example illustrates the video communication system architecture applicable to the embodiment of the present application. The video communication system architecture can be applied to video conferencing, or can be used for video calls, etc. Figure 1 As shown, the system architecture may include at least one terminal device (such as terminal device 101, terminal device 102 and terminal device 103, etc.) and a video cloud platform 200.

[0063] The video communication application in this application is loaded on a terminal device. The terminal device includes but is not limited to a terminal with data processing capabilities, including but not limited to electronic devices such as smart phones, tablet computers, desktop computers, laptops, etc., and can also be a terminal device such as a home brain.

[0064] The video cloud platform 200 has the functions of information processing and information forwarding. The video cloud platform 200 can be a single server or a server cluster. For example, the video cloud platform 200 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, cloud communications, domain name services, security services, and big data and artificial intelligence platforms. In addition, the video cloud platform 200 can receive audio and video data sent by any terminal device, and create a video communication service based on the video communication room address sent by the terminal device, so that each user can conduct video communication (such as video conferencing or video calls, etc.) in a timely manner.

[0065] Each terminal device and the video cloud platform 200 can be connected to each other through one or more networks. The network can be a wired network or a wireless network, for example, a wireless network can be a wireless fidelity (WIRE-FIELD) network, a mobile cellular network, or other possible networks, which are not limited in the embodiments of the present application.

[0066] In addition, the collector involved in this application ( Figure 1 The data collector (not shown) can be a camera configured on the terminal device, or it can be a collection device independent of the terminal device (such as a surveillance camera or a monitoring camera, etc.). The terminal device can call the collector to collect corresponding data.

[0067] It should be noted that the above Figure 1 The structure shown is only an example and is not limited to this embodiment of the present application.

[0068] Based on the above description, Figure 2 The flow of a video communication method provided in an embodiment of the present application is exemplarily shown, and the flow can be executed by a video communication device.

[0069] like Figure 2 As shown in the figure, the process specifically includes:

[0070] Step 201: After detecting that a user has selected a virtual image model, obtaining the user's portrait motion information and audio data based on data collected by a collector.

[0071] In the embodiments of the present application, in order to enable timely and effective video communication with other users, users need to load a video communication application on their terminal devices. After loading the video communication application, users can register on the video communication application, such as setting an account and password, setting identity verification, etc., to form the user's credentials for logging into the video communication application. During the user registration process, the video communication application collects the user's identity information and uploads the collected user's identity information to a corresponding server so that the server can store the user's identity information. For example, the collected identity information may include the user's real-life portrait data and verifiable identity information (such as the user's name, age, ID number, mobile phone number, etc.). The collected identity information can serve as the user's identity credential for using the video communication application, thereby ensuring the uniqueness and credibility of the user's identity. The collected real-life portrait data of the user can be used to construct an avatar model. Of course, users can also log in to the video communication application directly as a visitor without registering. However, logging in as a visitor does not ensure that the user's personal image data will not be leaked, nor can it ensure that the video communication content involved in the user's video communication process will not be leaked.

[0072] After registration, users can use the video communication application. When users use the video communication application, the video communication application will verify the user's identity information, that is, it can verify the account and password submitted by the user to determine whether the account and password submitted by the user are consistent with the pre-stored account and password. If they are consistent, the user is allowed to enter the video communication application. Alternatively, the user can also be subjected to portrait verification, that is, the user's first portrait image is collected by a collector to determine whether the first portrait image matches the pre-stored verification image (that is, the user's real portrait data). If they match, the user is allowed to enter the video communication application. In this way, illegal users can be prevented from impersonating and logging in to steal video communication content, and illegal users can be prevented from impersonating and logging in to steal information data stored by users in the video communication application, thereby ensuring the privacy and security of users.

[0073] For example, when using the video communication application, the user can log in to the video communication application based on the registered account and password, that is, the video communication application will provide the user with a login interface, which provides an account and password login method and a facial recognition login method. After the user selects the account and password login method, a user account and password login interface will pop up, and the user enters the registered account and password on the user account and password login interface. The video communication application will upload the account and password entered by the user to the corresponding server, and the corresponding server will verify the user's account and password to determine whether the account and password entered by the user are consistent with the pre-stored account and password. Alternatively, when using the video communication application, the user can log in to the video communication application based on facial recognition, that is, after the user selects the facial recognition login method, a facial recognition area will pop up, and the user's facial information (i.e., the user's portrait information) can be collected through the facial recognition area. That is, the user aligns his or her face with the facial recognition area, and the user's facial information can be captured by the collector, and then the user's facial information is uploaded to the corresponding server. The corresponding server will verify the user's facial information to determine whether the user's facial information matches the pre-stored user's real portrait.

[0074] After confirming that the user's entered account and password have been successfully verified, or after confirming that the user's facial information has been successfully verified, the video communication application allows the user to enter the video communication application. After the user enters the video communication application, the video communication application will provide the user with a video communication-related interface. This video communication-related interface may include an avatar function window for the user to use during the video communication. In this avatar function window, the user can choose to use or not use an avatar model during the video communication. The avatar model is generated using a 3D modeling method; the avatar model can be based on the user's real-life appearance through 3D human modeling, or it can be based on a 3D model of a cartoon character (such as various types of cartoon characters). The video communication-related interface may also include a window related to the video communication room involved in the video communication. In this video communication room window, the user can select the address of the video communication room to enter and configure the room function configuration for the video communication room. Alternatively, the user can create a new video communication room and configure the room function configuration for the newly created video communication room.

[0075] Taking the example of a virtual image model that is derived from the user's real image through three-dimensional human body modeling, if the user chooses to use the virtual image model during video communication, the video communication application will call the collector to collect the user's current portrait-related data after detecting that the user has chosen to use the virtual image model. The current portrait-related data includes the user's current portrait data and current audio data. The current portrait data is then analyzed to obtain the user's current portrait motion data. If the user chooses not to use the virtual image model during video communication, the video communication application will call the collector to collect the user's current portrait-related data after detecting that the user has chosen not to use the virtual image model. The current portrait-related data includes the user's current portrait data and current audio data. There is no need to analyze the current portrait data, and the current portrait data and current audio data are directly uploaded to the video cloud platform.

[0076] Taking the example of a virtual image model constructed by three-dimensional modeling based on a cartoon character, if the user chooses to use the virtual image model during video communication, the video communication application will call the collector to collect the user's current portrait-related data after detecting that the user has chosen to use the virtual image model. The current portrait-related data includes the user's current portrait data and current audio data. Then, the current portrait data is analyzed to obtain the user's current portrait motion data. If the user chooses not to use the virtual image model during video communication, the video communication application will call the collector to collect the user's current portrait-related data after detecting that the user has chosen not to use the virtual image model. The current portrait-related data includes the user's current portrait data and current audio data. There is no need to analyze the current portrait data, and the current portrait data and current audio data are directly uploaded to the video cloud platform. Alternatively, the video communication application can directly use a static cartoon character as the image of the user to be displayed to other users, so it only needs to call the collector to collect the user's current audio data; if the user chooses not to use the virtual image model during the video communication process, the video communication application will call the collector to collect the user's current portrait-related data after detecting that the user chooses not to use the virtual image model. The current portrait-related data includes the user's current portrait data and current audio data, and there is no need to analyze the current portrait data. The user's current portrait data and current audio data are directly uploaded to the video cloud platform.

[0077] It should be noted that the virtual image model obtained based on the user's real image through the human body three-dimensional modeling method is a human body three-dimensional model that is not equipped with clothing. The video communication application can provide users with various forms of clothing (such as formal wear, shirts, etc.) for users to choose a type of clothing to be configured on the human body three-dimensional model that is not equipped with clothing. Alternatively, the video communication application can also automatically configure a form of clothing for the human body three-dimensional model that is not equipped with clothing.

[0078] For example, the portrait motion tracking algorithm can be used to obtain the portrait motion of the user. In addition, the portrait motion tracking algorithm used in the embodiment of the present application is to capture the details of the portrait motion, such as head motion (such as nodding, shaking head, etc.), facial expression motion (such as lip motion, teeth, opening and closing eye motion, etc.), eyeball motion (such as eyeball moving left and right, eyeball moving up and down, etc.), limb and torso motion (such as arm swinging, arm bending posture, hand gesture, etc.). Specifically, the portrait tracking algorithm can meet the following forms:

[0079] Tracker=T(b,R,t)

[0080] Among them, Tracker represents the portrait tracking algorithm, b represents the user's portrait action, R represents the rotation data, and t represents the translation data.

[0081] Step 202 : Fitting the portrait motion information with the virtual image model to generate virtual image data of the user.

[0082] In an embodiment of the present application, character motion information may include facial expressions, eye movements, head movements, limb and torso movements, etc. When generating the user's virtual image data, the user's portrait motion information can be transferred to a virtual image model to construct an image for display to other users, so that the user can be displayed as the user's image in video communication based on the virtual image data. For example, during implementation, the user's real portrait motion can be transferred to a target constructed image (i.e., a spokesperson image). After the spokesperson image is constructed, the spokesperson image can be used to replace the video image data during the video communication process to ensure the privacy and security of the user's personal image data. Among them, when generating the spokesperson image, it is first necessary to distinguish between the source portrait motion and the target portrait motion, and the target portrait motion must be constructed using the source portrait expression and motion. The virtual portrait model is deformed and matched based on the portrait motion information obtained by the portrait tracking algorithm to generate a spokesperson image for display to other users. However, the existing technology is to modify the user's personal image based on virtual demand information after collecting the user's personal image, generate a virtual portrait model, and then replace the video image personal image with the virtual portrait model through augmented reality technology to achieve the purpose of using the virtual portrait for video chatting. That is, the existing technology does not pre-establish the user's virtual image model based on the user's real portrait data through three-dimensional modeling. Instead, it selects the virtual demand information used to modify the personal image from the pre-stored virtual demand information (such as anime characters, cartoon characters, celebrity characters, etc.) during use, and generates a virtual portrait model based on the collected user's personal image and virtual demand information. Moreover, the virtual portrait model established by the existing technology is a model used to hide the user's portrait, not just to hide the user's clothing image and facial image, etc., and cannot display the user's portrait feature information to the user in the video chat. Furthermore, existing technologies generate virtual portrait models based on the currently captured image of the user and the selected virtual requirement information. They do not generate the user's virtual image data in real time by capturing the user's portrait motion details (such as nodding and shaking their heads) and using a pre-established virtual image model to display the user's real-time original portrait motion in video communications with other users. Therefore, existing technologies cannot achieve real-time reproduction of the user's original portrait motion in virtual image data.

[0083] Taking the example of a virtual image model obtained by three-dimensional human body modeling based on the real image of the user as an example, before generating the user's virtual image data, it is necessary to construct a virtual image model (i.e., a portrait model). That is, the embodiment of the present application can perform three-dimensional human body modeling based on the real portrait data submitted by the user when registering to generate the user's virtual image model. For example, the embodiment of the present application extracts features of various parts of the human body based on the real portrait data submitted by the user when registering, and performs three-dimensional modeling based on the extracted features of various parts of the human body to generate the user's spokesperson portrait model. In other words, the embodiment of the present application constructs virtual high-definition portrait audio and video data based on the user's image, so as to interact in real time with other users participating in the same video communication, thereby also providing users with real-time interactive services such as privacy protection and interactive entertainment.

[0084] Taking the example of a 3D modeling of a cartoon character as an example, before generating the user's virtual image data, a 3D model (i.e., a cartoon portrait model) needs to be constructed. Specifically, embodiments of the present application can select a cartoon character based on the user's characteristics (e.g., gender, age, etc.), perform 3D modeling based on the cartoon character, and construct a 3D model representing the user's image. For example, embodiments of the present application extract features of various parts of the selected cartoon character, perform 3D modeling based on the extracted features of each part, generate a cartoon character model, and use the cartoon character model as the user's spokesperson portrait model.

[0085] After generating the spokesperson portrait model, if it is detected that the user has selected the avatar model, the user's current portrait motion data is obtained through a portrait tracking algorithm, and the user's current portrait motion data is fitted with the spokesperson portrait model. That is, the spokesperson image is generated by adjusting the various parameters in the spokesperson portrait model according to the user's current portrait motion data. For example, if the user opens his mouth, raises his lips, exposes his teeth, and raises his hands to his chest, the current portrait motion data may include raised lips, exposed teeth, and raised hands to his chest. At this time, these portrait motion data can be analyzed to generate corresponding portrait motion feature parameters, and these portrait motion feature parameters can be mapped to the spokesperson portrait model so that the corresponding parameters in the spokesperson portrait model can be adjusted to generate the spokesperson image of the user for video communication. During a video call, the video communication application captures the user's portrait motion details in real time and fits them to a spokesperson portrait model to generate a spokesperson image of the user in real time. This spokesperson image, which changes in real time with the user's real-time source portrait motion, effectively and timely displays the user's real-time source portrait motion in video calls with other users, allowing other users to promptly observe the user's real-time portrait motion changes. This allows the user's source portrait motion to be reproduced in real time in the spokesperson image. For example, suppose a user is speaking during a video call and the video communication application captures the user's nodding, mouth-opening, body tilting 30° to the left, and a hand-spreading gesture through a data collector. These portrait motion data can then be analyzed to generate corresponding portrait motion feature parameters, namely, nodding, mouth-opening, body tilting 30°, and hand-spreading. These portrait motion feature parameters are then mapped to the spokesperson portrait model to adjust the corresponding parameters in the spokesperson portrait model to generate the user's current spokesperson image. In this way, the real-time portrait action of the user can be displayed in real time on the terminal devices of other users participating in the same video communication.

[0086] Step 203: Fusing the virtual image data with the audio data to generate first audio and video stream data of the user.

[0087] In an embodiment of the present application, a video communication application fuses a user's virtual image with current audio data to obtain fused audio and video data, and then renders and compresses the fused audio and video data to generate the user's first audio and video stream data. For example, after generating a virtual, high-definition spokesperson image, the video conferencing application fuses the virtual, high-definition spokesperson image with the current audio data to obtain fused audio and video data, and then renders and encodes and compresses the fused audio and video data to generate spokesperson audio and video stream data. For example, two users, User A and User B, participate in the same video communication. Assume that User A is the spokesperson. User A has the video communication application installed on their terminal device. The video communication application on User A's terminal device fits User A's portrait motion data with User A's virtual image model to generate User A's spokesperson image, and then fuses User A's spokesperson image with User A's audio data to generate User A's spokesperson audio and video data. At the same time, user B can use the video communication application on the terminal device to generate user B's portrait audio and video data. When user A interacts with user B, user A can send user A's spokesperson audio and video data to user B's terminal device through the video cloud platform, so that the video communication application loaded in user B's terminal device can display and play user A's spokesperson audio and video data. At the same time, user B can also send user B's portrait audio and video data to user A's terminal device through the video cloud platform, so that the video communication application loaded in user A's terminal device can display and play user B's portrait audio and video data.

[0088] Step 204: Send the first audio and video stream data to a second terminal device participating in the same video communication with the user via a video cloud platform.

[0089] In an embodiment of the present application, after generating the user's first audio and video stream data, the video communication application will upload the user's first audio and video data to the video cloud platform, and will also upload the video communication room address and room function configuration to the video cloud platform. The video cloud platform will create a video communication service based on the video communication room address and room function configuration, and based on the video communication service, it will send the user's first audio and video data to a second terminal device participating in the same video communication with the user, so that the video communication application installed on the second terminal device participating in the same video communication can display and play the user's first audio and video data.

[0090] For example, the video communication application is used in a video conferencing scenario. Figure 3 , Figure 3 A communication diagram of a video conference provided in an embodiment of the present application. Figure 3The video communication application locally fits the user's portrait motion data obtained by the collector with the virtual image model to generate the user's spokesperson image data, and then fuses the user's spokesperson image data with the audio data to generate the user's spokesperson audio and video data. Then, the user's spokesperson audio and video data is uploaded to the video cloud platform, and the video conference room address and room function configuration are uploaded to the video cloud platform. The video cloud platform creates a video conferencing service based on the video conference room address and room function configuration, and based on the video conferencing service, sends the user's spokesperson audio and video data to other terminal devices participating in the same video conference with the user, so that the video communication applications installed in the other terminal devices participating in the same video conference can display and play the user's spokesperson audio and video data, thereby realizing real-time interaction between the user and other users participating in the same video conference.

[0091] For example, let's say three users, User A, User B, and User C, are participating in a video conference. Assume User A is the speaker. User A has a video communication application installed on their terminal device. The video communication application on User A's terminal device fits User A's portrait motion data with User A's avatar model to generate User A's speaker image. It also fuses User A's speaker image with User A's audio data to generate User A's speaker audio and video data. Simultaneously, User B can also use the video communication application on their terminal device to generate User B's portrait audio and video data. User C can also use the video communication application on their terminal device to generate User C's portrait audio and video data. When user A, user B and user C participate in the same video conference and interact with each other, user A can send user A's speaker audio and video data to user B's terminal device and user C's terminal device through the video cloud platform, so that the video communication applications loaded in user B's terminal device and user C's terminal device can each display and play user A's speaker audio and video data. At the same time, user B can also send user B's portrait audio and video data to user A's terminal device and user C's terminal device through the video cloud platform, so that the video communication applications loaded in user A's terminal device and user C's terminal device can each display and play user B's portrait audio and video data. At the same time, user C can also send user C's portrait audio and video data to user A's terminal device and user B's terminal device through the video cloud platform, so that the video communication applications loaded in user A's terminal device and user B's terminal device can each display and play user C's portrait audio and video data.

[0092] During the process of video communication interaction between a user and other users participating in the same video communication, in order to ensure the reliability of the user's portrait image data, the user in the video communication process will be periodically verified. For example, the user in the video communication process will be verified once every few minutes to determine whether the user is allowed to continue using the virtual image model. Specifically, during the video communication process, the video communication application obtains a third-party portrait image of the user collected by the collector, and determines whether the third-party portrait image matches the pre-stored verification image to verify whether the user has changed during the video communication process (for example, whether the user has become a person who is not the registered account), thereby ensuring the reliability of the user's portrait image data. Among them, when determining whether the third-party portrait image matches the pre-stored verification image, a feature comparison method or an image similarity method can be used to promptly and accurately determine whether the third-party portrait image matches the pre-stored verification image.

[0093] Among them, the feature comparison method can be: the video communication application periodically collects the user's portrait image during the video communication process, and uploads the user's portrait image to the corresponding server, and the corresponding server extracts features from the user's portrait image collected each time. For example, an image convolutional neural network can be used to extract features from the user's portrait image collected each time, so as to determine the portrait features of the user's portrait image collected each time. The portrait features of the user's portrait image collected each time are compared with the pre-stored portrait features to determine whether the portrait features of the user's portrait image collected each time are consistent with the pre-stored portrait features. If they are consistent, the video communication application allows the user to continue speaking based on the virtual image data (such as the speaker's image); if they are inconsistent, the user's speech is interrupted.

[0094] The image similarity method can be as follows: the video communication application periodically collects user portrait images during the video communication process and uploads each collected user portrait image to a corresponding server. The corresponding server compares each collected user portrait image with pre-stored portrait images to determine the similarity value between each collected user portrait image and the pre-stored portrait images. For each similarity value, if the similarity value is greater than or equal to a preset threshold, the video communication application allows the user to continue speaking based on the virtual image data (such as the speaker image); if the similarity value is less than the preset threshold, the user's speech is interrupted.

[0095] It should be noted that if the user does not select a virtual avatar model after logging into the video communication application, the user's second portrait image and audio data are directly obtained based on the collector, and based on the second portrait image and audio data, the user's second audio and video stream data is generated. Then, the second audio and video stream data is sent to the second terminal device participating in the same video communication with the user through the video cloud platform. In this way, the video communication application will directly generate high-definition second audio and video stream data based on the user's real image data (i.e., the second portrait image) and audio data, and display this high-definition second audio and video stream data to other users participating in the same video communication. In other words, the video communication application directly displays the user's real portrait data in the video communication with other users through the video cloud platform, without virtualizing the user's real portrait data, so that other users can view the user's real portrait in a timely manner. In this way, the solution can generate different audio and video data based on the different needs of users. That is, the displayed audio and video data can be flexibly adjusted according to the needs of the user, thereby making the solution more flexible in displaying audio and video data.

[0096] For example, if the user does not select an avatar model, the user's real portrait data and audio data are directly obtained based on the collector. After the real portrait data and audio data are integrated, the real high-definition spokesperson audio and video data is generated. This real high-definition spokesperson audio and video data is uploaded to the video cloud platform. The room address and room function configuration of the video call are also uploaded to the video cloud platform. The video cloud platform creates a video communication service based on the room address and room function configuration of the video call. Based on this video communication service, the user's real high-definition spokesperson audio and video data is sent to a second terminal device participating in the same video call with the user, so that the video communication application installed on the second terminal device participating in the same video call can display and play the user's real high-definition spokesperson audio and video data.

[0097] The above embodiment demonstrates that, taking a video communication application installed on a terminal device as an example, after detecting that a user has logged into the video communication application and selected an avatar model, the video communication application obtains the user's portrait motion information and audio data based on the data collected by the collector, and uses this to construct virtual image data for the user. Specifically, the portrait motion information is fitted with the avatar model to generate the user's virtual image data, and the virtual image data is then fused with the audio data to generate a high-definition first audio and video stream. This high-definition first audio and video stream is then transmitted via the video cloud platform to a second terminal device participating in the same video communication as the user. This allows the user's real-time portrait motion to be displayed to other users participating in the same video communication in a timely and effective manner through the avatar model. This solution, by enabling users to communicate using virtual image data during video communication, prevents the leakage of the user's real personal image data, eliminates the need for users to worry about their actual clothing or facial appearance, effectively ensures the security and privacy of the user's real personal image data, and greatly facilitates user experience.

[0098] Based on the same technical concept, Figure 4 A video communication device provided in an embodiment of the present application is exemplarily shown, and the device can execute the process of the video communication method.

[0099] like Figure 4 As shown, the device includes:

[0100] The acquisition unit 401 is configured to acquire the user's portrait motion information and audio data based on the data collected by the collector after detecting that the user has selected the avatar model; the avatar model is obtained by three-dimensional modeling;

[0101] Processing unit 402 is used to fit the portrait motion information with the virtual image model to generate virtual image data of the user; fuse the virtual image data with the audio data to generate the first audio and video stream data of the user; and send the first audio and video stream data to a second terminal device participating in the same video communication with the user through a video cloud platform.

[0102] In some exemplary embodiments, the virtual image model is obtained by three-dimensional human body modeling based on the user's real image;

[0103] The processing unit 402 is further configured to:

[0104] Before detecting that the user has selected the avatar model, obtaining a first portrait image captured by the collector;

[0105] Determine whether the first portrait image matches a pre-stored verification image; the pre-stored verification image is obtained when constructing the virtual image model.

[0106] In some exemplary embodiments, the processing unit 402 is further configured to:

[0107] When it is detected that the user has not selected the avatar model, obtaining the second portrait image of the user and the audio data collected by the collector;

[0108] generating second audio and video stream data of the user based on the second portrait image and the audio data;

[0109] The second audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform.

[0110] In some exemplary embodiments, the portrait motion information includes at least one of the following: facial expression motion, eye movement, head movement, and limb torso movement.

[0111] In some exemplary embodiments, the processing unit 402 is further configured to:

[0112] After the first audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform, the user is verified at least once during the video communication to determine whether the user is allowed to continue using the virtual image model.

[0113] In some exemplary embodiments, the processing unit 402 is specifically configured to:

[0114] During the video communication process, obtaining a third person portrait image of the user collected by the collector;

[0115] Determine whether the third portrait image matches a pre-stored verification image.

[0116] In some exemplary embodiments, the processing unit 402 is specifically configured to:

[0117] Based on a feature comparison method or an image similarity method, it is determined whether the third portrait image matches the pre-stored verification image.

[0118] Based on the same technical concept, the embodiment of the present application also provides a computing device, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5For example, the processor 501 and the memory 502 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0119] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the steps included in the aforementioned video communication method by executing the instructions stored in the memory 502.

[0120] The processor 501 is the control center of the computing device. It can connect the various parts of the computing device using various interfaces and lines, and realize data processing by running or executing instructions stored in the memory 502 and calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes the issuance of instructions. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0121] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiment of the method for video communication can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0122] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0123] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program that can be executed by a computing device. When the program runs on the computing device, the computing device executes the steps of the above-mentioned video communication method.

[0124] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0128] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0129] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A video communication method, characterized in that: include: After detecting that the user has selected the avatar model, obtaining the audio data of the user based on the data collected by the collector, and obtaining the portrait motion information of the user from the collected data using a portrait motion tracking algorithm; The virtual image model is obtained by three-dimensional modeling; Fitting the portrait motion information with the virtual image model to generate virtual image data of the user; Merging the virtual image data with the audio data to obtain merged audio and video data, and rendering and compressing the merged audio and video data to generate first audio and video stream data of the user; Sending the first audio and video stream data to a second terminal device participating in the same video communication with the user through a video cloud platform; Among them, during the video communication process, the third portrait image of the user collected by the collector is periodically obtained, and based on the feature comparison method or the image similarity method, it is determined whether the third portrait image matches the pre-stored verification image, so as to determine whether the user is allowed to continue to use the virtual image model.

2. The method according to claim 1, characterized in that The virtual image model is obtained by three-dimensional human body modeling based on the real image of the user; Before detecting that the user has selected the avatar model, the method further includes: Acquire a first portrait image collected by the collector; Determine whether the first portrait image matches a pre-stored verification image; the pre-stored verification image is obtained when constructing the virtual image model.

3. The method according to claim 1, characterized in that The method further comprises: When it is detected that the user has not selected the avatar model, acquiring the second portrait image of the user and the audio data collected by the collector; Generating second audio and video stream data of the user based on the second portrait image and the audio data; The second audio and video stream data is sent to a second terminal device participating in the same video communication with the user through the video cloud platform.

4. The method according to claim 1, characterized in that The portrait action information includes at least one of the following: facial expression action, eyeball action, head action, and limb torso action.

5. A video communication device, characterized in that: include: An acquisition unit, configured to acquire audio data of the user based on data collected by a collector after detecting that the user has selected a virtual image model, and to acquire portrait motion information of the user from the collected data using a portrait motion tracking algorithm; The virtual image model is obtained by three-dimensional modeling; A processing unit, configured to fit the portrait motion information with the virtual image model to generate virtual image data of the user; The virtual image data is merged with the audio data to obtain merged audio and video data, and the merged audio and video data is rendered and compressed to generate the first audio and video stream data of the user; the first audio and video stream data is sent to a second terminal device participating in the same video communication with the user through a video cloud platform; In which, during the video communication process, the processing unit is also used to: periodically obtain a third portrait image of the user collected by the collector, and determine whether the third portrait image matches a pre-stored verification image based on a feature comparison method or an image similarity method, thereby determining whether to allow the user to continue using the virtual image model.

6. A computing device, characterized in that The method comprises at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computing device, and when the program is run on the computing device, the computing device executes the method according to any one of claims 1 to 4.

Images