Video conference method and device, computer readable storage medium and electronic equipment

Abstract

The invention discloses a video conference method and device, a computer readable storage medium and electronic equipment. Relates to the field of artificial intelligence, and comprises the following steps: determining an initial video, and importing the initial video into video conference software, the initial video comprising image information of a target object participating in a conference; generating a target video according to the initial video and the speaking content under the condition that the speaking content to be speaked of the target object in the conference is received, the mouth shape of the target object in the target video being matched with the speaking content; and importing the target video and the target audio corresponding to the speaking content into the video conference software. The technical problem that users cannot be supported to participate in a plurality of video conferences at the same time in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a video conferencing method, device, computer-readable storage medium and electronic device. Background Art

[0002] In modern business and social environments, people increasingly rely on technology to improve communication efficiency and convenience. As an important remote communication method, video conferencing has been widely used in various occasions, such as corporate meetings, distance education, online medical consultation, etc.

[0003] Currently, when there is a meeting conflict, although users can join multiple meetings at the same time, this method is limited to telephone conferences. For video conferences that require users to appear on camera, users can only choose one video conference to participate in. Related technologies cannot support users to participate in multiple video conferences at the same time.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a video conferencing method, apparatus, computer-readable storage medium, and electronic device to at least solve the technical problem in the related art that users cannot participate in multiple video conferences at the same time.

[0006] According to one aspect of an embodiment of the present invention, a video conferencing method is provided, comprising: determining an initial video and importing the initial video into video conferencing software, wherein the initial video includes image information of a target object participating in the conference; upon receiving a speech content to be delivered by the target object in the conference, generating a target video based on the initial video and the speech content, wherein the lip shape of the target object in the target video matches the speech content; and importing the target video and target audio corresponding to the speech content into the video conferencing software.

[0007] Furthermore, the video conferencing method also includes: inputting the speech content into a speech generation model to obtain a target audio; and inputting the target audio and the initial video into a video generation model to obtain a target video.

[0008] Furthermore, the video conferencing method also includes: before generating the target video based on the initial video and the speech content, obtaining the speech content input by the target object, and determining the speech content input by the target object as the speech content to be delivered by the target object in the meeting, or, when receiving a question answering instruction input by the target object, inputting the question in the meeting into the target question and answer model, obtaining a target answer sentence matching the question, and determining the target answer sentence as the speech content to be delivered by the target object in the meeting.

[0009] Furthermore, the video conferencing method also includes: obtaining a character image of the target object; performing motion capture on the target object to obtain motion features, wherein the captured motion includes at least one of the following: body posture, muscle movement, facial expression; creating a three-dimensional model based on the character image, and determining the initial video based on the motion features and the three-dimensional model.

[0010] Furthermore, the video conferencing method also includes: obtaining an initial character image of the target object; obtaining the conference type of the conference; determining the clothing of the target object in the initial character image according to the conference type to obtain the target clothing; rendering the initial character image according to the target clothing to obtain the character image of the target object.

[0011] Furthermore, the video conferencing method also includes: obtaining the conference type of the conference; determining background information in the initial video to be generated according to the conference type; and determining the initial video according to the action features, the three-dimensional model and the background information.

[0012] Furthermore, the video conferencing method also includes: creating a skeletal system corresponding to the three-dimensional model; mapping the action features to the skeletal system of the three-dimensional model to obtain an animation sequence; based on the background information, creating a background in the animation sequence to obtain a target animation sequence; and determining the initial video based on the target animation sequence.

[0013] According to another aspect of an embodiment of the present invention, a video conferencing device is also provided, including: a first processing module, used to determine an initial video and import the initial video into video conferencing software, wherein the initial video includes image information of a target object participating in the meeting; a generation module, used to generate a target video based on the initial video and the speech content when receiving the speech content to be delivered by the target object in the meeting, wherein the lip shape of the target object in the target video matches the speech content; a second processing module, used to import the target video and target audio corresponding to the speech content into the video conferencing software.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned video conferencing method when running.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned video conferencing method when running.

[0016] In the embodiment of the present invention, a virtual digital human image of the target object is generated, and the virtual digital human image is used to replace the target object to attend the meeting. By determining the initial video and importing the initial video into the video conferencing software, and then receiving the speech content to be delivered by the target object in the meeting, a target video is generated according to the initial video and the speech content, thereby importing the target video and the target audio corresponding to the speech content into the video conferencing software. The initial video includes the image information of the target object attending the meeting, and the lip shape of the target object in the target video matches the speech content.

[0017] In the above process, by determining the initial video and importing the initial video into the video conferencing software, it is realized that the pre-generated video of the target object is displayed in the video conferencing software, which effectively replaces the real-time appearance of the target object, so that the target object can be displayed in real time in other conferences, or the initial video is used to appear in multiple conferences, thereby supporting users to participate in multiple video conferences at the same time. Further, by generating a target video according to the initial video and the speech content when receiving the speech content of the target object, and importing the target video and the target audio corresponding to the speech content into the video conferencing software, it is realized that when the target object needs to speak, a virtual digital person with the target object as the character image and the lip shape consistent with the lip shape of the speech content is displayed in the video conferencing software, thereby improving the naturalness and interactivity of the image of the target object displayed in the video conferencing software.

[0018] It can be seen that the solution provided in the present application achieves the purpose of generating a virtual digital human image of the target object and using the virtual digital human image to replace the target object to attend the meeting, thereby achieving the technical effect of supporting users to participate in multiple video conferences at the same time, and thus solving the technical problem in the related technology that cannot support users to participate in multiple video conferences at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional video conferencing method according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of an optional training process of a target adversarial generative network according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of an optional video conferencing device according to an embodiment of the present invention;

[0023] Figure 4is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a video conferencing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a schematic diagram of an optional video conferencing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, determining an initial video, and importing the initial video into the video conferencing software, wherein the initial video includes image information of a target object participating in the conference.

[0031] Optionally, electronic devices, application systems, servers and other devices may be used as the execution subject of the present application. In this embodiment, the target processing system is used as the execution subject to execute the video conferencing method.

[0032] In different application scenarios, the aforementioned target objects may also be different. For example, in a financial scenario, the target object may be an employee within the financial institution, in an online education scenario, the target object may be a teacher, and in an online shopping scenario, the target object may be a salesperson. In this embodiment, no specific limitation is made to the application scenarios and target objects.

[0033] Optionally, the initial video may be a video recorded in advance of the target object, or may be a video generated based on a character image of the target object. When the initial video is generated based on a character image, the character image in the initial video is equivalent to a virtual digital human.

[0034] In the process of importing the initial video into the video conferencing software, the target processing system can first open the video processing software, add a new "video acquisition device" in the video processing software, select "virtual camera", and then create a new video source in the video processing software and select the initial video. After that, open the video conferencing software and use the aforementioned video processing software as the video source, so that other participants in the conference system will be able to see the initial video, thereby realizing the import of the initial video into the video conferencing software.

[0035] Optionally, after importing the initial video into the video conferencing software, it is also possible to determine whether the image of the target object in the video conferencing software is clear, thereby executing step S102 if it is clear, and adjusting the video conferencing software or the aforementioned video processing software or the initial video if it is not clear.

[0036] Step S102, when receiving the speech content to be delivered by the target object in the meeting, generating a target video according to the initial video and the speech content, wherein the lip shape of the target object in the target video matches the speech content.

[0037] Optionally, in a meeting, the target object may need to speak, for example, give a speech, or answer questions raised by other participants. Therefore, the target object can send the speech content to be spoken in the meeting to the target processing system.

[0038] When the target processing system receives the speech content, the target processing system can generate a target video based on the initial video and the speech content. For example, the initial video and the speech content are directly input into the video generation model to obtain the target video. For another example, a target audio corresponding to the speech content is first generated, and then the target audio and the initial video are input into the video generation model to obtain the target video. Optionally, the lip shape of the target object in the target video matches the speech content, that is, the lip shape of the target object in the target video is the lip shape presented when speaking the speech content.

[0039] Step S103, importing the target video and the target audio corresponding to the speech content into the video conferencing software.

[0040] Optionally, the target processing system inputs the speech content into a speech generation model, and then the speech generation model generates the target audio.

[0041] After determining the target video and the target audio, the target processing system can import the target video and the target audio corresponding to the speech content into the video conferencing software. For example, the target processing system can replace the initial video in the video processing software with the target video in the aforementioned video processing software, create a corresponding audio source for the video source in the video processing software, select the target audio, and then use the aforementioned video processing software as the audio source in the video conferencing software. Since the aforementioned video processing software is also used as the video source in the video conferencing software, other participants in the conference system will be able to see the target video and hear the target audio, thereby realizing the import of the target video and the target audio into the video conferencing software.

[0042] Optionally, in this embodiment, the target video and the initial video are videos that do not contain audio content, and the character image in the target video is equivalent to a virtual digital human.

[0043] Based on the scheme defined in the above steps S101 to S103, it can be known that in the embodiment of the present invention, a virtual digital human image of the target object is generated, and the virtual digital human image is used to replace the target object to attend the meeting. By determining the initial video and importing the initial video into the video conferencing software, and then receiving the speech content to be delivered by the target object in the meeting, a target video is generated according to the initial video and the speech content, thereby importing the target video and the target audio corresponding to the speech content into the video conferencing software. Among them, the initial video includes the image information of the target object attending the meeting, and the lip shape of the target object in the target video matches the speech content.

[0044] It is easy to notice that in the above process, by determining the initial video and importing the initial video into the video conferencing software, it is realized that the pre-generated video of the target object is displayed in the video conferencing software, which effectively replaces the real-time appearance of the target object, so that the target object can be displayed in real time in other conferences, or the initial video is used to appear in multiple conferences, thereby supporting users to participate in multiple video conferences at the same time. Further, by generating a target video according to the initial video and the speech content when receiving the speech content of the target object, and importing the target video and the target audio corresponding to the speech content into the video conferencing software, it is realized that when the target object needs to speak, a virtual digital person with the target object as the character image and the mouth shape consistent with the mouth shape of the speech content is displayed in the video conferencing software, thereby improving the naturalness and interactivity of the image of the target object displayed in the video conferencing software.

[0045] It can be seen that the solution provided in the present application achieves the purpose of generating a virtual digital human image of the target object and using the virtual digital human image to replace the target object to attend the meeting, thereby achieving the technical effect of supporting users to participate in multiple video conferences at the same time, and thus solving the technical problem in the related technology that cannot support users to participate in multiple video conferences at the same time.

[0046] In an optional embodiment, in the process of generating a target video based on an initial video and speech content, the target processing system may input the speech content into a speech generation model to obtain target audio, and input the target audio and the initial video into a video generation model to obtain the target video.

[0047] Optionally, the training process of the speech generation model is described. First, a first training sample set is constructed, the first training sample set includes a first training sample and a first true label corresponding to the first training sample, the first training sample is a sample speech content, and the first true label is a sample target audio of the target object for the sample speech content, and then, the initial speech generation model is trained based on the first training sample set, so as to obtain a speech generation model when a preset iteration condition is met. The speech generation model can be a neural network model, and the preset iteration condition can be that the number of iterations reaches a preset number, the loss function value is less than a preset value, etc.

[0048] After the speech generation model is trained, the speech content can be input into the speech generation model to obtain the target audio output by the speech generation model.

[0049] Optionally, the training process of the video generation model is described. First, a second training sample set is constructed, the second training sample set includes a second training sample and a second true label corresponding to the second training sample, the second training sample is a sample video of a sample object and a sample audio corresponding to the speech content of the sample object, and the second true label is a sample target video whose lip shape matches that of the sample object and the sample audio, and then, the initial video generation model is trained based on the second training sample set, so as to obtain a video generation model when a preset iteration condition is met. The video generation model can be a neural network model, and the preset iteration condition can be that the number of iterations reaches a preset number, the loss function value is less than a preset value, etc.

[0050] After the video generation model is trained, the target audio and the initial video can be input into the video generation model. The video generation model can analyze the facial features in the initial video and then apply them to the video to be generated to achieve lip synchronization, thereby obtaining the target video.

[0051] It should be noted that, through the above method, on the one hand, the automatic generation of the target audio is realized, which can effectively replace the target object to speak, and on the other hand, the generation efficiency and accuracy of the target video are improved.

[0052] In an optional embodiment, when the speech content is used to answer questions in a meeting, before generating a target video based on the initial video and the speech content, the target processing system can obtain the speech content input by the target object, and determine the speech content input by the target object as the speech content to be delivered by the target object in the meeting, or, when receiving a question answering instruction input by the target object, input the question in the meeting into the target question and answer model, obtain a target answer sentence matching the question, and determine the target answer sentence as the speech content to be delivered by the target object in the meeting.

[0053] Optionally, the target processing system can provide the target object with a first window for inputting speech content, and the target object can input the content to be spoken in the first window, so that when the target processing system detects that the user has determined the speech content in the first window, the speech content will be determined as the speech content to be spoken by the target object in the meeting.

[0054] Optionally, the target processing system can also provide the target object with a second window for entering question answering instructions. The target object can enter question answering instructions in the second window, so that when the target processing system detects that a question answering instruction has been entered in the second window, it can determine the problem in the meeting and determine the content of the speech to be made based on the problem.

[0055] For example, the target processing system can record the speech content of each participant in the meeting in real time, and when a question answering instruction is detected, the speech content of the participant closest to the time point when the question answering instruction is detected is determined as the question. For another example, the target object can record the question to be answered in the question answering instruction, so that the target processing system can determine the question to be answered from the question answering instruction.

[0056] Afterwards, the target processing system can input the question into the target question-answering model, so that the target question-answering model outputs a target question-answering sentence, and determines the target answer sentence as the speech content to be spoken by the target object in the meeting. The target question-answering model can be a large language model.

[0057] In some embodiments, if the speech content is not used to answer questions in the meeting, for example, it is used to read a text, the target processing system can obtain the speech content input by the target object and determine the speech content input by the target object as the speech content to be delivered by the target object in the meeting.

[0058] It should be noted that, through the above method, the flexibility of determining the content of the speech to be made by the target object is improved.

[0059] In an optional embodiment, in the process of determining the initial video, the target processing system may obtain a character image of the target object, and then perform motion capture on the target object to obtain motion features, thereby creating a three-dimensional model based on the character image, and determining the initial video based on the motion features and the three-dimensional model. The captured motion includes at least one of the following: body posture, muscle movement, and facial expression.

[0060] Optionally, the person image may be an image obtained by capturing an image of the target object, or may be an image automatically generated according to facial features of the target object.

[0061] The target processing system can use motion capture technology to perform motion capture on at least one of the target object's body posture, muscle movement, and facial expression, thereby obtaining the target object's motion features. For example, the target processing system can perform motion capture on the target object's body posture, muscle movement, and facial expression, and then process the captured data through an inertial motion capture technology algorithm, such as cleaning, correction, smoothing, etc., to improve the accuracy and quality of the data. These data may include features such as real people's body posture, muscle movement, and facial expression, which need to be standardized to obtain motion features.

[0062] After obtaining the action features, the target processing system can create a three-dimensional model based on the character image. For example, the target processing system can input the character image into the three-dimensional model generation model, and then the three-dimensional model generation model outputs the three-dimensional model corresponding to the target object, wherein the three-dimensional model generation model can be a neural network model, and the three-dimensional model generation model can be trained according to the third training sample set, the third training sample set includes a third training sample and a third real label corresponding to the third training sample, the third training sample is a sample character image of the sample object, and the third real label is the three-dimensional model corresponding to the sample object.

[0063] For another example, the target processing system can use computer vision techniques, such as edge detection, corner detection, etc., to extract feature points from the character image of the target object, and then use human posture estimation technology to detect the key points of the character in the character image, such as the head, shoulders, elbows, wrists, hips, knees and ankles. After that, the information in the two-dimensional image is converted into a three-dimensional space to achieve three-dimensional reconstruction. For example, it can be achieved through stereo vision, multi-view geometry or deep learning-based methods, such as using a deep neural network (such as a convolutional neural network) to predict the three-dimensional coordinates corresponding to each key point and feature point in the image. Then, the target processing system can map the texture of the character image to the model surface obtained by three-dimensional reconstruction, for example, by projecting the image onto a three-dimensional grid, or using deep learning technology to achieve, thereby obtaining a three-dimensional model corresponding to the target task. Optionally, the target processing system can optimize the details of the generated three-dimensional model, such as smoothing the surface, adding texture details, etc., to improve the realism of the model.

[0064] After obtaining the three-dimensional model of the target object, the target processing system can determine the initial video according to the motion features and the three-dimensional model.

[0065] It should be noted that, through the above process, the automatic generation of the initial video is achieved, thereby improving the efficiency of video generation.

[0066] In an optional embodiment, in the process of obtaining the character image of the target object, the target processing system can obtain the initial character image of the target object, then obtain the meeting type of the meeting, and then determine the clothing of the target object in the initial character image according to the meeting type to obtain the target clothing, thereby rendering the initial character image according to the target clothing to obtain the character image of the target object.

[0067] Optionally, the initial character image of the target object may be an image obtained by capturing images of the target object, or may be an image automatically generated according to facial features of the target object.

[0068] The target processing system can request the target object to provide meeting information to determine the type of meeting based on the meeting information. The meeting information can refer to the type of meeting or the name of the meeting, etc. The target processing system has a preset correspondence between the meeting type and the clothing. Based on the correspondence, the target processing system can determine the clothing of the target object in the initial character image according to the meeting type to obtain the target clothing. For example, if the meeting is an important meeting, the target object needs to wear formal clothing. If the meeting is a training meeting, the target object can wear casual business-style clothes. If the meeting is an exchange meeting, the target object can wear casual clothing. If the meeting is a parent-teacher meeting, the target object can wear lively and youthful clothing.

[0069] Afterwards, the target processing system may render the initial character image according to the target clothing, so that the target character in the obtained character image wears the target clothing.

[0070] In some embodiments, the process of obtaining the initial character image of the target object is described. Optionally, the target processing system can collect a character video of the target object, and then input the character video into the target generation adversarial network, so that the target generation adversarial network outputs the initial task image of the target object.

[0071] Optional, Figure 2 is a schematic diagram of an optional target adversarial generative network training process according to an embodiment of the present invention, based on Figure 2 The training process of the target generative adversarial network is described. For example, a sample person video of the sample object is used as a training sample and processed, such as removing influencing factors such as background and noise. At the same time, the sample person video can be annotated, such as annotating the person's gender, age, expression, etc. Figure 2 As shown in the figure, the training samples are input into the initial generative adversarial network, which contains two neural networks, the generator and the discriminator. During the training process, the generator generates realistic pictures based on the random noise vector, and the discriminator distinguishes between real pictures (i.e., the image frames in the training samples) and fake pictures generated by the generator. Then, the loss of the discriminator on the real and fake images is calculated, and the parameters of the discriminator are updated through back propagation. The loss of the generator is calculated, and the parameters of the generator are updated through back propagation. Thus, through repeated iterations, the generator and the discriminator are continuously trained in adversarial training, so that the pictures generated by the generator become more and more realistic, and the target generative adversarial network is obtained.

[0072] It should be noted that, through the above process, the generated digital image is synthesized with the specific scene, which improves the flexibility of the character image and its applicability to conference scenes, thereby improving the applicability of the target video to conference scenes and making the target video more realistic.

[0073] In an optional embodiment, in the process of determining the initial video based on motion features and a three-dimensional model, the target processing system can obtain the meeting type of the meeting, and then determine the background information in the initial video to be generated based on the meeting type, thereby determining the initial video based on the motion features, the three-dimensional model and the background information.

[0074] Optionally, the target processing system has a preset correspondence between the meeting type and the background information. Based on the correspondence, the target processing system can determine the background information in the initial video to be generated according to the meeting type. For example, if the meeting is an important meeting, the background is the office background; if the meeting is a training meeting, the background can be a bookshelf; if the meeting is an exchange meeting, the background can be a leisure tea room; if the meeting is a parent-teacher meeting, the background can be a classroom. The background information can refer to the name of the background, the background picture, etc.

[0075] Afterwards, the target processing system can determine the initial video based on the motion characteristics, three-dimensional model and background information.

[0076] It should be noted that, through the above process, the generated video is synthesized with the specific scene, thereby improving the applicability of the target video to the conference scene and making the target video more realistic.

[0077] In an optional embodiment, in the process of determining the initial video based on action features, a three-dimensional model and background information, the target processing system can create a skeletal system corresponding to the three-dimensional model, and then map the action features to the skeletal system of the three-dimensional model to obtain an animation sequence, and then create a background in the animation sequence based on the background information to obtain a target animation sequence, thereby determining the initial video based on the target animation sequence.

[0078] Optionally, in the process of creating a skeletal system, the target processing system can create bones through 3D modeling and animation software, and place them according to the joint points of the 3D model, and then bind the bones to the vertices of the 3D model, for example, through a "skinning" or "rigging" process to ensure that when the bones move, the surface of the model will deform accordingly, thereby obtaining a skeletal system corresponding to the 3D model.

[0079] The target processing system can then use animation software to map the motion features to the skeletal system of the three-dimensional model to obtain an animation sequence.

[0080] After obtaining the animation sequence, the target processing system can create a background in the animation sequence based on the background information to obtain the target animation sequence. For example, the background image required is determined based on the background information, and then the background image is imported into the animation sequence to achieve the creation of the background, thereby obtaining the target animation sequence.

[0081] Afterwards, the target processing system can set rendering parameters, such as lighting, shadows, materials, etc., and then render the target animation sequence based on the rendering parameters to obtain the initial video, so that the generated video is visually more similar to the real scene.

[0082] It should be noted that, through the above process, accurate generation of the initial video is achieved.

[0083] In this embodiment, by determining the initial video and importing the initial video into the video conferencing software, it is realized that the pre-generated video of the target object is displayed in the video conferencing software, which effectively replaces the real-time appearance of the target object, so that the target object can be displayed in real time in other conferences, or the initial video is used to appear in multiple conferences, thereby supporting users to participate in multiple video conferences at the same time. Further, by generating a target video according to the initial video and the speech content when receiving the speech content of the target object, and importing the target video and the target audio corresponding to the speech content into the video conferencing software, it is realized that when the target object needs to speak, a virtual digital person with the target object as the character image and the mouth shape consistent with the mouth shape of the speech content is displayed in the video conferencing software, thereby improving the naturalness and interactivity of the image of the target object displayed in the video conferencing software.

[0084] It can be seen that the solution provided in the present application achieves the purpose of generating a virtual digital human image of the target object and using the virtual digital human image to replace the target object to attend the meeting, thereby achieving the technical effect of supporting users to participate in multiple video conferences at the same time, and thus solving the technical problem in the related technology that cannot support users to participate in multiple video conferences at the same time.

[0085] Example 2

[0086] According to an embodiment of the present invention, an embodiment of a video conferencing device is provided, wherein: Figure 3 is a schematic diagram of an optional video conferencing device according to an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0087] The first processing module 301 is used to determine an initial video and import the initial video into the video conferencing software, wherein the initial video includes image information of a target object participating in the conference;

[0088] A generating module 302 is used to generate a target video according to the initial video and the speech content when receiving the speech content to be delivered by the target subject in the meeting, wherein the lip shape of the target subject in the target video matches the speech content;

[0089] The second processing module 303 is used to import the target video and the target audio corresponding to the speech content into the video conferencing software.

[0090] In the embodiment of the present invention, the first processing module 301 is used to determine the initial video and import the initial video into the video conference software, the generation module 302 is used to generate the target video according to the initial video and the speech content when receiving the speech content to be delivered by the target object in the meeting, and the second processing module 303 imports the target video and the target audio corresponding to the speech content into the video conference software. Among them, the initial video includes the image information of the target object participating in the meeting, and the lip shape of the target object in the target video matches the speech content.

[0091] In the above process, by determining the initial video and importing the initial video into the video conferencing software, it is realized that the pre-generated video of the target object is displayed in the video conferencing software, which effectively replaces the real-time appearance of the target object, so that the target object can be displayed in real time in other conferences, or the initial video is used to appear in multiple conferences, thereby supporting users to participate in multiple video conferences at the same time. Further, by generating a target video according to the initial video and the speech content when receiving the speech content of the target object, and importing the target video and the target audio corresponding to the speech content into the video conferencing software, it is realized that when the target object needs to speak, a virtual digital person with the target object as the character image and the lip shape consistent with the lip shape of the speech content is displayed in the video conferencing software, thereby improving the naturalness and interactivity of the image of the target object displayed in the video conferencing software.

[0092] It can be seen that the solution provided in the present application achieves the purpose of generating a virtual digital human image of the target object and using the virtual digital human image to replace the target object to attend the meeting, thereby achieving the technical effect of supporting users to participate in multiple video conferences at the same time, and thus solving the technical problem in the related technology that cannot support users to participate in multiple video conferences at the same time.

[0093] It should be noted that the above-mentioned first processing module 301, generation module 302 and second processing module 303 correspond to steps S101 to S103 in the above-mentioned embodiment, and the examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0094] Optionally, the generation module also includes: a first processing submodule, used to input the speech content into the speech generation model to obtain the target audio; and a second processing submodule, used to input the target audio and the initial video into the video generation model to obtain the target video.

[0095] Optionally, the video conferencing device also includes: an acquisition module, which is used to acquire the speech content input by the target object, and determine the speech content input by the target object as the speech content to be delivered by the target object in the meeting, or a determination module, which is used to input the question in the meeting into the target question and answer model when receiving the question answering instruction input by the target object, obtain the target answer sentence matching the question, and determine the target answer sentence as the speech content to be delivered by the target object in the meeting.

[0096] Optionally, the first processing module also includes: an acquisition submodule, used to acquire a character image of the target object; a third processing submodule, used to capture the motion of the target object to obtain motion features, wherein the captured motion includes at least one of the following: body posture, muscle movement, facial expression; a determination submodule, used to create a three-dimensional model based on the character image, and determine the initial video based on the motion features and the three-dimensional model.

[0097] Optionally, the acquisition submodule also includes: a first acquisition unit, used to acquire an initial character image of the target object; a second acquisition unit, used to acquire the meeting type of the meeting; a first determination unit, used to determine the clothing of the target object in the initial character image according to the meeting type, and obtain the target clothing; a processing unit, used to render the initial character image according to the target clothing, and obtain the character image of the target object.

[0098] Optionally, the determination submodule also includes: a third acquisition unit, used to obtain the conference type of the meeting; a second determination unit, used to determine the background information in the initial video to be generated according to the conference type; and a third determination unit, used to determine the initial video based on action features, three-dimensional models and background information.

[0099] Optionally, the third determination unit also includes: a first creation subunit, used to create a skeletal system corresponding to the three-dimensional model; a mapping subunit, used to map action features to the skeletal system of the three-dimensional model to obtain an animation sequence; a second creation subunit, used to create a background in the animation sequence based on background information to obtain a target animation sequence; and a determination subunit, used to determine the initial video based on the target animation sequence.

[0100] Example 3

[0101] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned video conferencing method when running.

[0102] Example 4

[0103] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, wherein: Figure 4is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as Figure 4 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are used to run the programs, wherein the programs are set to execute the above-mentioned video conferencing method when they are run. When the processor executes the program, the following steps are implemented: determining an initial video, and importing the initial video into the video conferencing software, wherein the initial video includes image information of the target object participating in the meeting; when receiving the speech content to be delivered by the target object in the meeting, generating a target video according to the initial video and the speech content, wherein the lip shape of the target object in the target video matches the speech content; importing the target video and the target audio corresponding to the speech content into the video conferencing software.

[0104] Optionally, when the processor executes the program, the following steps are also implemented: inputting the speech content into the speech generation model to obtain the target audio; inputting the target audio and the initial video into the video generation model to obtain the target video.

[0105] Optionally, when the processor executes the program, it also implements the following steps: obtaining the speech content input by the target object, and determining the speech content input by the target object as the speech content to be delivered by the target object in the meeting; or, upon receiving a question answering instruction input by the target object, inputting the question in the meeting into the target question and answer model, obtaining a target answer sentence matching the question, and determining the target answer sentence as the speech content to be delivered by the target object in the meeting.

[0106] Optionally, when the processor executes the program, the following steps are also implemented: obtaining a character image of the target object; performing motion capture on the target object to obtain motion features, wherein the captured motion includes at least one of the following: body posture, muscle movement, facial expression; creating a three-dimensional model based on the character image, and determining an initial video based on the motion features and the three-dimensional model.

[0107] Optionally, when the processor executes the program, the following steps are also implemented: obtaining an initial character image of the target object; obtaining the meeting type of the meeting; determining the clothing of the target object in the initial character image according to the meeting type, and obtaining the target clothing; rendering the initial character image according to the target clothing, and obtaining the character image of the target object.

[0108] Optionally, when the processor executes the program, the following steps are also implemented: obtaining the conference type of the conference; determining the background information in the initial video to be generated according to the conference type; and determining the initial video according to the action features, the three-dimensional model and the background information.

[0109] Optionally, the processor also implements the following steps when executing the program: creating a skeletal system corresponding to the three-dimensional model; mapping motion features to the skeletal system of the three-dimensional model to obtain an animation sequence; creating a background in the animation sequence based on background information to obtain a target animation sequence; and determining an initial video based on the target animation sequence.

[0110] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0111] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0114] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0116] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video conferencing method, characterized in that: include: Determine an initial video, and import the initial video into the video conferencing software, wherein the initial video includes image information of a target object participating in the conference; Upon receiving the speech content to be delivered by the target object in the conference, generating a target video according to the initial video and the speech content, wherein the lip shape of the target object in the target video matches the speech content; Import the target video and the target audio corresponding to the speech content into the video conferencing software.

2. The method according to claim 1, characterized in that: Generating a target video according to the initial video and the speech content includes: Inputting the speech content into a speech generation model to obtain the target audio; The target audio and the initial video are input into a video generation model to obtain the target video.

3. The method according to claim 1, characterized in that In the case where the speech content is used to answer questions in the meeting, before generating a target video according to the initial video and the speech content, the method further includes: Acquire the speech content input by the target object, and determine the speech content input by the target object as the speech content to be delivered by the target object in the meeting, or, Upon receiving a question answering instruction input by the target object, the question in the meeting is input into a target question-answering model to obtain a target answer sentence matching the question, and the target answer sentence is determined as the speech content to be delivered by the target object in the meeting.

4. The method according to claim 1, characterized in that Identify the initial video, including: Acquire a person image of the target object; Performing motion capture on the target object to obtain motion features, wherein the captured motion includes at least one of the following: body posture, muscle movement, and facial expression; A three-dimensional model is created according to the character image, and the initial video is determined according to the action features and the three-dimensional model.

5. The method according to claim 4, characterized in that Acquiring a person image of the target object includes: Acquire an initial character image of the target object; Get the conference type of the conference; Determine the clothing of the target object in the initial person image according to the conference type, and obtain target clothing; The initial character image is rendered according to the target clothing to obtain the character image of the target object.

6. The method according to claim 4, characterized in that Determining the initial video according to the action feature and the three-dimensional model includes: Get the conference type of the conference; Determining background information in an initial video to be generated according to the conference type; The initial video is determined according to the action feature, the three-dimensional model and the background information.

7. The method according to claim 6, characterized in that Determining the initial video according to the action feature, the three-dimensional model, and the background information includes: Creating a skeletal system corresponding to the three-dimensional model; Mapping the motion features to the skeleton system of the three-dimensional model to obtain an animation sequence; Based on the background information, creating a background in the animation sequence to obtain a target animation sequence; The initial video is determined based on the target animation sequence.

8. A video conferencing device, characterized in that: include: A first processing module, configured to determine an initial video and import the initial video into the video conferencing software, wherein the initial video includes image information of a target object participating in the conference; A generating module, configured to generate a target video according to the initial video and the speech content when receiving the speech content to be delivered by the target object in the conference, wherein the lip shape of the target object in the target video matches the speech content; The second processing module is used to import the target video and the target audio corresponding to the speech content into the video conferencing software.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the video conferencing method described in any one of claims 1 to 7 when executed.

10. An electronic device, characterized in that: The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the video conferencing method described in any one of claims 1 to 7 when run.

Images