Virtual model display method, virtual model display device, equipment, medium and product

Abstract

The invention provides a virtual model display method, a virtual model display device, equipment, a medium and a product, which can be applied to the technical field of data processing and financial science and technology. The method comprises the following steps: under the condition that a first display page of a virtual conference is utilized to display a first virtual model, performing posture adjustment on the first virtual model in the first display page based on first control voice information of a spokesman so as to display the first display page to the spokesman and at least one participant; in response to a model control instruction of the participant, a second display page is formed in the first display page of the participant, and a second virtual model corresponding to the first virtual model is displayed in the second display page; acquiring first control information of participants; and based on the first control information, carrying out posture adjustment on a second virtual model in the second display page, so that the participants can check the first display page and the second display page at the same time.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a virtual model display method for a virtual meeting, a virtual model display device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] ‌Online virtual conference is a multimedia conference platform based on the Internet, which allows multiple users in different locations to share data and communicate in real time through the Internet. It breaks the limitations of time and space, allowing users to hold meetings like face-to-face communication.

[0003] However, when the speaker is presenting the virtual model through a virtual meeting, other participants cannot control the displayed virtual model, so it is difficult for the participants to observe the characteristics and advantages of the virtual model, resulting in a poor experience for the participants. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a virtual model display method for a virtual meeting, a virtual model display device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect of the present disclosure, a virtual model display method for a virtual meeting is provided, characterized in that the method comprises:

[0006] In the case of displaying the first virtual model using the first display page of the virtual conference, adjusting the posture of the first virtual model in the first display page based on the first control voice information of the speaker, so as to display the first display page to the speaker and at least one participant;

[0007] In response to a model control instruction of a participant, a second display page is formed in the first display page of the participant, wherein the second display page displays a second virtual model corresponding to the first virtual model;

[0008] Obtaining the first control information of the above-mentioned participant;

[0009] Based on the first control information, the posture of the second virtual model in the second display page is adjusted so that the participant can view the first display page and the second display page at the same time.

[0010] According to an embodiment of the present disclosure, adjusting the posture of the first virtual model in the first display page based on the first control voice information of the speaker includes:

[0011] Processing the first control voice information using a preset voice processing method to obtain the second control information of the speaker;

[0012] The posture of the first virtual model in the first display page is adjusted based on the second control information.

[0013] According to an embodiment of the present disclosure, the first control voice information is processed by a preset voice processing method to obtain the second control information of the speaker, including:

[0014] Processing the first control voice information using a voice recognition method to obtain first control text information;

[0015] Processing the first control text information using a keyword extraction method to obtain a plurality of control keywords;

[0016] The second control information corresponding to the plurality of control keywords is determined from the instruction matching table, wherein the instruction matching table stores a plurality of initial control information for adjusting the virtual model to different postures.

[0017] According to an embodiment of the present disclosure, different control keywords mentioned above have different time tags.

[0018] According to an embodiment of the present disclosure, determining the second control information corresponding to the plurality of control keywords from the instruction matching table includes:

[0019] Combining a plurality of the control keywords based on the time sequence to obtain at least one control statement;

[0020] The second control information corresponding to the control statement is determined from the instruction matching table.

[0021] According to an embodiment of the present disclosure, when there are multiple control statements, adjusting the posture of the first virtual model in the first display page based on the second control information includes:

[0022] For each of the control statements, determine the second control information corresponding to the control statement from the instruction matching table;

[0023] Sorting the plurality of second control information based on time to obtain a sorting result;

[0024] The plurality of sorted second control information in the sorting result is sequentially used as target control information, so as to adjust the posture of the first virtual model based on each of the target control information.

[0025] According to an embodiment of the present disclosure, the first control voice information is processed by a preset voice processing method to obtain the second control information of the speaker, including:

[0026] The first control voice information is processed by a target voice recognition network to obtain the second control information.

[0027] According to an embodiment of the present disclosure, the target speech recognition network is trained in the following manner:

[0028] Acquire a training set, wherein the training set includes a plurality of speech training information and control label information corresponding to each of the speech training information, and the control label information represents model posture adjustment information;

[0029] For each of the above-mentioned speech training information, input the above-mentioned speech training information into an initial speech recognition network, and output prediction control information;

[0030] Generate a loss result according to the predicted control information and the control label information corresponding to the speech training information;

[0031] The network parameters of the initial speech recognition network are iteratively adjusted according to the loss results to obtain a trained target speech recognition network.

[0032] According to an embodiment of the present disclosure, before adjusting the posture of the first virtual model, the method further includes:

[0033] Acquire context voice information corresponding to the first control voice information;

[0034] Processing the context voice information using a preset voice processing method to obtain context text information corresponding to the second control information;

[0035] The second control information is modified based on the context text information to obtain modified second control information, so as to adjust the posture of the first virtual model using the modified second control information.

[0036] According to an embodiment of the present disclosure, obtaining the first control information of the above-mentioned participant includes:

[0037] Obtaining the second control voice information of the above participant;

[0038] Processing the second control voice information using a preset voice processing method to obtain the first control information;

[0039] The posture of the second virtual model in the second display page is adjusted based on the first control information.

[0040] According to an embodiment of the present disclosure, the second display page is located in the target area of ​​the first display page, and at the same time, the first virtual model in the first display page is scaled so that the scaled first virtual model is located in the non-target area of ​​the first display page.

[0041] A second aspect of the present disclosure provides a virtual model display device for a virtual meeting, the device comprising:

[0042] A first adjustment module is used to adjust the posture of the first virtual model in the first display page of the virtual conference based on the first control voice information of the speaker, so as to display the first display page to the speaker and at least one participant;

[0043] A forming module, configured to form a second display page in the first display page of the participant in response to a model control instruction of the participant, wherein the second display page displays a second virtual model corresponding to the first virtual model;

[0044] An acquisition module, used to acquire the first control information of the above-mentioned participant;

[0045] The second adjustment module is used to adjust the posture of the second virtual model in the second display page based on the first control information, so that the participant can view the first display page and the second display page at the same time.

[0046] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0047] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0048] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0049] According to an embodiment of the present disclosure, when the speaker displays the first virtual model through the first display page of the virtual conference, the posture of the first virtual model in the first display page is adjusted based on the speaker's first control voice information, and in response to the model control instruction of the participant, a second display page displaying the second virtual model is formed in the first display page of the participant, and the posture of the second virtual model in the second display page is adjusted based on the first control information of the participant, so that the participant can view the first display page and the second display page at the same time. In this way, the posture of the second virtual model in the second display page can be controlled without affecting the speaker's speech, so that the participant can synchronously watch the speaker's speech content and the virtual model area of ​​his own interest, thereby improving the meeting participation experience of the virtual meeting and further improving the participants' understanding of the virtual meeting. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0051] Figure 1 A diagram schematically shows an application scenario of a virtual model display method or a virtual model display device according to an embodiment of the present disclosure;

[0052] Figure 2 A flowchart of a virtual model display method for a virtual meeting according to an embodiment of the present disclosure is schematically shown;

[0053] Figure 3 The training flow chart of the target speech recognition network according to the embodiment of the present disclosure is schematically shown;

[0054] Figure 4 A schematic diagram showing a structure of a virtual model display device according to an embodiment of the present disclosure is shown.

[0055] Figure 5 A structural block diagram schematically shows a target speech recognition network training device according to an embodiment of the present disclosure; and

[0056] Figure 6 A block diagram of an electronic device suitable for implementing the above method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0057] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0058] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0059] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0060] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0061] When multiple people collaborate to display a virtual model of a fixed object in a virtual meeting, if everyone can adjust the virtual model, the virtual model may become chaotic. This is because everyone may have different ideas and opinions, and they may try to adjust the virtual model to their liking, resulting in inconsistencies in the form and display of the virtual model. In addition, if there is no clear coordination and planning, each person may adjust the object at different times, resulting in inconsistencies in the objects seen by others, thus affecting the display effect of the entire virtual model. However, if it is set that only the speaker can adjust the object, it will affect other participants and prevent them from observing the object according to their own ideas, resulting in a poor experience for the participants.

[0062] In view of this, the embodiments of the present disclosure provide a virtual model display method, a virtual model display device, equipment, medium, and a product, the method including, when using a first display page of a virtual conference to display a first virtual model, adjusting the posture of the first virtual model in the first display page based on the first control voice information of the speaker, so as to display the first display page to the speaker and at least one participant; in response to the model control instruction of the participant, forming a second display page in the first display page of the participant, wherein the second display page displays a second virtual model corresponding to the first virtual model; obtaining the first control information of the participant; and based on the first control information, adjusting the posture of the second virtual model in the second display page, so that the participant can view the first display page and the second display page at the same time.

[0063] It should be noted that the virtual model display method and virtual model display device provided by the present disclosure can be used in the field of financial technology, such as banks and other financial institutions, and can also be used in any field other than the field of financial technology, such as hospitals and other medical fields. Therefore, the application field of the virtual model display method and virtual model display device provided by the present disclosure is not limited.

[0064] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0065] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0066] Figure 1 The application scenario diagram of the virtual model display method or the virtual model display device according to the embodiments of the present disclosure is schematically shown.

[0067] like Figure 1As shown, the application scenario 100 according to this embodiment may include a bank, etc. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0069] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc., on which virtual conference software for different users to participate in online meetings can be installed.

[0070] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0071] It should be noted that the virtual model display method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the virtual model display device provided in the embodiment of the present disclosure can generally be set in the server 105. The virtual model display method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the virtual model display device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] Figure 2 The flowchart of the virtual model display method for a virtual meeting according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 2 As shown, the virtual model display method for a virtual meeting in this embodiment includes operations S201 to S204.

[0075] In operation S201, when a first virtual model is displayed using a first display page of a virtual conference, a posture of the first virtual model in the first display page is adjusted based on a first control voice message of a speaker, so as to display the first display page to the speaker and at least one participant;

[0076] In operation S202, in response to a model control instruction of a participant, a second display page is formed in the first display page of the participant, wherein a second virtual model corresponding to the first virtual model is displayed in the second display page;

[0077] In operation S203, first control information of the participant is obtained;

[0078] In operation S204, based on the first control information, the posture of the second virtual model in the second display page is adjusted so that the participants can view the first display page and the second display page at the same time.

[0079] According to an embodiment of the present disclosure, the virtual conference can be an Internet conference provided by any type of remote conference software. When different users attend the Internet conference, the user who is giving a speech is regarded as a speaker, and other users are regarded as participants.

[0080] According to an embodiment of the present disclosure, the virtual model may be any object model, such as a virtual object model obtained by three-dimensionally reconstructing a table, a chair, a robot, or other object having a physical structure, or an object model drawn by three-dimensional software.

[0081] According to an embodiment of the present disclosure, in an Internet conference, the speaker can upload the virtual model to be explained in the virtual conference software in advance, or the conference software can automatically upload the speaker's file in real time during the speaker's speech. Thereafter, during the explanation, the speaker shows the first virtual model he is explaining to other participants through a shared first display page. During the explanation, the virtual conference software collects the speaker's audio information in real time, and the audio information includes the speaker's first control voice information of the control instructions for the first virtual model, such as "rotate the model 90°". The virtual conference software automatically adjusts the posture of the first virtual model according to the speaker's first control voice information.

[0082] According to the embodiments of the present disclosure, during the speaker's speech, some participants may want to adjust the first virtual model themselves so that the participants can understand the area of ​​interest, such as the specific structure on the back of the first virtual model. At this time, since the first virtual model is controlled by the speaker, if the participant operates the first virtual model, it may affect the subsequent explanations of other participants and dialect speakers. Therefore, in the embodiment of the present application, during the speaker's explanation, the virtual conference software of each participant will synchronously obtain the model control instructions of the corresponding participant in real time. If it is determined that the participant issues a model control instruction, the virtual conference software of the participant automatically forms a second display page in the first display page shared by the participant, and at the same time displays the file of the first virtual model, i.e., the second virtual model, in the second display page. At this time, the virtual conference software obtains the first control information of the participant in real time, and the virtual conference software of the participant adjusts the posture of the second virtual model in the second display page according to the first control information of the participant in real time, so that the participant can watch the area of ​​interest without affecting the speaker's speech.

[0083] According to an embodiment of the present disclosure, when the speaker displays the first virtual model through the first display page of the virtual conference, the posture of the first virtual model in the first display page is adjusted based on the speaker's first control voice information, and in response to the model control instruction of the participant, a second display page displaying the second virtual model is formed in the first display page of the participant, and the posture of the second virtual model in the second display page is adjusted based on the first control information of the participant, so that the participant can view the first display page and the second display page at the same time. In this way, the posture of the second virtual model in the second display page can be controlled without affecting the speaker's speech, so that the participant can synchronously watch the speaker's speech content and the virtual model area of ​​his own interest, thereby improving the meeting participation experience of the virtual meeting and further improving the participants' understanding of the virtual meeting.

[0084] According to an embodiment of the present disclosure, adjusting the posture of a first virtual model in a first display page based on first control voice information of a speaker includes:

[0085] Processing the first control voice information using a preset voice processing method to obtain second control information of the speaker;

[0086] The posture of the first virtual model in the first display page is adjusted based on the second control information.

[0087] According to an embodiment of the present disclosure, the preset speech processing method may be any method capable of converting language into text, such as a speech recognition method such as a neural network model.

[0088] According to an embodiment of the present disclosure, during a speaker's speech, the virtual conference software collects the speaker's voice information, and uses a preset voice processing method to identify the first control voice information, thereby obtaining the speaker's second control information, such as a control instruction to rotate 90° counterclockwise. In response to the second control information, the virtual conference software adjusts the posture of the first virtual model in the first display page, so that the speaker and the participants can watch the adjusted first virtual model while listening to the speaker's specific explanation of the first virtual model.

[0089] According to an embodiment of the present disclosure, by converting the first control voice information of the speaker into a control instruction for controlling the posture adjustment of the first virtual model, the timeliness of the explanation and the accuracy of the posture adjustment of the first virtual model caused by the speaker manually controlling the first virtual model are avoided, thereby improving the accuracy and convenience of the virtual model explanation.

[0090] According to an embodiment of the present disclosure, the first control voice information is processed by a preset voice processing method to obtain the second control information of the speaker, including:

[0091] Processing the first control voice information by using a voice recognition method to obtain first control text information;

[0092] Processing the first control text information using a keyword extraction method to obtain a plurality of control keywords;

[0093] Second control information corresponding to a plurality of control keywords is determined from an instruction matching table, wherein the instruction matching table stores a plurality of initial control information for adjusting different postures of the virtual model.

[0094] According to the embodiments of the present disclosure, the speech recognition method is also called Automatic Speech Recognition (ASR), which aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences. This is different from speaker recognition and speaker verification, which attempt to identify or verify the speaker who made the speech rather than the vocabulary content contained therein.

[0095] According to an embodiment of the present disclosure, the keyword extraction method may include a text frequency-inverse document frequency (TF-IDF) method, a graph-based ranking (TextRank) algorithm, a key sentence method, etc.

[0096] According to the embodiment of the present disclosure, since the first control language information of the speaker includes not only the control voice for adjusting the posture of the first virtual model, but also a large amount of explanation content about the virtual model, after the first control voice information is processed by the speech recognition method to obtain the first control text information, it is also necessary to perform keyword extraction on the first control text information to extract the control keywords related to the posture adjustment. For example, the first control text information of the speaker is "We now rotate the table model 90° counterclockwise, and you can see that there are many patterns on the side of the table... After rotating it again, you can see that there are many grooves on the top surface of the virtual model." At this time, by using the keyword extraction method, multiple control keywords can be obtained, namely "rotate 90° counterclockwise", "rotate, top surface", etc.

[0097] According to an embodiment of the present disclosure, after determining the control keyword, second control information corresponding to each control keyword is determined from the instruction matching table, so that the virtual conference software adjusts the posture of the first virtual model in the first display page based on each second control information.

[0098] It should be noted that when there are multiple control keywords, the virtual conference software needs to be executed in sequence. At the same time, the instruction matching table is pre-configured with multiple identical or different configuration keywords corresponding to each second control information. For example, when the second control information is "rotate 90 degrees counterclockwise", there can be multiple configuration keywords, such as "reverse 90 degrees", "flip 90 degrees to the left", "left side of the model", etc.

[0099] According to an embodiment of the present disclosure, a keyword extraction method can be used to extract control keywords for model control from the speaker's first control voice information containing a large amount of information irrelevant to model control, thereby improving the display accuracy of the virtual model.

[0100] According to an embodiment of the present disclosure, different control keywords have different time tags.

[0101] According to an embodiment of the present disclosure, determining second control information corresponding to a plurality of control keywords from an instruction matching table includes:

[0102] Combining multiple control keywords based on the time sequence to obtain at least one control statement;

[0103] The second control information corresponding to the control statement is determined from the instruction matching table.

[0104] According to an embodiment of the present disclosure, the time tag may be a time tag defined by using regional time, standard time, or by virtual conference software based on conference duration.

[0105] According to an embodiment of the present disclosure, during the speaker's explanation, within a longer time scale (for example, 1 minute), the speaker may issue multiple control keywords regarding model posture adjustment. At this time, the multiple control keywords are combined based on the time sequence to obtain at least one control statement, and the second control information corresponding to the control statement is determined from the instruction matching table to perform posture control on the first virtual model.

[0106] According to an embodiment of the present disclosure, when there are multiple control statements, adjusting the posture of the first virtual model in the first display page based on the second control information includes:

[0107] For each control statement, determining second control information corresponding to the control statement from the instruction matching table;

[0108] Sorting the plurality of second control information based on time to obtain a sorting result;

[0109] The plurality of sorted second control information in the sorting result is sequentially used as the target control information, so as to adjust the posture of the first virtual model based on each target control information.

[0110] According to an embodiment of the present disclosure, multiple second control information of the spokesperson within a time period are time-sequenced to obtain a sequence result including an instruction sequence, and the posture of the first virtual model is adjusted based on each target control information in the instruction sequence.

[0111] According to an embodiment of the present disclosure, for example, the speaker orders "rotate 90° counterclockwise" at the 10th second and "rotate, top surface" at the 12th second. At this time, the virtual conference software needs to display the first virtual model after counterclockwise rotation on the first display page at the 11th second, and display the top surface area of ​​the first virtual model after the 13th second, so as to achieve a quick response to the command.

[0112] According to an embodiment of the present disclosure, different second control information are sorted according to the speaker's speaking order for quick execution, thereby improving the display accuracy of the virtual model, allowing participants to view the area being explained in the first display page in a timely manner during the speaker's explanation, thereby improving the participants' understanding of the explanation content.

[0113] According to an embodiment of the present disclosure, the first control voice information is processed by a preset voice processing method to obtain the second control information of the speaker, including:

[0114] The first control voice information is processed by using a target voice recognition network to obtain second control information.

[0115] According to an embodiment of the present disclosure, the target speech recognition network may refer to a neural network, such as a deep neural network (DNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a convolutional neural network (CNN), and the like.

[0116] According to an embodiment of the present disclosure, by using a target speech recognition network to process the first control speech information to obtain the second control information, the speech recognition accuracy of the speaker can be improved, thereby improving the display accuracy of the virtual model.

[0117] Figure 3 The figure schematically shows a training flow chart of a target speech recognition network according to an embodiment of the present disclosure.

[0118] like Figure 3 As shown, the training method of the target speech recognition network includes operations S301 to S304:

[0119] In operation S301, a training set is obtained, wherein the training set includes a plurality of speech training information and control label information corresponding to each speech training information, and the control label information represents model posture adjustment information;

[0120] In operation S302, for each speech training information, the speech training information is input into an initial speech recognition network, and prediction control information is output;

[0121] In operation S303, a loss result is generated according to the predicted control information and the control label information corresponding to the speech training information;

[0122] In operation S304, the network parameters of the initial speech recognition network are iteratively adjusted according to the loss result to obtain a trained target speech recognition network.

[0123] According to an embodiment of the present disclosure, the speech training information may be speech information recorded by different speakers, such as speakers with different dialects and from different regions, which may improve the universality of the target speech recognition network.

[0124] According to an embodiment of the present disclosure, for each piece of speech training information, it is input into the initial speech recognition network, and the predicted control information corresponding to the text type of the speech training information can be obtained. The corresponding loss result is calculated based on the predicted control information and the control label information corresponding to the speech training information, and the network parameters of the initial speech recognition network are iteratively adjusted based on the loss result, thereby obtaining a trained target speech recognition network.

[0125] According to an embodiment of the present disclosure, the loss function may be a binary classification loss or a similarity-based ‌Jaccard similarity loss, ‌cosine embedding loss, and the like.

[0126] According to an embodiment of the present disclosure, before adjusting the posture of the first virtual model, the method further includes:

[0127] Acquiring context voice information corresponding to the first control voice information;

[0128] Processing the context voice information using a preset voice processing method to obtain context text information corresponding to the second control information;

[0129] The second control information is modified based on the context text information to obtain modified second control information, so as to use the modified second control information to adjust the posture of the first virtual model.

[0130] According to the embodiments of the present disclosure, during a speaker's speech, there may be inconsistencies between the previous and next words due to mistakes. In this case, if the posture is adjusted based on the second control information in the first control voice information of the speaker, it may cause inconsistencies between the explanation content and the display content of the virtual model. For example, the speaker's first sentence says to rotate 90° to the left, and the second sentence says that the right side of the virtual model can be seen... At this time, if the left side of the virtual model is displayed, it is inconsistent with the right side actually explained by the speaker.

[0131] According to an embodiment of the present disclosure, in order to avoid the above-mentioned problems, the embodiment of the present disclosure will also obtain contextual voice information corresponding to the first control voice information, process the contextual voice information using a preset voice processing method, obtain contextual text information corresponding to the second control information, and correct the second control information in combination with the contextual text information to obtain corrected second control information, so as to use the corrected second control information to adjust the posture of the first virtual model.

[0132] According to an embodiment of the present disclosure, the second control information is corrected by combining the context voice information corresponding to the first control voice information, thereby avoiding the problem of inconsistency between the virtual model display content and the actual explanation content caused by the speaker's slip of the tongue, thereby further improving the display accuracy of the virtual model.

[0133] According to an embodiment of the present disclosure, obtaining first control information of a participant includes:

[0134] Obtain the second control voice information of the participants;

[0135] Processing the second control voice information using a preset voice processing method to obtain first control information;

[0136] The posture of the second virtual model in the second display page is adjusted based on the first control information.

[0137] According to the embodiments of the present disclosure, in order to further improve the user experience of the virtual conference software and avoid the problem of inefficient model display caused by participants manually issuing posture adjustment instructions, the participant's virtual conference software can collect the participant's voice in real time. At this time, the participant can choose not to turn on audio sharing to avoid the participant's voice affecting the explanations of other participants and speakers.

[0138] According to an embodiment of the present disclosure, after obtaining the second control voice information of the participant, the first control information is recognized by the virtual conference software of the participant, so as to perform gesture control on the second virtual model on the second display page formed in the first display page based on the recognized first control information. For example, when the speaker explains the left side of the first virtual model, the participant wants to see the specific structure of the right side of the virtual model. At this time, in response to the first control information of the participant, the second virtual model in the second display page is gesture controlled, so that the participant can view the right side of the virtual model through the second virtual model, wherein the first virtual model and the second virtual model belong to the same virtual model.

[0139] According to an embodiment of the present disclosure, the second display page is located in the target area of ​​the first display page, and at the same time, the first virtual model in the first display page is scaled so that the scaled first virtual model is located in the non-target area of ​​the first display page.

[0140] According to an embodiment of the present disclosure, the target area may refer to any position in the first display page, such as the upper left corner, the lower right corner, etc. The area of ​​the target area should not be too large to prevent participants from ignoring or missing the details of the first virtual model presented by the speaker when watching the second virtual model. Preferably, the ratio of the area occupied by the first virtual model to the area occupied by the second virtual model is 6:4 or 7:3.

[0141] According to an embodiment of the present disclosure, since the second display page occupies part of the first display page, if it is displayed according to the previous first virtual model, the target area may block part of the first virtual model. Therefore, the first virtual model can be appropriately scaled so that the first virtual model can be completely located in the non-target area, thereby improving the effectiveness and accuracy of the model display.

[0142] Figure 4 The structural block diagram of the virtual model display device according to an embodiment of the present disclosure is schematically shown.

[0143] like Figure 4 As shown, the virtual model display device 400 for virtual conference of this embodiment includes a first adjustment module 410 , a formation module 420 , an acquisition module 430 , and a second adjustment module 440 .

[0144] A first adjustment module 410 is used to adjust the posture of the first virtual model in the first display page based on the first control voice information of the speaker when displaying the first virtual model using the first display page of the virtual conference, so as to display the first display page to the speaker and at least one participant;

[0145] A forming module 420 is used to form a second display page in the first display page of the participant in response to the model control instruction of the participant, wherein the second display page displays a second virtual model corresponding to the first virtual model;

[0146] The acquisition module 430 is used to acquire the first control information of the participant;

[0147] The second adjustment module 440 is used to adjust the posture of the second virtual model in the second display page based on the first control information, so that the participants can view the first display page and the second display page at the same time.

[0148] According to an embodiment of the present disclosure, when the speaker displays the first virtual model through the first display page of the virtual conference, the posture of the first virtual model in the first display page is adjusted based on the speaker's first control voice information, and in response to the model control instruction of the participant, a second display page displaying the second virtual model is formed in the first display page of the participant, and the posture of the second virtual model in the second display page is adjusted based on the first control information of the participant, so that the participant can view the first display page and the second display page at the same time. In this way, the posture of the second virtual model in the second display page can be controlled without affecting the speaker's speech, so that the participant can synchronously watch the speaker's speech content and the virtual model area of ​​his own interest, thereby improving the meeting participation experience of the virtual meeting and further improving the participants' understanding of the virtual meeting.

[0149] According to an embodiment of the present disclosure, the first adjustment module 410 includes:

[0150] A voice processing submodule, used for processing the first control voice information by using a preset voice processing method to obtain the second control information of the speaker;

[0151] The first adjustment submodule is used to adjust the posture of the first virtual model in the first display page based on the second control information.

[0152] According to an embodiment of the present disclosure, the speech processing submodule includes:

[0153] A speech recognition unit, used for processing the first control speech information by using a speech recognition method to obtain first control text information;

[0154] A keyword extraction unit, used to process the first control text information using a keyword extraction method to obtain a plurality of control keywords;

[0155] The instruction matching unit is used to determine second control information corresponding to multiple control keywords from an instruction matching table, wherein the instruction matching table stores multiple initial control information for adjusting different postures of the virtual model.

[0156] According to an embodiment of the present disclosure, different control keywords have different time tags.

[0157] According to an embodiment of the present disclosure, the instruction matching unit includes:

[0158] A keyword combination subunit, used to combine multiple control keywords based on the time sequence to obtain at least one control sentence;

[0159] The first matching subunit is used to determine the second control information corresponding to the control statement from the instruction matching table.

[0160] According to an embodiment of the present disclosure, when there are multiple control statements, the instruction matching unit includes:

[0161] A second matching subunit, configured to determine, for each control statement, second control information corresponding to the control statement from the instruction matching table;

[0162] A sorting subunit, used to sort the plurality of second control information based on time to obtain a sorting result;

[0163] The determination subunit is used to sequentially use the plurality of sorted second control information in the sorting result as target control information, so as to adjust the posture of the first virtual model based on each target control information.

[0164] According to an embodiment of the present disclosure, the speech processing submodule includes:

[0165] The network recognition unit is used to process the first control voice information using the target voice recognition network to obtain the second control information.

[0166] Figure 5 The structural block diagram of a target speech recognition network training device according to an embodiment of the present disclosure is schematically shown.

[0167] like Figure 5 As shown, the training device of the target speech recognition network includes a second acquisition module 510, a prediction module 520, a loss calculation module 530, and an iterative training module 540.

[0168] A second acquisition module 510 is used to acquire a training set, wherein the training set includes a plurality of speech training information and control label information corresponding to each speech training information, and the control label information represents model posture adjustment information;

[0169] A prediction module 520, for each speech training information, inputting the speech training information into an initial speech recognition network and outputting prediction control information;

[0170] A loss calculation module 530, configured to generate a loss result based on the predicted control information and the control label information corresponding to the speech training information;

[0171] The iterative training module 540 is used to iteratively adjust the network parameters of the initial speech recognition network according to the loss result to obtain a trained target speech recognition network.

[0172] According to an embodiment of the present disclosure, the first adjustment module 410 further includes:

[0173] A first acquisition submodule, used to acquire context voice information corresponding to the first control voice information;

[0174] A context submodule, used to process the context voice information using a preset voice processing method to obtain context text information corresponding to the second control information;

[0175] The correction submodule is used to correct the second control information based on the context text information to obtain the corrected second control information, so as to use the corrected second control information to adjust the posture of the first virtual model.

[0176] According to an embodiment of the present disclosure, the acquisition module 430 includes:

[0177] The second acquisition submodule is used to acquire the second control voice information of the participant;

[0178] An identification submodule, configured to process the second control voice information using a preset voice processing method to obtain the first control information;

[0179] The second adjustment submodule is used to adjust the posture of the second virtual model in the second display page based on the first control information.

[0180] According to an embodiment of the present disclosure, the second display page is located in the target area of ​​the first display page, and at the same time, the first virtual model in the first display page is scaled so that the scaled first virtual model is located in the non-target area of ​​the first display page.

[0181] According to an embodiment of the present disclosure, any multiple modules among the first adjustment module 410, the formation module 420, the acquisition module 430, the second adjustment module 440, or the second acquisition module 510, the prediction module 520, the loss calculation module 530, and the iterative training module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first adjustment module 410, the formation module 420, the acquisition module 430, the second adjustment module 440, or the second acquisition module 510, the prediction module 520, the loss calculation module 530, and the iterative training module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first adjustment module 410, the formation module 420, the acquisition module 430, the second adjustment module 440, or the second acquisition module 510, the prediction module 520, the loss calculation module 530, and the iterative training module 540 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0182] Figure 6 A block diagram of an electronic device suitable for implementing the above method according to an embodiment of the present disclosure is schematically shown.

[0183] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0184] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0185] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0186] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0187] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0188] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0189] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0191] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0192] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0193] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0195] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A virtual model display method for a virtual meeting, characterized in that: The method comprises: In the case of displaying the first virtual model using the first display page of the virtual conference, adjusting the posture of the first virtual model in the first display page based on the first control voice information of the speaker, so as to display the first display page to the speaker and at least one participant; In response to a model control instruction of a participant, forming a second display page in the first display page of the participant, wherein a second virtual model corresponding to the first virtual model is displayed in the second display page; Acquire the first control information of the participant; Based on the first control information, the posture of the second virtual model in the second display page is adjusted so that the participant can view the first display page and the second display page at the same time.

2. The method according to claim 1, characterized in that Adjusting the posture of the first virtual model in the first display page based on the first control voice information of the speaker includes: Processing the first control voice information by using a preset voice processing method to obtain second control information of the speaker; The posture of the first virtual model in the first display page is adjusted based on the second control information.

3. The method according to claim 2, characterized in that Processing the first control voice information by using a preset voice processing method to obtain second control information of the speaker includes: Processing the first control voice information by using a voice recognition method to obtain first control text information; Processing the first control text information using a keyword extraction method to obtain a plurality of control keywords; The second control information corresponding to the plurality of control keywords is determined from an instruction matching table, wherein the instruction matching table stores a plurality of initial control information for adjusting the virtual model to different postures.

4. The method according to claim 3, characterized in that Different control keywords have different time tags; Wherein, determining the second control information corresponding to the plurality of control keywords from the instruction matching table includes: Combining a plurality of the control keywords based on a time sequence to obtain at least one control statement; The second control information corresponding to the control statement is determined from the instruction matching table.

5. The method according to claim 4, characterized in that In the case where there are multiple control statements, adjusting the posture of the first virtual model in the first display page based on the second control information includes: For each of the control statements, determining second control information corresponding to the control statement from the instruction matching table; Sort the plurality of second control information based on time to obtain a sorting result; The plurality of sorted second control information in the sorting result is sequentially used as target control information, so as to adjust the posture of the first virtual model based on each of the target control information.

6. The method according to claim 2, characterized in that Processing the first control voice information by using a preset voice processing method to obtain second control information of the speaker includes: The first control voice information is processed using a target voice recognition network to obtain the second control information.

7. The method according to claim 6, characterized in that The target speech recognition network is trained in the following manner: Acquire a training set, wherein the training set includes a plurality of speech training information and control label information corresponding to each of the speech training information, and the control label information represents model posture adjustment information; For each of the speech training information, input the speech training information into an initial speech recognition network and output prediction control information; generating a loss result according to the predicted control information and the control label information corresponding to the speech training information; The network parameters of the initial speech recognition network are iteratively adjusted according to the loss result to obtain a trained target speech recognition network.

8. The method according to claim 2, characterized in that: Before adjusting the posture of the first virtual model, the method further includes: Acquire context voice information corresponding to the first control voice information; Processing the context voice information using a preset voice processing method to obtain context text information corresponding to the second control information; The second control information is modified based on the context text information to obtain modified second control information, so as to use the modified second control information to adjust the posture of the first virtual model.

9. The method according to claim 1, characterized in that: Acquiring the first control information of the participant, including: Acquire the second control voice information of the participant; Processing the second control voice information using a preset voice processing method to obtain the first control information; The posture of the second virtual model in the second display page is adjusted based on the first control information.

10. The method according to claim 1, characterized in that The second display page is located in the target area of ​​the first display page. At the same time, the first virtual model in the first display page is scaled so that the scaled first virtual model is located in the non-target area of ​​the first display page.

11. A virtual model display device for virtual conference, characterized in that: The device comprises: A first adjustment module is used to adjust the posture of the first virtual model in the first display page of the virtual conference based on the first control voice information of the speaker, so as to facilitate the display of the first display page to the speaker and at least one participant; A forming module, configured to form a second display page in the first display page of the participant in response to a model control instruction of the participant, wherein the second display page displays a second virtual model corresponding to the first virtual model; An acquisition module, used for acquiring the first control information of the participant; The second adjustment module is used to adjust the posture of the second virtual model in the second display page based on the first control information, so that the participant can view the first display page and the second display page at the same time.

12. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images