Interactive online conference implementation method and device based on augmented reality

Abstract

The invention relates to the technical field of augmented reality and financial science and technology, and particularly discloses an interactive online conference implementation method and device based on augmented reality. According to the method, the virtual conference scene can be created based on the augmented reality equipment, and the virtual portraits of the conference participants are projected into the scene, so that the participants can carry out a conference personally on the scene in the virtual conference scene, and the conference participating experience of the participants is improved; meanwhile, according to the hand position and the gesture action of the speaker, the interaction area or the interaction element is determined, the interaction instruction can be automatically triggered, real-time interaction is achieved, and the interaction effect is improved.

Description

Technical Field

[0001] The present application relates to the field of augmented reality and financial technology, and in particular to a method and device for implementing an interactive online conference based on augmented reality. Background Art

[0002] With the development of computer technology, most of the business in the financial field has shifted from offline to online and offline. For example, business personnel can conduct preliminary product promotion and after-sales services to customers through online meetings. The development of online business saves more time costs for business personnel, and business personnel can communicate with customers at any time. However, traditional online business lacks immersive participation, low collaboration efficiency and poor interaction effect, resulting in a low success rate of online business. Therefore, how to improve the interaction effect of online meetings and the experience of participants has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides an augmented reality-based interactive online conference implementation method and device to improve the interactive effect of online conferences and the conference experience of participants.

[0004] In a first aspect, the present application provides a method for implementing an interactive online conference based on augmented reality, the method comprising:

[0005] Based on a preset augmented reality device, a virtual conference scene is created, and virtual portraits and body postures of conference participants are obtained, and the virtual portraits and body postures are projected into the virtual conference scene;

[0006] When it is detected that a speaker is speaking and the speaker has an interaction requirement, based on a preset virtual interaction tool, the hand position and gesture movement of the speaker are identified;

[0007] generating a virtual pointer based on the hand position and the gesture action, and determining a target interaction area or interaction element based on the orientation of the virtual pointer;

[0008] Respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

[0009] In a second aspect, the present application further provides an augmented reality-based interactive online conference implementation device, the device comprising:

[0010] A scene creation module, used to create a virtual conference scene based on a preset augmented reality device, obtain virtual portraits and body postures of conference participants, and project the virtual portraits and body postures into the virtual conference scene;

[0011] An action recognition module is used to recognize the hand position and gesture of the speaker based on a preset virtual interaction tool when it is detected that a speaker is speaking and the speaker has an interaction requirement;

[0012] An interaction determination module, configured to generate a virtual pointer based on the hand position and the gesture action, and determine a target interaction area or interaction element based on the orientation of the virtual pointer;

[0013] The instruction response module is used to respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

[0014] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the above-mentioned augmented reality-based interactive online conference implementation method when executing the computer program.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the above-mentioned method for implementing an interactive online meeting based on augmented reality.

[0016] The present application discloses a method and device for implementing an interactive online conference based on augmented reality. Based on a preset augmented reality device, a virtual conference scene is created, and virtual portraits and body postures of conference participants are obtained, and the virtual portraits and body postures are projected into the virtual conference scene; when it is detected that a speaker is speaking and the speaker has an interactive need, the hand position and gesture of the speaker are identified based on a preset virtual interactive tool; a virtual pointer is generated based on the hand position and the gesture, and a target interactive area or interactive element is determined based on the direction of the virtual pointer; and instructions corresponding to the gesture are responded to on the interactive area or the interactive element to achieve conference interaction. The present application can create a virtual conference scene based on an augmented reality device, and project the virtual portraits of conference participants into the scene, so that participants can conduct meetings in an immersive manner in the virtual conference scene, thereby improving the participants' experience of attending meetings; at the same time, the interactive area or interactive element is determined according to the speaker's hand position and gesture, and the interactive instructions can be automatically triggered to achieve real-time interaction, thereby improving the interactive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a first embodiment of a method for implementing an interactive online conference based on augmented reality provided by an embodiment of the present application;

[0019] Figure 2 is a schematic flow chart of a second embodiment of a method for implementing an interactive online conference based on augmented reality provided in an embodiment of the present application;

[0020] Figure 3 is a schematic flow chart of a third embodiment of a method for implementing an interactive online conference based on augmented reality provided by an embodiment of the present application;

[0021] Figure 4 A schematic block diagram of an augmented reality-based interactive online conference implementation device provided in an embodiment of the present application;

[0022] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] The embodiments of the present application provide a method and device for implementing an interactive online conference based on augmented reality. The method can be applied to a server to create a virtual conference scene based on an augmented reality device, and project virtual portraits of conference participants into the scene, so that participants can conduct meetings in an immersive manner in the virtual conference scene, thereby improving the participants' experience; at the same time, the interactive area or interactive element is determined based on the speaker's hand position and gestures, and the interactive command can be automatically triggered to achieve real-time interaction and improve the interactive effect. The server can be an independent server or a server cluster.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] See also Figure 1 , Figure 1 This is a schematic flow chart of an implementation method of an interactive online conference based on augmented reality provided by an embodiment of the present application. The implementation method of an interactive online conference based on augmented reality can be applied to a server to create a virtual conference scene based on an augmented reality device, and project virtual portraits of conference participants into the scene, so that participants can conduct a meeting in an immersive way in the virtual conference scene, thereby improving the experience of participants; at the same time, according to the speaker's hand position and gestures, the interactive area or interactive element is determined, and the interactive command can be automatically triggered to achieve real-time interaction and improve the interactive effect.

[0030] like Figure 1 As shown, the method for implementing an interactive online conference based on augmented reality specifically includes steps S101 to S104.

[0031] S101, creating a virtual conference scene based on a preset augmented reality device, obtaining virtual portraits and body postures of conference participants, and projecting the virtual portraits and body postures into the virtual conference scene;

[0032] In this embodiment, a virtual conference room based on AR (Augmented Reality) technology is created and displayed, so that meeting participants can hold a meeting in an immersive way in a virtual space.

[0033] In one embodiment, the AR devices (such as AR helmets or smart glasses) of all meeting participants are calibrated to suit their respective visual needs.

[0034] In one embodiment, the graphics processing unit of the AR device or the cloud server is used to render the scene to ensure the realism and smoothness of the scene, and to add interactive areas and interactive elements such as a virtual whiteboard, presentation, video playback, etc. to the virtual scene.

[0035] In one embodiment, a camera and sensor built into the AR device or a camera and sensor external to the AR device (such as an inertial measurement unit) is used to capture the meeting status images and body postures of the meeting participants.

[0036] In one embodiment, the conference status image is processed and a virtual portrait of the conference participant is created. The acquired virtual portrait and body posture data are rendered in a virtual conference scene through an AR device.

[0037] Furthermore, the creation of a virtual conference scene based on a preset augmented reality device includes: obtaining the meeting theme and the number of participants of the current meeting based on the augmented reality device; determining the spatial size of the virtual conference scene based on the number of participants; determining the background environment of the virtual conference scene based on the meeting theme; and creating the virtual conference scene according to the spatial size and the background environment.

[0038] In one embodiment, natural language processing technology is used to analyze the meeting invitation, agenda or related documents to extract the meeting theme. Speech recognition technology can also be used to capture the host or organizer's verbal introduction in real time at the beginning of the meeting to extract the meeting theme.

[0039] In one embodiment, the number of participants entering the virtual conference scene is counted in real time using the camera and computer vision algorithm in the AR device. The number of participants who have joined the conference can also be automatically counted through user login information and device tracking.

[0040] In one embodiment, the required space size is analyzed according to the number of participants to ensure that each conference participant has sufficient activity space and viewing range. Interaction areas are divided in the virtual scene, such as lecture area, discussion area and rest area, and the sizes of these areas are adjusted according to the number of participants.

[0041] In one embodiment, a corresponding virtual background environment is designed according to the theme of the meeting. For example, if it is a business meeting, an office or conference room background can be selected; if it is a casual meeting, a natural environment such as a beach or a park can be selected. The organization's cultural and brand elements, such as corporate colors, logos, or specific decorative styles, can also be incorporated into the virtual background environment.

[0042] In a specific embodiment, a virtual conference scene is designed using 3D modeling software according to the determined space size and background environment. Necessary 3D models such as furniture, decorations and brand elements are imported. A graphics rendering engine is used to render the virtual conference scene.

[0043] Furthermore, obtaining the virtual portraits and body postures of the conference participants includes: obtaining a conference status image of each conference participant; extracting the body image of the conference participant from the conference status image based on a preset portrait segmentation algorithm; and performing posture detection on the conference status based on a posture estimation model to obtain the body posture of the conference participant.

[0044] In one embodiment, a built-in or external camera of the AR device is used to capture images of the meeting participants in real time, ensure that the camera covers all meeting participants, and adjust the viewing angle to obtain the best image quality.

[0045] In one embodiment, the captured image is preprocessed, including adjusting brightness, contrast, and color balance to improve the accuracy of subsequent processing. Image denoising techniques are applied to reduce noise in the image and improve image quality. The captured image is transmitted in real time to a processing device (such as a built-in processor of an AR helmet or a remote server) for further analysis.

[0046] In one embodiment, a portrait segmentation algorithm, such as Mask R-CNN, U-Net or other deep learning models, is applied to segment a human image from the meeting state image, and a corresponding foreground mask is generated to separate the human body from the background. The generated mask is optimized, such as edge smoothing and hole filling, to improve the quality of the human image.

[0047] In one embodiment, the segmented human body image is combined with a new virtual background to create a virtual portrait of the conference participant.

[0048] In one embodiment, a posture estimation model, such as OpenPose, PoseNet or other deep learning posture estimation algorithms, is used to detect the body posture of the meeting participants. Specifically, key points in the image, such as the head, shoulders, elbows, wrists, hips, knees, etc., and their relative positions and angles are identified. The changes of these key points over time are tracked to obtain the body posture and action sequence of the meeting participants. For multi-person scenarios, the key points of different meeting participants are distinguished and tracked independently.

[0049] In one embodiment, key point data is analyzed to identify body postures and movements of meeting participants, such as raising hands, standing, sitting, etc. Machine learning algorithms, such as hidden Markov models or long short-term memory networks, can also be applied to model and predict the behavior patterns of meeting participants.

[0050] S102, when it is detected that a speaker is speaking and the speaker has an interaction requirement, identifying the hand position and gesture of the speaker based on a preset virtual interaction tool;

[0051] Furthermore, when it is detected that a speaker is speaking and the speaker has an interaction need, based on a preset virtual interaction tool, the speaker's hand position and gesture movements are identified, including: when it is detected that the speaker is speaking, the speaker's speech content is obtained, the speaker's speech content is analyzed, and it is determined whether there are interaction keywords; when the interaction keywords are present, based on the virtual interaction tool, the speaker's body posture is identified, the body posture is compared with the preset interaction posture, and a matching degree is obtained; when the matching degree is greater than or equal to a preset threshold, it is determined that the speaker has an interaction need, and based on the virtual interaction tool, the speaker's hand position and gesture movements are identified.

[0052] In one embodiment, a built-in or external sound receiving device (such as a microphone) of the AR device is used to capture the speaker's voice, and the voice is converted into text using automatic speech recognition technology.

[0053] In one embodiment, the converted text is subjected to natural language processing, including word segmentation, part-of-speech tagging, and semantic analysis. A preset keyword library is used, and interactive keywords in the speech are identified through pattern matching or machine learning methods (such as classification algorithms). Specifically, interactive keywords can be defined by the user according to actual needs, such as "display", "switch", "enlarge", etc. It is detected whether these keywords are contained in the speech to determine whether the speaker is likely to perform interactive operations.

[0054] In one embodiment, the speaker's body key points are tracked and a pose estimation model, such as OpenPose, is applied to detect the speaker's body posture in real time.

[0055] In one embodiment, the speaker's real-time body posture is compared with the preset interaction posture. Specifically, a machine learning algorithm, such as a support vector machine or a deep learning model, can be used to match postures. The similarity or matching degree between the speaker's posture and the preset posture is calculated. When the matching degree exceeds a preset threshold, it is considered that the speaker has an interaction demand, and the speaker's hand position and gesture action are identified, and the corresponding interaction operation is further performed. Among them, the interaction posture and the preset threshold can be freely set by the user according to actual needs.

[0056] S103, generating a virtual pointer based on the hand position and the gesture action, and determining a target interaction area or interaction element based on the orientation of the virtual pointer;

[0057] In one embodiment, a built-in or external camera module (such as an RGB camera, a depth camera, or an infrared camera) of the AR device is used to capture the user's hand image. A built-in or external sensor (such as an IMU or a data glove) of the AR device is used to collect the position and motion data of the hand.

[0058] In one embodiment, a gesture recognition algorithm, such as a convolutional neural network based on deep learning, is applied to recognize the user's gesture movements.

[0059] In one embodiment, based on the user's hand position data, the starting point of the pointer is located in the virtual space, usually at the user's fingertips or the center of the palm. Based on the user's gestures and the hand posture estimation results, the direction of the pointer is determined. If the user's index finger points in a certain direction, the pointer should point in the same direction.

[0060] In one embodiment, a virtual ray is emitted from the tip of the virtual pointer toward it. The ray intersects with objects in the scene to determine the target area or element that the user intends to interact with. Specifically, based on the intersection of the ray with the interactive element or interactive area, the target that the user intends to operate is identified. If multiple elements or areas intersect with the ray, the target can be selected based on distance, size, or preset priority rules.

[0061] In one embodiment, when the virtual pointer targets an interactive element, visual feedback is provided, such as highlighting, border flashing, or dynamic effects. Auditory feedback, such as click sounds or prompt sounds, can also be provided to enhance the user's interactive experience. If the user wears a device that supports tactile feedback, such as a data glove or a VR controller, tactile feedback can be provided to simulate the sense of touch in the real world.

[0062] S104: Respond to an instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

[0063] In one embodiment, the correspondence between gestures and conference control instructions can be freely set by the user as needed. For example, waving a hand may correspond to "next page", and making a fist and then releasing it may correspond to "confirm selection".

[0064] In one embodiment, when a gesture action is recognized and an interactive area or interactive element is determined, the instruction corresponding to the gesture action is executed. For example, if the gesture is "zoom in", a zoom in operation will be performed on the document or image targeted by the user.

[0065] In another embodiment, the current conference context can also be understood to correctly execute the instructions corresponding to the gesture. For example, the same gesture may correspond to different operations in different conference stages or different application scenarios. Specifically, the gesture and instruction database corresponding to different stages can be called according to different conference stages to find the interaction instruction corresponding to the current gesture to achieve interaction.

[0066] The above embodiment provides a method and device for implementing an interactive online meeting based on augmented reality. Based on a preset augmented reality device, a virtual meeting scene is created, and virtual portraits and body postures of meeting participants are obtained, and the virtual portraits and body postures are projected into the virtual meeting scene; when it is detected that there is a speaker speaking and the speaker has an interactive need, the hand position and gesture of the speaker are identified based on a preset virtual interaction tool; a virtual pointer is generated based on the hand position and the gesture, and a target interactive area or interactive element is determined based on the direction of the virtual pointer; and instructions corresponding to the gesture are responded to on the interactive area or the interactive element to achieve conference interaction. The present application can create a virtual meeting scene based on an augmented reality device, and project virtual portraits of meeting participants into the scene, so that participants can conduct meetings in an immersive manner in the virtual meeting scene, thereby improving the meeting experience of participants; at the same time, the interactive area or interactive element is determined according to the speaker's hand position and gesture, and the interactive instructions can be automatically triggered to achieve real-time interaction, thereby improving the interactive effect.

[0067] See also Figure 2 , Figure 2 This is a schematic flow chart of an implementation method of an augmented reality-based interactive online conference provided by an embodiment of the present application. The augmented reality-based interactive online conference implementation method can be applied to a server to apply speech recognition and natural language processing technology to provide real-time subtitles for the conference, and generate personalized conference records according to the requirements of the conference records and the importance of the speech content, thereby improving the efficiency of conference records.

[0068] like Figure 2 As shown, the method for implementing an interactive online conference based on augmented reality specifically includes steps S201 to S204.

[0069] S201, when it is detected that a speaker is speaking, obtaining the speech audio of the speaker;

[0070] S202, performing speech recognition on the speech audio to obtain audio features corresponding to the speech audio;

[0071] S203, comparing the audio feature with the preset conference participant information to determine the identity information of the speaker;

[0072] S204: Perform voice conversion on the speech audio to obtain speech text, and generate subtitle content and meeting records based on the identity information of the speaker and the speech text.

[0073] In one embodiment, during a meeting, a built-in or external sound receiving device, such as a microphone, of the AR device is used to capture the speaker's voice in real time, and the collected audio data is synchronized with the speaker's identification (such as name or ID) and speech timestamp for subsequent processing.

[0074] In one embodiment, the captured audio stream is converted into text information using automatic speech recognition technology. A speech recognition model, such as a deep learning-based model, is applied to extract audio features and perform recognition. Key acoustic features, such as Mel-frequency cepstral coefficients, are extracted from the speech audio.

[0075] In one embodiment, the extracted audio features are compared with the pre-recorded voiceprint data of the conference participants, and voiceprint recognition technology is used to determine the identity information of the speaker.

[0076] In one embodiment, speech recognition technology is used to convert audio streams into text in real time to generate speech transcripts. Subtitles with speaker labels are generated based on the speaker's text and identity information. These subtitles can be displayed in real time on the shared screen of the meeting to improve the accessibility and participation of the meeting.

[0077] In one embodiment, the speech text and identity information are integrated into the conference record. If the conference has international conference participants, the speech text can be translated into different languages ​​and multi-language subtitles can be generated.

[0078] Furthermore, the speech audio is converted into speech to obtain speech text, and a meeting record is generated based on the speaker's identity information and the speech text, including: identifying the speech audio to obtain the speaker's voice tone change information; identifying the speaker's facial expression to obtain the speaker's facial expression change information; based on the voice tone change information and the facial expression change information, analyzing the importance of the speech text to obtain the importance level of each part of the speech text; based on the preset meeting record requirements and the importance level, recording the speech text to obtain the meeting record.

[0079] In one embodiment, audio analysis techniques such as pitch, volume, speech rate and rhythm are applied to identify changes in the speaker's tone of voice. Machine learning models such as sentiment analysis models are used to assess the speaker's tone of voice, such as positive, negative or neutral.

[0080] In one embodiment, a built-in or external camera of the AR device is used to capture the speaker's facial video in real time. A facial detection algorithm, such as a Haar cascade classifier or a deep learning model, is applied to locate the facial area. Facial expression recognition technology, such as FACS (Facial Action Coding System) based on deep learning, is used to identify and classify facial expressions. The main expressions include happiness, sadness, surprise, anger, etc., as well as more subtle expression changes.

[0081] In one embodiment, the emotional content of the speech is evaluated using multimodal sentiment analysis technology in combination with the information of tone changes and facial expression changes. The tone and expression data are analyzed to determine the emotional state of the speaker at different stages of the speech. Based on the sentiment analysis results, different parts of the speech text are scored for importance. For example, parts with higher emotional intensity may be marked as more important.

[0082] In one embodiment, the content is classified as "important", "normal" or "minor" according to the importance level of the speech text.

[0083] In one embodiment, based on the preset meeting record requirements and the importance level of the content, intelligently select which parts of the speech text to record. For example, the meeting record requirements may include recording "important" content in detail and marking it with emphasis, and recording "minor" content briefly.

[0084] In the above embodiments, speech recognition and natural language processing technologies can be applied to provide real-time subtitles for the meeting, and personalized meeting records can be generated according to the requirements of the meeting records and the importance of the speech content, thereby improving the efficiency of meeting records.

[0085] See also Figure 3 , Figure 3 This is a schematic flow chart of an implementation method of an interactive online conference based on augmented reality provided by an embodiment of the present application. The implementation method of an interactive online conference based on augmented reality can be applied to a server to intelligently arrange the positions of virtual portraits in a virtual conference scene according to the conference theme and the position information of the participants, thereby improving the interactivity of the conference and the experience of the participants.

[0086] like Figure 3 As shown, the method for implementing an interactive online conference based on augmented reality specifically includes steps S301 to S304.

[0087] S301, obtaining the meeting theme of the current meeting and the identity information table of the meeting participants, and extracting the position information of the meeting participants from the identity information table;

[0088] S302, analyzing the participation level of the conference participants in the current conference based on the conference theme and the position information;

[0089] S303: Determine the location information of the conference participant based on the participation level, and project the virtual portrait and the body posture into the virtual conference scene based on the location information.

[0090] In one embodiment, the subject of the current meeting can be obtained through a meeting arrangement system or a meeting invitation letter. The subject can be pre-defined by the user according to actual conditions, such as "annual budget review", "product launch meeting", etc.

[0091] In one embodiment, before the meeting starts, a table containing identity information of all participants is created, which can be provided by the meeting organizer or obtained when the meeting participants log in and join the meeting scene. The identity information table includes the name, position, department, and contact information of the meeting participants.

[0092] In one embodiment, the position information of each participant is extracted from the identity information table.

[0093] In one embodiment, the participation level of each participant is judged based on the meeting topic and the position information of the participants. For example, in a technical meeting, technicians and project managers may have a higher level of participation than administrative staff. Specifically, the pre-trained model can be trained by the position information of each meeting participant in the historical meeting records and the participation program to obtain a participation level evaluation model. The obtained meeting topic and the position information of the participants are used as input parameters of the participation level evaluation model to obtain a participation level score.

[0094] In one embodiment, the position of each participant in the virtual meeting scene is determined according to the participation score. For example, a person with a higher participation level may be placed in a more prominent position, such as the front row or the middle of the conference room.

[0095] In the above embodiment, the positions of the virtual portraits in the virtual conference scene are intelligently arranged according to the conference theme and the position information of the participants, thereby improving the interactivity of the conference and the conference experience.

[0096] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of an interactive online conference implementation device based on augmented reality, which is used to execute the aforementioned interactive online conference implementation method based on augmented reality. The interactive online conference implementation device based on augmented reality can be configured on a server.

[0097] like Figure 4 As shown, the interactive online conference implementation device 400 based on augmented reality includes:

[0098] The scene creation module 401 is used to create a virtual conference scene based on a preset augmented reality device, obtain virtual portraits and body postures of conference participants, and project the virtual portraits and body postures into the virtual conference scene;

[0099] The action recognition module 402 is used to recognize the hand position and gesture of the speaker based on a preset virtual interaction tool when it is detected that a speaker is speaking and the speaker has an interaction requirement;

[0100] An interaction determination module 403, configured to generate a virtual pointer based on the hand position and the gesture action, and determine a target interaction area or interaction element based on the orientation of the virtual pointer;

[0101] The instruction response module 404 is used to respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

[0102] Furthermore, the scene creation module 401 includes:

[0103] An image acquisition unit, used to acquire the conference status image of each conference participant;

[0104] A portrait segmentation unit, configured to extract a human body image of the conference participant from the conference attendance state image based on a preset portrait segmentation algorithm;

[0105] The posture acquisition unit is used to perform posture detection on the conference participant status based on a posture estimation model to obtain the body posture of the conference participant.

[0106] Furthermore, the augmented reality-based interactive online conference implementation device 400 further includes: a conference record generation module, the conference record generation module including:

[0107] An audio acquisition unit, configured to acquire the speech audio of the speaker when detecting that a speaker is speaking;

[0108] A feature acquisition unit, configured to perform speech recognition on the speech audio to obtain audio features corresponding to the speech audio;

[0109] An identity information determination unit, used to compare the audio feature with the preset conference participant information to determine the identity information of the speaker;

[0110] The record generation unit is used to perform voice conversion on the speech audio to obtain the speech text, and generate subtitle content and meeting records based on the identity information of the speaker and the speech text.

[0111] Furthermore, the record generation unit includes:

[0112] A voice change information obtaining subunit, used to recognize the speech audio and obtain the voice change information of the speaker;

[0113] An expression change information obtaining subunit, used for identifying the speaker's facial expression and obtaining the speaker's facial expression change information;

[0114] An importance level obtaining subunit, used to analyze the importance of the speech text based on the tone change information and the facial expression change information, and obtain the importance level of each part of the speech text;

[0115] The conference record obtaining subunit is used to record the speech text based on the preset conference record requirements and the importance level to obtain the conference record.

[0116] Furthermore, the action recognition module 402 includes:

[0117] An interactive keyword determination unit, configured to obtain the speech content of the speaker when detecting that the speaker is speaking, analyze the speech content of the speaker, and determine whether there is an interactive keyword;

[0118] a matching degree obtaining unit, configured to, when the interaction keyword exists, identify the speaker's body posture based on the virtual interaction tool, compare the body posture with a preset interaction posture, and obtain a matching degree;

[0119] The action recognition unit is used to determine that the speaker has an interaction demand when the matching degree is greater than or equal to a preset threshold, and to recognize the hand position and gesture action of the speaker based on the virtual interaction tool.

[0120] Furthermore, the scene creation module 401 includes:

[0121] A position information extraction unit, used to obtain the meeting theme of the current meeting and the identity information table of the meeting participants, and extract the position information of the meeting participants from the identity information table;

[0122] a participation degree analysis unit, configured to analyze the participation degree of the conference participants in the current conference based on the conference theme and the position information;

[0123] A position information obtaining unit is used to determine the position information of the conference participants based on the participation level, and project the virtual portrait and the body posture into the virtual conference scene based on the position information.

[0124] Furthermore, the scene creation module 401 includes:

[0125] A personnel quantity acquisition unit, used to acquire the meeting theme and the number of participants of the current meeting based on the augmented reality device;

[0126] A space size determination unit, configured to determine the space size of the virtual conference scene based on the number of participants;

[0127] A background environment determining unit, configured to determine the background environment of the virtual conference scene based on the conference theme;

[0128] A conference scene creation unit is used to create the virtual conference scene according to the space size and the background environment.

[0129] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0130] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0131] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0132] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0133] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for implementing an interactive online conference based on augmented reality.

[0134] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0135] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for implementing an interactive online meeting based on augmented reality.

[0136] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0138] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0139] Based on a preset augmented reality device, a virtual conference scene is created, and virtual portraits and body postures of conference participants are obtained, and the virtual portraits and body postures are projected into the virtual conference scene;

[0140] When it is detected that a speaker is speaking and the speaker has an interaction requirement, based on a preset virtual interaction tool, the hand position and gesture movement of the speaker are identified;

[0141] generating a virtual pointer based on the hand position and the gesture action, and determining a target interaction area or interaction element based on the orientation of the virtual pointer;

[0142] Respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

[0143] In one embodiment, when acquiring the virtual portrait and body posture of the conference participants, the processor is used to implement:

[0144] Obtain the meeting status image of each meeting participant;

[0145] Based on a preset portrait segmentation algorithm, extracting the human body image of the conference participant from the conference attendance state image;

[0146] Based on the posture estimation model, posture detection is performed on the conference participant status to obtain the body posture of the conference participant.

[0147] In one embodiment, after the processor creates a virtual conference scene based on a preset augmented reality device, obtains virtual portraits and body postures of conference participants, and projects the virtual portraits and body postures into the virtual conference scene, it is further used to implement:

[0148] When it is detected that a speaker is speaking, obtaining the speech audio of the speaker;

[0149] Performing speech recognition on the speech audio to obtain audio features corresponding to the speech audio;

[0150] Comparing the audio feature with the preset conference participant information to determine the identity information of the speaker;

[0151] The speech audio is converted into speech to obtain a speech text, and subtitle content and meeting records are generated based on the identity information of the speaker and the speech text.

[0152] In one embodiment, when the processor performs voice conversion on the speech audio to obtain the speech text, and generates the meeting record based on the identity information of the speaker and the speech text, it is used to implement:

[0153] Recognize the speech audio to obtain the speaker's tone change information;

[0154] Recognizing the speaker's facial expression and obtaining the speaker's facial expression change information;

[0155] Analyzing the importance of the speech text based on the tone change information and the facial expression change information to obtain the importance level of each part of the speech text;

[0156] Based on the preset meeting record requirements and the importance level, the speech text is recorded to obtain the meeting record.

[0157] In one embodiment, when the processor detects that a speaker is speaking and the speaker has an interaction requirement, based on a preset virtual interaction tool, identifying the speaker's hand position and gesture action, the processor is used to implement:

[0158] When it is detected that the speaker is speaking, obtaining the speech content of the speaker, analyzing the speech content of the speaker, and determining whether there is an interactive keyword;

[0159] When the interaction keyword exists, based on the virtual interaction tool, identifying the speaker's body posture, comparing the body posture with a preset interaction posture, and obtaining a matching degree;

[0160] When the matching degree is greater than or equal to a preset threshold, it is determined that the speaker has an interaction demand, and based on the virtual interaction tool, the hand position and gesture action of the speaker are identified.

[0161] In one embodiment, when the processor implements projecting the virtual portrait and body posture into the virtual conference scene, it is used to implement:

[0162] Obtaining the meeting subject of the current meeting and the identity information table of the meeting participants, and extracting the position information of the meeting participants from the identity information table;

[0163] Analyzing the participation level of the meeting participants in the current meeting based on the meeting theme and the position information;

[0164] The location information of the conference participants is determined based on the participation level, and based on the location information, the virtual portrait and the body posture are projected into the virtual conference scene.

[0165] In one embodiment, when creating a virtual conference scene based on a preset augmented reality device, the processor is used to implement:

[0166] Based on the augmented reality device, obtain the meeting topic and the number of participants of the current meeting;

[0167] Determining the spatial size of the virtual conference scene based on the number of participants;

[0168] Based on the conference theme, determining the background environment of the virtual conference scene;

[0169] The virtual conference scene is created according to the space size and the background environment.

[0170] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the augmented reality-based interactive online conference implementation methods provided in the embodiments of the present application.

[0171] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for implementing an interactive online conference based on augmented reality, characterized in that: include: Based on a preset augmented reality device, a virtual conference scene is created, and virtual portraits and body postures of conference participants are obtained, and the virtual portraits and body postures are projected into the virtual conference scene; When it is detected that a speaker is speaking and the speaker has an interaction requirement, based on a preset virtual interaction tool, the hand position and gesture movement of the speaker are identified; generating a virtual pointer based on the hand position and the gesture action, and determining a target interaction area or interaction element based on the orientation of the virtual pointer; Respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

2. The method for implementing an interactive online conference based on augmented reality according to claim 1, characterized in that: The step of obtaining the virtual portraits and body postures of the conference participants includes: Obtain the meeting status image of each meeting participant; Based on a preset portrait segmentation algorithm, extracting the human body image of the conference participant from the conference attendance state image; Based on the posture estimation model, posture detection is performed on the conference participant status to obtain the body posture of the conference participant.

3. The method for implementing an interactive online conference based on augmented reality according to claim 1, characterized in that: The method further includes: creating a virtual conference scene based on a preset augmented reality device, obtaining virtual portraits and body postures of conference participants, and projecting the virtual portraits and body postures into the virtual conference scene; When it is detected that a speaker is speaking, obtaining the speech audio of the speaker; Performing speech recognition on the speech audio to obtain audio features corresponding to the speech audio; Comparing the audio feature with the preset conference participant information to determine the identity information of the speaker; The speech audio is converted into speech to obtain a speech text, and subtitle content and meeting records are generated based on the identity information of the speaker and the speech text.

4. The method for implementing an interactive online conference based on augmented reality according to claim 3, characterized in that: The step of converting the speech audio into speech to obtain speech text, and generating a meeting record based on the identity information of the speaker and the speech text, includes: Recognize the speech audio to obtain the speaker's tone change information; Recognizing the speaker's facial expression and obtaining the speaker's facial expression change information; Analyzing the importance of the speech text based on the tone change information and the facial expression change information to obtain the importance level of each part of the speech text; Based on the preset meeting record requirements and the importance level, the speech text is recorded to obtain the meeting record.

5. The method for implementing an interactive online conference based on augmented reality according to claim 1, characterized in that: When it is detected that a speaker is speaking and the speaker has an interaction requirement, based on a preset virtual interaction tool, identifying the hand position and gesture action of the speaker includes: When it is detected that the speaker is speaking, obtaining the speech content of the speaker, analyzing the speech content of the speaker, and determining whether there is an interactive keyword; When the interaction keyword exists, based on the virtual interaction tool, identifying the speaker's body posture, comparing the body posture with a preset interaction posture, and obtaining a matching degree; When the matching degree is greater than or equal to a preset threshold, it is determined that the speaker has an interaction demand, and based on the virtual interaction tool, the hand position and gesture action of the speaker are identified.

6. The method for implementing an interactive online conference based on augmented reality according to claim 1, characterized in that: The projecting the virtual portrait and the body posture into the virtual conference scene includes: Obtaining the meeting subject of the current meeting and the identity information table of the meeting participants, and extracting the position information of the meeting participants from the identity information table; Analyzing the participation level of the meeting participants in the current meeting based on the meeting theme and the position information; The location information of the conference participants is determined based on the participation level, and based on the location information, the virtual portrait and the body posture are projected into the virtual conference scene.

7. The method for implementing an interactive online conference based on augmented reality according to any one of claims 1 to 6, characterized in that: The method of creating a virtual meeting scene based on a preset augmented reality device includes: Based on the augmented reality device, obtain the meeting topic and the number of participants of the current meeting; Determining the spatial size of the virtual conference scene based on the number of participants; Based on the conference theme, determining the background environment of the virtual conference scene; The virtual conference scene is created according to the space size and the background environment.

8. An interactive online conference implementation device based on augmented reality, characterized in that: include: A scene creation module, used to create a virtual conference scene based on a preset augmented reality device, obtain virtual portraits and body postures of conference participants, and project the virtual portraits and body postures into the virtual conference scene; An action recognition module is used to recognize the hand position and gesture of the speaker based on a preset virtual interaction tool when it is detected that a speaker is speaking and the speaker has an interaction requirement; An interaction determination module, configured to generate a virtual pointer based on the hand position and the gesture action, and determine a target interaction area or interaction element based on the orientation of the virtual pointer; The instruction response module is used to respond to the instruction corresponding to the gesture action in the interactive area or the interactive element to achieve conference interaction.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the augmented reality-based interactive online conference implementation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for implementing an augmented reality-based interactive online conference as described in any one of claims 1 to 7.

Images