AR-based conference interaction methods, devices, equipment, and storage media

By acquiring and processing meeting images and audio through an AR system, and constructing a point cloud structure to locate the sound source, the problem of users not being able to see the screen or hear the sound clearly during the meeting was solved, thus achieving an immersive meeting experience and efficient information exchange.

CN119225518BActive Publication Date: 2026-11-17GEER TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202310796756.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-30
Publication Date
2026-11-17
Estimated Expiration
2043-06-30

AI Technical Summary

Technical Problem

During meetings or presentations, due to differences in venue size and crowd density, users may not be able to clearly see the screen or hear the sound, resulting in a poor meeting experience and affecting learning and communication efficiency.

Method used

By acquiring multiple conference presentation images through an AR system, constructing a point cloud structure of the target speaker, locating the sound source of the presentation, and collecting and processing the presentation audio and the text displayed on the screen, a clear conference interaction can be achieved.

Benefits of technology

It improves the clarity of content access for users, enhances the conference experience, and ensures that users can intuitively and clearly access the text displayed on the screen and the audio content of the presentation, thereby improving learning and communication efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119225518B_ABST
    Figure CN119225518B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of wearable device technology and discloses a meeting interaction method, device, equipment, and storage medium based on an AR system. The method includes: acquiring multiple meeting presentation images when the current operating mode is detected as a meeting listening mode; determining the target speaker and screen-displayed text based on the multiple presentation images, and constructing a point cloud structure of the target speaker; locating the speaker's sound source based on the point cloud structure and determining a sound pickup beam area; collecting the speaker's audio based on the sound pickup beam area and determining the audio; and realizing meeting interaction based on the audio and the screen-displayed text. Through this method, the AR system accurately locates the sound source, ensuring accurate sound pickup, allowing users to intuitively and clearly understand the target speaker's screen-displayed text and speech content, thus enabling immersive meeting participation and improving learning and communication efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of wearable device technology, and in particular to a meeting interaction method, apparatus, device and storage medium based on an AR system. Background Technology

[0002] In modern society, meetings or speeches are an effective way to promote information exchange and knowledge transfer. However, in reality, due to differences in venue size and crowd density, users are sometimes too far from the speaker, making it difficult to see the screen or hear the speaker's voice. This inconveniences users' learning and communication, greatly weakens the effectiveness of meetings or speeches, and reduces users' learning efficiency. Summary of the Invention

[0003] The main objective of this invention is to provide a meeting interaction method, apparatus, device, and storage medium based on an AR system, aiming to solve the technical problem of how to improve the clarity of content accessed by users and enhance their meeting experience when participating in meetings.

[0004] To achieve the above objectives, the present invention provides a meeting interaction method based on an AR system, the meeting interaction method based on an AR system comprising: When the current operating mode is detected to be the meeting listening mode, multiple meeting presentation images are captured; The target speaker and the text displayed on the screen are determined based on multiple conference presentation images, and the point cloud structure of the target speaker is constructed. Based on the point cloud structure, the speech source of the target speaker is located, and the sound pickup beam area is determined; The speech audio of the target speaker is collected based on the pickup beam area to determine the speech audio; Meeting interaction is achieved based on the audio of the speech and the text displayed on the screen.

[0005] Optionally, the step of determining the target speaker and the text displayed on the screen based on multiple conference presentation images, and constructing the point cloud structure of the target speaker, includes: Based on the target object detection model, object detection is performed on each conference presentation image to determine the target speaker and the target feature value of the target speaker in each conference presentation image; The target feature values ​​are used to construct the object and determine the point cloud structure of the target speaker. Text detection is performed on images of presentations at various conferences to determine the text displayed on the screen of the target speaker.

[0006] Optionally, the step of locating the speech source of the target speaker based on the point cloud structure and determining the sound pickup beam region includes: The three-dimensional coordinates of the target speaker are determined based on the point cloud structure. The horizontal and vertical orientations of the speech sound source are determined based on the three-dimensional coordinates. The pickup beam area is determined based on the vertical and horizontal orientations.

[0007] Optionally, the step of acquiring the speech audio of the target speaker based on the pickup beam area and determining the speech audio includes: Directional audio pickup is performed based on the pickup beam area to determine the initial pickup audio; The initial picked-up audio is denoised to obtain denoised picked-up audio; Play the denoised audio to allow the target interactive user to determine the result based on the voiceprint feedback from the denoised audio. When the voiceprint determination result indicates that the speech voiceprint in the denoised audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected and the speech audio is determined.

[0008] Optionally, the step of enabling meeting interaction based on the speech audio and the text displayed on the screen includes: The speech audio is processed to identify the target audio data corresponding to the speech voiceprint of the target speaker; The target audio data is enhanced to obtain enhanced audio data; The enhanced audio data is played, and the screen display text is shown to enable conference interaction.

[0009] Optionally, displaying the text on the screen includes: Enable the focus adjustment mode and send a mode enable command to the target interactive user so that the target interactive user can adjust the focus according to the mode enable command and provide feedback on the adjusted focus. Upon receiving feedback from the target user regarding the target focal length adjustment, the current focal length is adjusted to the target focal length adjustment. The screen text is displayed by adjusting the focal length according to the target.

[0010] Optionally, before acquiring multiple conference presentation images when the current operating mode is detected to be the meeting listening mode, the method further includes: When environmental audio and environmental images are acquired, text conversion is performed based on the environmental audio to determine the environmental text. The environmental images are classified to determine the initial image scene; The current scene is determined based on the environmental text and the initial image scene; When the current scenario is a preset meeting scenario, a mode change reminder is sent to the target user. Upon receiving a mode change instruction from the target interactive user, the current operating mode is adjusted to the meeting and listening mode.

[0011] Furthermore, to achieve the above objectives, the present invention also proposes a conference interaction device based on an AR system, the conference interaction device based on an AR system comprising: The acquisition module is used to acquire multiple conference presentation images when the current operating mode is detected to be the attendee listening mode; A construction module is used to determine the target speaker and the text displayed on the screen based on multiple conference presentation images, and to construct the point cloud structure of the target speaker; The positioning module is used to locate the speech source of the target speaker based on the point cloud structure and determine the sound pickup beam area; The acquisition module is used to acquire the speech audio of the target speaker according to the pickup beam area and determine the speech audio; An interactive module is used to enable conference interaction based on the speech audio and the text displayed on the screen.

[0012] Furthermore, to achieve the above objectives, the present invention also proposes a conference interaction device based on an AR system. The conference interaction device based on an AR system includes: a memory, a processor, and a conference interaction program based on an AR system stored in the memory and executable on the processor. The conference interaction program based on an AR system is configured to implement the conference interaction method based on an AR system as described above.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a conference interaction program based on an AR system, wherein the conference interaction program based on an AR system is executed by a processor to implement the conference interaction method based on an AR system as described above.

[0014] This invention acquires multiple conference presentation images when the current operating mode is detected as a meeting listening mode; determines the target speaker and on-screen text based on these images, and constructs a point cloud structure of the target speaker; locates the speaker's audio source using the point cloud structure, determining a sound pickup beam region; collects the speaker's audio using the sound pickup beam region, determining the audio content; and finally, enables meeting interaction based on the audio and on-screen text. Through this method, the target speaker and on-screen text are determined based on multiple conference presentation images, and a point cloud structure of the target speaker is constructed. This allows for the accurate determination of a sound pickup beam region, which is then used to collect the audio. Finally, meeting interaction is achieved based on the audio and on-screen text. Utilizing an AR system, the accurate location of the sound source ensures accurate sound pickup, allowing users to intuitively and clearly understand the target speaker's on-screen text and audio content, thus enabling immersive meeting participation and improving learning and communication efficiency. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of an AR-based conference interaction device in the hardware operating environment of the embodiment of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the AR-based meeting interaction method of the present invention; Figure 3 This is a schematic diagram of the point cloud structure of an embodiment of the AR-based meeting interaction method of the present invention; Figure 4 This is a flowchart illustrating the second embodiment of the AR-based meeting interaction method of the present invention; Figure 5 This is a schematic diagram of the overall process of an embodiment of the AR-based meeting interaction method of the present invention; Figure 6 This is a structural block diagram of the first embodiment of the AR-based conference interaction device of the present invention.

[0016] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of an AR-based conference interaction device in the hardware operating environment of an embodiment of the present invention.

[0019] like Figure 1 As shown, the AR-based conference interaction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0020] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on AR-based conference interaction devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0021] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a conference interaction program based on an AR system.

[0022] exist Figure 1 In the AR-based conference interaction device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the AR-based conference interaction device of the present invention can be set in the AR-based conference interaction device, and the AR-based conference interaction device calls the AR-based conference interaction program stored in the memory 1005 through the processor 1001 and executes the AR-based conference interaction method provided in the embodiment of the present invention.

[0023] This invention provides a meeting interaction method based on an AR system, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a meeting interaction method based on an AR system according to the present invention.

[0024] The AR-based meeting interaction method includes the following steps: Step S10: When the current operating mode is detected to be the meeting listening mode, acquire multiple meeting presentation images.

[0025] It should be noted that the control processing unit in the AR system, the execution subject of this embodiment, can also be other modules or units that can achieve the same function, and this embodiment does not limit this.

[0026] Understandably, when the AR system is currently operating in the meeting listening mode, it means that the participants are attending a meeting or giving a speech. In the acoustic mode of the meeting listening mode, it is necessary to track and locate the speaker in the meeting, i.e., the target speaker. The sound waves in the space are directionally transmitted and focused through the microphone array to collect the audio of the target speaker's speech. The audio is then processed so that the participants can hear the speaker's voice clearly. In the focus adjustment mode of the meeting listening mode, the text content displayed on the large screen projection device used in the meeting can be displayed through the focus determined by the participants.

[0027] In practice, when the current operating mode is detected as the meeting listening mode, the AR wearable device will collect multiple two-dimensional photos of the target speaker from different angles. These multiple two-dimensional photos of the target speaker from different angles constitute the meeting presentation images.

[0028] It should be noted that, in order to save resources and improve processing efficiency, before data collection, it is first determined whether the current scenario is a preset meeting scenario. Furthermore, before acquiring multiple meeting presentation images when the current operating mode is detected to be the meeting listening mode, the process includes: when environmental audio and environmental images are acquired, performing text conversion on the environmental audio to determine the environmental text; classifying the environmental images into scenarios to determine the initial image scenario; determining the current scenario based on the environmental text and the initial image scenario; when the current scenario is a preset meeting scenario, sending a mode change reminder to the target interactive user; and when receiving the mode change instruction from the target interactive user, adjusting the current operating mode to the meeting listening mode.

[0029] Understandably, once the AR wearable device detects a human wearing it, it immediately activates and collects audio and surrounding images in real time. The audio in the environment is called ambient audio, and the surrounding images are called ambient images. After denoising the collected ambient audio, text conversion is performed to determine the corresponding ambient text. Then, scene recognition is performed based on the ambient images and a scene classification model to determine the initial image scene. The scene classification model is obtained after training on a large number of annotated scene images and a neural network model.

[0030] In the specific implementation, the initial image scene is verified based on the environmental text to further determine the current scene of the AR wearable device and determine whether the current scene is a meeting or speech scene. If so, it means that the current scene is the preset meeting scene. At this time, a mode change reminder is sent to the target interactive user, which refers to the meeting user wearing the AR wearable device.

[0031] It should be noted that the target user determines whether to enable the meeting listening mode based on the mode change reminder. The target user can interact through gestures, voice, or touch screen clicks. When the target user confirms that the meeting listening mode is enabled, a mode change command will be generated. At this time, when the AR system receives the mode change command, it will adjust the current operating mode to the meeting listening mode.

[0032] Step S20: Determine the target speaker and the text displayed on the screen based on multiple conference presentation images, and construct the point cloud structure of the target speaker.

[0033] It should be noted that the AR system utilizes object detection technology to perform object detection on multiple conference presentation images, determining the target speaker and their 2D coordinates in each image. Using these coordinates, the system tracks the target speaker, obtaining images of the presentation from different angles, and ultimately determining the speaker's feature values ​​at those angles. Simultaneously, it identifies the corresponding area on the large screen projection device used in the conference and the text content displayed on it. This text content is the screen display text. Furthermore, it uses the speaker's feature values ​​from different angles to construct an image, determining the speaker's point cloud structure, such as... Figure 3 As shown, Figure 3 This refers to the point cloud structure of the target speaker.

[0034] Understandably, to ensure the accuracy of the point cloud construction process, the step of determining the target speaker and the screen-displayed text based on multiple conference presentation images, and constructing the point cloud structure of the target speaker, further includes: performing object detection on each conference presentation image based on a target object detection model to determine the target speaker and the target feature values ​​of the target speaker in each conference presentation image; constructing objects based on the target feature values ​​to determine the point cloud structure of the target speaker; and performing text detection on each conference presentation image to determine the screen-displayed text of the target speaker.

[0035] In its implementation, the target object detection model is trained using YOLOv5 on images and labels within the conference scene. This model accurately outputs the two-dimensional coordinates of the target speaker and their location. By utilizing the two-dimensional coordinates and confidence scores of the target speaker in various conference presentation images, target speaker tracking and localization are performed, thereby obtaining conference presentation images of the target speaker from different angles and determining the target feature values ​​of the speaker at those angles.

[0036] It should be noted that, through 3D reconstruction technology, the point cloud structure of the target speaker in a 3D coordinate system is constructed using the target feature values ​​of the target speaker from different angles, and the position coordinates and depth of the target speaker in the 3D coordinate system are calculated. Furthermore, text detection is performed on each conference presentation image to determine the corresponding area of ​​the large-screen projection device used in the conference and the text content displayed on the large-screen projection device.

[0037] Step S30: Locate the speech source of the target speaker based on the point cloud structure and determine the sound pickup beam area.

[0038] It should be noted that the position coordinates and depth of the target speaker in the three-dimensional coordinate system are calculated by using the point cloud structure. The position coordinates and depth of the target speaker in the three-dimensional coordinate system are then used to locate the sound source of the target speaker's speech and determine the pickup beam area corresponding to the sound source when the target speaker is speaking.

[0039] Step S40: Collect the speech audio of the target speaker according to the pickup beam area and determine the speech audio.

[0040] It should be noted that beamforming technology is used to collect audio within the pickup beam area through a microphone array, thereby enabling directional transmission and focusing of sound waves within the pickup beam area. The audio of the target speaker is the audio of the speech.

[0041] Step S50: Implement meeting interaction based on the speech audio and the text displayed on the screen.

[0042] It should be noted that, in order to improve the interactive effect, the step of realizing conference interaction based on the speech audio and the screen display text further includes: performing signal processing on the speech audio to identify target audio data corresponding to the speech voiceprint of the target speaker; performing signal enhancement on the target audio data to obtain enhanced audio data; playing the enhanced audio data and displaying the screen display text to realize conference interaction.

[0043] Understandably, speech signal processing technology is used to process the speech audio, identify the target audio data corresponding to the speaker's voiceprint, increase the volume of the target audio data, and decrease the volume of other audio data in the speech audio, thereby obtaining enhanced audio data. The enhanced audio data is then played through the speaker on the AR wearable device.

[0044] In specific implementation, to ensure that the target interactive user can clearly see the text content displayed on the large screen projection device, the display of the screen text further includes: activating the focus adjustment mode and sending a mode activation command to the target interactive user, so that the target interactive user adjusts the focus according to the mode activation command and provides feedback on the target adjusted focus; upon receiving the target adjusted focus feedback from the target interactive user, adjusting the current focus to the target adjusted focus; and displaying the screen text according to the target adjusted focus.

[0045] It should be noted that in the focus adjustment mode of the meeting listening mode, the AR system can display the text content on the large screen projection device used in the meeting based on the focus adjusted by the participating user. The specific process is as follows: When the AR system detects text displayed on the screen, it activates the focus adjustment mode and sends a mode activation command to the target user. This command can be displayed as text to inform the user, or it can play audio messages such as "Mode Activation Command" or "Focus Adjustment Mode is Activated" to inform the user that the focus adjustment mode is enabled. The user can interact via gestures, voice, or touchscreen clicks, or by pushing physical controls on the AR wearable device to adjust the focus, thus confirming the desired adjustment focus. At this point, the focus adjustment mode is deactivated, and the AR system displays the text on the screen based on the adjusted focus.

[0046] It can be explained that when the interaction method involves the target user adjusting the focus by pushing a physical control on the AR wearable device, such as a scroll wheel, the AR system will display text on the screen in real time according to the adjusted focus while the target user touches the scroll wheel. When the target user determines that a certain focus is suitable, they can click the confirmation control to confirm that the current focus is selected as the target adjustment focus, at which point the AR system will turn off the focus adjustment mode.

[0047] This embodiment acquires multiple conference presentation images when the current operating mode is detected as a meeting listening mode; determines the target speaker and screen-displayed text based on these images, and constructs a point cloud structure of the target speaker; locates the speaker's audio source using the point cloud structure, determining a sound pickup beam region; collects the speaker's audio using the sound pickup beam region, determining the audio content; and finally, enables meeting interaction based on the audio and screen-displayed text. Through this method, the target speaker and screen-displayed text are determined based on multiple conference presentation images, and a point cloud structure of the target speaker is constructed. This allows for the accurate determination of a sound pickup beam region, which is then used to collect the audio. Finally, meeting interaction is achieved based on the audio and screen-displayed text. The AR system, through accurate sound source localization, ensures accurate sound pickup, allowing users to intuitively and clearly understand the target speaker's screen-displayed text and speech content, thus enabling immersive meeting participation and improving learning and communication efficiency.

[0048] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a meeting interaction method based on an AR system according to the present invention.

[0049] Based on the first embodiment described above, step S30 in the AR system-based meeting interaction method of this embodiment includes: Step S31: Determine the three-dimensional coordinates of the target speaker based on the point cloud structure.

[0050] It should be noted that after determining the point cloud structure of the target speaker, the three-dimensional coordinates are calculated using the point cloud structure to determine the position coordinates and depth of the target speaker in the three-dimensional coordinate system. The position coordinates and depth in the three-dimensional coordinate system are the three-dimensional coordinates of the target speaker.

[0051] Step S32: Determine the horizontal and vertical orientations of the speech sound source based on the three-dimensional coordinates.

[0052] It should be noted that the AR system identifies the horizontal and vertical orientation of the speech sound source in space based on the three-dimensional coordinates of the target speaker and beamforming technology. The speech sound source is the sound source corresponding to the target speaker when he / she is speaking.

[0053] Step S33: Determine the pickup beam area based on the vertical and horizontal orientations.

[0054] It should be noted that the vertical and horizontal orientations are used to form the sound pickup directionality, thereby determining the sound pickup beam area during audio acquisition.

[0055] Understandably, to ensure that the speech audio corresponds to the content spoken by the target speaker, the step of collecting the speech audio of the target speaker based on the pickup beam area and determining the speech audio further includes: performing directional audio pickup based on the pickup beam area to determine the initial pickup audio; performing audio denoising on the initial pickup audio to obtain denoised pickup audio; playing the denoised pickup audio so that the target interactive user can provide feedback on the voiceprint determination result based on the denoised pickup audio; and when the voiceprint determination result indicates that the speech voiceprint in the denoised pickup audio is the speech voiceprint of the target speaker, collecting the speech audio of the target speaker and determining the speech audio.

[0056] It should be understood that beamforming technology is used to collect audio within the pickup beam area through a microphone array, thereby enabling directional transmission and focusing of sound waves within the pickup beam area. The collected audio is the initial pickup audio.

[0057] In practical implementation, to ensure the accuracy of audio pickup, the voiceprint of the initial picked-up audio needs to be determined. At this time, the AR system performs audio denoising on the initial picked-up audio, extracts the audio corresponding to the human voice part in the denoised initial picked-up audio, and the audio corresponding to the human voice part in the denoised initial picked-up audio is the denoised picked-up audio. The denoised picked-up audio is played so that the target interactive user can determine whether the denoised picked-up audio is the speech audio of the target speaker, and thus feedback the corresponding voiceprint determination result. When the denoised video audio is the speech audio of the target speaker, the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker; when the denoised video audio is not the speech audio of the target speaker, the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is not the speech voiceprint of the target speaker.

[0058] It should be noted that when the voiceprint determination result shows that the speech voiceprint in the denoised audio is the speech voiceprint of the target speaker, the AR system locks the voiceprint and uses the locked speech voiceprint to collect the speech audio of the target speaker to determine the speech audio.

[0059] It is understandable that, such as Figure 5As shown, the AR system detects ambient audio and images in real time to determine if the current scene is a preset meeting scenario. If it is, a mode change notification is sent to the target user. When the user confirms the activation of the meeting listening mode, multiple conference presentation images are acquired to track and locate the speaker, and a point cloud structure is constructed. Based on the point cloud structure, the speaker's 3D coordinates are calculated, and a sound pickup beam area is determined. Initial audio is collected from this area. If the voiceprint in the initial audio is identified as the speaker's, the speaker's audio is further collected, and the volume is increased. Simultaneously, the AR system activates a focus adjustment mode, adjusting the focus to display text on the screen.

[0060] In this embodiment, the three-dimensional coordinates of the target speaker are determined based on the point cloud structure; the horizontal and vertical orientations of the speech sound source are determined based on the three-dimensional coordinates; and the sound pickup beam area is determined based on the vertical and horizontal orientations. Through this method, accurate localization of the speech sound source can be achieved, ensuring the clarity and accuracy of subsequent interactive processes.

[0061] In addition, refer to Figure 6 The present invention also proposes a conference interaction device based on an AR system, the conference interaction device based on an AR system comprising: The acquisition module 10 is used to acquire multiple conference presentation images when the current operating mode is detected to be the conference listening mode.

[0062] The construction module 20 is used to determine the target speaker and the text displayed on the screen based on multiple conference presentation images, and to construct the point cloud structure of the target speaker.

[0063] The positioning module 30 is used to locate the speech source of the target speaker based on the point cloud structure and determine the sound pickup beam area.

[0064] The acquisition module 40 is used to acquire the speech audio of the target speaker according to the pickup beam area and determine the speech audio.

[0065] The interaction module 50 is used to enable conference interaction based on the speech audio and the text displayed on the screen.

[0066] This embodiment acquires multiple conference presentation images when the current operating mode is detected as a meeting listening mode; determines the target speaker and screen-displayed text based on these images, and constructs a point cloud structure of the target speaker; locates the speaker's audio source using the point cloud structure, determining a sound pickup beam region; collects the speaker's audio using the sound pickup beam region, determining the audio content; and finally, enables meeting interaction based on the audio and screen-displayed text. Through this method, the target speaker and screen-displayed text are determined based on multiple conference presentation images, and a point cloud structure of the target speaker is constructed. This allows for the accurate determination of a sound pickup beam region, which is then used to collect the audio. Finally, meeting interaction is achieved based on the audio and screen-displayed text. The AR system, through accurate sound source localization, ensures accurate sound pickup, allowing users to intuitively and clearly understand the target speaker's screen-displayed text and speech content, thus enabling immersive meeting participation and improving learning and communication efficiency.

[0067] In one embodiment, the construction module 20 is further configured to perform object detection on each conference presentation image according to the target object detection model, and determine the target speaker and the target feature value of the target speaker under each conference presentation image; The target feature values ​​are used to construct the object and determine the point cloud structure of the target speaker. Text detection is performed on images of presentations at various conferences to determine the text displayed on the screen of the target speaker.

[0068] In one embodiment, the positioning module 30 is further configured to determine the three-dimensional coordinates of the target speaker based on the point cloud structure; The horizontal and vertical orientations of the speech sound source are determined based on the three-dimensional coordinates. The pickup beam area is determined based on the vertical and horizontal orientations.

[0069] In one embodiment, the acquisition module 40 is further configured to perform directional audio pickup based on the pickup beam area to determine the initial pickup audio; The initial picked-up audio is denoised to obtain denoised picked-up audio; Play the denoised audio to allow the target interactive user to determine the result based on the voiceprint feedback from the denoised audio. When the voiceprint determination result indicates that the speech voiceprint in the denoised audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected and the speech audio is determined.

[0070] In one embodiment, the interaction module 50 is further configured to perform signal processing on the speech audio to identify target audio data corresponding to the speech voiceprint of the target speaker; The target audio data is enhanced to obtain enhanced audio data; The enhanced audio data is played, and the screen display text is shown to enable conference interaction.

[0071] In one embodiment, the interaction module 50 is further configured to enable the focus adjustment mode and send a mode enable command to the target interactive user, so that the target interactive user adjusts the focus according to the mode enable command and provides feedback on the target adjusted focus. Upon receiving feedback from the target user regarding the target focal length adjustment, the current focal length is adjusted to the target focal length adjustment. The screen text is displayed by adjusting the focal length according to the target.

[0072] In one embodiment, the acquisition module 10 is further configured to perform text conversion based on the environmental audio when environmental audio and environmental images are acquired, and determine environmental text; The environmental images are classified to determine the initial image scene; The current scene is determined based on the environmental text and the initial image scene; When the current scenario is a preset meeting scenario, a mode change reminder is sent to the target user. Upon receiving a mode change instruction from the target interactive user, the current operating mode is adjusted to the meeting and listening mode.

[0073] Since this device adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0074] Furthermore, this embodiment of the invention also proposes a storage medium storing a conference interaction program based on an AR system. When the conference interaction program based on an AR system is executed by a processor, it implements the steps of the conference interaction method based on an AR system as described above.

[0075] Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0076] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0077] In addition, for technical details not described in detail in this embodiment, please refer to the AR-based meeting interaction method provided in any embodiment of the present invention, which will not be repeated here.

[0078] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0079] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0081] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A meeting interaction method based on an AR system, characterized in that, The AR-based meeting interaction method includes: When environmental audio and environmental images are acquired, text conversion is performed based on the environmental audio to determine the environmental text; scene classification is performed on the environmental images to determine the initial image scene. The current scene is determined based on the environmental text and the initial image scene. When the current scene is a preset meeting scene, a mode change reminder is sent to the target interactive user. Upon receiving a mode change instruction from the target interactive user, the current operating mode is adjusted to the meeting and listening mode; When the current operating mode is detected to be the meeting listening mode, multiple meeting presentation images are captured; The target speaker and the text displayed on the screen are determined based on multiple conference presentation images, and the point cloud structure of the target speaker is constructed. Based on the point cloud structure, the speech source of the target speaker is located, and the sound pickup beam area is determined; Directional audio is picked up according to the pickup beam area to determine the initial picked-up audio. The initial picked-up audio is then denoised to obtain denoised picked-up audio. The denoised picked-up audio is played so that the target interactive user can receive a voiceprint confirmation result based on the denoised picked-up audio. When the voiceprint determination result indicates that the speech voiceprint in the denoised audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected and the speech audio is determined. Meeting interaction is achieved based on the speech audio and the screen-displayed text, including: displaying the screen-displayed text; displaying the screen-displayed text includes: Enable the focus adjustment mode and send a mode enable command to the target interactive user so that the target interactive user can adjust the focus according to the mode enable command and provide feedback on the adjusted focus. Upon receiving feedback from the target user regarding the target focal length adjustment, the current focal length is adjusted to the target focal length adjustment; the text displayed on the screen is then shown according to the target focal length adjustment.

2. The meeting interaction method based on an AR system as described in claim 1, characterized in that, The step of determining the target speaker and the text displayed on the screen based on multiple conference presentation images, and constructing the point cloud structure of the target speaker, includes: Based on the target object detection model, object detection is performed on each conference presentation image to determine the target speaker and the target feature value of the target speaker in each conference presentation image; The target feature values ​​are used to construct the object and determine the point cloud structure of the target speaker; Text detection is performed on images of presentations at various conferences to determine the text displayed on the screen of the target speaker.

3. The meeting interaction method based on an AR system as described in claim 1, characterized in that, The step of locating the speech source of the target speaker based on the point cloud structure and determining the sound pickup beam region includes: The three-dimensional coordinates of the target speaker are determined based on the point cloud structure. The horizontal and vertical orientations of the speech sound source are determined based on the three-dimensional coordinates. The pickup beam area is determined based on the vertical and horizontal orientations.

4. The meeting interaction method based on an AR system as described in claim 1, characterized in that, The method of enabling meeting interaction based on the speech audio and the text displayed on the screen includes: The speech audio is processed to identify the target audio data corresponding to the speech voiceprint of the target speaker; The target audio data is enhanced to obtain enhanced audio data; The enhanced audio data is played, and the screen display text is shown to enable conference interaction.

5. A conference interaction device based on an AR system, characterized in that, The AR-based conference interaction device includes: The acquisition module is used to: convert environmental audio into text to determine environmental text when environmental audio and environmental images are acquired; classify environmental images into scenes to determine an initial image scene; determine the current scene based on the environmental text and the initial image scene; send a mode change reminder to the target interactive user when the current scene is a preset meeting scene; adjust the current operating mode to the meeting listening mode when the mode change instruction is received from the target interactive user; and acquire multiple meeting presentation images when the current operating mode is detected to be the meeting listening mode. A construction module is used to determine the target speaker and the text displayed on the screen based on multiple conference presentation images, and to construct the point cloud structure of the target speaker; The positioning module is used to locate the speech source of the target speaker based on the point cloud structure and determine the sound pickup beam area; The acquisition module is used to perform directional audio pickup based on the pickup beam area, determine the initial pickup audio, perform audio denoising on the initial pickup audio to obtain denoised pickup audio, and play the denoised pickup audio so that the target interactive user can provide feedback voiceprint determination results based on the denoised pickup audio. When the voiceprint determination result indicates that the speech voiceprint in the denoised audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected and the speech audio is determined. An interaction module is used to enable meeting interaction based on the speech audio and the screen-displayed text, including: displaying the screen-displayed text; the display of the screen-displayed text includes: activating a focus adjustment mode and sending a mode activation command to a target interactive user, so that the target interactive user adjusts the focus according to the mode activation command and provides feedback on the target adjusted focus; upon receiving the target adjusted focus feedback from the target interactive user, adjusting the current focus to the target adjusted focus; and displaying the screen-displayed text according to the target adjusted focus.

6. A conference interaction device based on an AR system, characterized in that, The device includes: a memory, a processor, and an AR-based conference interaction program stored in the memory and executable on the processor, the AR-based conference interaction program being configured to implement the AR-based conference interaction method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a conference interaction program based on an AR system, which, when executed by a processor, implements the conference interaction method based on an AR system as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Remote conference system

    CN218124814U