Conference interaction method, device and equipment based on AR (Augmented Reality) system and storage medium

Abstract

The invention belongs to the technical field of wearable equipment, and discloses a conference interaction method and device based on an AR system, equipment and a storage medium. The method comprises the steps of obtaining a plurality of conference speech images when it is detected that a current operation mode is a conference participation listening mode; determining a target speaker and a screen display text according to the plurality of conference speech images, and constructing a point cloud structure of the target speaker; positioning a speech sound source of the target speaker according to the point cloud structure, and determining a pickup beam area; acquiring speech audio of the target speaker according to the pickup beam area, and determining the speech audio; and realizing conference interaction according to the speech audio and the screen display text. Through the above mode, the AR system is utilized to accurately position the sound source, the pickup accuracy is ensured, and a user can visually and clearly know the screen display text and speech voice content of the target speaker, so that the user can participate in the conference in an immersive manner, and the learning and communication efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of wearable devices, and in particular to a conference interaction method, device, equipment and storage medium based on an AR system. Background Art

[0002] In modern society, meetings or speeches are an effective way to promote information exchange and knowledge transfer. However, in reality, due to the size of the venue and the density of the crowd, users are sometimes far away from the speaker, resulting in unclear viewing of the speech screen and unclear hearing of the speaker. This brings inconvenience to the user's learning and communication, greatly weakens the effect of the meeting or speech, and reduces the user's learning efficiency. Summary of the invention

[0003] The main purpose of the present invention is to provide a conference interaction method, device, equipment and storage medium based on an AR system, aiming to solve the technical problem of how to improve the clarity of content obtained by users when attending a meeting and enhance the user's conference experience.

[0004] To achieve the above object, the present invention provides a conference interaction method based on an AR system, and the conference interaction method based on an AR system includes: When it is detected that the current operation mode is a conference listening mode, a plurality of conference speech images are acquired; Determine a target speaker and screen display text according to a plurality of conference speech images, and construct a point cloud structure of the target speaker; Positioning the speech sound source of the target speaker according to the point cloud structure, and determining the pickup beam area; The speech audio of the target speaker is collected according to the sound pickup beam area to determine the speech audio; Meeting interaction is achieved based on the speech audio and the screen displayed text.

[0005] Optionally, determining a target speaker and screen display text according to a plurality of conference speech images, and constructing a point cloud structure of the target speaker includes: Perform object detection on each conference speech image according to the target object detection model to determine the target speaker and the target feature value of the target speaker under each conference speech image; Performing object construction according to the target feature value to determine the point cloud structure of the target speaker; Text detection is performed on each conference speech image to determine the screen display text of the target speaker.

[0006] Optionally, locating the speech sound source of the target speaker according to the point cloud structure and determining the pickup beam area includes: Determine the three-dimensional coordinates of the target speaker according to the point cloud structure; Determine the horizontal and vertical positions of the speech source according to the three-dimensional coordinates; A sound pickup beam area is determined according to the vertical orientation and the horizontal orientation.

[0007] Optionally, collecting the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio includes: Directional audio pickup is performed according to the pickup beam area to determine initial pickup audio; Performing audio denoising on the initially picked-up audio to obtain denoised picked-up audio; Play the denoised picked-up audio so that the target interactive user can determine the result based on the denoised picked-up audio feedback voiceprint; When the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected to determine the speech audio.

[0008] Optionally, implementing conference interaction according to the speech audio and the screen displayed text includes: Performing signal processing on the speech audio to identify target audio data corresponding to the speech voiceprint of the target speaker; Performing signal enhancement on the target audio data to obtain enhanced audio data; The enhanced audio data is played and the screen display text is displayed to achieve conference interaction.

[0009] Optionally, displaying the screen display text includes: Turning on a focus adjustment mode, and sending a mode turn-on instruction to a target interactive user, so that the target interactive user adjusts the focus according to the mode turn-on instruction, and feeds back the target adjusted focus; Upon receiving the target adjustment focal length fed back by the target interactive user, adjusting the current focal length to the target adjustment focal length; The screen display text is displayed by adjusting the focal length according to the target.

[0010] Optionally, when it is detected that the current operation mode is the conference listening mode, before acquiring the plurality of conference speech images, the method further includes: When the ambient audio and the ambient image are collected, text conversion is performed according to the ambient audio to determine the ambient text; Performing scene classification on the environment image to determine an initial image scene; Determine the current scene according to the environment text and the initial image scene; When the current scene is a preset conference scene, sending a mode change reminder to the target interactive user; When receiving the mode change instruction fed back by the target interactive user, the current operation mode is adjusted to the conference attending and listening mode.

[0011] In addition, to achieve the above-mentioned purpose, the present invention further proposes a conference interaction device based on an AR system, and the conference interaction device based on an AR system includes: The acquisition module is used to acquire multiple conference speech images when detecting that the current operation mode is the conference listening mode; A construction module, used to determine a target speaker and screen display text according to a plurality of conference speech images, and to construct a point cloud structure of the target speaker; A positioning module, used to locate the speech sound source of the target speaker according to the point cloud structure and determine the pickup beam area; A collection module, used for collecting the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio; An interaction module is used to realize conference interaction according to the speech audio and the screen display text.

[0012] In addition, to achieve the above-mentioned purpose, the present invention also proposes a conference interaction device based on an AR system, wherein the conference interaction device based on an AR system comprises: a memory, a processor, and a conference interaction program based on an AR system stored in the memory and executable on the processor, wherein the conference interaction program based on the AR system is configured to implement the conference interaction method based on the AR system as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a conference interaction program based on an AR system is stored. When the conference interaction program based on the AR system is executed by a processor, the conference interaction method based on the AR system as described above is implemented.

[0014] The present invention acquires a plurality of conference speech images when it is detected that the current operation mode is the conference listening mode; determines the target speaker and the text displayed on the screen according to the plurality of conference speech images, and constructs the point cloud structure of the target speaker; locates the speech sound source of the target speaker according to the point cloud structure, and determines the pickup beam area; collects the speech audio of the target speaker according to the pickup beam area, and determines the speech audio; and realizes conference interaction according to the speech audio and the text displayed on the screen. Through the above-mentioned method, the target speaker and the text displayed on the screen are determined based on a plurality of conference speech images, and the point cloud structure of the target speaker is further constructed, so that the accurate pickup beam area is determined based on the point cloud structure, and the speech audio is collected using the pickup beam area, and finally the conference interaction is realized according to the speech audio and the text displayed on the screen, and the AR system is used to accurately locate the sound source to ensure the accuracy of the sound pickup, so that the user can intuitively and clearly know the screen display text and speech voice content of the target speaker, so that the user can participate in the meeting immersively and improve the efficiency of learning and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural diagram of a conference interaction device based on an AR system in a hardware operating environment involved in an embodiment of the present invention; Figure 2 It is a flowchart of the first embodiment of the conference interaction method based on the AR system of the present invention; Figure 3 A schematic diagram of a point cloud structure of an embodiment of a conference interaction method based on an AR system of the present invention; Figure 4 It is a flowchart of the second embodiment of the conference interaction method based on the AR system of the present invention; Figure 5 It is a schematic diagram of the overall process of an embodiment of a conference interaction method based on an AR system of the present invention; Figure 6 This is a structural block diagram of the first embodiment of the conference interaction device based on the AR system of the present invention.

[0016] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0018] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a conference interaction device based on an AR system in the hardware operating environment involved in the embodiment of the present invention.

[0019] like Figure 1 As shown, the conference interaction device based on the AR system may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0020] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the conference interaction device based on the AR system, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0021] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a conference interaction program based on an AR system.

[0022] exist Figure 1 In the conference interaction device based on the AR system shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the conference interaction device based on the AR system of the present invention can be set in the conference interaction device based on the AR system, and the conference interaction device based on the AR system calls the conference interaction program based on the AR system stored in the memory 1005 through the processor 1001, and executes the conference interaction method based on the AR system provided in the embodiment of the present invention.

[0023] The embodiment of the present invention provides a conference interaction method based on an AR system, referring to Figure 2 , Figure 2 This is a flow chart of a first embodiment of a conference interaction method based on an AR system according to the present invention.

[0024] The conference interaction method based on the AR system includes the following steps: Step S10: when it is detected that the current operation mode is the conference listening mode, a plurality of conference speech images are acquired.

[0025] It should be noted that the control processing unit in the AR system, which is the execution subject of this embodiment, may also be other modules or units that can achieve the same function, and this embodiment is not limited to this.

[0026] It can be understood that when the current operating mode of the AR system is the conference listening mode, it means that the participating users are attending a meeting or giving a speech. At this time, in the acoustic mode of the conference listening mode, it is necessary to track and locate the speaker in the meeting, that is, the target speaker, and collect sound waves in the space through the microphone array to achieve the collection of the target speaker's speech audio, and process the speech audio so that the participating users can clearly hear the speaker's voice. In the focus adjustment mode of the conference listening mode, the text content displayed on the large-screen projection device used in the meeting can be displayed through the focal length determined by the participant user after adjustment.

[0027] In a specific implementation, when it is detected that the current operating mode is the conference listening mode, the AR wearable device will collect multiple two-dimensional photos of the target speaker taken at different angles. The multiple two-dimensional photos of the target speaker taken at different angles are the conference speech images.

[0028] It should be noted that in order to save resources and improve processing efficiency, before collecting data, it will be determined whether the current scene is a preset meeting scene. Furthermore, when it is detected that the current operating mode is a meeting listening mode, before acquiring multiple conference speech images, it also includes: when the ambient audio and ambient images are collected, text conversion is performed according to the ambient audio to determine the ambient text; scene classification is performed on the ambient image to determine the initial image scene; the current scene is determined according to the ambient text and the initial image scene; when the current scene is a preset meeting scene, a mode change reminder is sent to the target interactive user; when the mode change instruction feedback from the target interactive user is received, the current operating mode is adjusted to the meeting listening mode.

[0029] It can be understood that after the AR wearable device detects that someone is wearing it, it starts immediately and collects the audio and surrounding images in the environment in real time. The audio in the environment is the ambient audio, and the surrounding images are the ambient images. The collected ambient audio is denoised and then converted into text to determine the ambient text corresponding to the ambient audio. Scene recognition is performed based on the ambient image and scene classification model to determine the initial image scene. The scene classification model is obtained after training based on a large number of annotated scene images and a neural network model.

[0030] In the specific implementation, the initial image scene is verified based on the environmental text to further determine the current scene of the AR wearable device, and judge whether the current scene is a meeting or lecture scene. If so, it means that the current scene is a preset meeting scene. At this time, a mode change reminder is sent to the target interactive user. The target interactive user refers to the meeting user wearing the AR wearable device.

[0031] It should be noted that the target interactive user determines whether to turn on the meeting listening mode based on the mode change reminder. The target interactive user can interact through gestures, voice or touch screen clicks. When the target interactive user determines to turn on the meeting listening mode, a mode change instruction will be generated. At this time, when the AR system receives the mode change instruction, it adjusts the current operating mode to the meeting listening mode.

[0032] Step S20: determining a target speaker and screen display text according to a plurality of conference speech images, and constructing a point cloud structure of the target speaker.

[0033] It should be noted that the AR system uses target detection technology to perform object detection on multiple conference speech images, determine the target speaker under each conference speech image and the two-dimensional coordinates of the target speaker under each conference speech image, and use the two-dimensional coordinates of the target speaker under each conference speech image to track the target speaker, thereby obtaining the conference speech images of the target speaker at different angles, and finally determining the characteristic values ​​of the target speaker at different angles. At the same time, determine the corresponding area of ​​the large-screen projection device used in the meeting and the text content displayed on the large-screen projection device. The text content displayed on the large-screen projection device is the screen display text, and use the characteristic values ​​of the target speaker at different angles to construct an image and determine the point cloud structure of the target speaker, such as Figure 3 As shown, Figure 3 This is the point cloud structure of the target speaker.

[0034] It can be understood that in order to ensure the accuracy of the point cloud construction process, further, the target speaker and the screen display text are determined based on multiple conference speech images, and the point cloud structure of the target speaker is constructed, including: performing object detection on each conference speech image according to the target object detection model, and determining the target speaker and the target feature value of the target speaker under each conference speech image; performing object construction according to the target feature value to determine the point cloud structure of the target speaker; performing text detection on each conference speech image to determine the screen display text of the target speaker.

[0035] In the specific implementation, the target object detection model is a model obtained by training YOLOv5 on images and labels in the conference scene. The target object detection model can accurately output the two-dimensional coordinates of the target speaker and his position. The target speaker is tracked and located using the two-dimensional coordinates and confidence scores of the target speaker in each conference speech image, so as to obtain the conference speech images of the target speaker at different angles, thereby determining the target feature values ​​of the target speaker at different angles.

[0036] It should be noted that, through the three-dimensional reconstruction technology, the target feature values ​​of the target speaker at different angles are used to construct the point cloud structure of the target speaker in the three-dimensional coordinate system, and the position coordinates and depth of the target speaker in the three-dimensional coordinate system are calculated. Text detection is performed on each conference speech image to determine the corresponding area of ​​the large-screen projection device used in the meeting and the text content displayed on the large-screen projection device.

[0037] Step S30: locating the speech sound source of the target speaker according to the point cloud structure, and determining the pickup beam area.

[0038] It should be noted that the position coordinates and depth of the target speaker in the three-dimensional coordinate system are calculated through the point cloud structure, and the position coordinates and depth of the target speaker in the three-dimensional coordinate system are used to locate the target speaker's speech sound source, and determine the pickup beam area corresponding to the sound source when the target speaker is speaking.

[0039] Step S40: collecting the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio.

[0040] It should be noted that, by using beamforming technology, the audio in the pickup beam area is collected through a microphone array, thereby achieving directional transmission and focusing of sound waves in the pickup beam area, and the collected speech audio of the target speaker is the speech audio.

[0041] Step S50: realizing conference interaction according to the speech audio and the screen display text.

[0042] It should be noted that in order to improve the interaction effect, further, the conference interaction is realized based on the speech audio and the screen display text, including: performing signal processing on the speech audio to identify the target audio data corresponding to the speech voiceprint of the target speaker; performing signal enhancement on the target audio data to obtain enhanced audio data; playing the enhanced audio data and displaying the screen display text to realize conference interaction.

[0043] It can be understood that the speech audio is processed using voice signal processing technology to identify the target audio data corresponding to the target speaker's speech voiceprint, increase the sound volume of the target audio data, and reduce the sound volume of other audio data in the speech audio except the target audio data, thereby obtaining enhanced audio data, and playing the enhanced audio data using the speakers on the AR wearable device.

[0044] In a specific implementation, in order to ensure that the target interactive user can clearly see the text content displayed on the large-screen projection device, further, the display of the screen display text includes: turning on the focus adjustment mode, and sending a mode turn-on instruction to the target interactive user, so that the target interactive user adjusts the focus according to the mode turn-on instruction, and feeds back the target adjustment focus; when receiving the target adjustment focus fed back by the target interactive user, adjusting the current focus to the target adjustment focus; and displaying the screen display text according to the target adjustment focus.

[0045] It should be noted that in the focus adjustment mode of the conference listening mode, the AR system can display the text content displayed on the large-screen projection device used in the conference through the focus determined by the conference user after adjustment. The specific process is: when the AR system detects that the screen displays text, the AR system turns on the focus adjustment mode and sends a mode-on command to the target interactive user. The mode-on command can be displayed in text to inform the target interactive user, or the relevant voice content of "mode-on command" or "focus adjustment mode is on" can be played to let the target interactive user know that the focus adjustment mode is on. The target interactive user can interact through gestures, voice or touch screen clicks, or adjust the focus by pushing the physical controls on the AR wearable device, thereby feeding back the determined target adjustment focus. At this time, the focus adjustment mode is turned off, and the AR system displays the text displayed on the screen according to the target adjustment focus.

[0046] It can be explained that when the interaction method is that the target interactive user adjusts the focal length by pushing the physical control on the AR wearable device, such as the scroll wheel, while the target interactive user touches the scroll wheel, the AR system will display the text on the screen in real time according to the adjusted focal length. When the target interactive user determines that a certain focal length is suitable, he clicks the confirmation control to confirm the selection of the current focal length as the target adjustment focal length. At this time, the AR system closes the focal length adjustment mode.

[0047] This embodiment acquires multiple conference speech images when it is detected that the current operation mode is the conference listening mode; determines the target speaker and the text displayed on the screen based on the multiple conference speech images, and constructs the point cloud structure of the target speaker; locates the speech sound source of the target speaker based on the point cloud structure, and determines the pickup beam area; collects the speech audio of the target speaker based on the pickup beam area to determine the speech audio; and realizes conference interaction based on the speech audio and the screen text. Through the above method, the target speaker and the screen text are determined based on multiple conference speech images, and the point cloud structure of the target speaker is further constructed, so that the accurate pickup beam area is determined based on the point cloud structure, and the speech audio is collected using the pickup beam area, and finally the conference interaction is realized based on the speech audio and the screen text. The AR system is used to accurately locate the sound source to ensure the accuracy of the sound pickup, so that the user can intuitively and clearly know the screen text and speech voice content of the target speaker, so that the user can participate in the meeting immersively and improve the efficiency of learning and communication.

[0048] refer to Figure 4 , Figure 4 This is a flow chart of a second embodiment of a conference interaction method based on an AR system according to the present invention.

[0049] Based on the above first embodiment, step S30 in the conference interaction method based on the AR system in this embodiment includes: Step S31: determining the three-dimensional coordinates of the target speaker according to the point cloud structure.

[0050] It should be noted that after determining the point cloud structure of the target speaker, the three-dimensional coordinates are calculated using the point cloud structure to determine the position coordinates and depth of the target speaker in the three-dimensional coordinate system. The position coordinates and depth in the three-dimensional coordinate system are the three-dimensional coordinates of the target speaker.

[0051] Step S32: Determine the horizontal and vertical positions of the speech source according to the three-dimensional coordinates.

[0052] It should be noted that the AR system identifies the horizontal and vertical positions of the speech sound source in space based on the three-dimensional coordinates of the target speaker and beamforming technology. The speech sound source is the sound source corresponding to the target speaker when he speaks.

[0053] Step S33: determining a sound pickup beam area according to the vertical orientation and the horizontal orientation.

[0054] It should be noted that the sound pickup directivity is formed by utilizing the vertical orientation and the horizontal orientation, so as to determine the sound pickup beam area during audio collection.

[0055] It can be understood that in order to ensure that the speech audio is the content of the speech of the target speaker, further, the speech audio of the target speaker is collected according to the pickup beam area to determine the speech audio, including: performing directional audio pickup according to the pickup beam area to determine the initial picked-up audio; performing audio denoising on the initial picked-up audio to obtain denoised picked-up audio; playing the denoised picked-up audio so that the target interactive user can feedback the voiceprint determination result according to the denoised picked-up audio; when the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected to determine the speech audio.

[0056] It should be understood that, by using beamforming technology, the audio in the sound pickup beam area is collected through a microphone array, thereby achieving directional transmission and focusing of sound waves in the sound pickup beam area, and the collected audio is the initial picked-up audio.

[0057] In the specific implementation, in order to ensure the accuracy of the picked up audio, the voiceprint of the initial picked up audio needs to be determined. At this time, the AR system performs audio denoising on the initial picked up audio, extracts the audio corresponding to the vocal part in the denoised initial picked up audio, and the audio corresponding to the vocal part in the denoised initial picked up audio is the denoised picked up audio. The denoised picked up audio is played to enable the target interactive user to determine whether the denoised picked up audio is the speech audio of the target speaker, thereby feeding back the corresponding voiceprint determination result. When the denoised video audio is the speech audio of the target speaker, the voiceprint determination result is that the speech voiceprint in the denoised picked up audio is the speech voiceprint of the target speaker; when the denoised video audio is not the speech audio of the target speaker, the voiceprint determination result is that the speech voiceprint in the denoised picked up audio is not the speech voiceprint of the target speaker.

[0058] It should be noted that when the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker, the AR system locks the voiceprint, uses the locked speech voiceprint to collect the speech audio of the target speaker, and determines the speech audio.

[0059] It is understandable that if Figure 5As shown, the AR system detects the ambient audio and ambient image in real time, determines whether the current scene is the preset conference scene, and sends a mode change reminder to the target interactive user when the current scene is the preset conference scene. When the target interactive user determines to turn on the conference listening mode, multiple conference speech images are obtained to track and locate the target speaker, and the point cloud structure of the target speaker is constructed. The three-dimensional coordinates of the target speaker are calculated based on the point cloud structure, and the pickup beam area is determined based on the three-dimensional coordinates. The initial pickup audio in the pickup beam area is collected. When the voiceprint in the initial pickup audio is determined to be the target speaker's voiceprint, the target speaker's speech audio is further collected, and the volume of the target speaker's voice in the speech audio is increased. At the same time, the AR system turns on the focus adjustment mode, and displays the screen display text based on the target adjustment focus mode determined by the focus adjustment mode.

[0060] In this embodiment, the three-dimensional coordinates of the target speaker are determined according to the point cloud structure; the horizontal and vertical directions of the speech sound source are determined according to the three-dimensional coordinates; and the pickup beam area is determined according to the vertical and horizontal directions. In the above manner, the speech sound source can be accurately located to ensure the clarity and accuracy of the subsequent interaction process.

[0061] In addition, refer to Figure 6 The embodiment of the present invention further provides a conference interaction device based on an AR system, and the conference interaction device based on an AR system includes: The acquisition module 10 is used to acquire multiple conference speech images when it is detected that the current operation mode is the conference listening mode.

[0062] The construction module 20 is used to determine the target speaker and the screen display text according to the multiple conference speech images, and to construct a point cloud structure of the target speaker.

[0063] The positioning module 30 is used to locate the speech sound source of the target speaker according to the point cloud structure and determine the pickup beam area.

[0064] The collection module 40 is used to collect the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio.

[0065] The interaction module 50 is used to implement conference interaction according to the speech audio and the screen display text.

[0066] This embodiment acquires multiple conference speech images when it is detected that the current operation mode is the conference listening mode; determines the target speaker and the text displayed on the screen based on the multiple conference speech images, and constructs the point cloud structure of the target speaker; locates the speech sound source of the target speaker based on the point cloud structure, and determines the pickup beam area; collects the speech audio of the target speaker based on the pickup beam area to determine the speech audio; and realizes conference interaction based on the speech audio and the screen text. Through the above method, the target speaker and the screen text are determined based on multiple conference speech images, and the point cloud structure of the target speaker is further constructed, so that the accurate pickup beam area is determined based on the point cloud structure, and the speech audio is collected using the pickup beam area, and finally the conference interaction is realized based on the speech audio and the screen text. The AR system is used to accurately locate the sound source to ensure the accuracy of the sound pickup, so that the user can intuitively and clearly know the screen text and speech voice content of the target speaker, so that the user can participate in the meeting immersively and improve the efficiency of learning and communication.

[0067] In one embodiment, the construction module 20 is further used to perform object detection on each conference speech image according to the target object detection model, and determine the target speaker and the target feature value of the target speaker under each conference speech image; Performing object construction according to the target feature value to determine the point cloud structure of the target speaker; Text detection is performed on each conference speech image to determine the screen display text of the target speaker.

[0068] In one embodiment, the positioning module 30 is further used to determine the three-dimensional coordinates of the target speaker according to the point cloud structure; Determine the horizontal and vertical positions of the speech source according to the three-dimensional coordinates; A sound pickup beam area is determined according to the vertical orientation and the horizontal orientation.

[0069] In one embodiment, the acquisition module 40 is further used to perform directional audio pickup according to the pickup beam area to determine the initial picked-up audio; Performing audio denoising on the initially picked-up audio to obtain denoised picked-up audio; Play the denoised picked-up audio so that the target interactive user can determine the result based on the denoised picked-up audio feedback voiceprint; When the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected to determine the speech audio.

[0070] In one embodiment, the interaction module 50 is further used to perform signal processing on the speech audio to identify target audio data corresponding to the speech voiceprint of the target speaker; Performing signal enhancement on the target audio data to obtain enhanced audio data; The enhanced audio data is played and the screen display text is displayed to achieve conference interaction.

[0071] In one embodiment, the interaction module 50 is further used to start a focus adjustment mode and send a mode start instruction to a target interaction user, so that the target interaction user adjusts the focus according to the mode start instruction and feeds back the target adjusted focus; Upon receiving the target adjustment focal length fed back by the target interactive user, adjusting the current focal length to the target adjustment focal length; The screen display text is displayed by adjusting the focal length according to the target.

[0072] In one embodiment, the acquisition module 10 is further used to perform text conversion according to the environmental audio to determine the environmental text when the environmental audio and environmental image are collected; Performing scene classification on the environment image to determine an initial image scene; Determine the current scene according to the environment text and the initial image scene; When the current scene is a preset conference scene, sending a mode change reminder to the target interactive user; When receiving the mode change instruction fed back by the target interactive user, the current operation mode is adjusted to the conference attending and listening mode.

[0073] Since the present device adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0074] In addition, an embodiment of the present invention further proposes a storage medium, on which a conference interaction program based on an AR system is stored. When the conference interaction program based on the AR system is executed by a processor, the steps of the conference interaction method based on the AR system as described above are implemented.

[0075] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0076] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0077] In addition, for technical details not fully described in this embodiment, please refer to the conference interaction method based on the AR system provided in any embodiment of the present invention, and will not be repeated here.

[0078] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0079] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0081] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A conference interaction method based on an AR system, characterized in that: The conference interaction method based on the AR system includes: When it is detected that the current operation mode is a conference listening mode, a plurality of conference speech images are acquired; Determine a target speaker and screen display text according to a plurality of conference speech images, and construct a point cloud structure of the target speaker; Positioning the speech sound source of the target speaker according to the point cloud structure, and determining the pickup beam area; The speech audio of the target speaker is collected according to the sound pickup beam area to determine the speech audio; Meeting interaction is achieved based on the speech audio and the screen displayed text.

2. The conference interaction method based on the AR system according to claim 1, characterized in that: The step of determining a target speaker and screen display text according to a plurality of conference speech images, and constructing a point cloud structure of the target speaker includes: Perform object detection on each conference speech image according to the target object detection model to determine the target speaker and the target feature value of the target speaker under each conference speech image; Performing object construction according to the target feature value to determine the point cloud structure of the target speaker; Text detection is performed on each conference speech image to determine the screen display text of the target speaker.

3. The conference interaction method based on the AR system according to claim 1, characterized in that: The step of locating the speech sound source of the target speaker according to the point cloud structure and determining the pickup beam area includes: Determine the three-dimensional coordinates of the target speaker according to the point cloud structure; Determine the horizontal and vertical positions of the speech source according to the three-dimensional coordinates; A sound pickup beam area is determined according to the vertical orientation and the horizontal orientation.

4. The conference interaction method based on the AR system according to claim 1, characterized in that: The step of collecting the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio includes: Directional audio pickup is performed according to the pickup beam area to determine initial pickup audio; Performing audio denoising on the initially picked-up audio to obtain denoised picked-up audio; Play the denoised picked-up audio so that the target interactive user can determine the result according to the denoised picked-up audio feedback voiceprint; When the voiceprint determination result is that the speech voiceprint in the denoised picked-up audio is the speech voiceprint of the target speaker, the speech audio of the target speaker is collected to determine the speech audio.

5. The conference interaction method based on the AR system according to claim 1, characterized in that: The implementing conference interaction according to the speech audio and the screen displayed text includes: Performing signal processing on the speech audio to identify target audio data corresponding to the speech voiceprint of the target speaker; Performing signal enhancement on the target audio data to obtain enhanced audio data; The enhanced audio data is played and the screen display text is displayed to achieve conference interaction.

6. The conference interaction method based on the AR system according to claim 5, characterized in that: The displaying of the screen display text comprises: Turning on a focus adjustment mode, and sending a mode turn-on instruction to a target interactive user, so that the target interactive user adjusts the focus according to the mode turn-on instruction, and feeds back the target adjusted focus; Upon receiving the target adjustment focal length fed back by the target interactive user, adjusting the current focal length to the target adjustment focal length; The screen display text is displayed by adjusting the focal length according to the target.

7. The conference interaction method based on the AR system according to any one of claims 1 to 6, characterized in that: When the current operation mode is detected to be the conference listening mode, before acquiring the plurality of conference speech images, the method further includes: When the ambient audio and the ambient image are collected, text conversion is performed according to the ambient audio to determine the ambient text; Performing scene classification on the environment image to determine an initial image scene; Determine the current scene according to the environment text and the initial image scene; When the current scene is a preset conference scene, sending a mode change reminder to the target interactive user; When receiving the mode change instruction fed back by the target interactive user, the current operation mode is adjusted to the conference attending and listening mode.

8. A conference interaction device based on an AR system, characterized in that: The conference interaction device based on the AR system includes: The acquisition module is used to acquire multiple conference speech images when detecting that the current operation mode is the conference listening mode; A construction module, used to determine a target speaker and screen display text according to a plurality of conference speech images, and to construct a point cloud structure of the target speaker; A positioning module, used to locate the speech sound source of the target speaker according to the point cloud structure and determine the pickup beam area; A collection module, used for collecting the speech audio of the target speaker according to the sound pickup beam area to determine the speech audio; An interaction module is used to realize conference interaction according to the speech audio and the screen display text.

9. A conference interactive device based on an AR system, characterized in that: The device includes: a memory, a processor, and a conference interaction program based on an AR system stored in the memory and executable on the processor, wherein the conference interaction program based on the AR system is configured to implement a conference interaction method based on an AR system as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores an AR system-based conference interaction program, and when the AR system-based conference interaction program is executed by the processor, the AR system-based conference interaction method according to any one of claims 1 to 7 is implemented.

Images