Digital conference audio interaction system and method and storage medium

Abstract

The invention discloses a digital conference audio interaction system and method and a storage medium, and particularly relates to the technical field of audio interaction, and the system comprises a conference starting and authorization module which is used for starting a conference and managing the authority of participants; the conference scene virtualization module can be used for setting a conference scene and related elements; the participant identity verification module verifies the identity through face recognition and the like and provides personalized setting; the voice separation and recognition module can separate and extract personal audios when multiple persons speak; the interactive operation matching module is used for matching an operation instruction with a virtual object and processing an unidentified object; the system further comprises an audio processing module for optimizing the audio effect. According to the invention, the defects of the existing digital conference system in the aspects of organization management, scene setting, identity verification, voice processing, interactive operation and the like are overcome, the efficiency, quality and participating experience of the digital conference are improved, and the method is suitable for various business, academic and other conference scenes.

Description

Technical Field

[0001] The present invention relates to the field of audio interaction technology, and more particularly, to a digital conference audio interaction system, method and storage medium. Background Art

[0002] With the rapid development of information technology, digital conferencing is increasingly used in various business, academic, and social activities. However, traditional digital conferencing systems have gradually exposed many problems in practical applications.

[0003] In terms of meeting organization and management, there is a lack of efficient and comprehensive initiation and authorization mechanisms. Virtual meeting instructions cannot be received quickly and accurately, and the permissions for hosts and participants are not precisely defined, which easily leads to chaotic meeting processes. The management and presentation of meeting materials also lack a systematic approach, affecting the efficiency of meeting preparation and execution.

[0004] The construction of meeting scenes is relatively simple and lacks flexibility. Existing systems often fail to provide a rich variety of meeting scenes, background images, and the selection and configuration of various table, chair, and ornament models. This makes it difficult to meet the needs of different meeting themes and participants for personalized meeting environments, reducing participants' immersion and enthusiasm.

[0005] Vulnerabilities exist in the identity verification process. Traditional methods struggle to ensure the authenticity and accuracy of attendee identities. The lack of reliable facial recognition technology and mechanisms for verifying and completing relevant information can compromise meeting security and prevent attendees from having a personalized meeting experience, such as selecting a character model that matches their image or choosing their preferred seating position.

[0006] Speech processing when multiple people are speaking simultaneously has always been a challenge. Existing technologies struggle to effectively separate and identify individual participants' audio in complex speech environments, leading to speech confusion and severely impacting communication effectiveness and information accuracy.

[0007] The level of intelligent interaction is low. It's impossible to accurately match participants' commands with virtual objects in the meeting scene. There's a lack of effective labeling and manual identification for unrecognized objects, making interaction in meetings less convenient and smooth, impacting the overall efficiency and quality of the meeting.

[0008] Therefore, a digital conference audio interaction system, method and storage medium are proposed. Summary of the Invention

[0009] In order to overcome the above-mentioned defects of the prior art, the present invention provides a digital conference audio interaction system, method and storage medium to solve the problems raised in the above-mentioned background technology.

[0010] To achieve the above objectives, the present invention provides the following technical solutions: a digital conference audio interaction system, comprising:

[0011] The conference initiation and authorization module is used to receive virtual conference instructions to start the conference and authorize the host and participants, including setting the specific permissions of the host and participants;

[0012] The conference scene virtualization module has a model library for selecting and setting conference scenes, conference background images, conference table and chair models, and other decoration models;

[0013] The participant identity verification module verifies the identity of participants through facial recognition technology. Participants complete and verify their personal information, take a self-portrait through an external camera, select a character model that matches them, and choose their seating position for the meeting.

[0014] The speech separation and recognition module is used to process the currently collected real-time speech data when multiple people are speaking at the same time, and to separate, extract and recognize the individual audio of each participant;

[0015] The interactive operation matching module is used to match the operation instructions of participants with virtual objects in the conference scene, and to enable remote operators to mark and manually identify unrecognized objects.

[0016] Preferably, the conference initiation and authorization module further includes:

[0017] Conference material receiving unit, used to receive conference materials sent by participants;

[0018] A conference data storage unit, used to store the conference data;

[0019] The conference material display unit is used to display the conference material.

[0020] Preferably, the speech separation and recognition module further includes:

[0021] Perform short-time Fourier transform on the currently collected real-time speech data to obtain mixed spectrum features;

[0022] The mixed spectrum features are concatenated with the corresponding voiceprint features to obtain the reference spectrum features, which are then input into the dilated convolutional layer to obtain the basic features.

[0023] The basic features are input into the speech separation model, and the output is the spectrum mask, which is multiplied with the mixed spectrum features to obtain the individual spectrum;

[0024] The phase spectrum of real-time speech data is used to restore the personal spectrum to obtain personal audio.

[0025] Preferably, the system further comprises an audio processing module for performing acoustic echo cancellation, mixing near-end input audio signals to generate automatic mixing output signals, and generating audio output signals based on the input audio signals.

[0026] A digital conference audio interaction method comprises the following steps:

[0027] Use the meeting initiation and authorization module to receive virtual meeting instructions to start the meeting and authorize the host and participants;

[0028] Use the conference scene virtualization module to select and set the conference scene;

[0029] Use the participant authentication module to verify the identity of participants through facial recognition technology;

[0030] Use the speech separation and recognition module to separate, extract and recognize the individual audio of each participant when multiple people are speaking at the same time;

[0031] Use the interactive operation matching module to match the operation instructions of the participants with the virtual objects in the conference scene.

[0032] Preferably, it also includes:

[0033] Receive and store conference materials sent by participants;

[0034] Present the meeting materials.

[0035] A storage medium stores a computer program for executing the above-mentioned digital conference audio interaction method.

[0036] Technical effects and advantages of the present invention:

[0037] 1. The meeting initiation and authorization module enables rapid reception and response to virtual meeting commands, accurately sets the permissions of the host and participants, and ensures the orderly conduct of the meeting. The meeting material reception, storage, and display unit facilitates the centralized management and sharing of materials, improves the efficiency of the meeting preparation stage, and lays the foundation for the smooth progress of the meeting.

[0038] 2. The conference scene virtualization module, with its rich model library, allows users to freely select and set conference scenes, background images, table and chair models, and ornament models, etc., greatly enhancing the diversity and personalization of conference scenes, making participants feel as if they are in a real environment that fits the conference theme, and improving the immersion and appeal of the meeting.

[0039] 3. The participant identity verification module uses facial recognition technology to achieve accurate identity verification. At the same time, participants can complete their personal information, select appropriate character models and choose their seating positions. This not only ensures the security of the meeting, but also provides participants with an exclusive and personalized meeting experience, thereby improving participant satisfaction.

[0040] 4. The speech separation and recognition module demonstrates excellent performance in scenarios with multiple speakers. Through advanced technologies such as short-time Fourier transform, voiceprint feature splicing, dilated convolutional layers, and speech separation models, it accurately separates and identifies the individual audio of each participant, effectively resolving the problem of speech confusion, significantly improving the clarity and accuracy of voice interaction in meetings, and promoting efficient communication among participants.

[0041] 5. The interactive operation matching module achieves accurate matching between participants' operation instructions and virtual objects, and further improves the intelligence level of interactive operations through the remote operator's marking and manual recognition of unrecognized objects, making participants' operations in the meeting more convenient and smooth, reducing time wasted due to unsmooth operations, and improving the overall efficiency of the meeting.

[0042] 6. The acoustic echo cancellation, audio signal mixing and output functions performed by the audio processing module effectively reduce audio interference, improve audio quality, and provide participants with a clearer and purer auditory environment, helping participants to better understand the meeting content and improve the effectiveness of the meeting. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the digital conference audio interaction method of the present invention. DETAILED DESCRIPTION

[0044] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0045] A digital conference audio interaction system proposed in this embodiment includes:

[0046] The conference initiation and authorization module is used to receive virtual conference instructions to start the conference and authorize the host and participants, including setting the specific permissions of the host and participants;

[0047] The conference initiation and authorization module also includes:

[0048] Conference material receiving unit, used to receive conference materials sent by participants;

[0049] A conference data storage unit, used to store the conference data;

[0050] The conference material display unit is used to display the conference material.

[0051] In practice, when a user triggers a meeting start operation, the meeting start and authorization module receives the virtual meeting instruction. The system determines the host and participants based on preset rules or administrator settings, and assigns them corresponding permissions.

[0052] Specifically, the host has conference control permissions, such as starting / pausing the meeting and managing participants; participants have basic permissions such as speaking and viewing materials.

[0053] If the meeting initiation and authorization module includes a meeting material receiving unit, participants can use this unit to send meeting materials to the system. The meeting material storage unit in the system will classify and store the received materials. The storage method can be based on file type, meeting theme, or participant classification to facilitate subsequent search and management. When materials need to be presented during the meeting, the meeting material display unit will display the corresponding materials in the meeting interface in an appropriate manner according to the operating instructions of the host or participants, such as displaying them in the form of documents in a shared area or through screen projection.

[0054] The conference scene virtualization module has a model library for selecting and setting conference scenes, conference background images, conference table and chair models, and other decoration models.

[0055] In practice, the meeting scene virtualization module is activated after the meeting starts. Users can access a library of models through this module and choose from a variety of preset meeting scenes, meeting background images, meeting table and chair models, and other decorative models. Specifically, for business meetings, a formal conference room scene can be selected, while for academic meetings, a library scene with a scholarly atmosphere can be selected. Users can also customize elements in the scene according to their personal preferences, such as adjusting the placement of tables and chairs and changing the color of the background image, to create a personalized meeting environment.

[0056] The participant identity authentication module verifies the identity of participants through facial recognition technology. Participants complete and verify their personal information, take their own portraits through external cameras, select a character model that matches them, and choose their seating position during the meeting.

[0057] In practice, the participant identity verification module activates when a participant joins a meeting. An external camera captures the participant's facial image and uses facial recognition technology to compare it with pre-stored identity information to verify the participant's identity. After successful identity verification, the participant can further complete their profile information, such as adding contact information and position. The participant then takes a selfie. The system then matches or generates a matching persona model from a library based on the profile image's features for the participant to select. Finally, the participant selects their seat location for the meeting, completing their personalized meeting preparation.

[0058] The speech separation and recognition module is used to process the currently collected real-time speech data when multiple people are speaking at the same time, separate, extract, and recognize the individual audio of each participant. This module further includes:

[0059] Perform short-time Fourier transform on the currently collected real-time speech data to obtain mixed spectrum features;

[0060] The mixed spectrum features are concatenated with the corresponding voiceprint features to obtain the reference spectrum features, which are then input into the dilated convolutional layer to obtain the basic features.

[0061] The basic features are input into the speech separation model, and the output is the spectrum mask, which is multiplied with the mixed spectrum features to obtain the individual spectrum;

[0062] The phase spectrum of real-time speech data is used to restore the personal spectrum to obtain personal audio.

[0063] In practice, when multiple people speak simultaneously, the speech separation and recognition module begins operating. First, a short-time Fourier transform is performed on the currently collected real-time speech data, converting the time-domain speech signal into a mixed spectral feature. Next, the mixed spectral feature is concatenated with the corresponding voiceprint feature to form a reference spectral feature, which is then input into a dilated convolutional layer to obtain the basic feature. The basic feature is then input into the speech separation model, which outputs a spectral mask. This spectral mask is multiplied by the mixed spectral feature to obtain the individual spectrum. Finally, the phase spectrum of the real-time speech data is used to restore the individual spectrum, resulting in clearly identifiable individual audio for each participant, effectively achieving speech separation and recognition.

[0064] The interactive operation matching module is used to match the operation instructions of participants with virtual objects in the conference scene, and to enable remote operators to mark and manually identify unrecognized objects.

[0065] In specific implementation, during a meeting, as participants manipulate virtual objects in the scene, the interactive operation matching module monitors these commands in real time. Based on pre-set matching rules, the system accurately matches these commands with virtual objects, enabling corresponding operations such as opening documents and switching scene perspectives. When encountering unrecognized objects, remote operators can mark them through a specific interface, manually identify them, and define their operational attributes based on the actual situation, enabling subsequent participants to operate them normally.

[0066] Also included is an audio processing module for performing acoustic echo cancellation, mixing a near-end input audio signal to generate an automix output signal, and generating an audio output signal based on the input audio signal.

[0067] In practice, the audio processing module continuously processes audio signals. During the audio input phase, it performs acoustic echo cancellation to reduce echo interference caused by equipment or environmental factors. Simultaneously, it mixes the near-end input audio signals and generates an automatic mixing output signal based on specific mixing rules to balance the volume and sound quality of different audio sources. Based on the input audio signals, an audio processing algorithm generates a final audio output signal, ensuring that participants can hear clear and smooth audio content.

[0068] Throughout the meeting, all modules work together continuously, and the system monitors their operational status in real time to ensure stable and timely data transmission. After the meeting, the system automatically saves meeting records, including participant information, meeting material manipulation records, and voice interaction records, for subsequent query and analysis. The system also evaluates the performance of each module and optimizes and upgrades it based on the results to improve performance in subsequent meetings.

[0069] Example 1: Small Business Digital Conference

[0070] Meeting preparation stage

[0071] Meeting organizers use the meeting initiation and authorization module to initiate a small business meeting by sending virtual meeting instructions through a dedicated meeting management platform. During the authorization process, the host is granted comprehensive meeting control permissions, such as managing the attendee list, starting and ending the meeting, and controlling the presentation of meeting materials. Participants are granted basic permissions such as speaking, viewing materials, and voting. Participants send their business reports and other materials to the system through the meeting materials receiving unit. The system's meeting materials storage unit categorizes and stores them according to meeting topic and attendees.

[0072] The host uses the conference scene virtualization module to select a simple business conference room scene from the model library. The background image is set to the company logo and a blue business-style background. The table and chair models are selected as a standard conference table and chair layout. The ornament models are small models of the company's products and placed on the conference table.

[0073] When a participant joins a meeting, the participant identity verification module is activated. An external camera captures the participant's facial image and compares it with photos in the company's employee database to verify their identity. After successful authentication, the participant completes their personal information, such as their department, takes a headshot, selects a business suit model similar to their appearance, and selects their seat at the virtual conference table.

[0074] Meeting in progress

[0075] When two participants speak simultaneously to discuss a business cooperation plan, the speech separation and recognition module comes into play. It performs a short-time Fourier transform on the collected real-time speech data to obtain a mixed spectral feature. This feature is then combined with the pre-stored voiceprint features of the two speakers and input into a dilated convolutional layer to obtain the basic feature. The speech separation model then outputs a spectral mask, which is multiplied by the mixed spectral feature to obtain the individual spectrum. Finally, the phase spectrum of the real-time speech data is used to restore the individual audio, allowing other participants to accurately distinguish the speech content.

[0076] During a meeting, participants need to review and discuss an important business contract document, interacting with the virtual document object in the meeting scene through commands. The interactive operation matching module accurately identifies the commands and displays the document in the shared area in the appropriate format. Participants can then annotate and comment on the document. If a newly added, unrecognized encrypted document format is encountered during the operation, the remote operator marks the document, manually identifies its format, and installs the corresponding decryption and viewing plug-in, enabling participants to view and discuss the document normally.

[0077] The audio processing module continuously processes the audio to eliminate echoes in the conference room and ensure that the speakers are clear. At the same time, it mixes the audio signals of different participants and automatically adjusts the output according to the volume so that the sound heard by each participant is balanced.

[0078] End of meeting

[0079] After the meeting, the system automatically saves the meeting minutes, including information such as participants' speeches, document operation records, and voting results. The system evaluates the performance of each module, such as the accuracy of the speech separation and recognition modules and the response time of the interactive operation matching module. Based on the evaluation results, the system is optimized, such as updating the speech separation model algorithm to improve speech processing accuracy.

[0080] Example 2: Academic Discussion Digital Conference

[0081] Meeting preparation stage

[0082] The conference convener uses the conference initiation and authorization module to issue a virtual conference command on the academic conference platform, launching the academic conference. The host is assigned permissions such as guiding the meeting process and arranging expert presentations. Participants, including scholars and graduate students, have permissions to ask questions and share research findings. Participants upload their academic papers and other materials to the system, which are then stored in the conference materials storage unit, categorized by subject area and paper topic.

[0083] The host uses the conference scene virtualization module and selects the academic library scene from the model library. The background image is set to library bookshelves and academic posters. The table and chair models are selected as seminar tables and chairs with writing boards, and the ornament models are selected as academic elements such as books and globes.

[0084] When participants join a meeting, the identity verification module verifies their identity through facial recognition and compares it with the academic institution's personnel database. After their identity is confirmed, participants complete their academic profile, including their research interests, take a headshot, select a model of a scholar who matches their image, and choose their seat around the seminar table.

[0085] Meeting in progress

[0086] During the group discussion, multiple participants simultaneously presented their research perspectives. The speech separation and recognition module processed the real-time speech data. Through the aforementioned speech processing process, it accurately separated the audio of each participant, enabling other participants to clearly distinguish between the speeches of different scholars and promoting the exchange of academic ideas.

[0087] Participants will need to present their research data and charts during discussions. Using the interactive matching module, they can match operational instructions with objects in the virtual scene's display area, resulting in a clear presentation of the data and charts. If a chart in an unrecognized format is generated by new data analysis software, the remote operator will mark it, manually identify it, and convert it to a common format for presentation.

[0088] The audio processing module ensures audio quality, eliminates echoes and noise caused by differences in equipment used by different scholars, ensures stable audio output, and enables participants to focus on academic discussions.

[0089] End of meeting

[0090] After the meeting, the system will fully preserve the meeting records, including academic discussion content, document presentation records, etc. Each module of the system will be evaluated, such as checking the recognition accuracy of the participant identity verification module. Based on the evaluation results, the system will be improved, such as optimizing the identity verification algorithm to adapt to the personnel database formats of more academic institutions.

[0091] Example 3: Large-scale online training digital conference

[0092] Meeting preparation stage

[0093] Instructors use the meeting initiation and authorization module to initiate online training sessions on the training platform. Instructors are assigned permissions to teach, answer questions, and manage students; students are assigned permissions to attend classes, ask questions, and participate in interactive exercises. Instructors upload training courseware, case studies, and other materials to the system in advance. Meeting materials are stored in the meeting resource storage unit, categorized by training course chapter and type.

[0094] The instructor uses the conference scene virtualization module, selects a large training classroom scene, sets the background image to a picture related to the training theme, selects neatly arranged student tables and chairs as the table and chair models, and selects training-related props such as a laptop model as the ornament model.

[0095] When a student joins a meeting, the identity verification module verifies their identity through facial recognition and compares it with the student registration information database. After identity verification, the student completes their personal learning information, takes a headshot, selects a student character model that suits them, and is randomly assigned a seat.

[0096] Meeting in progress

[0097] During the interactive session, multiple students asked questions simultaneously. The speech separation and recognition module quickly separated the speech of each student, allowing the instructor to accurately understand the students' questions and provide answers.

[0098] During the lecture, the instructor needs to switch between different training courses and play different video materials. The interactive operation matching module accurately executes the operation instructions, achieving smooth switching between courseware and video. If a new video format cannot be played, the remote operator will mark it and install the corresponding decoder to enable normal video playback.

[0099] The audio processing module ensures that the instructor's voice is clearly transmitted to every student, while processing the students' audio feedback to ensure smooth audio communication during the training process.

[0100] End of meeting

[0101] After the meeting, the system saves the training records, including the participants' learning time, questions asked, and participation in the interaction. The system is evaluated, such as the performance of the speech separation and recognition modules when multiple people speak simultaneously. Based on the results, the system is optimized, such as improving the speech separation model's ability to recognize the speech of students with different accents.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A digital conference audio interaction system, characterized in that: include: The conference initiation and authorization module is used to receive virtual conference instructions to initiate the conference and authorize the host and participants, including setting the specific permissions of the host and participants; The conference scene virtualization module has a model library for selecting and setting conference scenes, conference background images, conference table and chair models, and other furnishing models; The participant identity verification module verifies the identity of the participant through facial recognition technology. The participant completes and verifies personal information, takes a self-portrait through an external camera, selects a character model that matches himself, and chooses the seat position for the meeting. The speech separation and recognition module is used to process the currently collected real-time speech data when multiple people are speaking at the same time, separate, extract and recognize the personal audio of each participant; Use the phase spectrum of real-time speech data to restore the personal spectrum and obtain personal audio; The interactive operation matching module is used to match the operation instructions of the participants with the virtual objects in the conference scene, and to enable the remote operator to mark and manually identify unrecognized objects.

2. A digital conference audio interaction system according to claim 1, characterized in that: The conference initiation and authorization module also includes: A conference material receiving unit, used to receive conference materials sent by conference participants; A conference data storage unit, used to store the conference data; The conference material display unit is used to display the conference material.

3. A digital conference audio interaction system according to claim 1, characterized in that: The speech separation and recognition module also includes: Perform short-time Fourier transform on the real-time speech data currently collected to obtain mixed spectrum features; The mixed spectrum feature is concatenated with the corresponding voiceprint feature to obtain the reference spectrum feature, which is then input into the dilated convolutional layer to obtain the basic feature. The basic features are input into the speech separation model, and the output is the spectrum mask, which is multiplied with the mixed spectrum features to obtain the personal spectrum; The phase spectrum of real-time speech data is used to restore the personal spectrum and obtain personal audio.

4. A digital conference audio interaction system according to claim 1, characterized in that: Also included is an audio processing module for performing acoustic echo cancellation, mixing near-end input audio signals to generate an automatic mixing output signal, and generating an audio output signal based on the input audio signal.

5. A digital conference audio interaction method, characterized in that: The following steps are involved: Use the meeting initiation and authorization module to receive virtual meeting instructions to start the meeting and authorize the host and participants; Use the meeting scene virtualization module to select and set the meeting scene; Use the participant authentication module to verify the identity of participants through facial recognition technology; Use the speech separation and recognition module to separate, extract and recognize the individual audio of each participant when multiple people are speaking at the same time; Use the interactive operation matching module to match the operation instructions of the participants with the virtual objects in the conference scene.

6. A digital conference audio interaction method according to claim 5, characterized in that: Also includes: Receive and store conference materials sent by participants; Display the meeting materials.

7. A storage medium, characterized in that: A computer program for executing the digital conference audio interaction method according to claim 5 or 6 is stored.

Images