Audio processing method and conference audio device
Patent Information
- Application Number
- CN202510569046.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2026-10-30
Smart Images
Figure CN123017037A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer application technology, and in particular to an audio processing method and a conference audio device. Background Technology
[0002] With the continuous development of smart devices and multimedia technologies, smart devices are increasingly being used in online communication scenarios such as meetings to collect and record communication content, generating multimedia files such as meeting minutes. This allows for review of the communication after the meeting and effectively prevents the possibility of communication leaks due to recording with personal devices. Currently, these smart devices typically send the collected user voice information to a server for voice recognition and meeting minutes generation. However, this process carries a risk of information leakage, allowing malicious actors to obtain users' voice characteristics and causing significant losses. Therefore, reducing this risk of information leakage during meeting minutes generation is a pressing technical problem that needs to be addressed.
[0003] Application content
[0004] To address the aforementioned technical problems, this application aims to provide an audio processing method and a conference audio device. It solves the current problem that, during the generation of conference minutes, audio information, including the user's voiceprint features, needs to be uploaded to a server for speech recognition. This application proposes a technical solution that performs speech recognition locally on the audio device to generate corresponding text content. Since the user's voice information does not need to be uploaded to a server for speech recognition and other processing, the risk of user voice feature leakage is reduced, effectively ensuring user information security.
[0005] The technical solution of this application is implemented as follows:
[0006] This application provides an audio processing method applied to a conference audio device, wherein the conference audio device interacts with a remote device via a terminal device, and the method includes:
[0007] Acquire audio stream data; wherein, the audio stream data includes uplink audio stream data obtained by the conference audio device from local audio acquisition of its environment, and downlink audio stream data sent by the terminal device;
[0008] Based on a preset processing order, speech recognition is performed on the uplink audio stream data and downlink audio stream data respectively to determine the audio text content.
[0009] The method in the above scheme further includes:
[0010] Based on the local voiceprint database, voiceprint recognition is performed on the audio stream data to determine the first user identity information corresponding to the audio text content;
[0011] Based on the first user identity information and the audio text content, audio log information is generated.
[0012] In the above solution, the conference audio device is connected to the conference video device, and the method further includes:
[0013] Receive video feeds captured by the video conferencing equipment;
[0014] Based on the video footage and the local voiceprint database, the first user identity information corresponding to the audio text content is determined.
[0015] In the above scheme, the step of performing speech recognition on the uplink audio stream data and downlink audio stream data respectively based on a preset processing order to obtain the audio text content includes:
[0016] Based on the preset processing order, the uplink audio stream data and the downlink audio stream data are sequentially input into the first thread for real-time audio stream speech recognition, and real-time text content is obtained by serial transmission.
[0017] The real-time text content is output so that a display device displays the real-time text content; wherein, the audio text content includes the real-time text content.
[0018] The method in the above scheme further includes:
[0019] Based on the preset processing order, the uplink audio stream data and downlink audio stream data are segmented into sentences in sequence to obtain at least one first audio sentence corresponding to the uplink audio stream data or at least one second audio sentence corresponding to the downlink audio stream data.
[0020] Store the at least one first audio statement to the uplink audio stream data storage queue, and store the at least one second audio statement to the downlink audio stream data storage queue;
[0021] Based on the preset processing order, the second thread sequentially retrieves sentences from the uplink audio stream data storage queue and the downlink audio stream data storage queue for sentence recognition, thereby obtaining the sentence text content; wherein, the audio text content includes the sentence text content.
[0022] The method in the above scheme further includes:
[0023] The text content of the statement is output to update the real-time text content displayed on the display device.
[0024] In the above scheme, the step of performing voiceprint recognition on the audio stream data based on a local voiceprint library to determine the first user identity information corresponding to the audio text content includes:
[0025] According to the preset processing order, the sentences to be recognized are sequentially obtained from the uplink audio stream data storage queue and the downlink audio stream data storage queue;
[0026] Extract the voiceprint features of the statement to be identified to obtain at least one voiceprint feature;
[0027] Store the at least one voiceprint feature to the voiceprint storage queue;
[0028] The local voiceprint library and the voiceprint storage queue are matched by a third thread to obtain the first user identity information corresponding to each statement to be identified.
[0029] In the above scheme, the step of matching the local voiceprint library and the voiceprint storage queue through a third thread to obtain the first user identity information corresponding to each statement to be identified includes:
[0030] If the third thread determines that the number of voiceprint features included in the voiceprint storage queue exceeds the first number, it performs cluster analysis on all voiceprint features in the voiceprint storage queue to obtain the clustering result.
[0031] The clustering results are matched with the voiceprints in the local voiceprint database to determine the first user identity information that matches each type of voiceprint feature in the clustering results, thus obtaining the first user identity information corresponding to each statement to be identified.
[0032] In the above scheme, the display device includes the touch-sensitive display screen of the conference audio device, or the terminal device, or the remote device includes at least one participating device that participates in the same conference as the terminal device, or the remote device includes at least a cloud device for providing minutes services.
[0033] The method in the above scheme further includes:
[0034] Determine the information feedback model;
[0035] If the information feedback mode is the first mode, the audio log information is sent to the terminal device, so that the terminal device uploads the audio log information to the cloud device, and then the cloud device generates meeting minutes based on the audio log information; wherein, the cloud device is the remote device;
[0036] If the information feedback mode is the second mode, the audio log information and the corresponding meeting audio information are sent to the terminal device, so that the terminal device uploads the audio log information and the corresponding meeting audio information to the cloud device, and then the cloud device performs speech recognition based on the meeting audio information to obtain corrected text information, and generates meeting minutes based on the audio log information and the corrected text information.
[0037] The method in the above scheme further includes:
[0038] If an update instruction is received from the terminal device, the second user identity information in the local voiceprint database is updated to the third user identity information based on the update instruction; wherein, the update instruction is used to instruct the second user identity information to be updated to the third user identity information, and the third user identity information is obtained by the user modifying the second user identity information displayed on the terminal device;
[0039] The second user identity information in the audio log information is modified to the third user identity information.
[0040] This application provides an audio processing apparatus applied to a conference audio device, wherein the conference audio device interacts with a remote device via a terminal device, and the apparatus includes: an acquisition unit and a recognition unit; wherein:
[0041] The acquisition unit is used to acquire audio stream data; wherein, the audio stream data includes uplink audio stream data obtained by the conference audio device from local audio acquisition of its environment, and downlink audio stream data sent by the terminal device;
[0042] The recognition unit is used to perform speech recognition on the uplink audio stream data and downlink audio stream data respectively based on a preset processing order to determine the audio text content.
[0043] This application provides a conference audio device, which includes at least: a microphone, a speaker, a memory, a processor, and a communication bus; wherein:
[0044] The microphone is used to collect local audio from the environment in which the conference audio equipment is located, and to obtain uplink audio stream data;
[0045] The speaker is used to play the acquired audio stream data;
[0046] The memory is used to store executable instructions;
[0047] The communication bus is used to realize the communication connection between the microphone, the speaker, the processor and the memory;
[0048] The processor is configured to execute the information processing program stored in the memory to implement the steps of the audio processing method as described in any of the preceding claims.
[0049] This application provides a storage medium storing an information processing program, which, when executed, implements the steps of the audio processing method as described in any of the preceding claims.
[0050] This application provides an audio processing method and a conference audio device. After acquiring audio stream data through the conference audio device, the device performs speech recognition on the upstream and downstream audio stream data according to a preset processing order to determine the audio text content. This solves the problem that current methods require uploading audio information, including user voiceprint features, to a server for speech recognition during meeting minutes generation. It proposes a technical solution for performing speech recognition locally on the audio device and generating corresponding text content. Since the user's voice information does not need to be uploaded to a server for speech recognition and other processing, the risk of user voice feature leakage is reduced, effectively ensuring user information security. Attached Figure Description
[0051] Figure 1 A flowchart illustrating an audio processing method provided in an embodiment of this application;
[0052] Figure 2 A schematic diagram of the structure of the conference audio device provided in the embodiments of this application;
[0053] Figure 3 This application provides a schematic diagram of the implementation process of an audio processing method.
[0054] Figure 4 A schematic diagram of the display interface of a conference audio device provided in an embodiment of this application;
[0055] Figure 5 A schematic diagram of a system architecture provided for an embodiment of this application;
[0056] Figure 6 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;
[0057] Figure 7 This is a schematic diagram of the structure of a conference audio device provided in an embodiment of this application. Detailed Implementation
[0058] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0059] The embodiments of this application provide an audio processing method, referring to... Figure 1 As shown, the method is applied to conference audio equipment, which interacts with remote devices through terminal devices. The method includes the following steps:
[0060] Step 101: Obtain audio stream data.
[0061] The audio stream data includes uplink audio stream data obtained by the conference audio equipment from local audio acquisition of its environment, and downlink audio stream data sent by the terminal equipment.
[0062] In this embodiment, the conference audio device is a device with at least audio acquisition and playback functions, and further may have a touch-operated display screen for quick interaction with the user. The terminal device is a device that can install a client corresponding to the conference audio device, such as a smart mobile terminal like a smartphone, desktop computer, tablet computer, or laptop computer, which has communication data network capabilities and can communicate with participating devices through the communication data network. The remote device may include participating devices, which are other terminal devices in the same conference as the terminal device connected to the conference audio device. The remote device may also include a cloud device for generating intelligent minutes files, and the cloud device may also have storage capabilities.
[0063] Audio stream data refers to the audio data transmitted between the conference audio equipment and the terminal equipment via the transmission channel. It includes uplink audio stream data obtained by the conference audio equipment from local audio acquisition of its environment, and downlink audio stream data sent by the terminal equipment.
[0064] For example, such as Figure 2 This application provides a schematic diagram of a conference audio device, including an audio processing module a1, a connecting component a2, and a support base a3 with a touch-sensitive display screen. The touch-sensitive display screen is mounted on the support base a3. The audio processing module a1 may include a microphone for audio acquisition and a speaker for audio playback, and is mounted on a cylindrical main component. It should be noted that the microphone and speaker in the audio processing module a1 can also be two independent modules, located at different positions on the same main component, depending on the specific circumstances.
[0065] The main body of the cylindrical structure housing the audio processing module a1 is connected to the supporting base a3 via a connecting component a2. This connecting component a2 can be a retractable component, internally housing a circuit board for connecting the touchscreen display, microphone, and speaker, thereby enabling communication between the touchscreen display and the speakers and microphone. The distance between the main body of the cylindrical structure and the supporting base a3 can be adjusted by changing the length of the retractable component, thus adjusting the overall height of the audio equipment. In some application scenarios, the bottom of the main body of the cylindrical structure can overlap with the touchscreen display, allowing for quick control of the touchscreen display to turn off or power off. It should be noted that... Figure 2 The structure of the conference audio equipment shown is only one implementation method and is not specifically limited here. In some application scenarios, the microphone and speaker in the audio processing module can also be set up separately, which can be determined according to the actual situation and is not specifically limited here.
[0066] Step 102: Based on the preset processing order, perform speech recognition on the uplink audio stream data and downlink audio stream data respectively to determine the audio text content.
[0067] In this embodiment of the application, the conference audio device distinguishes the acquired audio stream data, for example, according to the uplink channel. Then, according to a preset processing order, the distinguished uplink audio stream data and downlink audio stream data are processed locally on the conference audio device using speech recognition to obtain audio text content.
[0068] Based on the foregoing embodiments, in other embodiments of this application, the method further includes the following execution steps:
[0069] Based on the local voiceprint database, voiceprint recognition is performed on the audio stream data to determine the first user identity information corresponding to the audio text content.
[0070] Audio log information is generated based on the first user's identity information and the audio text content.
[0071] In this embodiment, the conference audio device also employs voiceprint recognition locally. Specifically, a local voiceprint database is established on the conference audio device, and then the acquired audio stream data is analyzed using this database to determine the identity of the currently speaking user, obtaining the first user identity information corresponding to the audio text content. The local voiceprint database is an experience database that records user identity information and one or more specific voiceprint features corresponding to each user identity. The first user identity information can be information used to uniquely identify the user, such as the user's name, ID number, or ID number.
[0072] Based on the identified user identity information and the corresponding audio text content, corresponding audio log information is generated. That is, the text content obtained from speech recognition and the corresponding user identity information are recorded. This allows for subsequent analysis of meeting minutes based on the audio log information, or users can query the audio log information to determine the content discussed in the meeting, thus achieving real-time recording of meeting content.
[0073] In some implementations, the conference audio device also includes a video capture component to capture video images; or it can connect to a conference video device and receive video images captured by that device. When identifying user identity information, the captured video image information can be compared with a local voiceprint database. This involves matching the captured audio stream data with the local voiceprint database while simultaneously performing image recognition on the speaker in the corresponding video image. This ensures the accuracy of the first user identity information corresponding to the audio text content. By combining audio recognition with image recognition, the possibility of failing to quickly determine user identity information when the audio stream data quality is poor or the analyzed voiceprint features show little difference is reduced, thus improving the accuracy of user identity information identification.
[0074] Based on the foregoing embodiments, in other embodiments of this application, after obtaining the audio stream data, the audio stream data is split locally on the conference audio device according to the data transmission direction to obtain uplink audio stream data and downlink audio stream data; wherein, the uplink audio stream data is the local audio stream data collected by the conference audio device, and the downlink audio stream data is the audio stream data sent by the terminal device.
[0075] In this embodiment, the preset processing order can be the order in which the uplink audio stream data and the downlink audio stream data are processed. For example, the uplink audio stream data can be processed first, followed by the downlink audio stream data, or vice versa. The preset processing order can be a default setting, or it can be configured by the user according to actual needs when the conference audio device is used. The specific order can be determined by the actual situation and is not specifically limited here.
[0076] When the conference audio equipment performs speech recognition, it first acquires the uplink audio stream data and the downlink audio stream data in real time according to the transmission direction.
[0077] Based on the foregoing embodiments, in other embodiments of this application, the steps of performing speech recognition on the uplink audio stream data and downlink audio stream data respectively to obtain audio text content, based on a preset processing order, can be implemented by the following steps:
[0078] Based on a preset processing order, the upstream audio stream data and the downstream audio stream data are sequentially input into the first thread for real-time audio stream speech recognition to obtain real-time text content.
[0079] Output real-time text content so that a display device can display the real-time text content; wherein, audio text content includes real-time text content.
[0080] In this embodiment of the application, assuming that the preset processing order is to process the uplink audio stream data first and then process the downlink audio stream data, the first thread of the conference audio device adopts a serial approach. First, it performs real-time speech recognition on the speech data in the uplink audio stream data that is currently recognized, and then performs real-time speech recognition on the speech data in the downlink audio stream data that is currently recognized to obtain real-time text content. The obtained real-time text content is usually one or two words in a sentence.
[0081] Based on the foregoing embodiments, in other embodiments of this application, the method further includes the following execution steps:
[0082] Based on a preset processing order, the uplink audio stream data and downlink audio stream data are segmented into sentences in turn to obtain at least one first audio sentence corresponding to the uplink audio stream data or at least one second audio sentence corresponding to the downlink audio stream data.
[0083] Store at least one first audio statement to the upstream audio stream data storage queue, and store at least one second audio statement to the downstream audio stream data storage queue;
[0084] Based on a preset processing order, the second thread sequentially retrieves sentences from the upstream audio stream data storage queue and the downstream audio stream data storage queue for sentence recognition, thereby obtaining the sentence text content; wherein, the audio text content includes the sentence text content.
[0085] In this embodiment, the uplink audio stream data is processed sequentially according to a preset processing order. When a certain duration interval is detected in the uplink audio stream data, a complete speech sentence is determined. At this point, a segment of speech from the start of the detected speech to the detected speech interval is defined as a sentence, and this sentence is extracted to obtain a first audio sentence. Similarly, the downlink audio stream data is processed to obtain the corresponding second audio sentence. The obtained first and second audio sentences are stored in different storage areas, such as different storage queues, so that the audio sentences can be quickly retrieved later.
[0086] The second thread retrieves sentences for recognition sequentially from the upstream and downstream audio stream data storage queues according to a preset processing order, based on the acquisition time sequence. For example, sentences are retrieved from the storage queues in a first-in-first-out order to perform speech recognition on the sentences, thereby obtaining the text content corresponding to each sentence.
[0087] For example, when performing speech recognition on audio stream data, the recognized sentence order is A1B1A2A3B2. After sentence segmentation, the data stored in the uplink audio stream data storage queue are A1, A2, and A3, and the data stored in the downlink audio stream data storage queue are B1 and B2. Therefore, when performing speech recognition, the second thread processes the uplink audio stream data first and then the downlink audio stream data. The recognition order should be: perform speech recognition on A1, then on B1, then on A2, then on A3, and finally on B2.
[0088] Based on the foregoing embodiments, in other embodiments of this application, the method further includes the following execution steps:
[0089] Output the statement text content to update the real-time text content displayed on the display device.
[0090] In this embodiment of the application, after speech recognition of each sentence, the corresponding text content of the sentence is immediately output for processing, so as to update the real-time text content displayed on the display device. The reason is that the recognition accuracy of the method to obtain real-time text content is relatively low compared with the accuracy of the sentence recognition method. In this way, it is ensured that the user can see the corresponding voice subtitles in a timely manner, while also ensuring the accuracy of the subtitles seen by the user.
[0091] Based on the foregoing embodiments, in other embodiments of this application, the step of performing voiceprint recognition on audio stream data based on a local voiceprint database to determine the first user identity information corresponding to the audio text content can be implemented by the following steps:
[0092] According to the preset processing order, the sentences to be recognized are obtained sequentially from the upstream audio stream data storage queue and the downstream audio stream data storage queue;
[0093] Extract the voiceprint features of the sentence to be identified to obtain at least one voiceprint feature;
[0094] Store at least one voiceprint feature to the voiceprint storage queue;
[0095] A third thread performs matching processing on the local voiceprint library and voiceprint storage queue to obtain the first user identity information corresponding to each statement to be identified.
[0096] In this embodiment of the application, when the second thread obtains the statement to be recognized from the uplink audio stream data storage queue and the downlink audio stream data storage queue according to a preset processing order for speech recognition, it can also perform voiceprint feature extraction processing on the statement to be recognized to obtain at least one voiceprint feature corresponding to the statement to be recognized, and store the at least one voiceprint feature in the voiceprint storage queue. In this way, the third thread can use the local voiceprint library to perform matching processing on the voiceprint features stored in the voiceprint storage queue, and then obtain the first user identity information corresponding to each statement to be recognized.
[0097] Based on the foregoing embodiments, in other embodiments of this application, the step of matching the local voiceprint database and the voiceprint storage queue through a third thread to obtain the first user identity information corresponding to each statement to be identified can be implemented through the following steps:
[0098] If the third thread determines that the number of voiceprint features included in the voiceprint storage queue exceeds the first number, it performs cluster analysis on all voiceprint features in the voiceprint storage queue to obtain the clustering results.
[0099] The clustering results are matched with the voiceprints in the local voiceprint database to determine the first user identity information that matches the voiceprint features of each clustering result. In this way, the first user identity information corresponding to each statement to be identified is obtained.
[0100] In this embodiment, the first quantity is an empirical value obtained from a large number of experiments, or an empirical value configured based on actual user experience. A third thread counts the number of voiceprint features stored in the voiceprint storage queue. When the count exceeds the first quantity, the third thread retrieves all voiceprint features from the queue and performs cluster analysis on them to obtain clustering results. These clustering results include at least one class of voiceprint features belonging to the same category. It should be noted that after the third thread retrieves all voiceprint features from the queue, the queue is empty to allow for the subsequent storage of new voiceprint features.
[0101] After obtaining the clustering results, the third thread determines the corresponding user identity information based on the clustering results. That is, for each type of voiceprint feature in the clustering results, the corresponding user identity information is determined from the local voiceprint database to obtain the first user identity information. The first user identity information is used to uniquely identify the user. Different users have different voiceprint features. In this way, a correspondence between identity information and voiceprint features can be established for each user.
[0102] Based on the foregoing embodiments, in other embodiments of this application, the display device includes a touch-sensitive display screen of a conference audio device, or a terminal device, or a remote device including at least one participating device that participates in the same conference as the terminal device, or a remote device including at least a cloud device for providing minutes services.
[0103] In this embodiment, when the conference audio device outputs conference log information, it can output it to the touch-sensitive display screen of the conference audio device, thereby displaying the conference log information on the touch-sensitive display screen of the conference audio device. Alternatively, the conference audio device can output the conference log information to a terminal device, so that the conference log information can be displayed on the terminal device. Or, the conference audio device can output the conference log information to the terminal device and simultaneously instruct the terminal device to output the conference log information to the participating devices, so that the participating devices can display or perform subsequent related processing based on the conference log information sent by the terminal device. Or, after the conference audio device outputs the conference log information to the terminal device, it instructs the terminal device to output the conference log information to the participating devices participating in the same conference as the terminal device, so that the participating devices can display the conference log information generated by the conference audio device or perform subsequent related processing on the conference log information. Alternatively, the terminal device can output the received conference log information to a cloud device, so that the cloud device can generate a smart minutes file based on the conference log information for users to view. It should be noted that after the conference audio equipment outputs the conference log information to the terminal device, the terminal device can determine whether to perform subsequent transmission processing based on the user's configuration or related operations; no specific restrictions are made here.
[0104] Based on the foregoing embodiments, in other embodiments of this application, the method further includes the following execution steps:
[0105] Determine the information feedback model;
[0106] If the information feedback mode is the first mode, audio log information is sent to the terminal device so that the terminal device can upload the audio log information to the cloud device, and then the cloud device can generate meeting minutes based on the audio log information; wherein, the cloud device is a remote device;
[0107] If the information feedback mode is the second mode, the audio log information and the corresponding meeting audio information are sent to the terminal device, so that the terminal device can upload the audio log information and the corresponding meeting audio information to the cloud device, and then the cloud device can perform speech recognition based on the meeting audio information to obtain the correction text information, and generate meeting minutes based on the audio log information and the correction text information.
[0108] In this embodiment, the information feedback mode can be the default mode set on the conference audio device, or it can be set by the user after configuring the conference audio device. The specific mode can be determined by the actual situation and is not specifically limited here.
[0109] The information feedback mode can be determined by the operating mode of the audio device. For example, if the audio device is operating in a mode that only generates audio log information, the information feedback mode can only be the first mode. In this scenario, the conference audio device can only send audio log information to the terminal device. If the audio device is operating in a mode that generates audio log information and records the meeting process, the information feedback mode can be determined not only by the audio device's operating mode but also by the user's configuration. If the user configures the information feedback mode to the first mode, the conference audio device will only send the generated audio log information to the terminal device. The terminal device then sends the audio log information to the cloud device. The cloud device can use a high-precision large model capable of generating smart minutes files to generate smart minutes files based on the received audio log information for user viewing. The cloud device can then send the generated smart minutes file to the terminal device for display, or send it from the terminal device to the conference audio device for display on its touchscreen display. If the user-configured information feedback mode is the second mode, the conference audio device will send the generated audio log information and the recorded conference audio information to the terminal device. Furthermore, the terminal device can send the audio log information and conference audio information to the cloud device. In this way, the cloud device performs comprehensive analysis based on the conference audio information and audio log information to generate an intelligent minutes file, ensuring the accuracy of the generated intelligent minutes file.
[0110] Based on the foregoing embodiments, in other embodiments of this application, the method further includes the following execution steps:
[0111] If an update instruction is received from the terminal device, the second user identity information in the local voiceprint database is updated to the third user identity information based on the update instruction; wherein, the update instruction is used to indicate that the second user identity information is updated to the third user identity information, and the third user identification information is obtained by the user modifying the second user identity information displayed on the terminal device;
[0112] Change the second user's identity information in the audio log information to the third user's identity information.
[0113] In this embodiment, if a user discovers during a meeting that the second user identity information displayed in the subtitles on the terminal device does not match the user corresponding to the actual speaking content, the user can modify the second user identity information to the correct third user identity information on the terminal device. Based on the user's operation, the terminal device generates an update instruction instructing the conference audio device to modify the second user identity information in its local voiceprint library to the third user identity information, and sends the update instruction to the conference audio device. This causes the conference audio device to respond to the update instruction and update the second user identity information in its local voiceprint library to the third user identity information, thus achieving automatic maintenance and updating of the local voiceprint library in the conference audio device and ensuring the accuracy and reliability of the information in the local voiceprint library. Furthermore, the conference audio device can also modify the second user identity information in the audio log information generated during the meeting to the third user identity information, ensuring the accuracy of the audio log information.
[0114] Based on the foregoing embodiments, this application provides a specific implementation process for the above-mentioned audio processing method, and the corresponding implementation flow can be as follows: Figure 3 As shown, the conference audio device includes a streaming (online) thread, a non-streaming (offline) thread, and a logging thread. The online thread corresponds to the first thread mentioned above, the offline thread corresponds to the second thread mentioned above, and the logging thread corresponds to the third thread mentioned above. The specific implementation process can be shown in the following steps:
[0115] Step a11: The conference audio equipment acquires the audio stream data transmitted in the audio output transmission channel.
[0116] Step a12: The conference audio equipment splits the received audio stream data into uplink audio stream data and downlink audio stream data.
[0117] When the conference audio equipment splits the audio stream data, it does so according to the transmission direction of the data stream relative to the conference audio equipment. The audio stream data transmitted from the terminal device to the audio equipment is determined as the downlink audio stream data, and the audio stream data transmitted from the audio equipment to the terminal device is determined as the uplink audio stream data.
[0118] Step a13: The conference audio equipment serially inputs the uplink audio stream data and downlink audio stream data to the online thread.
[0119] Step a14: The online thread processes the received uplink and downlink audio stream data in the order they are received. The processing includes two parts: The first part is Voice Activity Detection (VAD), which transmits the detected voice data to Streaming Automatic Speech Recognition (ASR). Streaming ASR extracts language features from the received voice data and then performs streaming decoding to output the streaming text content, which corresponds to the aforementioned real-time text content. The second part is that after VAD detects a complete sentence, it stores the complete sentence in the corresponding uplink or downlink storage queue and notifies the offline thread to process it.
[0120] In the online thread, since the upstream and downstream audio stream data are input serially to the end of the online thread, ASR can be performed through a streaming ASR in the online thread.
[0121] The online thread processes the recognized streaming text content, for example, by directly displaying it as subtitles on the touchscreen of the conference audio equipment. For instance, such as... Figure 4 The diagram illustrates the display of text content output by an online thread on a touch-sensitive screen of a conference audio device. The subtitle display area on the touch-sensitive screen includes two sub-display areas: one sub-display area b1 is the upward subtitle display area, and the other sub-display area b2 is the downward subtitle display area. This allows users to quickly identify which subtitle content corresponds locally to the conference audio device and which is the subtitle content of the audio received by the conference audio device, improving the user experience.
[0122] When VAD recognizes a complete sentence, if it detects a pause in the speech for a preset duration, such as 800 milliseconds (ms), it considers the user to have finished speaking. Therefore, the speech from the start of speech detection to the detection of a pause exceeding the preset duration can be considered a complete sentence. Thus, the audio data of this sentence can be extracted and stored in the corresponding storage queue. For example, if the sentence is uplink audio stream data, the audio data of the sentence is stored in the uplink audio stream data storage queue; if the sentence is downlink audio stream data, the audio data of the sentence is stored in the downlink audio stream data storage queue.
[0123] Step a15: The offline thread responds to the notification from the offline thread, sequentially retrieves the audio statements of the statements to be recognized from the notification storage queue, performs recognition through non-streaming ASR, obtains the text content of the statements, and outputs the text content of the statements.
[0124] The online thread directly outputs the recognized streaming text content to the touchscreen display of the conference audio equipment. When displayed as subtitles, the offline thread also outputs the corresponding sentence text content to the touchscreen display to update the real-time text content output by the online thread. Since the non-streaming ASR method used in the offline thread is usually more accurate than the streaming ASR method used in the online thread, the accuracy of the sentence text content output by the offline thread is higher than that of the real-time text content output by the online thread. However, the online thread takes less time to recognize the text. In this way, the streaming ASR provides the user with subtitle content in real time, while the non-streaming ASR corrects the streaming ASR content, ensuring the accuracy of the subtitle content and improving the user experience.
[0125] Step a16: While the offline thread recognizes the extracted audio sentences through non-streaming ASR, it also extracts the voiceprint features of the audio sentences through the voiceprint extraction module and stores the extracted voiceprint features in the voiceprint storage queue.
[0126] Step a17: The log thread monitors the number of voiceprints stored in the voiceprint storage queue. If the number of voiceprints in the voiceprint storage queue exceeds the preset number, the log thread extracts all voiceprint features in the voiceprint storage queue for cluster analysis. Then, the results of the cluster analysis are matched with the local voiceprint database to determine the user identity information corresponding to each cluster. In this way, the speaker's identity information corresponding to each audio statement can be determined. During this process, the log thread can clear the voiceprint storage queue for the next statistical analysis.
[0127] Step a18: The log thread establishes a relationship between the user identity information corresponding to each audio statement and the statement text content output by the non-streaming ASR, and generates audio log information based on this relationship.
[0128] The audio log information records each moment of speaking, the speaker's user identity information, and the corresponding audio text content.
[0129] After the conference audio device detects the end command, and obtains the final audio log information based on steps a11 to a18, the conference audio device can upload the audio log information to the cloud device according to the end-to-cloud protocol, so that the cloud device can store the uploaded content such as audio log information, or generate efficient intelligent meeting minutes for subsequent processing.
[0130] In some embodiments, the conference audio device only needs to upload the audio log information to the cloud according to the end-to-cloud protocol. In this way, the cloud device uses a pre-trained minutes generation model to intelligently analyze the audio log information and generate intelligent meeting minutes.
[0131] In some embodiments, the conference audio device, according to the end-to-cloud protocol, needs to upload audio log information and the corresponding recorded conference audio to the cloud device. The cloud device can then perform transcription processing based on the audio log information and conference audio to obtain the corresponding cloud-transcribed text content. Then, it uses a pre-trained minutes model to intelligently analyze the cloud-transcribed text content and generate intelligent meeting minutes. Because the cloud device performs cross-referencing transcription processing on the audio log information and conference audio, the accuracy of the cloud-transcribed text content is effectively guaranteed, resulting in more accurate intelligent meeting minutes. It should be noted that when the cloud device performs cross-referencing transcription processing on the audio log information and conference audio, it typically keeps the user identity information in the audio log information unchanged; that is, the cloud device does not modify the user identity information of the corresponding speaker during the transcription process.
[0132] In some embodiments, after the online thread and offline thread output the corresponding text content, or the log thread outputs the audio log information to the terminal device, when the terminal device displays the content on its display interface, if the user modifies the user identity information in the displayed content, the terminal device can directly feed back the modified user identity information to the conference audio device so that the conference audio device can update the user identity information in the local voiceprint library, thus ensuring the accuracy of the local voiceprint library.
[0133] In this way, through the above implementation process, the conference audio equipment achieves the effect of real-time output of uplink and downlink transcribed content, and also realizes dynamic updating of transcribed content to ensure the accuracy of output content; furthermore, after the conference ends, the conference audio equipment can immediately output the text log content of uplink and downlink speakers; moreover, by performing voiceprint recognition locally on the conference audio equipment, the risk of voiceprint privacy being leaked is reduced and the security of voiceprint features is improved.
[0134] Based on the foregoing embodiments, this application provides a system architecture for communication between an audio device with a touch-screen display, a terminal device, a meeting device, and a cloud device capable of intelligently generating minutes. A detailed system architecture diagram can be found in the provided text. Figure 5As shown in the diagram. Participating devices refer to other terminal devices located within the same meeting. During the application, the conference audio device acquires the audio data to be played from the terminal devices and sends the locally acquired audio data back to the terminal devices. The terminal devices decide whether to send the acquired data to the participating devices; that is, the conference audio device does not directly communicate with the participating devices, thus reducing the risk of information leakage at the conference audio device. Communication between the conference audio device and the terminal devices can be established via wired connection or near-field communication (NFC). To ensure information security, during the meeting, the conference audio device is typically configured not to communicate with data communication networks, such as wireless communication networks, or other devices besides the terminal devices. This effectively reduces the possibility of the conference audio device being controlled by malicious actors and having confidential meeting information obtained from it. Terminal devices can decide whether to synchronize the audio text content or audio log information output by the conference audio device to one or more participating devices, and can decide the content to be uploaded to the cloud device according to the protocol with the cloud device. After the cloud device generates the intelligent meeting minutes, it can be sent to the terminal devices and participating devices. The terminal devices can determine whether to send them to the conference audio devices and whether to display the intelligent meeting minutes on the display interface according to the configuration requirements.
[0135] The audio processing method provided in this application acquires audio stream data through a conference audio device, performs speech recognition on the audio stream data locally on the conference audio device to determine the audio text content, and performs voiceprint recognition on the audio stream data in a local voiceprint database to determine the first user identity information corresponding to the audio text content. Finally, based on the first user identity information and the audio text content, audio log information is generated. In this way, by performing speech and voiceprint recognition locally on the audio device and generating audio log information corresponding to the first user identity information of the specific speaking user, the method solves the problem that currently, in the process of generating meeting minutes, it is necessary to upload the user's voiceprint features to the server for voiceprint verification, which leads to the risk of voiceprint leakage. This method proposes a technical solution for performing voiceprint verification locally on the audio device and generating communication logs. Since the user's voice information does not need to be uploaded to the server for voiceprint comparison, the risk of user voice feature leakage is reduced, effectively ensuring user information security.
[0136] Based on the foregoing embodiments, embodiments of this application provide an audio processing apparatus applied to a conference audio device. The conference audio device interacts with a remote device via a terminal device. (Refer to...) Figure 6 As shown, the audio processing device 2 includes at least: an acquisition unit 21 and a recognition unit 22; wherein:
[0137] The acquisition unit 21 is used to acquire audio stream data; wherein, the audio stream data includes uplink audio stream data obtained by the conference audio device from local audio acquisition of its environment, and downlink audio stream data sent by the terminal device;
[0138] The recognition unit 22 is used to perform speech recognition on the uplink audio stream data and downlink audio stream data respectively based on a preset processing order to determine the audio text content.
[0139] The interaction process of other units or modules of the audio processing device can be referred to Figure 1 The implementation process of the audio processing method provided in the corresponding embodiments will not be repeated here.
[0140] Based on the foregoing embodiments, embodiments of this application provide a conference audio device that can be applied to... Figure 1 In the audio processing method provided in the corresponding embodiments, refer to Figure 7 As shown, the conference audio device 3 may include: a microphone 31, a speaker 32, a memory 33, a processor 34, and a communication bus 35; wherein:
[0141] Microphone 31 is used to collect local audio from the environment where the conference audio equipment is located, and obtain uplink audio stream data.
[0142] Speaker 32 is used to play the acquired audio stream data;
[0143] Memory 33 is used to store executable instructions;
[0144] The communication bus 35 is used to realize the communication connection between the microphone 31, the speaker 32, the processor 34 and the memory 33;
[0145] Processor 34 is used to execute information processing programs stored in memory, for implementing, etc. Figure 1 The implementation process of the audio processing method provided in the corresponding embodiments will not be repeated here.
[0146] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium, simply referred to as a storage medium, which stores one or more programs that can be executed by one or more processors to implement the reference. Figure 1 The implementation process of the audio processing method provided in the corresponding embodiments will not be repeated here.
[0147] Based on the foregoing embodiments, this application also provides a computer program product, including a computer program that can be executed by the processor 34 of the conference audio device 3 to complete any of the foregoing method steps.
[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0152] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.
Claims
1. An audio processing method, characterized in that, The method is applied to a conference audio device, which interacts with a remote device via a terminal device. The method includes: Acquire audio stream data; wherein, the audio stream data includes uplink audio stream data obtained by the conference audio device from local audio acquisition of its environment, and downlink audio stream data sent by the terminal device; Based on a preset processing order, speech recognition is performed on the uplink audio stream data and the downlink audio stream data respectively to determine the audio text content.
2. The method according to claim 1, characterized in that, The method further includes: Based on the local voiceprint database, voiceprint recognition is performed on the audio stream data to determine the first user identity information corresponding to the audio text content; Based on the first user identity information and the audio text content, audio log information is generated.
3. The method according to claim 2, characterized in that, The conference audio device is connected to the conference video device, and the method further includes: Receive video feeds captured by the video conferencing equipment; Based on the video footage and the local voiceprint database, the first user identity information corresponding to the audio text content is determined.
4. The method according to claim 1, characterized in that, The process involves performing speech recognition on the upstream and downstream audio stream data based on a preset processing order to obtain audio text content, including: Based on the preset processing order, the uplink audio stream data and the downlink audio stream data are sequentially input into the first thread for real-time audio stream speech recognition, and real-time text content is obtained by serial transmission. The real-time text content is output so that a display device displays the real-time text content; wherein, the audio text content includes the real-time text content.
5. The method according to claim 4, characterized in that, The method further includes: Based on the preset processing order, the uplink audio stream data and downlink audio stream data are segmented into sentences in sequence to obtain at least one first audio sentence corresponding to the uplink audio stream data or at least one second audio sentence corresponding to the downlink audio stream data. Store the at least one first audio statement to the uplink audio stream data storage queue, and store the at least one second audio statement to the downlink audio stream data storage queue; Based on the preset processing order, the second thread sequentially retrieves sentences from the uplink audio stream data storage queue and the downlink audio stream data storage queue for sentence recognition, thereby obtaining the sentence text content; wherein, the audio text content includes the sentence text content.
6. The method according to claim 5, characterized in that, The method further includes: The text content of the statement is output to update the real-time text content displayed on the display device.
7. The method according to claim 2, characterized in that, The step of performing voiceprint recognition on the audio stream data based on a local voiceprint database to determine the first user identity information corresponding to the audio text content includes: According to the preset processing order, the sentences to be recognized are sequentially obtained from the uplink audio stream data storage queue and the downlink audio stream data storage queue; Extract the voiceprint features of the statement to be identified to obtain at least one voiceprint feature; Store the at least one voiceprint feature to the voiceprint storage queue; The local voiceprint library and the voiceprint storage queue are matched by a third thread to obtain the first user identity information corresponding to each statement to be identified.
8. The method according to claim 7, characterized in that, The process of matching the local voiceprint database and the voiceprint storage queue using a third thread to obtain the first user identity information corresponding to each statement to be identified includes: If the third thread determines that the number of voiceprint features included in the voiceprint storage queue exceeds the first number, it performs cluster analysis on all voiceprint features in the voiceprint storage queue to obtain the clustering result. The clustering results are matched with the voiceprints in the local voiceprint database to determine the first user identity information that matches each type of voiceprint feature in the clustering results, thus obtaining the first user identity information corresponding to each statement to be identified.
9. The method according to claim 4 or 6, characterized in that, The display device includes a touch-sensitive display screen of the conference audio device, or the terminal device, or the remote device includes at least one participating device that participates in the same conference as the terminal device, or the remote device includes at least a cloud device for providing minutes services.
10. The method according to any one of claims 2-8, characterized in that, The method further includes: Determine the information feedback model; If the information feedback mode is the first mode, the audio log information is sent to the terminal device, so that the terminal device uploads the audio log information to the cloud device, and then the cloud device generates meeting minutes based on the audio log information; wherein, the cloud device is the remote device; If the information feedback mode is the second mode, the audio log information and the corresponding meeting audio information are sent to the terminal device, so that the terminal device uploads the audio log information and the corresponding meeting audio information to the cloud device, and then the cloud device performs speech recognition based on the meeting audio information to obtain corrected text information, and generates meeting minutes based on the audio log information and the corrected text information.
11. The method according to claim 2, characterized in that, The method further includes: If an update instruction is received from the terminal device, the second user identity information in the local voiceprint database is updated to the third user identity information based on the update instruction; wherein, the update instruction is used to instruct the second user identity information to be updated to the third user identity information, and the third user identity information is obtained by the user modifying the second user identity information displayed on the terminal device; The second user identity information in the audio log information is modified to the third user identity information.
12. A conference audio device, characterized in that, The device includes at least: a microphone, a speaker, a memory, a processor, and a communication bus; wherein: The microphone is used to collect local audio from the environment in which the conference audio equipment is located, and to obtain uplink audio stream data; The speaker is used to play the acquired audio stream data; The memory is used to store executable instructions; The communication bus is used to realize the communication connection between the microphone, the speaker, the processor and the memory; The processor is configured to execute the information processing program stored in the memory to implement the steps of the audio processing method as described in any one of claims 1 to 11.