A control method and processing device for microphone and speaker, and microphone and speaker
By integrating AI chips and large language models in microphone speakers, automatic transcription of voice data and extraction of conference content is achieved, the problem of single functions in the existing technology is solved, meeting efficiency and quality is improved, and efficient intelligent conferences are achieved Record.
Patent Information
- Application Number
- CN202411243063.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-05
AI Technical Summary
The existing microphone speakers have a single function and cannot meet the application scenarios and functional requirements of increasingly rich dialogue human-computer interaction products. It is difficult to achieve efficient voice transcription and automatic assembly of conference content in conference scenarios.
By integrating AI chips into microphone speakers, the voice data of conference users are obtained, converted into text data, and the conference content is extracted using a large language model, meeting minutes are generated, and meeting records and transcription are realized.
It enriches the intelligent functions of microphone speakers, improves meeting efficiency and quality, saves labor and time costs, and realizes fast transcription and efficient recording of meetings.
Smart Images

Figure CN118764762B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent voice technology, and in particular to a control method, a processing device and a microphone speaker. Background Art
[0002] Microphone speakers are audio devices designed for conferences and other scenarios. They integrate the functions of microphones and speakers to provide clear voice input and output and facilitate communication in meetings. Microphone speakers (especially conference microphone speakers) provide significant advantages in improving meeting efficiency and quality with their high-quality audio capabilities, wide coverage, and lightweight design, especially in large conference rooms or multi-participant scenarios.
[0003] However, as the forms of conversational human-computer interaction products become increasingly diverse, the application scenarios and functions implemented are gradually increasing, and the functions of existing microphones and speakers are relatively simple, not smart enough, and insufficient to meet user needs. Summary of the invention
[0004] The embodiments of the present application provide a control method, a processing device and a microphone speaker to at least solve the above-mentioned technical problems in the prior art.
[0005] In a first aspect, an embodiment of the present invention provides a method for controlling a microphone speaker, including:
[0006] While the microphone and speaker are in operation, multiple voice data of conference users are obtained;
[0007] Convert each segment of the voice data into text data to obtain multiple segments of the text data;
[0008] Extracting the meeting content from each of the text data using a large language model to obtain a plurality of the meeting contents;
[0009] After receiving the instruction that the microphone and speaker are in the stopped state, a meeting minutes is generated according to the multiple meeting contents, and the meeting minutes is sent to the terminal device.
[0010] Based on the same inventive concept, in a second aspect, the present invention further provides a processing device to implement the steps of the microphone speaker control method described in the first aspect.
[0011] Based on the same inventive concept, in the third aspect, the present invention also provides a microphone speaker, comprising: an AI chip, and a memory chip connected to the AI chip, a noise reduction microphone array, a power amplifier chip, a Wifi chip, and a speaker connected to the power amplifier chip, wherein the AI chip is used to execute the steps of the microphone speaker control method described in the first aspect.
[0012] Optionally, it also includes: buttons, peripheral interfaces and a power management chip, the AI chip, the memory chip, the noise reduction microphone array, the power amplifier chip and the Wifi chip are all connected to the power management chip, and the buttons and peripheral interfaces are all connected to the AI chip.
[0013] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages:
[0014] The control method of the microphone speaker of the embodiment of the present invention is applied to the microphone speaker, especially the conference microphone speaker, that is, the control method of the embodiment of the present invention is realized by the microphone speaker (especially the AI chip in the microphone speaker). When the microphone speaker is in operation, multiple segments of voice data of the conference users are first obtained. Each segment of the voice data is then converted into text data to obtain multiple segments of the text data, so that the microphone speaker realizes the speech-to-text application. Then, through the large language model, the meeting content is extracted from each segment of the text data to obtain multiple meeting contents, so as to add a large language model to the microphone speaker so that the microphone speaker can automatically sort out the meeting content and meeting minutes. In this way, through the control method of the embodiment of the present invention, the intelligent functions of the microphone speaker are enriched, the rapid transcription of the meeting is realized, the meeting efficiency and quality of the meeting users are improved, and labor costs and time costs are saved. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Also, throughout the accompanying drawings, the same reference figures are used to represent the same components. In the drawings:
[0016] Figure 1 A schematic flow chart showing the steps of a method for controlling a microphone speaker in an embodiment of the present invention is shown;
[0017] Figure 2 A schematic diagram of the structure of an AI chip in an embodiment of the present invention is shown;
[0018] Figure 3 A data flow relationship diagram between the core and the TPU in the AI chip in an embodiment of the present invention is shown;
[0019] Figure 4 A schematic flow chart showing an example of a method for controlling a microphone speaker in an embodiment of the present invention;
[0020] Figure 5 A schematic structural diagram of a microphone speaker in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0021] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0022] Embodiment 1
[0023] The first embodiment of the present invention provides a method for controlling a microphone speaker. Figure 1 As shown, including:
[0024] S101, while the microphone and speaker are in operation, obtaining multiple voice data of conference users;
[0025] S102, converting each segment of voice data into text data to obtain multiple segments of text data;
[0026] S103, extracting the meeting content from each piece of text data through a large language model to obtain multiple meeting contents;
[0027] S104, after receiving the instruction that the microphone and speaker are in the stopped state, generate meeting minutes according to the contents of multiple meetings, and send the meeting minutes to the terminal device.
[0028] The control method of the microphone speaker of the present embodiment is applied to the microphone speaker, especially the conference microphone speaker, that is, the control method of the present embodiment is implemented by the microphone speaker (especially the AI chip in the microphone speaker). When the microphone speaker is in operation, multiple voice data of the conference users are first obtained. Each voice data is then converted into text data to obtain multiple text data, so that the microphone speaker realizes the speech-to-text application. Then, through the large language model, the meeting content is extracted from each text data to obtain multiple meeting contents, so that the large language model is added to the microphone speaker, so that the microphone speaker can automatically sort out the meeting content and meeting minutes. In this way, through the control method of this embodiment, the intelligent functions of the microphone speaker are enriched, the rapid transcription of the meeting is realized, the meeting efficiency and quality of the conference users are improved, and labor costs and time costs are saved.
[0029] Next, combine Figure 1 The specific implementation steps of the microphone speaker control method provided in this embodiment are described in detail:
[0030] First, step S101 is executed to obtain multiple voice data of conference users while the microphone and speaker are in operation.
[0031] Specifically, the microphone speaker being in the running state means that after the microphone speaker is powered on and each module of the microphone speaker is initialized, the microphone speaker is in the working state. The working state can be optionally the working state of connecting to the self-built local area network, that is, the offline working state. While the microphone speaker is in the running state, the conference audio stream of the conference user is first obtained. Then, the conference audio stream is subjected to noise reduction processing through the noise reduction microphone array of the microphone speaker to obtain the conference audio stream after noise reduction. The conference audio stream after noise reduction is segmented and processed to obtain multiple segments of voice data, that is, the conference audio stream after noise reduction is segmented into segments of voice. In this embodiment, the audio stream after noise reduction is segmented into segments of voice, which facilitates the processing of voice and improves processing efficiency and computing speed.
[0032] If the number of conference users and the corresponding conversation content of each conference user need to be counted in the conference content and minutes, it is also necessary to add speech recognition technology to the acquisition of multiple voice data of conference users to identify the audio voiceprint of each conference user, mark it in the conference audio stream and voice data after noise reduction, so that the microphone speaker can clearly record the specific conversation content of the conference user, further improve the efficiency and quality of the meeting, and facilitate the transcription of the meeting. Among them, the speech recognition technology can adopt the Transformer model, the hidden Markov model HMM (Hidden MarkovModels), etc. The specific operation is that after obtaining the conference audio stream after noise reduction, before the conference audio stream after noise reduction is segmented, the audio voiceprint in the conference audio stream after noise reduction is analyzed to obtain the corresponding audio voiceprint identifier in the conference audio stream after noise reduction. Then, the specific process of segmenting the conference audio stream after noise reduction to obtain multiple voice data is to segment the conference audio stream after noise reduction in order to obtain multiple voice data, wherein each voice data segment is voice data carrying at least one corresponding audio voiceprint identifier. The order of segmenting the conference audio stream after noise reduction can be set according to actual needs, and the conference audio stream after noise reduction can be segmented according to the time sequence. The specific implementation form of the audio voiceprint identification can be set according to actual needs.
[0033] For example, suppose there is a conference audio stream with the content "Conference user A said that the meeting has started, and conference user B said that everyone should keep quiet". The conference audio stream is subjected to noise reduction processing through a noise reduction microphone array to obtain a noise-reduced conference audio stream. Then, the audio voiceprint in the noise-reduced conference audio stream is analyzed to obtain the audio voiceprint identifier A0 of A and the audio voiceprint identifier B0 of B. Then, the noise-reduced conference audio stream is divided in sequence to obtain multiple segments of voice data. If the multiple segments of voice data are "The meeting has started" and "Everyone keep quiet", then the voice data of "The meeting has started" carries the audio voiceprint identifier A0, and the voice data of "Everyone keep quiet" carries the audio voiceprint identifier B0. If the multiple segments of voice data are "Everyone in the meeting" and "Keep quiet", then the voice data of "Everyone in the meeting" carries the audio voiceprint identifiers A0 and B0, among which the audio voiceprint identifier of "The meeting has started" is A0, and the audio voiceprint identifier of "Everyone" is B0. The voice data of "Keep quiet" carries the audio voiceprint identifier B0.
[0034] Next, step S102 is executed to convert each segment of voice data into text data to obtain multiple segments of text data.
[0035] Specifically, each segment of voice data is converted into readable text data, thereby obtaining multiple segments of text data. For example, the conference audio stream after noise reduction is segmented and processed to obtain 5 segments of voice data. Each segment of voice data is converted into text data to obtain 5 segments of text data. The order of converting each segment of voice data into readable text data can be set according to actual needs, such as converting voice data into text data in chronological order, or a sequential rule of segmenting first and converting first. For example, the technology of converting voice data to text data adopts open source whisper, wenet or other open source artificial neural network models.
[0036] In this embodiment, voice data is converted into text data to facilitate subsequent large language model processing, so that the large language model can understand the complex language environment and context based on the natural language text of the text data, and output the meeting content and meeting minutes as a whole. In this way, the microphone speaker automatically organizes the meeting minutes, improves the meeting efficiency and quality of the meeting users, facilitates the meeting users to quickly review the meeting points, and subsequently search and quote the meeting content, which is helpful for the discussion and decision-making of the meeting, reduces the burden of the meeting recorder, and greatly reduces the time for manual recording and sorting.
[0037] Then, step S103 is executed to extract the conference content from each piece of text data through the large language model to obtain multiple conference contents.
[0038] Specifically, the Large Language Model of this embodiment is used for reasoning services. It is used to parse user input, understand context, intent and entity, and provide structured information for subsequent processing. According to the meeting content or user request, it automatically generates meeting minutes, answers questions, provides information or suggestions. For example, the large language model uses open source Qwen2, Mistral 7b or other models.
[0039] The AI chip in the microphone speaker implements the control method of this embodiment. Here is an introduction to the structure of the AI chip. Figure 2 As shown, AI chips include: core (such as Figure 2 The A53 core shown in the figure), TPU (Tensor Processing Unit), Global Direct Memory Access Engine (Global DMAEngine), Video Post Processing, Video Decoder, Video Encoder, JPEG Decoder, JPEG Encoder, Security Engine, and some interface modules. For example, the interface LPDDR4 / 4X for connecting DDRSORAM (Double Data Rate Synchronous Dynamic Random Access Memory), the interface SDIO 3.0 for connecting SD card (ie SD Card), etc., will not be repeated here. Figure 2 The AI chip shown has a connection relationship and control relationship between the core and each module, which will not be described here. The AI chip can be selected with a computing power greater than 32TOPS and a memory greater than 8GB.
[0040] Figure 3 This is a diagram of the data flow relationship between the core and TPU in the AI chip. Figure 3As shown in the figure, the HOST processor running in the kernel is connected to the TPU through the underlying driver module driver. The driver module driver acts as a communication bridge, performing handshake, verification and other operations between the two. The HOST processor calls the large language model in the double-rate synchronous dynamic random access memory DDR in the TPU through the TPU-RUNTIME instruction runtime library and the BMLib toolset, and inputs each piece of text data into the DDR of the TPU. Among them, the TPU-RUNTIME instruction runtime library is a programming interface related to the TPU or a series of instruction sets used to control the TPU to perform specific tasks. The BMLib toolset is a toolset for driving the TPU hardware.
[0041] The TPU includes: a microcontroller unit (MCU), on-chip memory LOCAL MEM, DDR, and a dynamic information module dyn_info, a direct memory access module GDMA (Global Direct Memory Access), and an integrated tensor processing module TIU (Tensor Integration unit) connected to the microcontroller unit. The dynamic information module dyn_info is used to record the model data executed by the large language model. The model data includes model execution instructions, as well as intermediate states such as splicing sub-segments, splitting sub-segments, the order of splicing sub-segments, and the order of splitting sub-segments. The direct memory access module GDMA is used to execute the process of moving data in the DDR of the TPU to the on-chip memory LOCAL MEM. The direct memory access management module GDMAPMU is used to manage the direct memory access module GDMA. The direct memory access data module GDMA PMU DATA is used to store the management data of the direct memory access management module GDMA PMU, which includes the timing, amount of data to be moved, and destination of data movement when moving data in the DDR of the TPU to the on-chip memory LOCAL MEM. TIU is used to execute the calculation process of data in the on-chip memory LOCALMEM. The integrated tensor processing management module TIU PMU is used to manage the TIU module. The integrated tensor processing data module TIU PMU DATA is used to store the management data of the TIU PMU, which includes the calculation mode of the data in the on-chip memory LOCAL MEM, the calculation object, and the destination for storing the result output by the calculation object.
[0042] Combination Figure 3 The detailed process of extracting the meeting content from each piece of text data by using a large language model in one embodiment is described as follows:
[0043] For each piece of text data, a. Through the HOST processor, the text data is extracted and stored in the double-rate synchronous dynamic random access memory DDR of the tensor processor TPU.
[0044] b. In each character of the text data, the dynamic information module of the TPU is used to segment the characters into multiple sub-segments in the DDR. It should be noted that the characters are specifically represented in the DDR in the form of a string of numbers, vectors, or matrices. Due to the specific representation of the characters, the characters can be segmented into multiple sub-segments.
[0045] c. In each sub-segment, the sub-segment is moved from the DDR to the on-chip memory of the TPU through the general direct memory access module GDMA of the TPU, and the sub-segment in the on-chip memory is processed by the integrated tensor processing module TIU of the TPU for a preset number of blocks of the large language model to obtain the processed sub-segment, and the processed sub-segment is moved to the DDR, and then multiple processed sub-segments are stored in the DDR.
[0046] d. Through the dynamic information module, multiple processed sub-segments are spliced into text identifiers, and then the identifier of each character in the text data is obtained.
[0047] e. Generate conference content based on the identification of each character in the text data, and transmit and store the conference content in the vector database.
[0048] After performing the above-mentioned operation of generating meeting content for each piece of text data, that is, after executing step ae for each piece of text data, multiple meeting contents are stored in the vector database. It should be noted that one meeting content is generated for each piece of text data. The vector database can quickly realize similarity search of meeting content by vectorizing the features of these meeting contents. The vector data block helps the microphone speaker to better understand the semantics of the meeting content. Through the vectorized expression of different topics, emotions or opinions, the microphone speaker can automatically classify the topics of meeting discussion, or identify key viewpoints and differences in the discussion, providing a basis for post-meeting summary and decision support.
[0049] The specific process of performing block processing of a preset number of blocks of a large language model on the sub-segments in the on-chip memory through the TIU of the TPU to obtain the processed sub-segments is as follows:
[0050] The sub-segment is processed by each block of the large language model in turn to obtain a processed sub-segment; wherein the block processing process of each block is as follows:
[0051] Attention(Q,K,V)=softmax( )V;
[0052] Q represents the mask matrix, Q , K represents the position partial code matrix, K , V represents the vector matrix of the sub-segment, V , Indicates the matrix dimension, for example, m× represents a two-dimensional matrix. If we ignore the softmax activation function, they are actually three , , Multiply the matrices to get a The matrix of The sequence Q is encoded into a new sequence.
[0053] In the process of extracting the meeting content from each paragraph of text data, DMA technology is used to move data, avoiding the CPU processor from moving data, improving the processing speed and process of text data, and reflecting the high efficiency and high precision of the microphone speaker in extracting the meeting content from each paragraph of text data. Through the advantages of high-performance computing, high-bandwidth memory, low power consumption, low latency and flexible deployment of the TPU module, the intelligent functions of the microphone speaker are richer and the execution efficiency is high, which improves the user experience.
[0054] Through the large language model in the AI chip and the data stream processing process in the AI chip, the meeting content is extracted from each piece of text data, so that the microphone speaker can complete the inference of the meeting content and meeting minutes without being connected to the Internet, thereby realizing the large language model inference process and language processing process of the terminal. In this way, not only the intelligent functions of the microphone speaker are enriched, but also the microphone speaker can generate meeting content and meeting minutes offline, so that the microphone speaker can improve the quality of meeting records and provide more valuable functions for enterprises and individual users, thereby improving work efficiency and collaboration effects.
[0055] Next, step S104 is executed, after receiving the instruction that the microphone and speaker are in the stopped state, a meeting minutes is generated according to the contents of the multiple meetings, and the meeting minutes is sent to the terminal device.
[0056] Specifically, multiple meeting contents are extracted from the vector database by sequential query. The sequence can be set according to actual needs, for example, in chronological order. Multiple meeting contents are merged to obtain merged meeting contents. According to the preset prompt words, the merged meeting contents are processed by the large language model to obtain the meeting minutes. The meeting minutes are then sent to the terminal device so that the terminal device displays the meeting minutes. The preset prompt words can be set according to actual needs. A large language model is added to the microphone speaker so that the microphone speaker can automatically sort out the meeting content and meeting minutes. After the meeting minutes are generated, the large language model translates the meeting minutes according to the preset language so that the meeting minutes are translated into the preset language text to meet user needs, improve user experience, and enhance practicality. In this way, through the control method of this embodiment, the intelligent functions of the microphone speaker are enriched, the rapid transcription of the meeting is realized, the meeting efficiency and quality of the meeting users are improved, and labor costs and time costs are saved.
[0057] In addition, the control method of this embodiment can also implement the basic functions of the microphone and speaker, for example, implement the key service function, which processes input events from physical keys. It can be used to trigger different behaviors, such as adjusting the speaker volume or muting the microphone or speaker. The specific process of the key service function is to obtain the input event of the key and perform corresponding control according to the input event. Implement USB Audio (Universal Serial Bus Audio, which refers to the technology of transmitting audio signals through the USB (Universal Serial Bus) interface, that is, USB audio interface) service, which handles USB audio device-related matters, such as recording or playing sound. Implement Web service, which is used for the microphone and speaker to connect to a terminal device through a local area network, so that the meeting minutes generated by the microphone and speaker are displayed on the web page of the display screen of the terminal device.
[0058] The following is an example of the control method of the microphone speaker of this embodiment.
[0059] like Figure 4As shown, when the microphone speaker is in operation, the conference audio stream of the conference user is obtained. The conference audio stream is subjected to noise reduction processing by the noise reduction microphone array to obtain the conference audio stream after noise reduction. Then, the audio voiceprint in the conference audio stream after noise reduction is analyzed to obtain the corresponding audio voiceprint identifier in the conference audio stream after noise reduction. The conference audio stream after noise reduction is divided in sequence to obtain multiple segments of voice data. Then, each segment of voice data is converted into text data through the voice-to-text service to obtain multiple segments of text data. Assume that the text data content converted from the first segment of voice data is "Hello everyone, welcome to the X conference. The theme of today's meeting is how to protect the ecological environment. Next, please let Mr. Li speak. Hello everyone, I am honored to speak on behalf of everyone...", and the content of this text data is represented by chunk. The text "Hello everyone, welcome to the X conference. The theme of today's meeting is how to protect the ecological environment. Next, please let Mr. Li speak." in the text data carries the audio voiceprint identifier A1. The text "Hello everyone, I am honored to speak on behalf of everyone..." carries the audio voiceprint identifier A2.
[0060] According to the prompt words of the meeting content, "You are a meeting recorder, you are holding a meeting, you hear a meeting content, the content is ${chunk}, you count how many people are in the meeting and what this person said", the meeting content is extracted from the chunk. The specific process is that the text data chunk is extracted and stored in the double rate synchronous dynamic random access memory DDR of the tensor processor TPU through the HOST processor. In each word of the chunk, taking the word "big" as an example, the dynamic information module of the TPU divides "big" into multiple sub-segments in the DDR.
[0061] In each sub-segment of "large", the sub-segment is moved from the DDR to the on-chip memory of the TPU through the general direct memory access module GDMA of the TPU, and the sub-segments in the on-chip memory are block processed by the preset number of blocks of the large language model through the integrated tensor processing module TIU of the TPU to obtain the processed sub-segments, and the processed sub-segments are moved to the DDR, and then multiple processed sub-segments of "large" are stored in the DDR.
[0062] Through the dynamic information module, multiple processed sub-segments of "big" are spliced into the "big" character identification. In this way, the identification of each character in the text data chunk is obtained. According to the identification of each character in the text data chunk and the prompt words of the meeting content, the meeting content is generated, and the meeting content is transmitted and stored in the vector database.
[0063] After executing the above-mentioned operation of generating meeting content for each piece of text data, multiple meeting contents are stored in the vector database. After receiving the instruction that the microphone speaker is in the stop state, that is, the end signal, multiple meeting contents are sequentially queried and extracted from the vector database through the large language model. The multiple meeting contents are merged to obtain the merged meeting content. According to the preset prompt words of the meeting minutes, the merged meeting content is processed by the large language model to obtain the meeting minutes. Then the meeting minutes are sent to the terminal device so that the terminal device displays the meeting minutes. Among them, the prompt words of the meeting minutes can be as follows: Figure 4 As shown in the example, "You are a meeting recorder. You are holding a meeting. You hear a meeting content. At ${time1}, you hear ${content}. At ${timeN}, you hear ${contentN}. Now you need to summarize the meeting content of this period. According to the format within #, output the text ###1. Meeting topic 2. Participants 3. Key points 4. Decision items: ###". Each meeting content carries time data. The meeting content record is represented by contentN, and the time data of the meeting content is represented by timeN. N represents the number of the meeting content.
[0064] Through the control method of the microphone speaker of this embodiment, the intelligent functions of the microphone speaker are enriched, the rapid transcription of the meeting is realized, the meeting efficiency and quality of the meeting users are improved, and the labor cost and time cost are saved. During the use of the microphone speaker, new information is absorbed and dynamically adjusted to better adapt to user needs and language evolution. The microphone speaker uses advanced artificial intelligence, especially large language models, to enhance its ability to understand and generate natural language, so that it gradually adapts to the needs of specific users or scenarios, provides personalized experience, and realizes more autonomous, flexible and humanized service and interaction mode.
[0065] Embodiment 2
[0066] Based on the same inventive concept, the third embodiment of the present invention further provides a processing device to implement the steps of the microphone and speaker control method as described in the first embodiment.
[0067] The processing device of this embodiment may be a chip, such as an IA chip, or may be other devices or apparatuses, such as a conference assistant agent.
[0068] Embodiment 2
[0069] Based on the same inventive concept, the second embodiment of the present invention further provides a microphone speaker, such as Figure 5As shown, it includes: an AI chip, a memory chip connected to the AI chip, a noise reduction microphone array, an amplifier chip, a Wifi chip, and a speaker connected to the amplifier chip, wherein the AI chip is used to execute the steps of the microphone speaker control method of embodiment 1. The speaker can be selected as a high-fidelity speaker.
[0070] As an optional embodiment, the microphone speaker also includes: buttons, peripheral interfaces and a power management chip. The AI chip, memory chip, noise reduction microphone array, power amplifier chip and Wifi chip are all connected to the power management chip, and the buttons and peripheral interfaces are all connected to the AI chip.
[0071] Specifically, the power management chip is used to provide power signals to each module of the microphone and speaker. The memory chip is used to store the model data, intermediate data and program code of the microphone and speaker for the processor to read and execute. The capacity of the memory module can be set according to actual needs. The power amplifier chip is used to convert the digital audio signal into an analog signal, amplify the output and send it to the speaker for sound. The noise reduction microphone array is used to achieve audio signal noise reduction, eliminate background noise, venue echo, automatic voice gain, and improve the quality of conference audio through a multi-microphone array. The Wifi chip is used to establish a Wifi hotspot, provide connections, and provide users with a Web operation interface. The AI chip is used to execute LLM algorithms, speech-to-text algorithms, and web services. The peripheral interface is a USBAudio service interface, which is used to handle transactions related to USB audio devices, such as recording or playing sound.
[0072] The microphone speaker extracts the meeting content and meeting minutes from each piece of text data through the large language model in the AI chip and the data stream processing process in the AI chip, so that the microphone speaker can complete the reasoning of the meeting content and meeting minutes without being connected to the Internet, thereby realizing the large language model reasoning process and language processing process of the terminal. In this way, not only the intelligent functions of the microphone speaker are enriched, but also the microphone speaker can process and generate meeting content and meeting minutes offline, so that the microphone speaker can improve the quality of meeting records and provide more valuable functions for enterprises and individual users, thereby improving work efficiency and collaboration effects.
[0073] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, microphone speakers, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0075] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0077] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0078] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for controlling a microphone speaker, characterized in that: include: While the microphone and speaker are in operation, multiple voice data of conference users are obtained; Convert each segment of the voice data into text data to obtain multiple segments of the text data; By using a large language model, the conference content is extracted from each piece of text data to obtain a plurality of conference contents, including: In each segment of the text data, the text data is extracted and stored in the double rate synchronous dynamic random access memory DDR of the tensor processor TPU through the HOST processor; In each character of the text data, the character is segmented into a plurality of sub-segments in the DDR by a dynamic information module of the TPU; In each of the sub-segments, the sub-segments are moved from the DDR to the on-chip memory of the TPU through the general direct memory access module GDMA of the TPU, the sub-segments in the on-chip memory are subjected to block processing of a preset number of blocks of the large language model through the integrated tensor processing module TIU of the TPU to obtain processed sub-segments, and the processed sub-segments are moved to the DDR, and then a plurality of the processed sub-segments are stored in the DDR; By means of the dynamic information module, the plurality of processed sub-segments are spliced into the identifier of the character, thereby obtaining the identifier of each character in the character data; Generate the conference content according to the identifier of each character in the text data, and transmit and store the conference content in a vector database; After performing the above-mentioned operation of generating conference content on each piece of text data, a plurality of conference contents are stored in the vector database; After receiving the instruction that the microphone and speaker are in the stopped state, a meeting minutes is generated according to the multiple meeting contents, and the meeting minutes is sent to the terminal device.
2. The method according to claim 1, characterized in that The step of performing block processing of a preset number of blocks of the large language model on the sub-segments in the on-chip memory by the integrated tensor processing module TIU of the TPU to obtain processed sub-segments includes: The sub-segment is sequentially processed by each of the blocks of the large language model to obtain the processed sub-segment; wherein the block processing process of each of the blocks is as follows: Attention(Q,K,V)=softmax( )V; Among them, Q represents the mask matrix, Q , K represents the position encoding matrix, K , V represents the vector matrix of the sub-segment, V , Represents the matrix dimensions.
3. The method according to claim 1, characterized in that Generating meeting minutes according to the plurality of meeting contents includes: Extracting a plurality of the conference contents from the vector database in sequence; Merging the plurality of conference contents to obtain merged conference content; According to the preset prompt words, the merged meeting content is processed by the large language model to obtain the meeting minutes.
4. The method according to claim 1, characterized in that The step of obtaining multiple voice data of conference users includes: Obtaining a conference audio stream of the conference user; The conference audio stream is subjected to noise reduction processing by a noise reduction microphone array to obtain a noise-reduced conference audio stream; The conference audio stream after noise reduction is segmented to obtain multiple segments of voice data.
5. The method according to claim 4, characterized in that After obtaining the conference audio stream after noise reduction and before segmenting the conference audio stream after noise reduction, the method further includes: By analyzing the audio voiceprint in the conference audio stream after noise reduction, the corresponding audio voiceprint identifier in the conference audio stream after noise reduction is obtained, so as to count the number of conference users and the corresponding conversation content.
6. The method according to claim 5, characterized in that The segmenting of the conference audio stream after noise reduction to obtain multiple segments of voice data includes: The conference audio stream after noise reduction is segmented in sequence to obtain a plurality of segments of voice data, wherein each segment of voice data is voice data carrying at least one corresponding audio voiceprint identifier.
7. A processing device, characterized in that: Implement the steps of the microphone speaker control method as described in any one of claims 1-6.
8. A microphone speaker, characterized in that: include: An AI chip, and a memory chip, a noise reduction microphone array, a power amplifier chip, a Wifi chip connected to the AI chip, and a speaker connected to the power amplifier chip, wherein the AI chip is used to execute the steps of the microphone and speaker control method as described in any one of claims 1 to 6.
9. The microphone speaker according to claim 8, characterized in that: Also includes: Buttons, peripheral interfaces and a power management chip, the AI chip, the memory chip, the noise reduction microphone array, the power amplifier chip and the Wifi chip are all connected to the power management chip, and the buttons and peripheral interfaces are all connected to the AI chip.
