Audio recognition method and electronic device

Abstract

The application discloses an audio recognition method and an electronic device, and the method comprises the following steps: calling an AudioWorklet component to collect audio data, and alternately writing the audio data into at least two webassembly buffer areas of a shared memory; calling a webassembly component to read the audio data from the at least two webassembly buffer areas, performing slicing processing on the audio data, and storing a plurality of audio segments obtained in the shared memory; calling a webworker component to read the plurality of audio segments from the shared memory, converting valid segments in the plurality of audio segments into segment texts in parallel, assigning an ID to each segment text, and storing the ID and the segment text in the shared memory; and calling the webassembly component to splice the segment texts into a text string according to the corresponding ID. The audio recognition method provided by the application can realize efficient audio recognition.

Description

Technical Field

[0001] This application relates to the field of audio recognition technology, and in particular to an audio recognition method and electronic device. Background Technology

[0002] Natural Language Processing (NLP) is an important research area in the field of artificial intelligence, aiming to enable machines to understand, interpret, and generate human language.

[0003] Among related technologies, speech recognition solutions using NLP in a Web environment suffer from technical bottlenecks such as high latency, main thread blocking, and low computational efficiency. Summary of the Invention

[0004] This application provides an audio recognition method and electronic device to solve the problems of high latency, main thread blocking, and low computational efficiency in Web speech recognition schemes.

[0005] In a first aspect, embodiments of this application provide an audio recognition method, the method comprising:

[0006] Call the Audio Worklet component to collect audio data and write the audio data alternately to at least two webassembly buffers in shared memory;

[0007] The webassembly component is invoked to read audio data from at least two webassembly buffers. Based on the audio energy of each audio frame in the audio data, the audio data is sliced ​​to obtain multiple audio segments, and the multiple audio segments are stored in shared memory.

[0008] The webworker component is invoked to read multiple audio segments from shared memory, and the valid segments from the multiple audio segments are converted into segment text in parallel. Each segment text is assigned an ID, and the ID and segment text are stored together in shared memory.

[0009] The webassembly component is invoked to concatenate the read fragment text into a text string according to the corresponding ID.

[0010] The audio recognition method provided in this application uses the Audio Worklet component to collect audio data. Since the Audio Worklet component is an audio processing API provided by the web, it can process audio data in real time in an independent thread, avoiding blocking of the main thread, thereby achieving low-latency audio input / output. The audio data is alternately written to at least two webassembly buffers in shared memory, achieving fast audio data writing while avoiding stuttering. In addition, this application uses shared memory to improve the data transmission efficiency during audio recognition. Since the webassembly component is called to perform segmentation processing on the audio data, and the webassembly component has efficient instruction set compilation characteristics, it can provide near-native computing performance, which can accelerate the segmentation processing of audio data. This application combines the above-mentioned efficiency optimizations in audio data acquisition, transmission, and processing to achieve high-efficiency audio recognition, which can bring a good user experience in real-time voice interaction application scenarios.

[0011] Optionally, after calling the Audio Worklet component to acquire audio data, the method also includes:

[0012] The number of frame samples in the audio data is actively obtained through the onMessage method of the Audio Worklet component; the number of frame samples is sent to the webassembly component by calling the postMessage method of the Audio Worklet component, so that the webassembly component can determine the number of segments when performing segmentation processing on the audio data.

[0013] Based on this, the audio recognition method provided in this application embodiment can obtain the number of frame samples of audio data based on the Audio Worklet component and send the number of frame samples to the webassembly component, so that the webassembly component can calculate the audio energy based on the number of frame samples, thereby determining the number of segments when slicing the audio data based on the audio energy.

[0014] Optionally, audio data is alternately written to at least two webassembly buffers in shared memory, including:

[0015] Audio data collected by the Audio Worklet component is received through the browser's global Audio Context object;

[0016] The audio data’s different phonemes are alternately written to at least two webassembly buffers via the postMessage function in the Audio Worklet component.

[0017] And / or,

[0018] Create at least two webassembly buffers in shared memory. Write audio data to any one of the at least two webassembly buffers using postMessage from the Audio Worklet component. When that buffer is full, write the audio data to the other of the at least two webassembly buffers.

[0019] Based on this, by using `postMessage`, phonemes from the audio data are alternately written to at least two WebAssembly buffers. This allows the WebAssembly and WebWorker components to easily read data from shared memory, improving data transfer efficiency and potentially increasing the processing speed of the audio recognition method. Furthermore, audio data can be written to another buffer only after one buffer is full, reducing the frequency of buffer switching and thus lowering system overhead.

[0020] Optionally, audio data is alternately written to at least two webassembly buffers in shared memory, including:

[0021] Audio data collected by the Audio Worklet component is received through the browser's global Audio Context object;

[0022] The average length of an existing audio segment is obtained through the onMessage method of the Audio Worklet component, and then the average length of the segment is sent to postMessage.

[0023] The audio data is alternately written to at least two webassembly buffers using the postMessage function in the Audio Worklet component, with the average segment length being the single write length of the current audio data.

[0024] Based on this, when writing audio data alternately to at least two webassembly buffers, the average length of the existing audio segments is used as the single write length of the current audio data. In this way, when the webassembly component performs audio data slicing, it only needs to make minor adjustments to the audio data to complete the slicing work, which reduces the workload of the webassembly component in audio data slicing and can improve the processing speed of the audio recognition method to a certain extent.

[0025] Optionally, the audio data is sliced ​​based on the audio energy of each audio frame, including:

[0026] For each audio frame in the audio data, the mean square energy of the audio frame is calculated based on the sample value of each audio frame, and the absolute value of the energy corresponding to the no speech type is determined based on the change of the mean square energy.

[0027] The starting point when the mean square energy of an audio frame changes from no speech type to other types, and the ending point when the mean square energy of an audio frame changes from other types to no speech type, are marked as endpoints.

[0028] The audio data is sliced ​​using the endpoints as cutting points.

[0029] Based on this, mean square energy can be used to mark the start and end points of valid speech segments (i.e., the start point when the mean square energy of an audio frame changes from no speech type to other types, and the end point when the mean square energy of an audio frame changes from other types to no speech type). This allows the part of the audio data that corresponds to no speech type (such as silence or only background noise) to be separated from the valid speech segments. In subsequent audio recognition, only valid speech segments can be recognized, reducing the workload of audio recognition and improving the processing efficiency of audio recognition.

[0030] Optionally, the webworker component is invoked to read multiple audio segments from shared memory, and the valid segments from the audio segments are converted into segment text in parallel, including:

[0031] The webworker component reads the segments to be converted from multiple audio segments in shared memory, and uses the STT (Speech-to-Text) library to determine whether the segment to be converted is a valid segment based on the audio energy of the segment to be converted.

[0032] The webworker component creates at least two worker child threads and distributes multiple valid fragments to at least two worker child threads;

[0033] Call at least two worker child threads in parallel to convert multiple valid fragments into multiple fragment texts.

[0034] Based on this, the audio recognition method provided in this application creates at least two worker sub-threads through the webworker component, and performs text conversion on effective segments in parallel through multiple threads, thereby improving the processing efficiency of audio-to-text conversion.

[0035] Optionally, the webworker component creates at least two worker child threads and distributes multiple valid fragments to at least two worker child threads, including:

[0036] After identifying 8 valid segments, the webworker component distributes these 8 valid segments as an audio array to the currently idle worker child threads.

[0037] The worker thread that receives the audio array converts the audio array into a text array, stores the text array along with the IDs of the corresponding 8 valid segments into shared memory, and then changes the current state of the worker thread to idle.

[0038] Based on this, the audio recognition method provided in this application stores eight valid segments as an audio array in shared memory. By marking the status of worker sub-threads, idle worker sub-threads are used cyclically to perform text conversion on different audio arrays to obtain text arrays. This achieves efficient and parallel text conversion of different audio arrays while consuming less system memory resources, thus improving the processing efficiency of audio-to-text conversion.

[0039] Optionally, the webassembly component is invoked to concatenate the read fragment text into a text string according to the corresponding ID, including:

[0040] The webassembly component reads a text array from shared memory, which contains 8 text fragments. The text fragments in the text array are concatenated into a text string according to their ID order.

[0041] The natural language processing model is invoked to optimize the text string. The optimization process includes at least one of the following: syntax correction, semantic coherence checking, deduplication, and word order adjustment.

[0042] Based on this, the audio recognition method provided in this application optimizes the concatenated text string using a natural language processing model, making the audio recognition results more accurate.

[0043] Optionally, after optimizing the text string, the method also includes:

[0044] Display the optimized text string on the UI page.

[0045] Based on this, the audio recognition method provided in this application can display the text obtained by audio recognition and conversion on the UI page, realizing the effect of human-computer interaction. This method can be applied to web application scenarios that require real-time voice interaction, such as online conferencing systems, browser-based voice assistants, instant caption generation, and voice control interfaces of IoT devices, providing users with a good voice recognition service experience.

[0046] Secondly, embodiments of this application provide an electronic device, including: at least one memory and at least one processor, wherein the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-described audio recognition method.

[0047] The audio recognition method provided in this application uses the Audio Worklet component to collect audio data. Since the Audio Worklet component is an audio processing API provided by the web, it can process audio data in real time in an independent thread, avoiding blocking of the main thread, thereby achieving low-latency audio input / output. The audio data is alternately written to at least two webassembly buffers in shared memory, achieving fast audio data writing while avoiding stuttering. In addition, this application uses shared memory to improve the data transmission efficiency during audio recognition. Since the webassembly component is called to perform segmentation processing on the audio data, and the webassembly component has efficient instruction set compilation characteristics, it can provide near-native computing performance, which can accelerate the segmentation processing of audio data. This application combines the above-mentioned efficiency optimizations in audio data acquisition, transmission, and processing to achieve high-efficiency audio recognition, which can bring a good user experience in real-time voice interaction application scenarios. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a web-based audio recognition architecture provided in one embodiment of this application;

[0049] Figure 2 This is a flowchart of an audio recognition method provided in one embodiment of this application;

[0050] Figure 3 A schematic diagram illustrating the transmission of frame sample numbers by an Audio Worklet component according to an embodiment of this application;

[0051] Figure 4 A schematic diagram illustrating the connection relationship between the Audio Worklet component and the browser's global AudioContext object, provided in one embodiment of this application;

[0052] Figure 5 A flowchart illustrating the alternating writing of audio data to at least two shared memory locations is provided as one embodiment of this application.

[0053] Figure 6 This application provides a flowchart of one embodiment of distributing multiple valid fragments to two worker sub-threads;

[0054] Figure 7This is a schematic diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0055] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0056] This application provides an audio recognition method, which includes: calling an Audio Worklet component to collect audio data, and alternately writing the audio data into at least two webassembly buffers in shared memory;

[0057] The webassembly component is invoked to read audio data from at least two webassembly buffers. Based on the audio energy of each audio frame in the audio data, the audio data is sliced ​​to obtain multiple audio segments, and the multiple audio segments are stored in shared memory.

[0058] The webworker component is invoked to read multiple audio segments from shared memory, and the valid segments from the multiple audio segments are converted into segment text in parallel. Each segment text is assigned an ID, and the ID and segment text are stored together in shared memory.

[0059] The webassembly component is invoked to concatenate the read fragment text into a text string according to the corresponding ID.

[0060] Specifically, such as Figure 1 As shown, it illustrates a schematic diagram of a web-based audio recognition architecture provided in one embodiment of this application. Figure 1 As shown, the web-based audio recognition architecture includes an Audio Worklet component, a webassembly component, a webworker component, and shared memory. The shared memory includes at least two webassembly buffers. The Audio Worklet component provides an audio processing API for the browser, allowing real-time processing of the audio stream in a separate thread, avoiding blocking the main thread and achieving low-latency audio input / output. The webassembly component uses a binary instruction format that can be efficiently executed in a browser (Web), providing near-native computational performance and accelerating the inference process of the speech recognition model. The webworker component is used to create independent threads for execution in the Web, avoiding blocking the main thread and improving application performance and responsiveness.

[0061] like Figure 1As shown, the Audio Worklet component establishes a communication connection with shared memory, enabling the audio data collected by the Audio Worklet component to be stored in the webassembly buffer of shared memory. The web-based audio recognition architecture provided in this embodiment includes at least two shared memory webassembly buffers. The Audio Worklet component alternately stores audio data into different webassembly buffers to prevent stuttering during audio data storage. Both the webassembly component and the webworker component establish communication connections with each shared memory instance to facilitate data retrieval from the shared memory webassembly buffers in subsequent audio recognition steps, eliminating the need for parameter passing via API calls and improving data transfer efficiency during audio recognition.

[0062] Based on such Figure 1 The web-based audio recognition architecture shown is as follows: Figure 2 As shown, this application provides an audio recognition method to achieve highly efficient audio-to-text conversion. The method includes the following steps:

[0063] The Audio Worklet component collects audio data and alternately writes the audio data to at least two webassembly buffers in shared memory.

[0064] The webassembly component reads audio data from at least two webassembly buffers, slices the audio data to obtain multiple audio segments, and stores these multiple audio segments into at least two webassembly buffers. As an optional implementation, when slicing the audio data, the audio frames can be sliced ​​based on their audio energy; the specific slicing method will be described later.

[0065] The webworker component reads multiple audio segments from shared memory, converts the valid segments into segment text, assigns an ID to each segment text, and stores the segment text and its ID together in shared memory.

[0066] The ID of a text segment indicates the temporal position of the corresponding audio segment in the audio data. The webassembly component reads the text segment and its ID, and then concatenates the text segment into a string based on the ID, thus realizing natural language processing for audio-to-text conversion.

[0067] In the audio recognition method provided in this application embodiment, shared memory is used to improve the data transmission efficiency during the audio recognition process. Furthermore, the method calls the Audio Worklet component to acquire audio data. The Audio Worklet component is an audio processing API provided by the web client, capable of processing audio data in real time in an independent thread, avoiding main thread blocking and thus achieving low-latency audio input / output. In addition, the method also calls the WebAssembly component to slice the audio data. The WebAssembly component has efficient instruction set compilation characteristics, providing near-native computational performance and accelerating the audio data slicing process. Through these efficiency optimizations in audio data acquisition, transmission, and processing, the audio recognition method provided in this application achieves high-efficiency audio recognition. Using this method, a better user experience can be provided in real-time voice interaction application scenarios.

[0068] As an optional implementation, after calling the Audio Worklet component to collect audio data, the method also includes:

[0069] The number of frame samples in the audio data is actively obtained through the onMessage method of the Audio Worklet component; the number of frame samples is then sent to the webassembly component so that the webassembly component can determine the number of segments when performing segmentation processing on the audio data.

[0070] Specifically, such as Figure 3 The diagram illustrates how the Audio Worklet component transmits the number of frame samples. As shown, the Audio Worklet component uses `postMessage` and `onMessage` to transmit the number of frame samples. `postMessage` and `onMessage` are methods for implementing message passing; `postMessage` represents the sender, and `onMessage` represents the receiver.

[0071] The frame sample count refers to the number of samples in an audio frame. After the frame sample count is set in the main thread of the web application, the main thread sends the frame sample count to the Audio Worklet component via postMessage. After receiving the frame sample count via onMessage, the Audio Worklet component then passes the frame sample count to the webassembly component via postMessage. The webassembly component receives the frame sample count via onMessage within its own component.

[0072] Based on this, in the audio recognition method provided by the embodiments of the present application, the number of frame samples of audio data can be obtained based on the Audio Worklet component, and the number of frame samples is sent to the webassembly component, so that the webassembly component can calculate the audio energy of each audio frame according to the number of frame samples, thereby determining the number of segments when slicing the audio data according to the audio energy.

[0073] As an optional implementation manner, alternately writing audio data into at least 2 webassembly buffers of the shared memory comprises:

[0074] receiving the audio data collected by the Audio Worklet component through the browser-global Audio Context object;

[0075] alternately writing different phonemes in the audio data into at least 2 webassembly buffers through postMessage;

[0076] and / or,

[0077] creating at least 2 webassembly buffers in the shared memory, writing the audio data into any one of the at least 2 webassembly buffers through postMessage in the Audio Worklet component, and when the buffer is full, writing the audio data into another one of the at least 2 webassembly buffers.

[0078] Specifically, a phoneme is the smallest speech unit divided according to the natural attributes of speech. In Chinese speech recognition applications, phonemes can be understood as pinyin representations of pronunciations. For example, the Chinese character "中" corresponds to "zhong1", which can be regarded as a valid syllable phoneme or a phoneme sequence, and "zhong1" includes 6 phonemes.

[0079] as shown in Figure 4 , an exemplary schematic diagram of the connection relationship between the Audio Worklet component and the browser-global AudioContext object is provided. Based on the connection relationship between the Audio Worklet component and the browser-global AudioContext object, in the audio recognition method provided by the embodiments of the present application, the Audio Worklet component can alternately write different phonemes in the audio data into at least 2 webassembly buffers of the shared memory through postMessage.

[0080] As an alternative implementation, when writing different phonemes alternately to different shared memories, two or more phonemes that are sequentially adjacent can be written to the first shared memory. After writing a preset number of phonemes to the first shared memory, subsequent phonemes are written to the second shared memory, and so on, alternating between the first and second shared memories. For example, after one of at least two webassembly buffers is full, subsequent audio data can be written to the other of the at least two webassembly buffers. This method reduces the frequency of buffer switching, thereby reducing system overhead.

[0081] Based on the above description, the audio recognition method provided in this application embodiment uses the postMessage and onMessage methods to alternately write the phonemes in the audio data collected by the Audio Worklet component into different shared memories. This allows the webassembly component and webworker component to easily read data from the shared memory, improving the efficiency of data transmission and increasing the processing speed of the audio recognition method to a certain extent.

[0082] As an optional implementation, the audio recognition method provided in this application embodiment alternately writes audio data to at least two webassembly buffers in shared memory, including:

[0083] Audio data collected by the Audio Worklet component is received through the browser's global Audio Context object;

[0084] The average length of the segments processed by webassembly for slicing existing audio segments is obtained through the onMessage method of the Audio Worklet component, and the average length of the segments is sent to postMessage.

[0085] The audio data is alternately written to at least two webassembly buffers in shared memory using the postMessage method of the Audio Worklet component, with the average fragment length being the single write length of the current audio data.

[0086] Specifically, message passing between the Audio Context object and the WebAssembly component is implemented through the `postMessage` and `onMessage` methods. For example... Figure 5As shown, the Audio Context object, acting as the message receiver, receives audio data sent by the Audio Worklet component and the average segment length sent by the webassembly component via its onMessage method. The average segment length can be calculated by the webassembly component based on the average length of existing audio segments. After receiving the audio data and the average segment length, the Audio Context object, acting as the message sender, uses postMessage to alternately write the audio data to at least two webassembly buffers in shared memory, according to the single-write length of the average segment length. Based on this, when alternately writing audio data to at least two webassembly buffers in shared memory, the average segment length of existing audio segment processing is used as the single-write length of the current audio data. In this way, when the webassembly component performs segmentation processing on the audio data subsequently, it only needs to make minor adjustments to the audio data to complete the segmentation, reducing the workload of the webassembly component in audio data segmentation processing and thus improving the processing speed of the audio recognition method to a certain extent.

[0087] As an optional implementation, existing audio segments are sliced ​​based on the audio energy of each audio frame in the audio data, including:

[0088] For each audio frame in the audio data, the mean square energy of each audio frame is calculated, and the absolute value of the energy corresponding to the no speech type is determined based on the change of the mean square energy.

[0089] The starting point when the mean square energy of an audio frame changes from no speech type to other types, and the ending point when the mean square energy of an audio frame changes from other types to no speech type, are marked as endpoints.

[0090] The audio data is sliced ​​using the endpoints as cutting points.

[0091] Specifically, the mean square energy of each audio frame can be calculated using the following formula:

[0092] E=(1 / N)*Σ(x[i]^2)(i=0 to N-1);

[0093] In the formula, x represents a sample array of an audio frame in the audio data, x[i] is the sample value of the sampling point in the audio frame sample array (usually a floating-point number in [-1,1]), and N is the length of the audio frame sample array (i.e., the number of frame samples).

[0094] Based on the absolute value of the mean square energy of the calculated audio frame, we can refer to the following table of the correspondence between the mean square energy value range and the audio type to determine which of the calculated absolute values ​​of the mean square energy of the audio frame correspond to the absolute values ​​of the energy without speech type, and thus determine the position of the invalid segment in the audio frame.

[0095] The table below shows the correspondence between mean square energy range and audio type:

[0096]

[0097] In this embodiment of the application, the mean square energy absolute value in the case of silence or only background noise is defined as the energy absolute value without speech type. When the mean square energy absolute value calculated by two adjacent audio sample arrays in an audio frame changes from lower than the energy absolute value without speech type to higher than it, it indicates that the audio data in the audio frame has changed from no speech type to other types. The starting phoneme of the latter audio sample array is marked as the endpoint, and the audio frame is sliced ​​using the endpoint as the cutting point.

[0098] For example, suppose we acquire audio data with 10 audio frames. This audio data can be represented as follows:

[0099] const audio Frames=[new Float32Array([0.03,-0.02,0.01,-0.03,0.02]),new Float32Array([0.02,0.01,-0.02,0.01,-0.02]),new Float32Array([0.05,-0.04,0.03,-0.05,0.04]),new Float32Array([0.07,-0.05,0.03,-0.06,0.04]),newFloat32Array([0.08,-0.07,0.05,-0.06,0.05]),new Float32Array([0.1,-0.08,0.06,-0.07,0.05]),new Float32Array([0.14,-0.12,0.08,-0.1,0.08]),new Float32Array([0.12,-0.1,0.07,-0.09,0.06]),new Float32Array([0.1,-0.08,0.06,-0.07,0.05]),new Float32Array([0.01,-0.005,0.008,-0.006,0.004])];

[0100] Here, `const audio Frames` represents audio data, and `new Float32Array` represents a sample array of audio frames.

[0101] The mean square energy (MSE) results for each frame are: [0.00054, 0.00028, 0.00182, 0.0027, 0.00398, 0.00548, 0.0100928, 0.0082, 0.00548, 0.0000482]. According to the MSE results, the MSE energy values ​​exceed the threshold of 0.001 (i.e., the energy range during silence / background noise) at the 3rd and 9th audio frames, respectively. The audio type corresponding to the value changes from noise to normal speech in the 3rd audio frame, and then becomes noise again after the 9th audio frame. Therefore, the 3rd and 9th audio frames can be used as endpoints to divide the original audio data into three audio segments: [0.00054, 0.00028], [0.00182, 0.0027, 0.00398, 0.00548, 0.0100928, 0.0082, 0.00548], and [0.0000482].

[0102] Based on this, mean square energy can be used to mark the start and end points of valid speech segments (i.e., the start point when the mean square energy of an audio frame changes from no speech type to other types, and the end point when the mean square energy of an audio frame changes from other types to no speech type). This allows the part of the audio data that corresponds to no speech type (such as silence or only background noise) to be separated from the valid speech segments. In subsequent audio recognition, only valid speech segments can be recognized, reducing the workload of audio recognition and improving the processing efficiency of audio recognition.

[0103] As an optional implementation, the webworker component is invoked to read multiple audio segments from shared memory and convert the valid segments in the audio array into segment text in parallel, including:

[0104] The webworker component reads the segments to be converted from multiple audio segments in shared memory, and determines whether the segment to be converted is a valid segment based on the STT speech-to-text library and the audio energy of the segment to be converted.

[0105] The webworker component creates at least two worker child threads and distributes multiple valid fragments to at least two worker child threads;

[0106] Call at least two worker child threads in parallel to convert multiple valid fragments into multiple fragment texts.

[0107] Specifically, in this embodiment, audio sample arrays with mean square energy values ​​within the range of silence / background noise are considered invalid segments, while those outside this range are considered valid segments. The audio recognition method provided in this embodiment creates at least two worker threads using a webworker component, enabling parallel text conversion of valid segments. This improves the efficiency of audio-to-text conversion, and further efficiency can be enhanced by setting more worker threads.

[0108] As an optional implementation, in the audio recognition method provided in this application embodiment, the webworker component creates at least two worker child threads and distributes multiple valid segments to at least two worker child threads, including:

[0109] After identifying 8 valid segments, the webworker component distributes these 8 valid segments as an audio array to the currently idle worker child threads.

[0110] The worker thread that receives the audio array converts the audio array into a text array, stores the text array along with the IDs of the corresponding 8 valid segments into shared memory, and then changes the current state of the worker thread to idle.

[0111] Specifically, the number of segments in the audio array can be dynamically adjusted according to the client configuration. In the audio recognition method provided in this application embodiment, eight valid segments are stored in a shared memory as an audio array, and the creation of two worker child threads is used as an example for illustration.

[0112] like Figure 6 As shown, distributing multiple valid fragments to two worker child threads includes the following steps:

[0113] Check if the first worker thread is idle. If so, distribute the 8 valid segments as an audio array to the first worker thread. The first worker thread performs text conversion on the audio array and marks itself as working until it finishes text conversion. Then mark the first worker thread as idle again.

[0114] If the query to see if the first worker thread is idle is negative, then the query to see if the second worker thread is idle is performed. If it is, the eight valid segments are distributed as an audio array to the second worker thread. The second worker thread performs text conversion on the audio array and marks itself as working until it finishes text conversion. Then the second worker thread is marked as idle again.

[0115] If neither of the two worker child threads is idle, the process will wait until one of the worker child threads becomes idle.

[0116] By marking the status of worker child threads, idle worker child threads are used in a loop to convert different audio arrays into text arrays. This achieves efficient and parallel text conversion of different audio arrays while consuming less system memory resources, thus improving the processing efficiency of audio-to-text conversion.

[0117] As an optional implementation, the audio recognition method provided in this application embodiment calls the webassembly component to concatenate the read fragment text into a text string according to the corresponding ID, including:

[0118] The webassembly component reads a text array from shared memory, which contains 8 text fragments. The text fragments in the text array are concatenated into a text string according to their ID order.

[0119] The natural language processing model is invoked to optimize the text string. The optimization process includes at least one of the following: syntax correction, semantic coherence checking, deduplication, and word order adjustment.

[0120] Based on this, the audio recognition method provided in this application optimizes the concatenated text string using a natural language processing model, making the audio recognition results more accurate.

[0121] As an optional implementation, after optimizing the text string, the method also includes:

[0122] Display the optimized text string on the UI page.

[0123] Based on this, the audio recognition method provided in this application can display the text obtained by audio recognition and conversion on the UI page, realizing the effect of human-computer interaction. This method can be applied to web application scenarios that require real-time voice interaction, such as online conferencing systems (e.g., using ordinary PCs / laptops + microphones as peripherals to quickly generate meeting minutes text for online meetings), browser-side voice assistants, real-time caption generation, and voice control interfaces of IoT devices (e.g., in the voice-to-text scenario of IoT device voice control, converting voice commands into text for users to confirm the correctness of the commands). This provides users with a good user experience of voice recognition services.

[0124] Based on the same inventive concept, embodiments of this application also provide a voice recognition device (e.g., a voice recognition text pen, a voice recognition display screen, etc.), which may include at least one memory storing executable code to implement the above-described audio recognition method.

[0125] Based on the same inventive concept, such as Figure 7 As shown in the figure, this application embodiment also provides an electronic device, including: at least one memory and at least one processor, wherein the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-described audio recognition method.

[0126] Based on the electronic device provided in this application embodiment, audio data can be acquired through the Audio Worklet component. The audio data is quickly and alternately written to at least two webassembly buffers in shared memory. The webassembly component is then called to read audio data from these buffers, slices the audio data, and puts it back into the shared memory buffers. The webworker component is then called to read the sliced ​​audio data from the shared memory, converts the sliced ​​audio data into text fragments, stores these text fragments in the shared memory buffers, and concatenates the text fragments using the webassembly component. This achieves natural language processing for audio-to-text conversion. In the audio recognition method provided in this application embodiment, shared memory is used to improve data transmission efficiency during audio recognition. Furthermore, the method calls the Audio Worklet component to acquire audio data. The Audio Worklet component is an audio processing API provided by the web client, capable of processing audio data in real time in an independent thread, avoiding main thread blocking, thereby achieving low-latency audio input / output. Furthermore, the method provided in this application also calls the WebAssembly component to perform segmentation processing on the audio data. The WebAssembly component has efficient instruction set compilation characteristics, providing near-native computational performance and accelerating the segmentation process of audio data. Through the efficiency optimizations in audio data acquisition, transmission, and processing described above, the audio recognition method provided in this application can achieve highly efficient audio recognition. Using the method provided in this application, a better user experience can be provided in real-time voice interaction application scenarios.

[0127] At least one of the aforementioned memories can be used to store a computer program, which may include instructions and data to implement the steps of any of the methods described above. The memory may be random access memory, read-only memory, non-volatile, programmable ROM, erasable PROM, electrically erasable, flash memory, optical memory, and registers, etc. The processor may be a general-purpose processor, which is a processor that performs specific steps and / or operations by reading and executing the computer program stored in the memory. The general-purpose processor may use the memory stored in the memory during the execution of the steps and / or operations. The general-purpose processor may be a central processing unit, ASIC, and FPGA, etc. The electronic device may also include a communication interface, which may include input / output interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The methods disclosed in the embodiments of this application can be directly implemented by a hardware processor or by a combination of hardware and software modules in the processor.

[0128] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0129] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0130] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via infrared, microwave, or other means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), solid-state drives, etc.

[0131] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0132] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0134] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0135] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An audio recognition method, characterized in that, include: The Audio Worklet component is invoked to collect audio data, and the audio data is alternately written to at least two webassembly buffers in shared memory; The webassembly component is invoked to read audio data from the at least two webassembly buffers. Based on the audio energy of each audio frame in the audio data, the audio data is sliced ​​to obtain multiple audio segments, and the multiple audio segments are stored in the shared memory. The webworker component is invoked to read the multiple audio segments from the shared memory, and the valid segments in the multiple audio segments are converted into segment text in parallel. Each segment text is assigned an ID, and the ID and segment text are stored together in the shared memory. The webassembly component is invoked to concatenate the read fragment text into a text string according to the corresponding ID; The audio data is alternately written to at least two webassembly buffers in shared memory, including: The audio data collected by the Audio Worklet component is received through the browser's global Audio Context object; The audio data is written alternately to the at least two webassembly buffers via the postMessage function of the Audio Worklet component; And / or, Create at least two webassembly buffers in shared memory, and write the audio data to any one of the at least two webassembly buffers via postMessage in the Audio Worklet component. When the buffer is full, write the audio data to the other buffer of the at least two webassembly buffers. Based on the audio energy of each audio frame in the audio data, the audio data is sliced, including: For each audio frame in the audio data, the mean square energy of the audio frame is calculated based on the sample value of each audio frame, and the absolute value of the energy corresponding to the no speech type is determined based on the change of the mean square energy. The starting point when the mean square energy of an audio frame changes from no speech type to other types, and the ending point when the mean square energy of an audio frame changes from other types to no speech type, are marked as endpoints. The audio data is sliced ​​using the endpoints as cutting points. The step of calling the webworker component to read the multiple audio segments from the shared memory and converting the valid segments from the audio segments into segment text in parallel includes: The webworker component reads the segment to be converted from the multiple audio segments in the shared memory, and determines whether the segment to be converted is a valid segment based on the STT speech-to-text library and the audio energy of the segment to be converted. The webworker component creates at least two worker child threads and distributes multiple valid fragments to the at least two worker child threads; The process involves calling at least two worker child threads to convert multiple valid fragments into multiple fragment texts in parallel.

2. The method as described in claim 1, characterized in that, After calling the Audio Worklet component to acquire audio data, the method further includes: The number of frame samples in the audio data is actively obtained through the onMessage method of the Audio Worklet component; the number of frame samples is sent to the webassembly component through the postMessage method of the Audio Worklet component, so that the webassembly component can determine the number of segments when performing segmentation processing on the audio data.

3. The method as described in claim 1, characterized in that, The audio data is alternately written to at least two webassembly buffers in shared memory, including: The audio data collected by the Audio Worklet component is received through the browser's global Audio Context object; The average length of an existing audio segment is obtained through the onMessage method of the Audio Worklet component, and the average length of the segment is sent to the postMessage method of the Audio Worklet component. The audio data is alternately written to the at least two webassembly buffers using the postMessage function in the Audio Worklet component, with the average length of the segment being the single write length of the current audio data.

4. The method as described in claim 1, characterized in that, The webworker component creates at least two worker child threads and distributes multiple valid fragments to the at least two worker child threads, including: After identifying 8 valid segments, the webworker component distributes the 8 valid segments as an audio array to the worker child threads that are currently idle. The worker thread that receives the audio array converts the audio array into a text array, stores the text array along with the IDs of the corresponding 8 valid segments into shared memory, and then changes the current state of the worker thread to idle.

5. The method as described in claim 4, characterized in that, The webassembly component is invoked to concatenate the read fragment text into a text string according to the corresponding ID, including: The webassembly component reads a text array from the shared memory, wherein the text array includes 8 text fragments, and concatenates the text fragments in each text array into a text string according to the ID order; The text string is optimized by calling a natural language processing model, wherein the optimization includes at least one of the following: syntax correction, semantic coherence checking, deduplication, and word order adjustment.

6. The method as described in claim 5, characterized in that, After optimizing the text string, the method further includes: Display the optimized text string on the UI page.

7. An electronic device, characterized in that, include: An electronic device includes: at least one memory and at least one processor. The at least one memory stores executable code, and the at least one processor executes the executable code in the at least one memory to implement the method as described in any one of claims 1-6.

Images