Distributed audio transcription method and system

Abstract

The application discloses a distributed audio transcription method and system, comprising: after collecting audio in an earphone charging bin and before transmission, performing sliding window processing on the audio to obtain a plurality of audio segments; performing content analysis on the plurality of audio segments to determine a content value label of each audio segment, the content value label at least including a first label representing valid content and a second label representing non-valid content; converting the audio segment with the content value label as the first label into acoustic embedding data through an acoustic encoder; transmitting the acoustic embedding data from the earphone charging bin to a terminal device through a wireless link; and decoding the acoustic embedding data locally in the terminal device to generate text corresponding to the valid content. Thus, the earphone side filters valid content and only transmits low-data-volume acoustic embedding data, and the terminal locally decodes.

Description

Technical Field

[0001] This application relates to the technical field of audio processing, and more specifically, to a distributed audio transcription method and system. Background Technology

[0002] In related technologies, a typical audio transcription process involves: the recording device capturing complete audio, transmitting the complete audio wirelessly to the user terminal, forwarding it to a cloud server to complete speech recognition, and finally returning the recognized text to the terminal for display.

[0003] However, transmitting the complete audio to the user terminal results in wasted bandwidth and storage pressure, and the reliance on the cloud for translation makes it necessary to ensure that the terminal is always connected to the internet, and there is a risk of privacy leakage when transmitting to a third-party server. Summary of the Invention

[0004] In view of the above problems, this application proposes a distributed audio transcription method and system that can solve the above problems.

[0005] In a first aspect, embodiments of this application provide a distributed audio transcription method, comprising: after collecting audio in the earphone charging case and before transmission, performing sliding window processing on the audio to obtain multiple audio segments; performing content analysis on the multiple audio segments to determine the content value tag of each audio segment, wherein the content value tag includes at least a first tag representing valid content and a second tag representing invalid content; converting the audio segments with the first content value tag into acoustic embedding data through an acoustic encoder; transmitting the acoustic embedding data from the earphone charging case to a terminal device through a wireless link; and decoding the acoustic embedding data locally on the terminal device to generate text corresponding to the valid content.

[0006] Secondly, embodiments of this application also provide a distributed audio transcription system, which includes headphones and a terminal device, wherein: the headphones are configured to perform sliding window processing on the audio after it is collected in the headphone charging case and before it is transmitted, to obtain multiple audio segments; the headphones are configured to perform content analysis on the multiple audio segments to determine the content value tag of each audio segment, wherein the content value tag includes at least a first tag representing valid content and a second tag representing invalid content; the headphones are configured to convert the audio segments with the first content value tag into acoustic embedding data through an acoustic encoder; the headphones are configured to transmit the acoustic embedding data from the headphone charging case to the terminal device through a wireless link; and the terminal device is configured to decode the acoustic embedding data locally on the terminal device to generate text corresponding to the valid content.

[0007] The technical solution provided in this application includes the following method: after the audio is collected in the earphone charging case and before transmission, the audio is processed by a sliding window to obtain multiple audio segments; content analysis is performed on the multiple audio segments to determine the content value tag of each audio segment, wherein the content value tag includes at least a first tag representing valid content and a second tag representing invalid content; the audio segments with the content value tag as the first tag are converted into acoustic embedding data by an acoustic encoder; the acoustic embedding data is transmitted from the earphone charging case to the terminal device via a wireless link; and the acoustic embedding data is decoded locally on the terminal device to generate text corresponding to the valid content. Thus, by performing content analysis and filtering out valid content with the content value tag as the first tag on the earphone charging case side, and only extracting acoustic features from the valid content and transmitting low-volume acoustic embedding data, the indiscriminate transmission of complete audio is avoided, alleviating bandwidth waste and storage pressure; at the same time, the terminal device deploys a preset language decoding model and completes decoding locally without relying on a cloud server, thus not requiring the terminal to be constantly connected to the internet, and reducing the risk of privacy leakage because the original audio is not uploaded to a third-party server. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments and drawings obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0009] Figure 1 A flowchart illustrating a distributed audio transcription method according to an embodiment of this application is shown.

[0010] Figure 2 A schematic diagram of the structure of a distributed audio transcription system provided in an embodiment of this application is shown.

[0011] Figure 3 A schematic diagram of another distributed audio transcription system provided in an embodiment of this application is shown.

[0012] Figure 4 A schematic diagram of the structure of a distributed audio transcription device provided in an embodiment of this application is shown.

[0013] Figure 5 This illustration shows a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0015] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0016] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0017] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0020] Please see Figure 2 and Figure 3 , Figure 2 This illustration shows a schematic diagram of the structure of a distributed audio transcription system provided in an embodiment of this application. Figure 3 A schematic diagram of another distributed audio transcription system provided in an embodiment of this application is shown. Figure 2 and Figure 3 As shown, the distributed audio transcription system 200 includes an earphone charging case 210 and a terminal device 220, wherein: The earphone charging case 210 is configured to perform sliding window processing on the audio after acquisition and before transmission to obtain multiple audio segments; the earphone charging case 210 is configured to perform content analysis on the multiple audio segments to determine the content value tag of each audio segment, wherein the content value tag includes at least a first tag representing valid content and a second tag representing invalid content; the earphone charging case 210 is configured to convert the audio segments with the first content value tag into acoustic embedding data through an acoustic encoder; the earphone charging case 210 is configured to transmit the acoustic embedding data from the earphone charging case 210 to the terminal device 220 via a wireless link; the terminal device 220 is configured to decode the acoustic embedding data to generate text corresponding to the valid content.

[0021] In this embodiment, the device 210 may be, but is not limited to, the earphone charging case 210, and the terminal device 220 may be, but is not limited to, a smartphone, tablet, laptop, or other electronic device with display and computing capabilities. A communication connection is established between the earphone charging case 210 and the terminal device 220 via a wireless link. The specific form of the wireless link may include, but is not limited to, Wi-Fi Aware, Bluetooth Low Energy (BLE), Wi-Fi Direct, or other methods suitable for establishing point-to-point connections between two devices in close proximity.

[0022] The earphone charging case 210 integrates a low-power processor, a microphone array, a wireless communication module, and non-volatile memory. The low-power processor may include an embedded processor supporting neural network inference, such as a SoC chip with an NPU acceleration unit; the microphone array is used to collect ambient sound and generate audio signals; the wireless communication module is used for data interaction with the terminal device 220; and the non-volatile memory is used to cache audio data, acoustic embedding vectors, and locally generated summary data. The user terminal 220 has a companion application installed and possesses local neural network inference capabilities.

[0023] Please see Figure 1 , Figure 1 This diagram illustrates a flowchart of a distributed audio transcription method according to an embodiment of this application. The method can be jointly executed by an earphone charging case 210 and a terminal device 220. In the overall process, the earphone charging case 210 continuously performs audio acquisition, sliding window processing, content value filtering, acoustic feature extraction of valid content, and acoustic embedding vector transmission; the terminal device 220 continuously performs acoustic embedding vector reception and local language decoding. These two parts belong to the front-end node process and the terminal node process, respectively, and can be executed asynchronously and in parallel, forming a pipelined processing. Figure 1 As shown, the distributed audio transcription method includes steps 110 to 150, specifically: In step 110, after the audio is collected in the earphone charging case and before it is transmitted, the audio is processed by a sliding window to obtain multiple audio segments.

[0024] After the earphone charging case collects the audio to be processed, it performs sliding window processing on the audio to be processed before transmitting the audio to the mobile terminal, resulting in multiple audio segments. The earphone charging case can be a front-end device with audio acquisition and processing capabilities, such as a TWS earphone charging case, which has a built-in microphone array, a low-power processor with a certain computing power (e.g., an embedded processor that supports NPU acceleration), a wireless communication module (e.g., supporting Wi-Fi Aware and Bluetooth Low Energy BLE), and non-volatile storage devices.

[0025] In one specific implementation, the earphone charging case uses its microphone array to collect audio signals from the environment in real time to obtain the audio to be processed. Then, according to a preset duration sliding window, the continuously collected audio stream is processed by frame or segmentation to obtain multiple audio segments that are continuous or partially overlapping in time. This allows for subsequent real-time analysis of each segment according to the window, ensuring that the processing process is synchronized with the audio acquisition process and meeting the requirements of real-time stream processing.

[0026] The microphone array of the earphone charging case 210 acquires mono PCM audio signals at a sampling frequency of 16,000 times per second and segments the audio stream according to a preset sliding window. For example, the window length and step size of the sliding window can be set to 500 milliseconds, meaning that 500-millisecond audio segments are taken each time for content value determination, and adjacent windows do not overlap to meet the needs of real-time processing. In another optional implementation, the window length can be set to 300 milliseconds and the step size to 100 milliseconds, meaning that adjacent windows partially overlap to improve the accuracy of capturing the boundaries of effective content. This embodiment does not limit the specific window length and step size.

[0027] In step 120, content analysis is performed on the plurality of audio segments to determine the content value tag for each audio segment. The content value tag includes at least a first tag representing valid content and a second tag representing invalid content.

[0028] In some implementations, the earphone charging case operates multiple neural network models at different activation levels, including a keyword trigger and a speech activity detection module at a first activation level, an acoustic behavior analyzer at a second activation level, and an acoustic encoder at a third activation level. The content value analysis is performed by the acoustic behavior analyzer, which is activated when the keyword trigger detects a keyword in a preset vocabulary or when the speech activity detection module continuously detects speech activity for more than a preset duration. The acoustic encoder is activated only when the acoustic behavior analyzer outputs the first tag.

[0029] In this system, the keyword trigger and speech activity detection module in the first activation level are always running to continuously monitor the audio environment; the acoustic behavior analyzer in the second activation level is activated when preset conditions are met; and the acoustic encoder in the third activation level is only activated when the acoustic behavior analyzer outputs the first tag representing valid content. By allocating computational resources to the three models in a hierarchical manner according to information density and content value, the excessive power consumption caused by all models being always running can be avoided, allowing the earphone charging case to support a relatively long period of meeting recording even with a limited battery capacity.

[0030] Specifically, the keyword trigger at the first activation level can be implemented based on a quantized neural network model with approximately 80KB of parameters. It performs real-time inference on audio features using a sliding window of approximately 200 milliseconds. A trigger signal is generated when the activation probability of any meeting keyword in a preset vocabulary exceeds a preset threshold. The preset vocabulary can include high-frequency keywords from the meeting scenario, such as "the conclusion is," "next step," "agree," "decided," and "start discussion." The speech activity detection module is used to label speech segments in the audio stream. This can be implemented using a rule-based method based on energy thresholds and zero-crossing rates, or a VAD method based on lightweight neural networks.

[0031] The acoustic behavior analyzer at the second activation level is used to perform content value filtering. This acoustic behavior analyzer can be implemented based on the aforementioned preset neural network model, or it can employ other lightweight content classifiers. The wake-up conditions for the acoustic behavior analyzer include at least one of the following: a keyword trigger generates a trigger signal; or the speech activity detection module continuously detects speech activity for more than a preset duration, such as 2 consecutive seconds. By waking up the acoustic behavior analyzer only when the above conditions are met, unnecessary computational overhead can be avoided in continuous silence, pure noise, or obviously non-conversational environments.

[0032] An acoustic behavior analyzer extracts multidimensional acoustic behavior features from input audio segments and determines the content value of the audio segments based on these features. These multidimensional acoustic behavior features include Mel-spectral features, and at least one of the following extracted through parametric signal processing: speaker switching density features, energy envelope temporal regularity features, and segment duration to gap ratio features. Specifically, speaker switching density features describe the frequency of alternation between different speakers per unit time; energy envelope temporal regularity features describe the structured nature of audio signal energy changes over time; and segment duration to gap ratio features describe the ratio between the average duration of consecutive speech segments and the average duration of the silence gap between adjacent speech segments. By inputting these multidimensional features and Mel-spectral features into the acoustic behavior analyzer, the content value of audio can be determined solely through acoustic behavior patterns without relying on semantic understanding, balancing reliability and low computational cost.

[0033] In this implementation, the single inference latency of the preset neural network model is less than the step time of the sliding window. Valid content includes topic discussions, statements of opinion, decision-making speeches, or paragraphs containing preset keywords. Invalid content includes at least silent segments, ambient noise, noise interference, or paragraphs unrelated to the topic. Furthermore, invalid content can be further divided into low-priority content and discarded content. Low-priority segments in the second tag, such as transitional chatter, can be temporarily stored locally in the earphone charging case and transmitted in the background as needed when transmission resources are idle. Discarded content in the second tag, such as continuous ambient noise, is directly terminated and does not enter the encoding and transmission process. This significantly saves wireless link bandwidth and reduces the pressure on user terminal storage and decoding. Low-priority content is temporarily stored locally on the device and does not preempt network resources required for the transmission of valid content. Compared with the existing solution that transmits all audio content indiscriminately, this implementation achieves precise allocation of resources based on content value.

[0034] In one specific implementation, a preset neural network model takes the original audio waveform data of the audio segment, or the Mel-spectral features obtained after a simple transformation of the original audio waveform data, as input to determine the content value of each audio segment and output the corresponding content value label. The earphone charging case 210 first performs preprocessing on the audio segments. The preprocessing steps include: calculating 128-dimensional Mel-spectral features from a 16kHz mono PCM stream with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. Then, the calculated Mel-spectral features are used as input to the preset neural network model. The model output is the probability distribution of different content value levels corresponding to the audio segment, and the level with the highest probability is determined as the content value label of the audio segment.

[0035] In some embodiments, the distributed audio transcription method further includes: dynamically switching the operating modes of the multiple neural network models according to the remaining battery level of the headphone charging case, wherein the operating modes include full-function mode, equalization mode, and battery life mode.

[0036] When the battery level is above a first preset threshold, the system operates in full-function mode. In this mode, all modules work according to optimal performance strategies. The acoustic behavior filter uses complete multi-dimensional acoustic behavior features, and the acoustic encoder outputs high-quality high-dimensional embedding vectors. When the battery level drops below the first preset threshold but above the second preset threshold, the system switches to a balanced mode. For example, it reduces the activation frequency of the acoustic behavior filter or decreases the dimension of the embedding vector output by the acoustic encoder. When the battery level falls below the second preset threshold, the system enters a battery life mode. For example, it retains only the basic functions of the voice activity detection module and the acoustic behavior filter, while disabling or reducing the output quality of the acoustic encoder to extend the device's usability. Through these multi-level degradation strategies, the system's workload can be dynamically adjusted under different battery levels to meet the battery life requirements of long-duration meeting recording scenarios.

[0037] Since the pre-defined neural network model is deployed in the low-power embedded processor of the earphone charging case, a neural network model architecture with extremely low parameter count can be used, such as the MobileNet series models or Tiny Transformer variants, to ensure that its single inference latency is much smaller than the step time of the sliding window, thus not affecting the real-time streaming of audio.

[0038] As described above, the earphone charging case primarily relies on its built-in preset neural network model to automatically determine the content value of each audio segment. However, since the preset neural network model makes real-time judgments based on a sliding window, in real-world scenarios, the start and end boundaries of valid content may not perfectly match the window division. For example, a speaker's tone may be relatively flat before entering the formal discussion, causing the preset neural network model to fail to promptly determine it as valid content at that stage. Similarly, paragraphs that users subjectively consider important, but which may not contain obvious characteristic words, may also be judged as invalid content by the classification model.

[0039] To compensate for potential boundary misjudgments or omissions that may occur when relying solely on automatic classification, in some embodiments, the earphone charging case is equipped with a pressable operating component. The distributed audio transcription method may also include the following steps: (1) Receive manual marking instructions from the user through the pressable operation component provided on the earphone charging case; (2) According to the manual marking instruction, the audio data within the preset time period before the current trigger time and the audio data within the preset time period after the current trigger time are forcibly marked as the first tag.

[0040] In response to a pressable component, the earphone charging case marks the audio data within a preset time period before the current trigger time and the audio data within a preset time period after the current trigger time as a first tag.

[0041] If a user subjectively believes that the content being marked is of significant value, they can actively intervene in the marking process by performing a trigger operation (e.g., clicking or double-clicking) on ​​the pressable component.

[0042] In response to detecting a trigger operation on the pressable operating part (i.e., a manual marking command issued by the pressable operating part), the earphone charging case uses the time when the trigger operation occurs as the current trigger time, and marks the audio data within a preset time period before the current trigger time and the audio data within a preset time period after the current trigger time as the first tag, so that they can enter the subsequent acoustic feature extraction processing flow.

[0043] It should be noted that this labeling process is triggered by the user and is not limited by the automatic judgment results of the preset neural network model. Even if a piece of content was originally judged as the second label by the preset neural network model, it will still be relabeled as the first label as long as it falls within the preset time range before and after the aforementioned triggering time. This method compensates for the risk of missed judgments that may occur in automatic classifiers.

[0044] This allows users to directly intervene in the labeling results based on their subjective judgment of the importance of the meeting content, making up for important content that the preset neural network model cannot fully capture and that can only be identified through context or subjective judgment, thereby further reducing the possibility of missing effective content.

[0045] In some implementations, when the keyword trigger detects a trigger word, the distributed audio transcription method may further include the following steps: (1) The cached audio within a preset backtracking time before the triggering time is forcibly marked as the first tag and sent to the acoustic encoder; (2) Extract the multidimensional acoustic behavior features of the audio within the traceback period and add them to the recent positive sample queue; (3) Offset the decision threshold of the acoustic behavior analyzer online according to the recent positive sample queue.

[0046] When outputting content value labels, the preset neural network model determines the final content value label of an audio segment by comparing the judgment result (e.g., confidence score) obtained from judging the content value of the audio segment with a decision threshold. Therefore, the decision threshold of the online offset acoustic behavior analyzer based on the recent positive sample queue can make the judgment result of the preset neural network model more accurate.

[0047] Considering that the inherent regularities of audio content in different speaking scenarios can affect the difficulty and accuracy of the preset neural network model in making validity judgments—for example, in scenarios where multiple people take turns speaking in an orderly manner, the content is usually highly relevant to the topic; however, in scenarios where multiple people speak simultaneously and their voices overlap, the resulting audio content is often difficult to form into valid and usable transcribed text. Therefore, in some implementations, this distributed audio transcription method may further include the following steps: (1) Based on the multidimensional acoustic behavior characteristics of the audio segment, dynamically adjust the decision threshold of the acoustic behavior analyzer: (2) When an orderly multi-person turn-by-turn speaking pattern is detected, the decision threshold is lowered to expand the range of those identified as the first label; (3) When a long period of continuous single-person speech is detected, restore the default sensitivity; (4) When multiple people are detected speaking at the same time and it is difficult to separate them, the decision threshold is increased and the range of the first label is narrowed.

[0048] During the content analysis of the audio being processed, the earphone charging case can also detect the speaker alternation frequency (i.e., the number of times different speakers take turns speaking per unit time) and energy envelope characteristics (i.e., the distribution pattern of audio signal energy over time) in the audio being processed, and dynamically adjust the decision threshold based on the detection results.

[0049] Multidimensional acoustic behavior features may include Mel spectrum features, as well as at least one of the following extracted through parametric signal processing: speaker switching density features, energy envelope temporal regularity features, and segment duration to gap ratio features.

[0050] When multidimensional acoustic behavior characteristics indicate that multiple people are speaking in an orderly manner, the decision threshold is lowered to below the default decision threshold. When multidimensional acoustic behavior characteristics indicate that a single person is speaking continuously for a long time, the default sensitivity is restored, that is, the decision threshold is adjusted to the default decision threshold. When multidimensional acoustic behavior characteristics indicate that multiple people are speaking at the same time and separation is difficult, the decision threshold is increased to above the default decision threshold.

[0051] When the headphone charging case detects that there is an orderly multi-person turn-by-turn speaking situation based on the frequency of speaker alternation, it indicates that the audio content in the current scenario has high availability and topic relevance. At this time, the headphone charging case lowers the decision threshold to below the default decision threshold, so that more audio segments can be identified as the first label, thereby minimizing the omission of valid speech content while ensuring the availability of the transcribed content.

[0052] When the earphone charging case detects that multiple people are speaking simultaneously based on the energy envelope features, it means that even if the audio content in the current scenario is transcribed, it is difficult to form semantically clear and usable text. At this time, the earphone charging case raises the decision threshold to a level higher than the default decision threshold, so that only audio segments with more significant content value features will be judged as the first label. This avoids the generation of unusable text due to the transcription of a large amount of overlapping and chaotic speech content, and also reduces unnecessary acoustic feature extraction processing and data transmission overhead.

[0053] Therefore, the earphone charging case can adaptively balance the recall of effective content and the usability of transcribed content according to changes in the actual speaking scenario.

[0054] The above describes how the earphone charging case can supplement and correct the content value tags automatically determined by the preset neural network model through three methods: keyword detection-triggered backtracking tags, user-initiated press tags, and dynamic adjustment of decision thresholds based on speech scene characteristics. It should be noted that these methods are not independent or unaffected in practical applications; rather, they may overlap or be adjacent in time. For example, the time period during which the user presses the operating component for tagging may overlap with the time period of keyword-triggered backtracking tags; or, for instance, a boundary moment affected by the dynamic adjustment of the decision threshold may fall near the time interval covered by keyword backtracking tags or button tags.

[0055] If the aforementioned overlapping or proximity relationships are not further processed, inconsistent or redundant content value tags may appear within the same time period. Therefore, in some embodiments, this distributed audio transcription method may further include the following steps: (1) Obtain the backtracking marking time interval corresponding to the hit preset keyword, the trigger marking time interval corresponding to the manual marking instruction, and the candidate boundary time point indicated by the dynamic adjustment of the decision threshold; (2) If the backtracking mark time interval overlaps with the trigger mark time interval or the time interval is less than a preset interval threshold, the union of the two is determined as the time interval for the final mark as the first tag; (3) If the candidate boundary time point falls within the preset proximity range of the backtracking mark time interval or the trigger mark time interval, the content value tag corresponding to the candidate boundary time point shall be verified a second time based on the backtracking mark time interval or the trigger mark time interval.

[0056] The backtracking marking time interval can be the time range of backtracking marking triggered by preset keywords, the triggering marking time interval can be the time range of user-initiated marking, and the candidate boundary time point can be the time point where the content value tag changes near that moment due to the adjustment of the decision threshold.

[0057] After obtaining the backtracking marking time interval determined after hitting the preset keyword, the trigger marking time interval corresponding to the manual marking command, and the candidate boundary time point generated during the dynamic adjustment based on the decision threshold, the earphone charging case further processes these three types of time information.

[0058] If there is a temporal overlap between the backtracking marker time interval and the trigger marker time interval, or if the time interval between the two is less than a preset interval threshold, that is, the two are relatively close in time and may correspond to different judgment sources of the same valid content, then the earphone charging case will take the union of these two time intervals, that is, take the smallest continuous time range that can simultaneously cover these two time intervals as the final time interval marked as the first tag.

[0059] By using the union method, we can avoid the situation where the effective content is incomplete or truncated in the final labeling result because each of the two labeling mechanisms only covers a part of the same effective content time range.

[0060] If the candidate boundary time point falls within the preset proximity range of the backtracking marking time interval or the trigger marking time interval, that is, the candidate boundary time point is relatively close to the boundary of the aforementioned two marking time intervals, it indicates that the determination result of the content value tag corresponding to the candidate boundary time point may conflict with or be inconsistent with the two more explicit and reliable marking bases of keyword triggering or user active marking.

[0061] At this point, the earphone charging case uses the backtracking or triggering time interval as a reference to perform a secondary verification of the content value label of the audio segment corresponding to the candidate boundary time point. That is, based on the preliminary judgment result given by the dynamic adjustment of the decision threshold, it further refers to the result of keyword triggering or user active labeling to correct whether the audio segment should be ultimately labeled as the first label or the second label. In this way, the candidate judgment result obtained by indirect inference based on statistical features is corrected with a clearer labeling basis directly indicated by the user or keywords, thereby improving the accuracy of the final labeling result at the effective content boundary.

[0062] In step 130, the audio segment with the content value tag of the first tag is converted into acoustic embedding data by an acoustic encoder.

[0063] For audio segments that are determined to be valid content (i.e. marked as the first tag), the earphone charging case inputs them into an acoustic encoder for acoustic feature extraction, converting the original high-dimensional audio signal into low-dimensional acoustic embedding data.

[0064] In one specific implementation, the acoustic encoder can be implemented using a miniature sequence encoder, such as a quantized Conformer encoder or a lightweight Transformer encoder.

[0065] Acoustic embedding data is typically much smaller in dimensionality than the original audio, for example, it can be controlled between 32 and 128 dimensions, and the time step can be set to 10 to 20 milliseconds per vector. This makes the equivalent bit rate carried by the acoustic embedding data sequence significantly lower than the original audio bit rate, and the compression ratio can reach tens of times or even higher, while still retaining enough acoustic information to support subsequent language decoding.

[0066] In some implementations, the second tag in the content value tag is further divided into low-priority content and discarded content: the acoustic embedding data corresponding to the audio segments determined to be valid content is transmitted in real time through a high-priority transmission queue; the acoustic embedding data corresponding to the audio segments determined to be low-priority content is temporarily stored in the front-end device and transmitted in the background as needed; the audio segments determined to be discarded content are terminated.

[0067] In step 140, the acoustic embedding data is transmitted from the earphone charging case to the terminal device via a wireless link.

[0068] After generating acoustic embedding data, the earphone charging case adds it to a transmission queue for sending data to the terminal device and sets the priority of the transmission queue to the highest priority. The earphone charging case then transmits the acoustic embedding data in the transmission queue to the terminal device in sequence via a wireless link to ensure that the acoustic embedding data corresponding to the effective content can be sent to the terminal device in a priority and timely manner, thereby reducing the presentation delay of the effective content transcription results.

[0069] The terminal device can be a user device with mobile neural network inference capabilities, such as a mobile phone, tablet, or personal computer, and it has a corresponding application installed. The earphone charging case establishes a communication connection with the terminal device via a wireless link, such as Wi-Fi Aware or Bluetooth Low Energy.

[0070] However, during the process of the earphone charging case adding the generated acoustic embedding data to the transmission queue and sending it to the terminal device via the wireless link, due to the possibility of out-of-order arrival or brief packet loss in wireless transmission, in order to ensure that the terminal device can ultimately obtain the acoustic embedding data for speech decoding completely and in order, in some embodiments, the terminal device includes a receiving buffer with sequence number verification, and the earphone charging case includes a feature data buffer; the terminal device stores the received acoustic embedding data into the receiving buffer; sorts and checks the integrity of the acoustic embedding data according to the sequence number verification; if it is determined that a certain acoustic embedding data is missing, a retransmission request for that acoustic embedding data is generated. The earphone charging case stores the generated acoustic embedding data in the feature data buffer and retains it for a preset time; when a retransmission request is received, it retrieves the target acoustic embedding data corresponding to the retransmission request from the feature data buffer and retransmits the target acoustic embedding data to the terminal device.

[0071] When the earphone charging case sends acoustic embedding data to the terminal device via a wireless link, it can assign a corresponding sequence number to each piece of acoustic embedding data to identify its temporal order within the entire acoustic embedding data sequence. After receiving the acoustic embedding data, the terminal device first stores it in a locally maintained receive buffer. Based on the sequence number carried by the acoustic embedding data, it sorts the multiple pieces of acoustic embedding data already stored in the receive buffer. At the same time, based on the continuity of the sequence numbers, it checks for any missing sequence numbers, i.e., cases where a piece of acoustic embedding data was not successfully delivered.

[0072] If the terminal device determines that a certain acoustic embedding data (i.e., the acoustic embedding data corresponding to a certain sequence number) is missing, it generates a corresponding retransmission request for the missing acoustic embedding data and sends the retransmission request to the earphone charging case via the wireless link to request the earphone charging case to retransmit the acoustic embedding data.

[0073] At the same time, while generating acoustic embedding data and adding it to the transmission queue for transmission, the earphone charging case also stores the acoustic embedding data in a local feature data cache and keeps it in the feature data cache for a preset time, instead of discarding it immediately after transmission, in order to cope with possible retransmission needs.

[0074] When the earphone charging case receives a retransmission request from the terminal device, it retrieves the target acoustic embedding data corresponding to the retransmission request from the feature data buffer according to the sequence number information indicated in the retransmission request, and retransmits the target acoustic embedding data to the terminal device through the wireless link. This allows the terminal device to complete the missing acoustic embedding data in the receive buffer, thereby ensuring the integrity and continuity of the acoustic embedding data sequence that ultimately participates in language decoding in terms of time order. This avoids significant content omissions or semantic interruptions in the transcribed text due to the loss or out-of-order order of individual data packets.

[0075] Considering that data transmission between the earphone charging case and the terminal device is conducted via a wireless link, and that the bandwidth of the wireless link, the data processing capability of the terminal device, and the battery life of the earphone charging case itself may all affect the actual operation of the entire system, in some implementations, the terminal device generates a threshold adjustment command based on at least one of the following: the current bandwidth of the wireless link, the backlog of the receiving buffer, or the remaining battery power of the earphone charging case, and sends the threshold adjustment command to the earphone charging case; the earphone charging case adjusts the output decision threshold or adjusts the dimension of the acoustic embedding data obtained by acoustic feature extraction processing according to the threshold adjustment command.

[0076] During system operation, the terminal device can continuously acquire at least one of the following status information: First, the bandwidth status of the current wireless link, that is, the actual available transmission rate that the current wireless link can provide when data is transmitted between the earphone charging case and the terminal device; Second, the backlog of the receive buffer, that is, the quantity or proportion of acoustic embedded data stored in the receive buffer but not yet decoded by the preset language decoding model; Third, the remaining battery power information of the earphone charging case, which can be reported by the earphone charging case to the terminal device through the wireless link.

[0077] Based on at least one of the above status information, the terminal device determines whether the current system is in a state of tight transmission resources, limited processing power, or insufficient battery life, and generates a corresponding threshold adjustment command accordingly, and sends the threshold adjustment command to the earphone charging case through a wireless link.

[0078] After receiving a threshold adjustment instruction from the terminal device, the earphone charging case adjusts its processing parameters accordingly based on the specific content of the instruction. This can include at least one of the following methods: First, adjusting the output decision threshold, that is, adjusting the decision threshold used to determine the value label of audio segment content. For example, when bandwidth is tight, buffer backlog is high, or the remaining battery is low, the decision threshold is increased so that fewer audio segments are identified as the first label, thereby reducing the amount of data that needs to be extracted and transmitted for acoustic features. After the resource situation improves, the decision threshold can be restored or reduced to improve the recognition coverage of effective content.

[0079] Secondly, adjust the dimension of the acoustic embedding data obtained from acoustic feature extraction processing. That is, adjust the dimension of the acoustic embedding data output by the preset neural network model or acoustic encoder. For example, reduce the dimension when resources are limited to further compress the amount of data that needs to be transmitted per unit time; when resource conditions improve, the dimension can be increased to obtain richer acoustic information and improve the accuracy of subsequent language decoding.

[0080] By dynamically adjusting the processing parameters on the earphone charging case side based on status information such as link bandwidth, buffer backlog, and remaining battery power of the earphone charging case, the system can adaptively balance the integrity of the transcribed content with system resource consumption under different wireless transmission conditions, processing load, and battery life.

[0081] In step 150, the acoustic embedding data is decoded locally on the terminal device to generate text corresponding to the valid content.

[0082] After receiving the acoustic embedding data sent by the earphone charging case via a wireless link, the terminal device inputs it into a pre-deployed local language decoding model. This model then decodes the acoustic embedding data into natural language text segment by segment. The pre-deployed language decoding model can employ a distilled and compressed Transformer decoder or a recurrent neural network language model, or other model structures suitable for local inference on mobile devices. Its parameter count can be controlled within a range suitable for local deployment and inference on the terminal device.

[0083] The entire decoding process of acoustically embedded data is completed locally on the terminal device, without relying on an internet connection, thus enabling the conversion from acoustically embedded data to natural language text.

[0084] Therefore, audio acquisition, content value judgment, and acoustic feature extraction are completed in the earphone charging case. Only the acoustic embedded data corresponding to the content that has been screened and determined to be valid, which has a data volume much smaller than the original audio, is transmitted to the terminal device. Language decoding is then completed locally on the terminal device. This reduces the amount of data transmitted and improves the system response speed. At the same time, since the original audio data never leaves the earphone charging case, it also reduces the risk of the original voice content being acquired during transmission and processing.

[0085] In some implementations, the earphone charging case extracts summary data of audio segments with the content value tag as the second tag, or discards audio segments with the content value tag as the second tag; the audio segments with the content value tag as the second tag are added to the non-transmission queue or the lowest priority transmission queue.

[0086] For audio segments with the second content value tag, the earphone charging case can adopt at least one of the following two processing methods: First, extract the summary data of the audio segment. For example, simplify the original audio waveform of the audio segment by downsampling, etc., to obtain summary audio data with a data volume much smaller than the original audio but still retaining a certain degree of listenability. The summary data is stored locally in the earphone charging case so that when the user needs to review a segment of audio that has been judged as invalid content, they can still obtain the corresponding simplified version. Second, directly discard the audio segment without extracting a summary or storing it locally. This is suitable for application scenarios with high storage resource requirements or where it is clear that invalid content does not need to be retained.

[0087] Meanwhile, for audio segments with the second content value tag (including audio segments after the summary data has been extracted, or audio segments that have not yet been discarded), when deciding whether to send them to the terminal device, the earphone charging case either adds them to the non-transmission queue, meaning that this part of the content will not be transmitted, or adds them to the lowest priority transmission queue, meaning that the data corresponding to this part of the content will only be considered for transmission if the acoustic embedding data in the high priority transmission queue corresponding to the first tag has been sent and there are still spare transmission resources.

[0088] In some implementations, the earphone charging case downsamples the audio segment with the content value tag as the second tag to obtain summary data, and stores the summary data in local storage after establishing an index relationship between the summary data and the corresponding timestamp. In response to a retrieval request sent by the terminal device for a specified timestamp, the earphone charging case retrieves the corresponding summary data according to the index relationship and sends the summary data to the terminal device.

[0089] After receiving the summary data of the audio segment with the second content value tag, the earphone charging case establishes an index relationship between the summary data and the timestamp of its corresponding original audio segment on the timeline. The summary data, along with the established index relationship, is then stored in the earphone charging case's local non-volatile storage. By establishing the index relationship between the summary data and the timestamp, when it is necessary to retrieve summary data for a specific time period, the corresponding summary data can be quickly located and retrieved based on the timestamp corresponding to that time period, without having to traverse and search through all the locally stored summary data.

[0090] After the earphone charging case completes the storage and indexing of summary data, when a user on the terminal device wants to further view the content corresponding to a time period that was determined to be invalid and therefore not fully transmitted, the terminal device generates a retrieval request for the specified timestamp and sends the retrieval request to the earphone charging case.

[0091] Upon receiving the retrieval request, the earphone charging case retrieves the summary data corresponding to the specified timestamp based on the specified timestamp and the previously established index relationship between the summary data and the timestamp. The retrieved summary data is then transmitted to the terminal device via a wireless link for playback or display.

[0092] In one specific implementation, the workflow of the distributed audio transcription system 200 of this application may include the following steps: Step 1: The user presses the record button on the earphone charging case 210, triggering the earphone charging case 210 to start audio acquisition.

[0093] Step two: The earphone charging case 210 begins to collect audio. In the first 10 seconds after the meeting begins, the collected audio content mainly consists of the host's greetings and environmental noise generated during the seating of attendees. The audio content classification model classifies this audio segment as LOW_PRIORITY or DISCARD (i.e., the second label), so this audio segment will not be sent to the acoustic encoder, meaning that the acoustic feature extraction process is not activated.

[0094] Step 3: When the host says "Let's start discussing Project X", the keyword trigger captures the preset keyword "Project X" and triggers the backtracking tag for the previous 5 seconds. The cached audio segment containing the phrase "Let's start discussing" within 5 seconds before the detection time is forcibly marked as a TRANSMIT tag (i.e., the first tag). The audio segment is then sent to the acoustic encoder for acoustic feature extraction to obtain the corresponding acoustic embedding data. This acoustic embedding data is sent to the mobile phone with high priority.

[0095] Step four: After receiving the aforementioned acoustic embedding data, the terminal device 220 decodes the data using its locally deployed preset language decoding model to obtain and display the natural language text "We begin to discuss Project X".

[0096] Step 5: As the meeting continues, the earphone charging case 210 continuously filters the subsequently collected audio segments based on the classification results of the audio content classification model. For audio segments identified as having the first tag, corresponding acoustic embedding data is continuously generated and transmitted via wireless links such as Wi-Fi Aware. Occasionally, casual conversation segments are classified as having the LOW_PRIORITY tag. These audio segments are temporarily stored locally in the earphone charging case 210 and are not used for transmission or transcription.

[0097] Step Six: During a technical debate at a certain point in the meeting, the user actively marks the current moment of the double-click operation and the audio content within the next 30 seconds as a TRANSMIT tag (i.e., the first tag) by double-clicking the pressable operation part on the earphone charging case 210. This makes the content unaffected by the automatic judgment result of the audio content classification model and directly enter the acoustic feature extraction and transmission process.

[0098] Step 7: After the meeting, the user can view the transcript of the meeting in the accompanying application on the terminal device 220 through the aforementioned "Highlights" mode. The key conclusions of the meeting have been fully recorded, while non-valid content such as casual conversation has been automatically filtered out.

[0099] Step 8: If the user needs to view the content that was identified as the second tag during the meeting and therefore not fully transcribed, they can switch to the aforementioned full mode in the application. At this time, the user terminal will send a retrieval request for the corresponding time period to the headphone charging case 210 through a low-priority transmission link, and retrieve the cached acoustic embedding data (or the aforementioned summary data) corresponding to the previously temporarily stored LOW_PRIORITY audio segment from the headphone charging case 210 for the user to view further.

[0100] In some implementations, for earphone charging cases 210 with sufficient hardware computing power, the acoustic encoder can be replaced with a coding model with a deeper structure and a larger number of parameters to improve the accuracy of acoustic feature extraction; correspondingly, the replaced coding model can output acoustic embedding data with lower dimensionality, while ensuring or even improving the ability to express acoustic information, and further reducing the amount of data transmission.

[0101] In some implementations, if the terminal device 220 has internet connectivity, the acoustic embedded data can be selectively uploaded to a cloud server in addition to being decoded locally by a preset language decoding model. The cloud server can then perform higher-quality language decoding or multilingual translation, thereby achieving a hybrid working mode with local decoding as the primary method and cloud processing as a secondary method. This ensures the offline availability of the system while providing enhanced transcription and translation capabilities for users with network access.

[0102] In some implementations, the audio content classification model can also employ a rule-based lightweight pre-filtering mechanism. For example, it can use energy threshold detection of audio signals combined with voice activity detection (VAD) to perform preliminary screening of the collected audio segments. Only audio segments that are determined to be potentially valid content after preliminary screening are further activated by a neural network classifier for more refined content value determination. This two-level determination method, which combines pre-filtering and neural network classification, avoids running the neural network classifier on all audio segments, thereby further reducing the power consumption of the headphones.

[0103] Please see Figure 4 , Figure 4 The diagram shows a structural schematic of a distributed audio transcription device provided in an embodiment of this application. The distributed audio transcription device 300 in this application may include one or more of the following components: a processor 310, a memory 320, and one or more application programs. The one or more application programs may be stored in the memory 320 and configured to be executed by one or more processors 310. The one or more programs are configured to execute the distributed audio transcription method as described in the foregoing method embodiments.

[0104] Processor 310 may include one or more processing cores. Processor 310 connects to various parts within the distributed audio transcription device 300 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 320, and by calling data stored in memory 320. Optionally, processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 310 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 310 and may be implemented separately using a communication chip.

[0105] The memory 320 may include random access memory (RAM) or read-only memory (ROM). The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created during use by the distributed audio transcription device 300.

[0106] Please see Figure 5 , Figure 5 This illustration shows a schematic diagram of a computer-readable storage medium 400 provided in an embodiment of this application. The computer-readable storage medium 400 stores program code, which can be called by a processor to execute the distributed audio transcription method described in the above method embodiment.

[0107] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code can be read from or written to one or more computer program devices. The program code 410 may be compressed, for example, in a suitable form.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A distributed audio transcription method, characterized in that, include: After the audio is collected from the earphone charging case and before it is transmitted, the audio is processed by a sliding window to obtain multiple audio segments; Content analysis is performed on the multiple audio segments to determine the content value tag for each audio segment. The content value tag includes at least a first tag representing valid content and a second tag representing invalid content. Audio segments with the first content value tag are converted into acoustic embedding data using an acoustic encoder; The acoustic embedded data is transmitted from the earphone charging case to the terminal device via a wireless link; The acoustic embedding data is decoded locally on the terminal device to generate text corresponding to the valid content.

2. The distributed audio transcription method according to claim 1, characterized in that, The earphone charging case operates multiple neural network models, which are at different activation levels, including a keyword trigger and a voice activity detection module at the first activation level, an acoustic behavior analyzer at the second activation level, and an acoustic encoder at the third activation level. Content value analysis is performed by the acoustic behavior analyzer, which is activated when the keyword trigger detects a keyword in a preset vocabulary or when the voice activity detection module continuously detects voice activity for more than a preset duration. The acoustic encoder is activated only when the acoustic behavior analyzer outputs the first tag.

3. The distributed audio transcription method according to claim 2, characterized in that, When the keyword trigger detects a trigger word, the method further includes a collaborative calibration process: The cached audio within a preset backtracking duration prior to the trigger time is forcibly marked as the first tag and sent to the acoustic encoder; Extract the multidimensional acoustic behavior features of the audio within the retracing duration and add them to the recent positive sample queue; The decision threshold of the acoustic behavior analyzer is offset online based on the recent positive sample queue.

4. The distributed audio transcription method according to claim 2, characterized in that, The earphone charging case is provided with a pressable operating component, and the method further includes: Receives manual marking commands from the user via a pressable operating component on the earphone charging case; According to the manual marking instruction, the audio data within a preset time period before the current trigger time and the audio data within a preset time period after the current trigger time are forcibly marked as the first tag.

5. The distributed audio transcription method according to claim 2, characterized in that, The method further includes: Based on the multidimensional acoustic behavior characteristics of the audio segment, the decision threshold of the acoustic behavior analyzer is dynamically adjusted: When an orderly multi-person turn-taking speaking pattern is detected, the decision threshold is lowered, and the range of those identified as the first label is expanded. When a prolonged period of continuous single-person speaking is detected, restore the default sensitivity. When multiple people are detected speaking simultaneously and separation is difficult, the decision threshold is increased, and the range of those identified as the first label is narrowed.

6. The distributed audio transcription method according to claim 5, characterized in that, The multidimensional acoustic behavior features include: Mel spectrum features, and at least one of the following extracted through parametric signal processing: speaker switching density features, energy envelope temporal regularity features, and segment duration to gap ratio features.

7. The distributed audio transcription method according to any one of claims 2 to 4, characterized in that, The method further includes: Obtain the backtracking marking time interval corresponding to the hit preset keyword, the trigger marking time interval corresponding to the manual marking instruction, and the candidate boundary time point indicated by the dynamic adjustment of the decision threshold; If the backtracking marking time interval overlaps with the trigger marking time interval or the time interval is less than a preset interval threshold, then the union of the two is determined as the time interval for the final marking as the first tag. If the candidate boundary time point falls within a preset adjacent range of the backtracking mark time interval or the trigger mark time interval, then the content value tag corresponding to the candidate boundary time point is verified a second time based on the backtracking mark time interval or the trigger mark time interval.

8. The distributed audio transcription method according to claim 2, characterized in that, The operating modes of the multiple neural network models are dynamically switched based on the remaining battery level of the earphone charging case. The operating modes include full-function mode, balanced mode, and battery life mode.

9. The distributed audio transcription method according to claim 1, characterized in that, The second tag in the content value tag is further divided into low-priority content and discarded content: Among them, the acoustic embedding data corresponding to the audio segments that are determined to be valid content are transmitted in real time through a high-priority transmission queue; The acoustic embedding data corresponding to audio segments that are determined to be low-priority content is temporarily stored on the front-end device and transmitted in the background as needed. Audio segments deemed as discardable content will have their processing terminated.

10. A distributed audio transcription system, characterized in that, Includes the earphone charging case and terminal device, among which: The earphone charging case is configured to perform sliding window processing on the audio after audio acquisition and before transmission to obtain multiple audio segments; The earphone charging case is configured to perform content analysis on the plurality of audio segments and determine a content value tag for each audio segment. The content value tag includes at least a first tag representing valid content and a second tag representing invalid content. The earphone charging case is configured to convert an audio segment with a content value tag of the first tag into acoustic embedding data through an acoustic encoder. The earphone charging case is configured to transmit the acoustic embedding data from the earphone charging case to a terminal device via a wireless link; The terminal device is configured to decode the acoustic embedding data and generate text corresponding to the valid content.

Images