An audio data processing method and system

Abstract

The application provides an audio data processing method and system, and relates to the field of audio data processing. The method comprises the following steps: acquiring an audio stream and marking information; segmenting the audio stream based on the marking information to obtain a plurality of audio segments; performing a transcription operation on each audio segment to generate corresponding text data; calculating an integrity check value based on the plurality of audio segments and the text data; and storing the integrity check value in association with the plurality of audio segments and the text data. The technical problems of difficulty in quickly locating key content, privacy leakage risk, and easy tampering of the transcription text content of the long recording data in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of audio data processing, and in particular to an audio data processing method and system. Background Technology

[0002] In the field of intelligent recording peripherals, existing intelligent recording devices typically use local capture and storage of complete audio files. Users bookmark the audio by pressing buttons to record time points. The device then uploads the entire audio segment to a cloud server via the network. The cloud-based automatic speech recognition system transcribes the entire audio segment and returns plain text. Users can manually drag the progress bar or jump to bookmarks to locate key points when listening back to the text on the terminal. However, in practical applications, this processing method results in bookmarks only serving as playback jump points and having no linkage with transcription, retrieval, or export. Furthermore, the audio and text are stored as ordinary files without any integrity verification mechanism, making them easy to edit and replace, and their authenticity is difficult to verify. Therefore, there is an urgent need for a method to solve the technical problems of difficulty in quickly locating key content in long recording data, privacy leakage risks, and the ease with which recording data and transcribed text can be tampered with in existing technologies. Summary of the Invention

[0003] This application provides an audio data processing method and system, which solves the technical problem that recording data and transcribed text are easily tampered with in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, an audio data processing method is provided, comprising: acquiring an audio stream and tagging information; the tagging information includes a tag identifier, a timestamp, a tag type, a trigger source, and a tag label; wherein the tag type includes an important tag, a pending tag, a question tag, and an end tag; segmenting the audio stream using the tagging information as boundaries to obtain multiple audio segments; performing a transcription operation on each audio segment to generate corresponding text data; calculating an integrity check value based on the multiple audio segments and the text data; and associating and storing the integrity check value with the multiple audio segments and the text data.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, the audio stream is segmented using the marker information as a boundary to obtain multiple audio segments. This includes: sorting the marker information by timestamp, merging adjacent marker information with a timestamp interval less than a preset time interval threshold to obtain preprocessed marker information; segmenting the audio stream using the preprocessed marker information as a boundary to obtain initial audio segments; and removing invalid audio segments from the initial audio segments whose duration is less than a preset duration threshold to obtain the remaining multiple audio segments.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, a transcription operation is performed on each audio segment to generate corresponding text data, including: extracting overlapping areas on both sides of the segment boundary of adjacent audio segments; participating the audio data of the overlapping areas in the transcription operation of two adjacent audio segments respectively to obtain boundary area recognition results; performing alignment and deduplication processing on the boundary area recognition results, and generating text data based on the deduplication results.

[0007] In conjunction with the first aspect above, in one possible implementation, before performing the transcription operation on each audio segment, the method further includes: performing speech activity detection on each audio segment and removing the silent regions at the beginning and end of each audio segment; and inputting the audio segment after removing the silent regions into an automatic speech recognition model for transcription.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the boundary area recognition results are aligned and deduplicated, and text data is generated based on the deduplicated results. This includes: extracting text features and time information from the boundary area recognition results; calculating the text matching relationship between adjacent boundary area recognition results based on the text features, and determining the time overlap range of the corresponding text content based on the time information; when the text content in adjacent boundary area recognition results meets the preset matching conditions and the corresponding time range overlaps, the text content is identified as duplicate recognition content, and one of the text results is retained; the boundary area recognition results after removing duplicate recognition content and the non-overlapping area recognition results of the corresponding audio segment are concatenated to generate the text data corresponding to the audio segment.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the integrity verification value is calculated based on multiple audio segments and text data, including: obtaining the audio data, text data, and segment association information corresponding to each audio segment respectively; combining the audio data, text data, and segment association information according to a preset data combination rule to generate data to be verified; performing a digest calculation on the data to be verified based on a preset hash algorithm to obtain the integrity verification value corresponding to each audio segment; and associating the integrity verification value corresponding to each audio segment with the segment identifier of the audio segment to obtain segment-level integrity verification information.

[0010] In conjunction with the first aspect mentioned above, one possible implementation involves associating the integrity check value with multiple audio segments and text data, including: obtaining the segment identifier, start time, end time, text data, and integrity check value corresponding to each audio segment; associating the segment identifier, start time, end time, text data, and corresponding integrity check value into an index file to establish a correspondence between the integrity check value, audio segments, and text data; generating an integrity check file based on the integrity check value corresponding to each audio segment, the integrity check file including the integrity check value corresponding to each audio segment and recording metadata; and associating the index file and the integrity check file on the device.

[0011] In conjunction with the first aspect above, in one possible implementation, after generating the corresponding text data, the method further includes: generating structured input data based on the text data and corresponding tagging information; switching the audio data processing device to the HID mode of the human-machine interface device; converting the structured input data into an HID key sequence and sending it to the electronic device according to the time sequence corresponding to the text data, wherein the electronic device generates corresponding text based on the HID key sequence.

[0012] In a second aspect, an audio data processing system is provided, which performs the methods described in the first aspect and any possible implementation thereof. The system includes: a microphone or microphone array for acquiring audio streams; a tag input unit for receiving tag operations and generating corresponding tag information, the tag input unit including physical buttons, a motion sensor, a voice keyword detection unit, and a host computer communication interface; a main control unit connected to the microphone or microphone array and the tag input unit, which performs the methods described in the first aspect and any possible implementation thereof; a storage unit connected to the main control unit for storing audio segments, text data, index files, and integrity verification files; and a communication interface connected to the main control unit for exporting audio segments, text data, index files, and integrity verification files, the communication interface including a USB interface and supporting USB mass storage mode and HID human-machine interface device mode.

[0013] In conjunction with the second aspect mentioned above, in one possible implementation, the main control unit includes: a tag processing module, used to parse the tag information generated by the tag input unit, obtain the corresponding timestamp and tag type, and determine the audio segment division boundary based on the tag information; a segmentation processing module, used to segment the audio stream according to the division boundary, generating multiple audio segments with corresponding segment identifiers; an end-side transcription module, used to call the automatic speech recognition model deployed on the device to transcribe the multiple audio segments and generate corresponding text data; an integrity verification module, used to generate corresponding integrity verification values ​​based on the audio segments, text data, and segment association information; and an index generation module, used to generate an index file containing audio segment information, text data, and integrity verification values ​​to establish the association between audio segments, text data, and integrity verification values.

[0014] This application provides an audio data processing method and system. It segments the audio stream using marker information as boundaries, performs transcription on each audio segment to generate text data, and then calculates an integrity check value based on multiple audio segments and text data. Finally, it associates and stores the integrity check value with multiple audio segments and text data. By combining audio and text data to calculate a hash digest and writing it to an index file, any tampering with the audio segment or transcribed text will result in a discrepancy between the recalculated hash value and the stored integrity check value. This allows for the detection of content tampering locally on the device without requiring a network connection or third-party services, effectively ensuring the authenticity and reliability of the recorded data and transcribed text. Simultaneously, the marker information is integrated throughout the segmentation, transcription, verification, and storage process, achieving end-to-end closed-loop processing within a single device. This avoids the cumbersome process of manually connecting multiple software programs and eliminates the privacy leakage risks associated with uploading audio and text to the cloud.

[0015] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0016] Figure 1A system architecture diagram of an audio data processing system provided in this application embodiment; Figure 2 A flowchart illustrating an audio data processing method provided in this application embodiment. Figure 1 ; Figure 3 A flowchart illustrating an audio data processing method provided in this application embodiment. Figure 2 ; Figure 4 A flowchart illustrating an audio data processing method provided in this application embodiment. Figure 3 ; Figure 5 A flowchart illustrating an audio data processing method provided in this application embodiment. Figure 4 ; Figure 6 A flowchart illustrating an audio data processing method provided in this application embodiment. Figure 5 . Detailed Implementation

[0017] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0018] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0019] Example 1: The audio data processing method provided in this application embodiment can be applied to, for example, Figure 1 In the audio data processing system shown, such as Figure 1As shown, the system includes: a microphone or microphone array 101 for acquiring audio streams; a tag input unit 102 for receiving tag operations and generating corresponding tag information, the tag input unit including physical buttons, motion sensors, a voice keyword detection unit, and a host computer communication interface; a main control unit 103 connected to the microphone or microphone array and the tag input unit, for executing any of the following methods; a storage unit 104 connected to the main control unit, for storing audio segments, text data, index files, and integrity verification files; and a communication interface 105 connected to the main control unit, for exporting audio segments, text data, index files, and integrity verification files, the communication interface including a USB interface and supporting USB mass storage mode and HID human-machine interface device mode.

[0020] In one possible implementation, the main control unit 103 includes: a tag processing module, used to parse the tag information generated by the tag input unit, obtain the corresponding timestamp and tag type, and determine the audio segment division boundary based on the tag information; a segmentation processing module, used to segment the audio stream according to the division boundary and generate multiple audio segments with corresponding segment identifiers; an end-side transcription module, used to call the automatic speech recognition model deployed on the device to transcribe the multiple audio segments and generate corresponding text data; an integrity verification module, used to generate corresponding integrity verification values ​​based on the audio segments, text data, and segment association information; and an index generation module, used to generate an index file containing audio segment information, text data, and integrity verification values ​​to establish the association between audio segments, text data, and integrity verification values.

[0021] Example 2: To address the technical problem that recorded data and transcribed text are easily tampered with in the prior art, embodiments of this application provide an audio data processing method. Figure 2 A flowchart illustrating the audio data processing method provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the method includes: S201. Obtain the audio stream and tagging information.

[0022] The audio stream refers to a continuous digital audio signal acquired by an audio acquisition device and arranged in a time sequence. Tagging information refers to structured data triggered by the user or system during or after the audio stream acquisition process to identify specific points in time or events. Tagging information includes tag identifiers, timestamps, tag types, trigger sources, and tag labels. Tagging types include priority tags, pending tags, question tags, and end tags. The tag identifier is a unique number for the tag information; the timestamp is the absolute time point at which the tag was generated; the tag type is the semantic classification set by the user for the recorded content; the trigger source is the physical or software channel that generated the tag; and the tag label is a user-defined extended text annotation.

[0023] In one possible implementation, the system continuously collects ambient sound and generates an audio stream using a built-in microphone or microphone array at a preset sampling rate. Simultaneously, it monitors various trigger sources, generating timestamp-infused tag information upon detection of a trigger. The audio stream and tag information are time-aligned based on a unified system clock. When generating tag information, the system assigns a globally unique tag number, records the current system time timestamp, parses the type field (e.g., important, pending task) based on user actions or voice commands, records the source field (e.g., physical button, KWS), and receives the label text input by the user. In subsequent processing, the system writes this multi-dimensional tag information as part of the segment association information into an index file, enabling users to filter and retrieve recording segments by tag type or label.

[0024] It should be noted that the tagging information is not limited to manual triggering, but can also include tags automatically detected based on voice keywords or automatically generated based on external commands. The acquisition of tagging information and the acquisition of audio streams are parallel and continuous processes.

[0025] S202. The audio stream is segmented using the marker information as the boundary to obtain multiple audio segments.

[0026] Segmentation refers to the process of dividing a continuous audio stream into several independent and continuous audio data segments based on the position of the marker information on the time axis.

[0027] In one possible implementation, the system extracts the timestamps of all the marker information, arranges them in chronological order, and extracts the audio stream data between two adjacent timestamps as an independent audio segment. The first segment runs from the start of recording to the first timestamp, and the last segment runs from the last timestamp to the end of recording.

[0028] It should be noted that segmenting based on the marked information makes each audio segment semantically correspond to a specific meeting segment or topic, rather than simply dividing them into segments of equal duration. This segmentation method preserves the semantic integrity of the audio content.

[0029] Based on the above steps, the system uses tagging information to drive segmentation, transforming lengthy linear audio into structured segments with semantic boundaries, thereby improving the efficiency of locating key content in long recordings.

[0030] S203. Perform a transcription operation on each audio segment to generate the corresponding text data.

[0031] Transcription refers to the process of converting speech signals in audio segments into corresponding text using automatic speech recognition technology.

[0032] In one possible implementation, the system deploys a lightweight automatic speech recognition model locally on the device. Multiple segmented audio segments are input into the model sequentially or in parallel, and the model outputs the text sequence corresponding to each audio segment as text data.

[0033] It should be noted that the transcription operation is performed entirely locally on the device. Neither the audio data nor the generated text data leaves the device, and there is no need to rely on a network connection or cloud server.

[0034] As an example, the system calls the built-in quantization Conformer model to perform inference on a 15-minute audio segment, outputting text data containing sentence-by-sentence text and corresponding timestamps.

[0035] Based on the above steps, the system completes the conversion from audio to text in a closed loop on the device side, avoiding the risk of privacy leakage caused by uploading audio data to the cloud. At the same time, segmented transcription reduces the computational pressure of a single inference.

[0036] S204. Calculate the integrity check value based on multiple audio segments and text data.

[0037] The integrity check value refers to a fixed-length digest value generated by performing specific cryptographic operations on the data to verify whether the data has been tampered with.

[0038] In one possible implementation, the system acquires the audio data and its corresponding text data for each audio segment, combines the audio data and text data according to a preset byte stream concatenation rule, then uses a hash algorithm to perform a digest calculation on the combined data, and uses the output hash value as the integrity verification value of the segment.

[0039] It should be noted that the calculation of the integrity check value is not limited to audio data, but rather the audio is calculated in conjunction with the transcribed text. This means that any editing of the audio waveform or modification of the transcribed text will result in the recalculated check value not matching the original value.

[0040] As an example, the system uses the SHA-256 algorithm to concatenate the PCM audio data of an audio segment with the UTF-8 encoded text data and then calculates a digest to obtain a 256-bit integrity check value.

[0041] Based on the above steps, the system binds audio content with text content using a cryptographic digest algorithm, generating a unique digital fingerprint for each piece of data, enabling subsequent data tampering to be accurately detected.

[0042] S205. Associate and store the integrity check value with multiple audio segments and text data.

[0043] Among them, associated storage refers to the process of establishing a mapping relationship between the integrity verification value and its corresponding audio segments and text data in the storage medium and persistently saving them.

[0044] In one possible implementation, the system creates a structured index file in the local storage medium, writes the identifier, start and end times, text data and corresponding integrity check value of each audio segment into the same data record, and saves the original audio file of the audio segment and the index file in the same storage directory.

[0045] It should be noted that associated storage makes the integrity check value and the audio data physically independent but logically tightly bound. When a third party performs verification, it only needs to read the check value in the index file and compare it with the recalculation result.

[0046] Based on the above steps, the system achieves deep integration and storage of verification information and audio-text data. Without needing to connect to the internet or rely on third-party services, it can detect content tampering locally on the device, effectively ensuring the authenticity and reliability of the recording data and transcribed text.

[0047] This application embodiment acquires the audio stream and tagging information, segments the audio stream using the tagging information as boundaries, and performs a transcription operation on each audio segment to generate text data. Then, it calculates an integrity check value based on multiple audio segments and text data, and finally associates and stores the integrity check value with multiple audio segments and text data. This achieves end-to-end closed-loop processing from recording acquisition, tagging and segmentation, end-side transcription to local evidence storage, effectively preventing audio and text data from being tampered with while ensuring data privacy.

[0048] Example 3: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S202 can be specifically implemented through the following S301, S302 and S303, which are explained in detail below: S301. Sort the tag information by timestamp, merge adjacent tag information with timestamp intervals less than a preset time interval threshold, and obtain preprocessed tag information.

[0049] The timestamp refers to the absolute time point bound to the generated tag information, and the preset time interval threshold refers to the time difference limit value pre-set by the system to determine whether two adjacent tag information belong to the same event.

[0050] In one possible implementation, after the recording ends, the system extracts the timestamps of all the marker information, arranges them in ascending order according to their time sequence, and calculates the time difference between two adjacent marker information in turn. If the time difference is less than a preset time interval threshold, it is determined that the two marker information are caused by the continuous actions generated by the same triggering event, and they are merged into one marker information as preprocessed marker information.

[0051] Preferably, a preset time interval threshold Satisfy the following formula:

[0052] in, This represents the timestamp of the i-th marker after being sorted by timestamp. This represents the timestamp of the (i+1)th tag after sorting by timestamp. When the time interval is less than a preset threshold, the i-th and (i+1)-th markers are merged. The preset time interval threshold can range from 200 milliseconds to 500 milliseconds. If the threshold is set too small, such as below 200 milliseconds, redundant markers generated by key bounce or continuous accidental touches cannot be effectively merged, resulting in the same semantic content being incorrectly segmented. If the threshold is set too large, such as above 500 milliseconds, two independent markers continuously pressed by the user for different topics may be incorrectly merged, destroying semantic boundaries. Therefore, choosing a range of 200 milliseconds to 500 milliseconds can filter out accidental touches while preserving the true semantic boundaries.

[0053] It should be noted that the merging operation does not delete redundant tag information, but rather uses the timestamp of the first tag information as the timestamp of the preprocessed tag information after merging, while retaining attribute information such as tag type.

[0054] Based on the above steps, the system eliminates redundant marker interference caused by continuous triggering by sorting timestamps and merging adjacent markers, ensuring that the segment boundaries correspond to the true semantic event starting points.

[0055] S302. Using the preprocessing marker information as the boundary, the audio stream is segmented to obtain the initial audio segments.

[0056] The initial audio segment refers to the unfiltered audio data segment directly segmented based on the preprocessing tagging information.

[0057] In one possible implementation, the system maps the preprocessed tagging information to the timeline of the audio stream by timestamp, and uses the interval between every two adjacent preprocessed tagging information as a segment to cut the audio stream, thereby generating several initial audio segments.

[0058] It should be noted that the number of initial audio segments is one more than the number of preprocessed markers. The first segment starts at the beginning of the audio stream and ends at the first preprocessed marker, while the last segment starts at the last preprocessed marker and ends at the end of the audio stream.

[0059] Based on the above steps, the system uses the preprocessed clean tagging information to complete the structured segmentation of the audio stream, providing basic data units for subsequent segment-level processing.

[0060] S303. Remove invalid audio segments from the initial audio segments whose duration is lower than the preset duration threshold to obtain the remaining audio segments.

[0061] Among them, the preset duration threshold refers to the minimum duration limit set by the system to determine whether an audio segment has processing value, and an invalid audio segment refers to a segment that is too short to contain effective semantic information.

[0062] In one possible implementation, the system calculates the difference between the end time and the start time of each initial audio segment to obtain the segment duration. The segment duration is then compared with a preset duration threshold. If the segment duration is less than the preset duration threshold, the initial audio segment is marked as an invalid audio segment and discarded. Segments with a duration greater than or equal to the preset duration threshold are retained as the final multiple audio segments.

[0063] Preferably, the duration of the initial audio segment Satisfy the following formula:

[0064] in, Indicates the end time of the initial audio segment. Indicates the start time of the initial audio segment. When Audio segments shorter than a preset duration threshold are considered invalid. The preset duration threshold can range from 0.3 seconds to 0.8 seconds. If the threshold is set too low, for example, below 0.3 seconds, fragmented segments containing only key presses or coughs cannot be filtered out, causing the subsequent transcription model to perform invalid calculations on these meaningless audio segments. If the threshold is set too high, for example, above 0.8 seconds, valid segments containing short confirmation phrases may be mistakenly identified as invalid and removed, resulting in semantic loss.

[0065] It should be noted that the time intervals of the invalid audio segments that are removed are marked in the subsequent index file to ensure the continuity and traceability of the audio stream timeline.

[0066] Based on the above steps, the system filters out fragmented audio without effective semantics by using a duration threshold, avoiding waste of edge computing resources and improving the audio quality of the input transcription model.

[0067] This application embodiment achieves accurate and denoised segmentation of the audio stream by sorting and merging the marker information by timestamp to eliminate redundant triggers and removing invalid audio segments with too short a duration. This improves the effectiveness of subsequent transcription and indexing and avoids interference to the end-side transcription model caused by erroneous segmentation.

[0068] Example 4: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S203 can be specifically implemented through the following S401, S402 and S403, which are explained in detail below: S401. Extract the overlapping area on both sides of the segment boundary of adjacent audio segments.

[0069] The overlapping area refers to the audio data interval formed by extending a preset duration before and after the segment boundary of two adjacent audio segments. The audio data within this interval belongs to the processing range of both audio segments.

[0070] In one possible implementation, after determining the segmentation time point between two adjacent audio segments, the system uses that time point as the center to extract audio data of a first preset duration from the end of the preceding audio segment and extracts audio data of a second preset duration from the beginning of the following audio segment. These two audio data segments together form the overlapping area of ​​the boundary. Furthermore, the first and second preset durations can be the same or different, typically ranging from 50 milliseconds to 200 milliseconds, to ensure coverage of complete word pronunciations that may cross the boundary.

[0071] It should be noted that the purpose of extracting the overlapping area is to provide redundant data for subsequent boundary alignment. Since human speech is continuous, a complete word or phrase is often cut off by segment boundaries. If it is transcribed independently according to the segmentation points, the words at the boundaries will not be correctly recognized due to audio truncation. By extracting the overlapping area, the acoustic features of the context before and after the boundary are preserved.

[0072] Based on the above steps, the system establishes an audio redundancy region spanning the preceding and following segments at the segment boundaries, so that the acoustic features cut off by the boundaries can be fully presented in the overlapping area, providing a data foundation for subsequent continuous semantic recognition.

[0073] S402. The audio data of the overlapping area are used in the transcription operation of the two adjacent audio segments respectively to obtain the boundary area recognition result.

[0074] Among them, the boundary area recognition result refers to the text content and time information corresponding to the overlapping area output by the automatic speech recognition model after the audio data of the overlapping area is attached to the two adjacent audio segments for transcription.

[0075] In one possible implementation, when the system inputs the Nth audio segment into the transcription model, it also inputs the audio data of the overlapping area at the end of the segment. This allows the model to continuously identify the ending portion of the Nth segment using the context information of the overlapping area. Simultaneously, when the (N+1)th audio segment is input into the transcription model, the audio data of the overlapping area at the beginning of the segment is also input. This allows the model to continuously identify the beginning portion of the (N+1)th segment using the context information of the overlapping area. Thus, the audio data of the same overlapping area is used for two transcriptions, generating boundary region identification results belonging to the preceding audio segment and those belonging to the following audio segment, respectively.

[0076] It should be noted that this sequential logic of using overlapping data to transcribe adjacent segments means that when the model processes the boundary, it no longer faces the truncated audio in isolation, but obtains the complete acoustic context. This enables it to accurately identify words cut off by the boundary, avoiding recognition errors or word loss caused by hard segmentation.

[0077] Based on the above steps, the system uses bidirectional participation of overlapping area data in transcription, enabling adjacent segments to obtain complete acoustic context at the boundary, effectively avoiding the loss and misidentification of boundary words, and improving the overall accuracy of segmented transcription.

[0078] S403. Align and deduplicate the boundary area identification results, and generate text data based on the deduplicated results.

[0079] Among them, alignment and deduplication refers to comparing the recognition results of overlapping areas generated by two transcriptions, identifying duplicate content and removing it, and retaining only one accurate result.

[0080] In one possible implementation, the system extracts the boundary region recognition results generated by transcribing two adjacent audio segments. By comparing the text content and time information, it determines whether there is duplicate text pointing to the same audio interval in the two results. When duplicate recognition content is determined, the system retains the text result with higher confidence or more accurate timestamp, and concatenates the deduplicated boundary region recognition result with the non-overlapping region recognition result of the corresponding audio segment to generate the final text data.

[0081] It should be noted that since the overlapping area is transcribed twice, some duplicate text content will inevitably be generated. If the text is directly spliced, it will result in text duplication. Through alignment and deduplication processing, the system can eliminate redundant text while preserving the complete semantics of the boundary, ensuring a one-to-one correspondence between the generated text data and the audio content on the timeline.

[0082] Based on the above steps, the system accurately deduplicates and splices the repeated content in the boundary area, ensuring the semantic integrity of the boundary while avoiding text redundancy, generating coherent and accurate text data, and achieving the equivalent effect of segmented transcription and whole-segment transcription in terms of semantic coherence.

[0083] This application embodiment solves the problem of semantic fragmentation at the boundary caused by independent segmented transcription by extracting overlapping areas on both sides of the segment boundary, using the overlapping area data to participate in the transcription operation of two adjacent audio segments, and performing alignment and deduplication processing on the boundary area recognition results, thus ensuring the continuity and integrity of the segmented transcribed text.

[0084] Example 5: In one possible implementation of this application embodiment, before S401, the audio data processing method provided in this application embodiment further includes the following method: In one possible implementation, speech activity detection is performed on each audio segment, and the silent regions at the beginning and end of each audio segment are removed. After the system completes the segmentation of the audio stream, a speech activity detection unit deployed at the input end of each audio segment is used to determine the speech state of each audio frame in the audio segment. Specifically, the audio frames can be classified by extracting acoustic features such as short-time energy, zero-crossing rate, or Mel-frequency cepstral coefficients, or by using neural network feature extraction methods, to determine whether each frame belongs to a speech frame. Based on the speech detection results of each audio frame, the interval from the start position to the first speech frame in the audio segment is located as the head silence region, and the interval from the last speech frame to the end position is located as the tail silence region. The aforementioned head and tail silence regions are then trimmed and removed from the audio segment.

[0085] Among them, speech activity detection is a technique used to distinguish speech segments from non-speech segments (such as silence and environmental noise) in an audio signal. The silence region refers to the interval in an audio segment that does not contain valid human speech signals.

[0086] Furthermore, the audio segment after removing silence is input into the automatic speech recognition model for transcription. The system directly sends the clean audio segment, after silence removal, into the automatic speech recognition model inference engine on the edge. The model decodes the effective speech signal based on acoustic features and language models, and outputs the corresponding text sequence. Since the input data has been freed from silence interference, the model can focus more on the extraction and mapping of effective speech features.

[0087] Among them, the automatic speech recognition model refers to a lightweight algorithm model deployed locally on the device to convert speech signals into text.

[0088] It should be noted that after removing the silent regions at the beginning and end of the audio segment, the processed audio segment will have a time offset relative to the original audio segment. The system records the time offset corresponding to the silent region at the beginning and performs time correction based on the time offset when generating the timestamp corresponding to the text data, so that the text time information can be mapped to the corresponding position in the original audio stream.

[0089] Based on the above technical solution, the system detects speech activity in each audio segment and removes the silent areas at the beginning and end, inputting the clean audio into the automatic speech recognition model. This achieves the saving of transcription computing resources and the improvement of recognition accuracy, effectively adapting to the operating environment where the resources of the end device are limited.

[0090] Example 6: In one possible implementation of the embodiments of this application, combined with Figure 4 ,like Figure 5As shown, the above S403 can be specifically implemented through the following S501 to S504, which are explained in detail below: S501. Extract text features and time information from the boundary area recognition results.

[0091] Among them, text features refer to the data used to characterize the attributes of the transcribed text content, and time information refers to the time interval corresponding to the text content in the audio stream.

[0092] In one possible implementation, after the system obtains the boundary area recognition results generated by transcribing the overlapping area of ​​two adjacent audio segments, it extracts the text string corresponding to each word or character from the structured data output by the transcription model as text features, and at the same time extracts the time interval formed by the start timestamp and end timestamp corresponding to each word or character as time information.

[0093] S502. Calculate the text matching relationship between the recognition results of adjacent boundary areas based on text features, and determine the time overlap range of the corresponding text content based on time information.

[0094] Among them, text matching relationship is used to quantify the similarity between two text features at the semantic or literal level, and time overlap range is used to quantify the size of the intersection of two time intervals on the time axis.

[0095] In one possible implementation, the system inputs the text features from the identification results of two adjacent boundary regions into a string matching algorithm or a semantic similarity calculation model, outputs a matching degree value to characterize the text matching relationship, and calculates the ratio of the intersection duration to the union duration of two time intervals, using this ratio as the time overlap range.

[0096] Preferably, the text matching relationship satisfies the following formula:

[0097] in, Indicates the degree of text matching. This represents the feature vector of the i-th word or character in the previous boundary region recognition result. s represents the feature vector of the i-th word or character at the corresponding position in the subsequent boundary region recognition result. This represents the similarity calculation function, where n represents the number of words or characters involved in the comparison. When M is greater than a preset matching threshold, the preset matching condition is satisfied. The preset matching threshold can range from 0.8 to 0.95. If the threshold is set too low, words with similar pronunciations but different semantic meanings may be misjudged as duplicates; if the threshold is set too high, slight differences in the model's output may cause duplicate content to be missed. Simultaneously, the time overlap range is determined when the intersection duration accounts for more than 50% of the total duration of the shorter time interval, indicating an overlap in the corresponding time range.

[0098] It should be noted that relying solely on text matching or solely on time overlap cannot accurately determine duplicates, because the same text may appear at different time points, while different text may appear at the same time point due to noise. Therefore, both methods must be combined for cross-validation.

[0099] S503. When the text content in the recognition results of adjacent boundary areas meets the preset matching conditions and the corresponding time ranges overlap, the text content is identified as duplicate recognition content, and one of the text results is retained.

[0100] Among them, duplicate content refers to redundant text generated by the transcription of adjacent segments from the same audio interval, and retaining one text result means selecting and retaining a single text according to a certain strategy to eliminate redundancy.

[0101] In one possible implementation, when the system determines that the recognition results of adjacent boundary areas simultaneously meet the text matching threshold and the time overlap threshold, the text content is marked as duplicate recognition content. Subsequently, the system obtains the confidence scores of the two text results, retains the text result with higher confidence, or retains the text result with more accurate timestamp, and discards the other result.

[0102] S504. The boundary area recognition results after removing duplicate recognition content and the non-overlapping area recognition results of the corresponding audio segment are concatenated to generate the text data corresponding to the audio segment.

[0103] Among them, the non-overlapping region identification result refers to the transcribed text of the part of the audio segment other than the overlapping area, and splicing refers to the process of combining the deduplicated boundary text with the non-overlapping area text in chronological order.

[0104] In one possible implementation, after the system completes deduplication and retains a single boundary text result, it concatenates the boundary text result with the non-overlapping region identification result of the previous audio segment in timestamp order, and concatenates it with the non-overlapping region identification result of the subsequent audio segment in timestamp order, thereby generating a complete text data that is coherent from beginning to end.

[0105] It should be noted that during the splicing process, the system will synchronously adjust the timestamps corresponding to the text to ensure that the spliced ​​text data is continuous and unbroken on the timeline, and maintains a precise mapping with the physical location of the original audio stream.

[0106] Based on the above technical solution, the system extracts text features and time information, calculates text matching relationships and time overlap ranges, and retains a single result and splices it when both conditions are met. This achieves accurate deduplication of boundary duplicate recognition content, effectively prevents the loss and duplication of boundary words, and improves the accuracy and coherence of text data.

[0107] Example 7: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 6 As shown, the above S204 can be specifically implemented through the following S601 to S605, which are explained in detail below: S601. Obtain the audio data, text data, and segment association information corresponding to each audio segment.

[0108] The segment association information refers to metadata used to describe the attributes and contextual relationships of audio segments, including but not limited to segment identifier, start and end timestamps, tag type, and transcription confidence. This application uses millisecond-level timestamps.

[0109] In one possible implementation, after completing the end-side transcription, the system reads the original PCM audio data of each audio segment from the local storage medium, extracts the corresponding UTF-8 encoded text data from the output of the transcription module, and extracts the segment identifier, start time, end time and other related information from the log of the tagging processing module. The above three types of data are temporarily stored in the memory buffer for subsequent combination.

[0110] It should be noted that the fragment association information is not limited to time information, but can also include device identifiers, recording scene tags, etc. The introduction of this information makes the subsequently generated verification value not only bound to the audio and text content itself, but also bound to the content's production environment, further enhancing the dimensionality and tamper-proof strength of the evidence data.

[0111] Based on the above steps, the system completed the classification and extraction of the verification source data, providing a complete input foundation for subsequent data combination and summary calculation.

[0112] S602. According to the preset data combination rules, combine the audio data, text data and segment association information to generate the data to be verified.

[0113] Among them, the preset data combination rules refer to the rules predefined by the system for concatenating multiple types of data into a unified byte stream.

[0114] In one possible implementation, the system converts the audio data, text data, and segment association information into binary byte streams, concatenates them in the order of segment association information, text data, and audio data, and inserts preset delimiters and length markers between each data segment. The resulting continuous byte stream is the data to be verified. It should be understood that the data combination order is not limited to the above method; audio data can also be used first, followed by text data, as long as the same combination rules are applied at the verification end.

[0115] It should be noted that the purpose of introducing delimiters and length markers is to prevent ambiguity at the splicing boundaries of different types of data, ensure that the data to be verified has a unique parsing structure, and thus guarantee the determinism and reproducibility of subsequent hash calculations.

[0116] Based on the above steps, the system integrates multidimensional data into a single data stream through structured splicing, so that any data tampering in any dimension can be reflected in the final byte stream changes, achieving comprehensive coverage of the tamper-proof object.

[0117] S603. Calculate the digest of the data to be verified based on the preset hash algorithm to obtain the integrity verification value corresponding to each audio segment.

[0118] Among them, the preset hash algorithm refers to a cryptographic function that can map input data of arbitrary length to a fixed-length output digest, including but not limited to SHA-256, SHA-3 or SM3 algorithms.

[0119] In one possible implementation, the system calls the device's hardware encryption engine or software hash library to input the generated data to be verified into the SHA-256 algorithm. The algorithm performs padding, block division, and multiple rounds of compression operations on the data, and finally outputs a 256-bit hexadecimal string, which is the integrity verification value of the audio segment.

[0120] It should be noted that hash algorithms are one-way and collision-resistant, which means that attackers cannot reverse-engineer the original data from the check value, nor can they find two different audio and text data that produce the same check value, thus ensuring the unforgeability of the data at the cryptographic level.

[0121] Based on the above steps, the system uses a cryptographic digest algorithm to generate a unique digital fingerprint for each audio segment, enabling subsequent data tampering to be accurately detected.

[0122] S604. Associate the integrity check value corresponding to each audio segment with the segment identifier of the audio segment to obtain segment-level integrity check information.

[0123] Among them, fragment-level integrity verification information refers to the key-value pair mapping relationship established between integrity verification values ​​and fragment identifiers.

[0124] In one possible implementation, the system binds the calculated integrity check value to its corresponding fragment identifier, constructs a data structure such as "{seg_id: hash_value}", and temporarily stores this structure in memory as fragment-level integrity check information so that it can be directly called when generating the index file.

[0125] It should be noted that associating segments through segment identifiers enables the rapid location and verification of the integrity of specific segments within massive amounts of segment data, improving the flexibility and efficiency of the verification process.

[0126] Based on the above steps, the system establishes a correspondence between check values ​​and specific segments, providing structured data units for the subsequent construction of a global check file.

[0127] S605. Obtain the segment identifier, start time, end time, text data, and integrity check value corresponding to each audio segment. Write the segment identifier, start time, end time, text data, and corresponding integrity check value into an index file. Establish the correspondence between integrity check value, audio segment, and text data. Generate an integrity check file based on the integrity check value corresponding to each audio segment. The integrity check file includes the integrity check value corresponding to each audio segment and recording metadata. Store the index file and integrity check file together on the device.

[0128] The index file is a structured file that records the mapping relationship between audio segments and text data. The integrity verification file is an independent verification dataset that summarizes the verification values ​​of each segment and global metadata. The recording metadata includes the recording start and end time, the total number of segments, the device identifier, and the tag summary, etc.

[0129] In one possible implementation, the system creates a JSON-formatted index file on the local storage medium, writing the segment identifier, start time, end time, text data, and integrity check value of each audio segment as a data object into this file; at the same time, the system creates a separate integrity check file, summarizing the integrity check values ​​and recording metadata of all segments into it, and associates the index file and the integrity check file in the storage directory through filename hashes or directory references.

[0130] It should be noted that by writing the integrity check value into both the index file and a separate integrity check file, a double-redundant check data network is formed. During third-party verification, single-segment check values ​​can be extracted from the index file for local verification, or the global data can be verified as a whole through the integrity check file. The verification method is flexible and has a high fault tolerance rate.

[0131] Based on the above steps, the system achieves deep integration and associated storage of verification information and audio-text index, which facilitates independent verification by third parties locally, and can detect content tampering without network connection or reliance on third-party services.

[0132] Example 8: In one possible implementation of this application, after generating the corresponding text data in S203, the method provided by this application further includes: generating structured input data based on the text data and the corresponding tag information; switching the audio data processing device to the human-machine interface device HID mode; converting the structured input data into an HID key code sequence and sending it to the electronic device in the time order corresponding to the text data; and the electronic device generating the corresponding text based on the HID key code sequence.

[0133] In one possible implementation, the system acquires the tagging information corresponding to the text data and performs structured processing on the text data based on the tag type, tag label, and timestamp information in the tagging information. Based on the text data and tagging information, structured input data is generated according to preset data organization rules.

[0134] The tag type is used to characterize the attributes of the corresponding text content, such as emphasis tag, to-do tag, question tag, or end tag; the timestamp information is used to determine the positional relationship of the text data in the original audio stream.

[0135] As an example, multiple text data are sorted according to timestamp information, and the tag type, tag label and text content corresponding to each text data are combined to form a structured input sequence with time order, so that the text content output to external electronic devices remains consistent with the original audio recording process.

[0136] Furthermore, when it is necessary to send structured input data to external electronic devices, the audio data processing device is controlled to switch to the HID (Human Interface Device) working mode. The main control unit controls the communication interface to switch from data transmission mode to HID device mode, and generates the corresponding HID key code sequence based on the text characters in the structured input data.

[0137] The HID key code sequence includes the key codes corresponding to the text characters and control information for controlling the input order. The control information includes at least character input order information and text segment switching information.

[0138] Furthermore, the HID keycode sequence is sorted according to the time order in the structured input data and sent sequentially to the external electronic device via the communication interface. Upon receiving the HID keycode sequence, the external electronic device recognizes the audio data processing device as a standard human-machine input device and simulates keyboard input operations based on the HID keycode sequence to input the corresponding text content into the current application.

[0139] The above method enables text transmission between audio data processing devices and external electronic devices without the need to deploy dedicated data receiving programs in external electronic devices. At the same time, it utilizes tagging information to perform structured management of text content, improving the usability and interactive efficiency of the transcription output process.

[0140] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0141] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. An audio data processing method, characterized in that, include: Acquire audio stream and tagging information; the tagging information includes tag identifier, timestamp, tag type, trigger source, and tag label; wherein, the tag type includes important tag, to-do tag, question tag, and end tag; The audio stream is segmented using the marked information as boundaries to obtain multiple audio segments; Perform a transcription operation on each audio segment to generate the corresponding text data; Calculate the integrity check value based on the multiple audio segments and the text data; The integrity verification value is associated with and stored in conjunction with the multiple audio segments and the text data.

2. The method according to claim 1, characterized in that, The audio stream is segmented using the marked information as boundaries to obtain multiple audio segments, including: The tagging information is sorted by timestamp, and adjacent tagging information with a timestamp interval less than a preset time interval threshold is merged to obtain preprocessed tagging information; Using the preprocessed marker information as boundaries, the audio stream is segmented to obtain initial audio segments; Invalid audio segments with a duration lower than a preset duration threshold are removed from the initial audio segments to obtain the remaining audio segments.

3. The method according to claim 1, characterized in that, The process of transcribing each audio segment to generate corresponding text data includes: Extract the overlapping area on both sides of the segment boundary of adjacent audio segments; The audio data of the overlapping area is used in the transcription operation of two adjacent audio segments to obtain the boundary area recognition result; The boundary region identification results are aligned and deduplicated, and text data is generated based on the deduplicated results.

4. The method according to claim 1, characterized in that, Before performing the transcription operation on each audio segment, the method further includes: Speech activity detection is performed on each audio segment, and the silent regions at the beginning and end of each audio segment are removed; The audio segment after removing silent areas is input into an automatic speech recognition model for transcription.

5. The method according to claim 3, characterized in that, The boundary region identification results are aligned and deduplicated, and text data is generated based on the deduplicated results, including: Extract text features and time information from the boundary region recognition results; The text matching relationship between the recognition results of adjacent boundary areas is calculated based on the text features, and the time overlap range of the corresponding text content is determined based on the time information. When the text content in the recognition results of adjacent boundary areas meets the preset matching conditions and the corresponding time ranges overlap, the text content is identified as duplicate recognition content, and one of the text results is retained; The boundary area recognition results after removing duplicate recognition content are combined with the non-overlapping area recognition results of the corresponding audio segment to generate the text data corresponding to the audio segment.

6. The method according to claim 1, characterized in that, The calculation of the integrity verification value based on the multiple audio segments and the text data includes: Obtain the audio data, text data, and segment association information corresponding to each audio segment; According to the preset data combination rules, the audio data, text data and the segment association information are combined to generate data to be verified; The data to be verified is digested based on a preset hash algorithm to obtain the integrity verification value corresponding to each audio segment.

7. The method according to claim 1, characterized in that, The step of associating and storing the integrity verification value with the multiple audio segments and the text data includes: Obtain the segment identifier, start time, end time, text data, and integrity check value corresponding to each audio segment; The segment identifier, start time, end time, text data, and corresponding integrity check value are associated and written into the index file to establish the correspondence between the integrity check value, the audio segment, and the text data; An integrity verification file is generated based on the integrity verification value corresponding to each audio segment. The integrity verification file includes the integrity verification value corresponding to each audio segment and the recording metadata. The index file and the integrity verification file are associated and stored on the device.

8. The method according to claim 1, characterized in that, After generating the corresponding text data, the method further includes: Generate structured input data based on text data and corresponding tagging information; Switch the audio data processing device to HID (Human Machine Interface) mode; The structured input data is converted into an HID key sequence and sent to an electronic device in chronological order corresponding to the text data. The electronic device generates corresponding text based on the HID key sequence.

9. An audio data processing system for performing the method according to any one of claims 1 to 8, characterized in that, The system includes: A microphone or microphone array used to capture audio streams; A tag input unit is used to receive tagging operations and generate corresponding tagging information. The tag input unit includes physical buttons, motion sensors, a voice keyword detection unit, and a host computer communication interface. A main control unit, connected to the microphone or microphone array and the tag input unit, is used to execute the method according to any one of claims 1 to 8; The storage unit, connected to the main control unit, is used to store audio clips, text data, index files, and integrity verification files. A communication interface, connected to the main control unit, is used to export the audio segments, text data, index files, and integrity verification files. The communication interface includes a USB interface and supports USB mass storage mode and HID human-machine interface device mode.

10. The system according to claim 9, characterized in that, The main control unit includes: The tagging processing module is used to parse the tagging information generated by the tagging input unit, obtain the corresponding timestamp and tag type, and determine the audio segment division boundary based on the tagging information; The segmentation module is used to segment the audio stream according to the division boundary to generate multiple audio segments with corresponding segment identifiers. The device-side transcription module is used to call the automatic speech recognition model deployed on the device to transcribe the multiple audio segments and generate corresponding text data. The integrity verification module is used to generate a corresponding integrity verification value based on the audio segment, the text data, and the segment association information. The index generation module is used to generate an index file containing audio segment information, text data, and integrity check values, in order to establish the association between audio segments, text data, and integrity check values.

Images