Multi-modal data correction method and device, computer equipment and storage medium

Abstract

The invention relates to a multi-modal data correction method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining target multi-modal data, and generating a corresponding initial text based on the target multi-modal data, the target multi-modal data at least comprising voice data; sending the initial text to a plurality of terminals, so that the plurality of terminals perform real-time labeling operation based on the initial text to obtain collaborative labeling data; receiving collaborative annotation data sent by the plurality of terminals, and generating a collaborative annotation result according to the collaborative annotation data; and based on the collaborative labeling result, correcting the initial text to generate a corrected text. Therefore, a multi-user real-time collaborative labeling and dynamic correction mechanism is introduced in a multi-mode voice transcription process, and adaptive optimization and elastic computing power scheduling are combined, so that real-time editability, collaborative consistency and high accuracy of a transcription text are realized, and meanwhile, stable operation of a system in a high-concurrency scene is guaranteed.

Description

Multimodal data correction method, apparatus, computer equipment and storage medium Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimodal data correction method, apparatus, computer device and storage medium. Background Technology

[0002] With the increasing popularity of applications such as remote conferencing and online education, real-time speech-to-text, as an important application in the field of speech information processing, continues to grow in both frequency of use and scale of application. Existing speech-to-text systems typically rely on a single speech recognition model to automatically transcribe speech data and provide text results after transcription, which is insufficient to meet the requirement of real-time availability of transcription results during the generation process.

[0003] In practical applications, users have placed higher demands on real-time speech-to-text systems, including the ability to acquire editable text content during the transcription process, maintain high accuracy for professional content such as industry terminology and names of people and places, and support collaborative correction of text content by multiple users in the same transcription task. However, in existing technologies, speech-to-text functionality and collaborative text editing functionality are usually independent of each other. Speech-to-text systems lack effective linkage with multi-user collaborative annotation mechanisms and cannot dynamically adjust the transcription process based on users' real-time annotation behavior.

[0004] Furthermore, existing speech-to-text systems suffer from shortcomings in real-time performance and consistency in multi-user collaborative scenarios. Concurrent annotation by multiple users can easily lead to editing conflicts, resulting in overwritten or lost annotations. On the other hand, the workload of transcription tasks varies significantly across different time periods, and traditional fixed-power computing configurations struggle to respond promptly to these concurrent changes, easily causing transcription delays or task failures under high load. Therefore, there is an urgent need for a technical solution that supports real-time speech-to-text, multi-user collaborative annotation, and elastic resource scheduling capabilities to address the difficulties in real-time correction, poor collaborative consistency, and low resource utilization efficiency during the transcription process. Summary of the Invention

[0005] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the present invention provides a multimodal data correction method, apparatus, computer equipment and storage medium.

[0006] In a first aspect, embodiments of the present invention provide a multimodal data correction method, comprising: acquiring target multimodal data and generating corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; sending the initial text to multiple terminals, so that the multiple terminals perform real-time annotation operations based on the initial text to obtain collaborative annotation data; receiving the collaborative annotation data sent by the multiple terminals and generating collaborative annotation results based on the collaborative annotation data; and correcting the initial text based on the collaborative annotation results to generate corrected text.

[0007] In one possible implementation, generating collaborative annotation results based on the collaborative annotation data includes: determining the annotation position, annotation content, and annotation type corresponding to each annotation operation based on the collaborative annotation data, wherein the annotation type includes at least: error correction annotation and semantic completion annotation; when multiple annotation operations are detected to be performed on the same annotation position of the initial text, generating a globally consistent annotation operation sequence for the multiple annotation operations based on the annotation type, wherein the annotation operation sequence represents the execution order of each annotation operation; and generating the collaborative annotation results based on the annotation operation sequence, the annotation position, and the annotation content.

[0008] In one possible implementation, the step of correcting the initial text based on the collaborative annotation results includes: determining, based on the collaborative annotation results, the target text positions in the initial text that need to be corrected, as well as the correction content and correction order corresponding to the target text positions; and applying the correction content sequentially to the corresponding target text positions according to the correction order to update the corresponding text content in the initial text.

[0009] In one possible implementation, after correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains error correction annotations, generating an error correction record based on the error correction annotations for writing into a target database, wherein the error correction record is used to characterize the mapping relationship between erroneous text and corresponding correct text; and correcting erroneous text in the generated initial text based on the target database when repeatedly executing the step of generating the corresponding initial text based on the target multimodal data.

[0010] In one possible implementation, after correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains semantic completion annotations, inputting the semantic completion annotations into a context understanding model to optimize the context understanding model, wherein the optimization includes adjusting the text sentence generation weights based on the semantic completion annotations, and the context understanding model is used to cooperate with the initial text generation model to generate corresponding initial text based on the target multimodal data.

[0011] In one possible implementation, the method further includes: when it is detected that the concurrency of the task generating the initial text reaches a preset concurrency threshold and the utilization rate of the computing resources used to execute the task generating the initial text exceeds a preset resource utilization threshold, adding a computing node for executing the task generating the initial text; and releasing the computing node when the task generating the initial text is completed or the utilization rate of the computing resources is lower than a preset release threshold.

[0012] In one possible implementation, the method further includes: performing frequency statistics on the collaboratively labeled data to filter out high-frequency labeled data and low-frequency labeled data; after generating the initial text, using a rule engine to immediately correct the erroneous text corresponding to the high-frequency labeled data; constructing a training sample set from the erroneous text, corrected content, and contextual text corresponding to the low-frequency labeled data; incrementally training the target model based on the training sample set; and using the target model to correct the initial text.

[0013] Secondly, embodiments of the present invention provide a multimodal data correction device, comprising: an acquisition module, configured to acquire target multimodal data and generate corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; a sending module, configured to send the initial text to multiple terminals, so that the multiple terminals perform real-time annotation operations based on the initial text to obtain collaborative annotation data; a receiving module, configured to receive the collaborative annotation data sent by the multiple terminals and generate collaborative annotation results based on the collaborative annotation data; and a correction module, configured to correct the initial text based on the collaborative annotation results to generate corrected text.

[0014] Thirdly, embodiments of the present invention provide a computer device, including: a processor and a memory, wherein the processor is configured to execute a multimodal data correction program stored in the memory to implement the multimodal data correction method described in any one of the first aspects above.

[0015] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the multimodal data correction method described in any one of the first aspects.

[0016] The multimodal data correction scheme provided in this invention acquires target multimodal data and generates corresponding initial text based on it, enabling speech information to be instantly converted into text form after acquisition, providing a unified text foundation for subsequent annotation and correction. The initial text is then sent to multiple terminals to support real-time annotation operations based on the same initial text, forming collaborative annotation data. This allows for concurrent participation and real-time intervention by multiple users in the transcribed text. Furthermore, the collaborative annotation data sent by multiple terminals is centrally processed to generate collaborative annotation results, unifying and integrating the annotation behaviors of different users to reduce conflicts and data loss caused by concurrent annotation. Finally, the initial text is corrected based on the collaborative annotation results to generate corrected text, ensuring continuous optimization of the transcribed text during the generation process. This improves the accuracy, completeness, and real-time availability of the transcription results, meeting the application needs for real-time speech-to-text collaborative processing in scenarios such as meetings and teaching. Attached Figure Description

[0017] Figure 1 is a flowchart illustrating a multimodal data correction method provided in an embodiment of the present invention; Figure 2 is a structural diagram illustrating a multimodal data correction device provided in an embodiment of the present invention; Figure 3 is a structural diagram illustrating a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0020] Figure 1 is a flowchart of a multimodal data correction method provided by an embodiment of the present invention. As shown in Figure 1, the method specifically includes: S11, acquiring target multimodal data and generating corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data.

[0021] The multimodal data correction method provided in this invention is applicable to application scenarios requiring real-time transcription of speech content and support for multi-user collaborative correction, such as remote conferencing, online education, video conference recordings, interview recordings, and speech content organization. In these scenarios, the speech content is synchronously converted into editable text during the acquisition process, allowing multiple participating users to annotate and correct the text content in real time during transcription, thereby improving the accuracy and usability of the transcription results.

[0022] The executing entity of this invention can be a computer device, including but not limited to servers, desktop computers, etc., for example, applied to a cloud-based transcription processing system or server-side processing platform. This executing entity is used to acquire target multimodal data, generate initial text, receive collaborative annotation data from multiple terminals, and generate collaborative annotation results and corrected text based on the collaborative annotation data. The multiple terminals can be conference participation terminals, teaching terminals, or other user terminals with display and input functions, used to display the initial text and receive real-time annotation operations from users. By having the executing entity uniformly complete the transcription processing and annotation result integration, the multi-user collaborative annotation process possesses good consistency and real-time performance, thereby supporting the real-time speech-to-text collaborative processing requirements in the aforementioned application scenarios.

[0023] In this embodiment, target multimodal data refers to the raw data set used for transcription into text. The core data type is speech data (e.g., audio signals from conference speeches), and auxiliary data types may include image recognition text, video subtitle data, and audio streams from course lectures. Multimodal data collaboration can improve the accuracy of transcribed text. The initial text is the raw text generated after the target multimodal data is transcribed by an initial transcription model (e.g., a speech recognition model). It has not been manually annotated and corrected, and may contain issues such as terminology errors, semantic incompleteness, and accent mismatch. It serves as the foundation text for subsequent multi-user collaborative annotation.

[0024] Specifically, taking the cloud service layer as the executing entity, it can collect target multimodal data in real time through collection devices (e.g., conference microphones, mobile phone recording modules, web-based audio and video collection tools). The data includes at least voice data and can also collect image recognition text (e.g., text in screenshots, handwritten text recognition on whiteboards, etc.). The collected multimodal data is uploaded to the cloud service layer via the network, where the real-time data transmission module temporarily stores and forwards it to the initial transcription model. Data compression optimization is used during transmission to reduce bandwidth usage.

[0025] After receiving multimodal data, the initial transcription model first performs noise reduction on the speech data: filtering environmental noise (such as air conditioning noise in the conference room, background conversation noise) and echo cancellation (such as equipment echo in remote conferencing) through spectrum analysis algorithms to improve the clarity of the speech signal; and formatting the image recognition text data acquired simultaneously: correcting image recognition errors (such as tilted text correction, blurred character repair), and extracting key text fragments (such as titles, core data) as auxiliary reference information for speech transcription.

[0026] The initial transcription model can be built based on a pre-trained Automatic Speech Recognition (ASR) model. Its core uses a Transformer-based speech recognition network, providing end-to-end speech-to-text capabilities. The model extracts features from pre-processed speech data, combines them with contextual information, and uses a decoding algorithm to map the speech signal into a corresponding Chinese text sequence, thus transing multimodal data into text. The transcribed text sequence is then formatted (e.g., segmented into sentences according to pauses, automatically added punctuation) to generate a structured initial text (i.e., a draft). This draft retains the temporal features of the speech (e.g., arranged according to the order of speech).

[0027] S12. Send the initial text to multiple terminals so that the multiple terminals can perform real-time annotation operations based on the initial text to obtain collaborative annotation data.

[0028] In this embodiment, after receiving the initial text, the real-time data transmission module of the cloud service layer compresses the text data using a lightweight data compression protocol (such as Protocol Buffers) to reduce transmission bandwidth consumption. Simultaneously, it pre-defines communication links with each terminal using connection pooling technology to avoid delays caused by temporary connection establishment. Based on the WebSocket protocol, the real-time data transmission module initiates initial text pushes to multiple terminals (web, mobile, etc.) that have joined the collaborative annotation scenario. The push latency is strictly controlled within 200ms to ensure that all participating users receive the initial text synchronously, avoiding annotation deviations caused by text asynchrony.

[0029] Each terminal user can view the initial text through the multi-terminal collaborative annotation interface and perform annotation operations based on the annotation functions provided by the interface. The annotation types of annotation operations can include, but are not limited to: terminology correction annotation (such as selecting "love" and annotating "should be AI"), semantic completion annotation (such as inserting "Q3 online" after the text) and name / place name deviation annotation.

[0030] The annotation interface supports real-time preview of operations. After annotating, users can immediately see their own annotation traces (such as highlighting error text and displaying annotation content) while waiting for other users' annotation operations to be synchronized in the cloud.

[0031] Collaborative annotation data is generated based on information such as annotation content, annotation location, annotation type, and terminal identifier of the annotation operation.

[0032] Collaborative annotation data from each terminal is uploaded to the cloud service layer in real time via a persistent WebSocket connection. During transmission, a data verification mechanism is used to ensure the integrity of the annotation data and prevent loss or tampering during transmission.

[0033] S13. Receive collaborative annotation data sent by multiple terminals and generate collaborative annotation results based on the collaborative annotation data.

[0034] In this embodiment, the real-time data transmission module of the cloud service layer receives collaborative annotation data uploaded by multiple terminals in parallel through a persistent WebSocket connection. Upon receiving the data, it immediately performs dual verification: one is data integrity verification, and the other is legality verification. Data that fails the verification will be returned to the terminal to prompt for resubmission, ensuring that the annotation data is valid.

[0035] The collaborative editing module extracts all valid annotation data, initially sorts it by operation timestamp, and combines the OT (Operational Transformation) algorithm with short text operation transformation rules adapted to the meeting scenario to handle concurrent conflicting operations. For example, conflict determination: when the target text segments of two or more annotation data overlap (such as two users simultaneously modifying the "Project Progress" paragraph), it is determined to be a conflict; conflict resolution: through the OT algorithm, the annotation operations of each user are converted into transformation instructions based on a global text version, and a globally consistent execution order is generated according to the priority rule of "basic error correction (terminology / name errors) first, followed by semantic completion" (for example: execute the terminology error correction of "AI misidentified as love" first, and then execute the semantic completion after "Project Progress") to avoid data overwriting or loss; after processing, a globally unique execution ID is assigned to each annotation operation to ensure that the operation is traceable.

[0036] The collaborative editing module precisely associates the globally ordered annotation data with the initial text, embedding the annotation content into the corresponding positions in the initial text (e.g., terminology correction annotations directly replace erroneous text, semantic completion annotations are inserted into specified positions, and commentary annotations are associated with corresponding text segments and displayed on the side). During the fusion process, a complete annotation trace is preserved, including the annotator's identifier, operation timestamp, and original text content, facilitating subsequent traceability or reversal of operations (e.g., the moderator can see which user initiated a correction). After fusion, a structured collaborative annotation result is generated, which includes at least: the annotated and corrected initial text, the original initial text, and a complete annotation log (including details of all users' annotation operations and conflict resolution records). The collaborative annotation result is synchronously stored in the cloud database and pushed to all participating terminals via a real-time data transmission module, ensuring that the text version seen by each terminal user is completely consistent. The final generated collaborative annotation result is used by users to view the correction effect in real time and is also synchronously transmitted to the dynamic correction engine of the AI ​​model layer for subsequent model optimization and final text correction.

[0037] In one possible implementation, the annotation position, annotation content, and annotation type corresponding to each annotation operation are determined based on the collaborative annotation data, wherein the annotation type includes at least: error correction annotation and semantic completion annotation; when multiple annotation operations are detected to be performed on the same annotation position of the initial text, a globally consistent annotation operation sequence is generated for the multiple annotation operations based on the annotation type, the annotation operation sequence representing the execution order of each annotation operation; and a collaborative annotation result is generated based on the annotation operation sequence, annotation position, and annotation content.

[0038] In this embodiment, after receiving collaborative annotation data uploaded by multiple terminals, the collaborative editing module of the cloud service layer determines the annotation location and clarifies the specific paragraph or sentence to which the annotation operation applies in the initial text; it extracts the annotation content (such as the correct text for error correction and supplementary information for semantic completion) as the core basis for correction; it determines the annotation type, which includes at least error correction annotations (for scenarios that need to be replaced and corrected, such as incorrect terminology or discrepancies in personal and place names) and semantic completion annotations (for scenarios that need to be supplemented and improved, such as missing text or incomplete logic, such as adding "Q3 launch" after "project progress"), while also being compatible with other auxiliary annotation types such as comments.

[0039] By comparing the annotation positions of each annotation operation, it is determined whether there are multiple annotation operations pointing to the same position in the initial text (such as two users simultaneously modifying statements related to "blockchain"). If a conflict is detected, a globally consistent annotation operation sequence is generated based on the priority rules of the annotation type: error correction annotations are executed first, followed by semantic completion annotations, ensuring text accuracy first and then improving text integrity. At the same time, combined with the OT algorithm, the annotation operations of each user are converted into transformation instructions based on the global text version, eliminating conflicts caused by differences in operation timing, ensuring that the execution logic of the annotation operation sequence conforms to the collaborative intent of multiple users, and avoiding data overwriting or correction failure.

[0040] The collaborative editing module sequentially associates and merges the annotation content of each operation with the initial text according to the generated annotation operation sequence: for error correction annotations, it directly replaces the erroneous content in the initial text based on the annotation position; for semantic completion annotations, it inserts supplementary information according to the annotation position. During the fusion process, the original initial text, user identifiers of each annotation operation, operation timestamps, and conflict handling records are simultaneously preserved to ensure that the annotation traceability is maintained. Finally, the corrected text after the orderly execution of annotation operations, the original text, and the complete annotation log are integrated to generate a structured collaborative annotation result.

[0041] S14. Based on the collaborative annotation results, the initial text is corrected to generate corrected text.

[0042] In this embodiment, after receiving the collaborative annotation results generated by the collaborative editing module, the text correction module of the cloud service layer corrects the text one by one based on the initial text, following the execution order of error correction annotation first and semantic completion annotation second: for error correction annotation, the error segment in the initial text is accurately located according to the annotation position, and the error text is replaced with the annotation content; for semantic completion annotation, supplementary information is inserted into the corresponding paragraph of the initial text according to the annotation position. During the correction process, annotation traces and backups of the original text are preserved simultaneously to ensure traceability; after the correction is completed, the system automatically formats the text (such as adjusting punctuation marks and optimizing paragraph layout), and finally generates corrected text that is structurally complete, accurate in content, and conforms to the user's collaborative annotation intention. At the same time, the corrected text is pushed to all terminals participating in collaborative annotation in real time for users to view, confirm, or perform further supplementary operations.

[0043] This invention achieves continuous, real-time optimization of transcription results across multiple terminals by constructing a closed-loop mechanism encompassing initial transcription, real-time collaborative annotation across multiple terminals, aggregation of collaborative results, and dynamic correction. After acquiring target multimodal data and generating initial text, the initial text is sent to multiple terminals in real time for parallel annotation, allowing users with different roles to correct and supplement the text in real time from different dimensions such as terminology accuracy, semantic completeness, and contextual logic. The collaborative annotation data returned from multiple terminals is then fused and processed for consistency, generating a unified collaborative annotation result, which is used to structurally correct the initial text. Because multi-source annotation information can improve the error recognition coverage and confidence of the initial text, the corrected text is continuously optimized during use, effectively improving transcription accuracy.

[0044] Meanwhile, this invention introduces a real-time annotation feedback mechanism during the transcription process, enabling semantic completion, terminology correction, and contextual adjustments to take effect immediately during text generation, rather than requiring batch modifications afterward. This ensures that the transcribed content can be used directly during the meeting. Combined with low-latency data transmission and concurrent collaborative processing mechanisms, the system ensures consistency across multiple user operations while enabling simultaneous editing and real-time output of usable text. This significantly improves the completeness and real-time availability of the transcription results, meeting the comprehensive requirements of high accuracy, high completeness, and immediate editability in scenarios such as remote conferencing and online education.

[0045] Secondly, when multiple terminals are annotating the initial text in real time, all annotation operations are sent to the cloud in the form of operation instructions. The collaborative editing module uniformly receives and sorts the editing operations from each terminal, enabling position recalculation and conflict resolution for concurrent modifications, avoiding text overwriting, data loss, or inconsistency issues. By transforming decentralized concurrent editing into a centralized, ordered transformation processing mechanism, the system can maintain text version consistency and data integrity even in high-concurrency scenarios, fundamentally solving the problem of editing conflicts.

[0046] In one possible implementation, the initial text is corrected based on the collaborative annotation results, including: determining the target text positions that need to be corrected in the initial text, as well as the correction content and correction order corresponding to the target text positions, based on the collaborative annotation results; and applying the correction content to the corresponding target text positions in sequence according to the correction order to update the corresponding text content in the initial text.

[0047] In this embodiment, a dynamic correction engine is activated based on the collaborative annotation results to uniformly schedule and control the correction process of the initial text. The dynamic correction engine is used to maintain the text structure state in real time during the text correction process and dynamically adjust subsequent correction strategies according to the execution status of the correction operations.

[0048] First, the dynamic correction engine parses the collaborative annotation results, associating each annotation position in the results with the corresponding text positions in the initial text that need correction. These text positions can be identified through a mapping between character indexes, word numbers, or timestamps and text content. This method clearly identifies which text positions in the initial text require correction.

[0049] For each defined target text location, the corresponding correction content is extracted from the collaborative annotation results, and the type and scope of the correction content are recorded. Since different correction content may involve different correction operations such as insertion, replacement, or deletion, to avoid text misalignment or overwriting issues during multiple corrections, the dynamic correction engine generates an initial correction order based on the target text location and its corresponding correction content. The correction order can be determined based on the order of text locations in the initial text, the priority of annotation activation, or the annotation time order. When multiple correction operations are performed on the same target text location in the initial text, the correction is performed on that location according to the order of the annotation operation sequence.

[0050] During text updates, corrections are applied sequentially to the corresponding target text locations, replacing, supplementing, or deleting existing text content in the initial text. The text index information is updated synchronously after each correction to ensure that subsequent corrections accurately apply to the corresponding text locations. This approach ensures the stability and consistency of the text update process even when multiple corrections coexist, ultimately achieving ordered correction of the initial text based on collaborative annotation results.

[0051] In one possible implementation, after correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains error correction annotations, generating error correction records based on the error correction annotations for writing to the target database, wherein the error correction records are used to characterize the mapping relationship between the error text and the corresponding correct text; and when repeatedly performing the step of generating the corresponding initial text based on the target multimodal data, correcting the error text in the generated initial text based on the target database.

[0052] In this embodiment, when the collaborative annotation data contains error correction annotations, the dynamic correction engine first extracts the core information of such annotations from the collaborative annotation results: accurately identifying the erroneous text in the original initial text corresponding to the error correction annotation, the correct text annotated by the user (such as the erroneous text "love" and the correct text "AI"), and at the same time associating the scene tag of the annotation operation (such as meeting, medical consultation, etc.), user identifier and operation timestamp, to generate a structured error correction record. Each record contains mapping fields such as: erroneous text, correct text, scene tag, effective time, etc., to ensure that the mapping relationship is clear and traceable.

[0053] The dynamic correction engine verifies the validity of error correction records through data validation mechanisms (such as plagiarism detection and format validation), avoiding the writing of duplicate or invalid records. Then, it synchronously writes the validated records into the target database (such as a cloud-based structured database) in a preset data format (such as JSON) and creates an index to improve subsequent query matching efficiency. When repeatedly executing the step of transcribing the target multimodal data to generate initial text, the initial transcription model, during the process of generating the initial text draft, calls the error correction records in the target database in real time. It uses a text matching algorithm to search for whether there is a corresponding erroneous text mapping in the initial text. If a matching erroneous text is detected, the corresponding correct text is automatically called for replacement and correction, eliminating the need for manual re-annotation. This achieves one-time error annotation and permanent correction, continuously improving the transcription accuracy of the initial text.

[0054] In one possible implementation, after correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains semantic completion annotations, inputting the semantic completion annotations into the context understanding model to optimize the context understanding model, wherein the optimization includes adjusting the text sentence generation weights based on the semantic completion annotations, and the context understanding model is used to cooperate with the initial text generation model to generate the corresponding initial text based on the target multimodal data.

[0055] In this embodiment, when the collaborative annotation data contains semantic completion annotations, the dynamic correction engine first extracts the complete structured information of such annotations from the collaborative annotation results, including missing semantic paragraphs in the initial text, semantic completion content added by the user, context text corresponding to the completion position (such as 3-5 sentences of text before and after the completion content), and scene tags (such as meeting, education, professional office, etc.), and constructs a training sample set.

[0056] Subsequently, the completed training sample set is input into the context understanding model (which works in conjunction with the initial transcription model (initial text generation model), and its core purpose is to improve the semantic coherence and logical integrity of the transcribed initial text). The model optimization process is then initiated: the completion cases in the sample set are learned through reinforcement learning algorithms, focusing on analyzing the semantic relationship logic between the completion content and the context (such as the causal relationship of completing "Q3 complete core function development" after "project progress", and the subordinate relationship of completing "including architecture design and interface specifications" after "technical solution"). Based on the learning results, the sentence generation weights of the model are dynamically adjusted. The weights of the corresponding sentences are increased for semantic relationship types that match the completion scenario (such as progress supplementation in a meeting scenario and detail supplementation in a professional scenario), while the weights of logically unrelated generation directions are reduced. At the same time, the model's original basic semantic generation capabilities are preserved to avoid overfitting to a single completion scenario.

[0057] The optimization process employs a lightweight training strategy, combined with batch gradient descent to iteratively update model parameters, ensuring high efficiency without impacting the system's real-time transcription performance. The optimized contextual understanding model will work closely with the initial transcription model. When generating initial text based on target multimodal data, it can automatically supplement semantic content that conforms to contextual logic based on adjusted sentence weights (e.g., in conference speech transcription, based on high-weight progress association rules, it automatically predicts and supplements expressions like "expected Q4 launch" after "project progress plan"), reducing semantic gaps in the initial text and improving the completeness and usability of the transcribed text from the source.

[0058] In one possible implementation, if the concurrency of the task of generating initial text reaches a preset concurrency threshold and the utilization rate of computing resources used to execute the task of generating initial text exceeds a preset resource utilization threshold, a computing node for executing the task of generating initial text is added; if the task of generating initial text is completed or the utilization rate of computing resources is lower than a preset release threshold, the computing node is released.

[0059] In this embodiment, the elastic computing power scheduling module pre-configures task concurrency thresholds, computing resource utilization thresholds, and release thresholds, and monitors in real time the task concurrency of generating initial text (such as the number of simultaneous transcription tasks) and the resource status of the computing nodes undertaking the task, such as GPU / CPU utilization. When the task concurrency reaches the preset concurrency threshold and the computing resource utilization exceeds the preset resource utilization threshold, the module immediately triggers a scaling process through Kubernetes container orchestration technology. Idle resources are called from the cloud resource pool to quickly deploy computing nodes containing the initial transcription model runtime environment, dependency libraries, and data interfaces. After the new nodes are deployed, they are automatically included in the task scheduling system to distribute subsequent newly added initial text generation tasks, avoiding transcription delays caused by excessive load on a single node.

[0060] As the task of generating the initial text is gradually completed, or when the elastic computing power scheduling module detects that the utilization rate of computing resources remains below the preset release threshold, the resource release process is initiated. The system first stops allocating new tasks to the nodes to be released through a load balancing mechanism, ensuring that the tasks already received on the nodes are executed normally. After the nodes complete all tasks, the container instances on the nodes are automatically destroyed, releasing the occupied GPU / CPU, memory, and other computing resources, and returning the resources to the cloud resource pool for reuse by subsequent tasks, thereby reducing resource waste while ensuring transcription performance.

[0061] In one possible implementation, frequency statistics are performed on the collaboratively labeled data to filter out high-frequency and low-frequency labeled data; after generating the initial text, the error text corresponding to the high-frequency labeled data is corrected in real time through a rule engine; a training sample set is constructed by constructing the error text corresponding to the low-frequency labeled data, the corrected content, and the contextual text; the target model is incrementally trained based on the training sample set, and the target model is used to correct the initial text.

[0062] In this embodiment, the dynamic correction engine first performs structured parsing on the received collaborative annotation data, extracting the annotation type, error text, correction content and context information of each annotation, and then counts all error correction annotations through a frequency statistics algorithm, sets a frequency threshold (such as ≥5 times of cumulative annotation for a single error text), and filters out high-frequency annotation data (common terminology errors repeatedly annotated by multiple users) and low-frequency annotation data (scattered accent deviations, personalized expression errors, such as mispronunciations caused by dialects).

[0063] After generating the initial text, the rule engine calls the error mapping relationship corresponding to the high-frequency labeled data in real time to build an instant correction rule base. When the initial transcription model outputs the initial text, it automatically matches the erroneous text in the rule base and replaces it with the corresponding correct text content, realizing instant correction of high-frequency errors. For low-frequency labeled data, the system constructs a standardized training sample set according to the format of erroneous text, corrected content, and contextual text. After removing duplicate and invalid samples, the sample set is input into a lightweight target model (such as TinyBERT) for incremental training. By fine-tuning the model parameters, the system optimizes accent adaptation and personalized error recognition capabilities. The trained target model will simultaneously empower the initial transcription process to correct similar low-frequency errors in subsequent initial texts, forming a dual-drive correction mechanism of instant correction of high-frequency rules and incremental optimization of low-frequency models.

[0064] The multimodal data correction method provided in this invention introduces a multi-user real-time collaborative annotation mechanism based on acquiring target multimodal data and generating initial text. This allows the transcribed text to be synchronously annotated and corrected by multiple terminals during the generation process. A dynamic correction engine uniformly schedules and orderly applies the collaborative annotation results, thereby avoiding text misalignment and correction conflicts when multiple correction contents coexist. At the same time, the collaborative annotation results can continuously influence the subsequent text generation and correction process, enabling the system to have adaptive optimization capabilities when facing industry terminology, accent differences, and complex semantic scenarios. Combined with an elastic computing power scheduling mechanism, the computing resource configuration is dynamically adjusted according to the transcription task load, improving resource utilization efficiency while ensuring real-time performance. Ultimately, this achieves the technical effect of real-time usability, collaborative consistency, and high accuracy of transcribed text, making it suitable for application scenarios with high requirements for real-time speech transcription and collaborative processing, such as meetings and teaching.

[0065] As an example, the speech data is first transcribed using AI (generating initial text): After the meeting begins, the speech stream is uploaded to the cloud in real time, and the initial transcription model (such as an ASR model based on Transformer) generates a draft text, which is pushed to the user interface via WebSocket (latency <200ms); multi-user real-time annotation: the host and the recorder operate simultaneously through the annotation interface: the recorder discovers that "AI" is misspelled as "love", selects the incorrect text and annotates "Terminology correction: should be 'AI'"; the host supplements semantics: annotates "Complete: Core function development completed in Q3" after "Project Progress"; collaborative editing module The block handles concurrency through the OT algorithm: converting two-person operations into a global sequence of "terminology correction first, then semantic supplementation" to avoid conflicts; dynamic deviation correction: labeled data is fed into the dynamic correction engine in real time; terminology correction records trigger terminology database updates: "love→AI" is added to the user-defined terminology database, and subsequent transcriptions prioritize matching this rule; semantic completion data is used to optimize the context understanding model: sentence generation weights are adjusted through reinforcement learning to improve the coherence of long sentences; elastic computing power scheduling: when the meeting concurrency reaches 1000+, the elastic scheduling module detects that the GPU utilization rate is >80%, automatically expands by 3 computing nodes, and releases resources after the task is completed.

[0066] This invention also provides a multimodal data correction system, comprising three major modules: a user layer, a cloud service layer, and an AI model layer. The user layer provides a multi-terminal collaborative annotation interface (Web / mobile) that offers text editing, error annotation (highlighting / annotation), and terminology database management access, supporting simultaneous operation by multiple users.

[0067] The cloud service layer includes: a real-time data transmission module, a collaborative editing module, and an elastic computing power scheduling module; the real-time data transmission module: establishes a long connection based on the WebSocket protocol to transmit annotation instructions (latency <100ms) to achieve real-time data transmission; the collaborative editing module: uses the OT algorithm to handle concurrent annotation conflicts and ensures eventual consistency of multi-user operations; the elastic computing power scheduling module: automatically scales up and down GPU / CPU resources based on the amount of transcription tasks through Kubernetes container orchestration.

[0068] The AI ​​model layer includes: an initial transcription model, a dynamic correction engine, and a terminology database. The initial transcription model generates a draft text based on a pre-trained ASR model. The dynamic correction engine receives labeled data, optimizes the accent adaptation model through incremental training, and updates the personalized terminology database (e.g., automatically replacing "K8s" with "Kubernetes"). It adopts a "dual-drive approach of incremental fine-tuning + rule engine": high-frequency errors (such as terminology) are corrected instantly by the rule engine, while low-frequency errors (such as accents) are incrementally trained using a lightweight model (TinyBERT) to reduce the time spent on each update. The terminology database supports user-uploaded industry dictionaries, which are synchronized to the AI ​​model in real time.

[0069] Figure 2 illustrates a multimodal data correction device provided in an embodiment of the present invention. As shown in Figure 2, the device specifically includes: an acquisition module 21, used to acquire target multimodal data and generate corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; a sending module 22, used to send the initial text to multiple terminals, so that the multiple terminals can perform real-time annotation operations based on the initial text to obtain collaborative annotation data; a receiving module 23, used to receive the collaborative annotation data sent by the multiple terminals and generate collaborative annotation results based on the collaborative annotation data; and a correction module 24, used to correct the initial text based on the collaborative annotation results to generate corrected text.

[0070] In one possible implementation, the generation module 25 is configured to determine the annotation position, annotation content, and annotation type corresponding to each annotation operation based on the collaborative annotation data, wherein the annotation type includes at least: error correction annotation and semantic completion annotation; when multiple annotation operations are detected to be performed on the same annotation position of the initial text, a globally consistent annotation operation sequence is generated for the multiple annotation operations based on the annotation type, the annotation operation sequence representing the execution order of each annotation operation; and the collaborative annotation result is generated based on the annotation operation sequence, the annotation position, and the annotation content.

[0071] In one possible implementation, the correction module is specifically used to determine, based on the collaborative annotation results, the target text position in the initial text that needs to be corrected, as well as the correction content and correction order corresponding to the target text position; and to apply the correction content to the corresponding target text position in sequence according to the correction order, so as to update the corresponding text content in the initial text.

[0072] In one possible implementation, the correction module is further configured to generate an error correction record based on the error correction label when the collaborative annotation data contains error correction labels, for writing to the target database, wherein the error correction record is used to characterize the mapping relationship between the error text and the corresponding correct text; and when repeatedly executing the step of generating the corresponding initial text based on the target multimodal data, correcting the error text in the generated initial text based on the target database.

[0073] In one possible implementation, the correction module is further configured to input the semantic completion annotations into the context understanding model when the collaboratively labeled data contains semantic completion annotations, so as to optimize the context understanding model. The optimization includes adjusting the text sentence generation weights based on the semantic completion annotations. The context understanding model is used to cooperate with the initial text generation model to generate corresponding initial text based on the target multimodal data.

[0074] In one possible implementation, the processing module 26 is configured to add a computing node for executing the task of generating the initial text when it is detected that the concurrency of the task of generating the initial text reaches a preset concurrency threshold and the utilization rate of the computing resources used to execute the task of generating the initial text exceeds a preset resource utilization threshold; and to release the computing node when the task of generating the initial text is completed or the utilization rate of the computing resources is lower than a preset release threshold.

[0075] In one possible implementation, the correction module is further configured to perform frequency statistics on the collaboratively labeled data, and filter out high-frequency and low-frequency labeled data; after generating the initial text, the error text corresponding to the high-frequency labeled data is corrected in real time by a rule engine; a training sample set is constructed by the error text corresponding to the low-frequency labeled data, the corrected content, and the contextual text; the target model is incrementally trained based on the training sample set, and the target model is used to correct the initial text.

[0076] The multimodal data correction device provided in this embodiment can be the device shown in Figure 2, which can execute all the steps of the multimodal data correction method shown in Figure 1, thereby achieving the technical effect of the multimodal data correction method shown in Figure 1. For details, please refer to the relevant description in Figure 1. For the sake of brevity, it will not be elaborated here.

[0077] Figure 3 is a schematic diagram of a computer device according to an embodiment of the present invention. The computer device 300 shown in Figure 3 includes at least one processor 301, a memory 302, at least one network interface 304, and other user interfaces 303. The various components in the computer device 300 are coupled together through a bus system 305. It is understood that the bus system 305 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 305 in Figure 3.

[0078] The user interface 303 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0079] It is understood that the memory 302 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 302 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0080] In some implementations, memory 302 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 3021 and application program 3022.

[0081] The operating system 3021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 3022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 3022.

[0082] In this embodiment of the invention, by calling the program or instructions stored in the memory 302, specifically the program or instructions stored in the application program 3022, the processor 301 is used to execute the method steps provided in each method embodiment, such as: acquiring target multimodal data and generating corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; sending the initial text to multiple terminals so that the multiple terminals can perform real-time annotation operations based on the initial text to obtain collaborative annotation data; receiving the collaborative annotation data sent by the multiple terminals and generating collaborative annotation results based on the collaborative annotation data; and correcting the initial text based on the collaborative annotation results to generate corrected text.

[0083] The methods disclosed in the above embodiments of the present invention can be applied to processor 301, or implemented by processor 301. Processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 301 or by instructions in the form of software. The processor 301 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 302. The processor 301 reads the information in memory 302 and, in conjunction with its hardware, completes the steps of the above method.

[0084] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0085] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0086] The computer device provided in this embodiment can be the computer device shown in Figure 3, which can execute all the steps of the multimodal data correction method shown in Figure 1, thereby achieving the technical effect of the multimodal data correction method shown in Figure 1. For details, please refer to the relevant description in Figure 1. For the sake of brevity, it will not be elaborated here.

[0087] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0088] When one or more programs in the storage medium can be executed by one or more processors to implement the above-described multimodal data correction method executed on the device side.

[0089] The processor is configured to execute a multimodal data correction program stored in the memory to implement the following steps of a multimodal data correction method executed on the device side: acquiring target multimodal data and generating corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; sending the initial text to multiple terminals so that the multiple terminals perform real-time annotation operations based on the initial text to obtain collaborative annotation data; receiving collaborative annotation data sent by the multiple terminals and generating collaborative annotation results based on the collaborative annotation data; and correcting the initial text based on the collaborative annotation results to generate corrected text.

[0090] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0091] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0092] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal data correction method, characterized in that, include: Acquire target multimodal data and generate corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; send the initial text to multiple terminals so that the multiple terminals can perform real-time annotation operations based on the initial text to obtain collaborative annotation data; receive the collaborative annotation data sent by the multiple terminals and generate collaborative annotation results based on the collaborative annotation data; Based on the collaborative annotation results, the initial text is corrected to generate corrected text.

2. The method according to claim 1, characterized in that, The step of generating collaborative annotation results based on the collaborative annotation data includes: determining the annotation position, annotation content, and annotation type corresponding to each annotation operation based on the collaborative annotation data, wherein the annotation type includes at least: error correction annotation and semantic completion annotation; when multiple annotation operations are detected to be performed on the same annotation position of the initial text, generating a globally consistent annotation operation sequence for the multiple annotation operations based on the annotation type, wherein the annotation operation sequence represents the execution order of each annotation operation; and generating the collaborative annotation results based on the annotation operation sequence, the annotation position, and the annotation content.

3. The method according to claim 1, characterized in that, The step of correcting the initial text based on the collaborative annotation results includes: determining the target text position that needs to be corrected in the initial text, as well as the correction content and correction order corresponding to the target text position, based on the collaborative annotation results; and applying the correction content to the corresponding target text position in sequence according to the correction order to update the corresponding text content in the initial text.

4. The method according to claim 1, characterized in that, After correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains error correction annotations, generating error correction records based on the error correction annotations for writing into the target database, wherein the error correction records are used to characterize the mapping relationship between error text and corresponding correct text; when repeatedly executing the step of generating the corresponding initial text based on the target multimodal data, correcting the error text in the generated initial text based on the target database.

5. The method according to claim 1, characterized in that, After correcting the initial text based on the collaborative annotation results, the method further includes: if the collaborative annotation data contains semantic completion annotations, inputting the semantic completion annotations into a context understanding model to optimize the context understanding model, wherein the optimization includes adjusting the text sentence generation weights based on the semantic completion annotations, and the context understanding model is used to cooperate with the initial text generation model to generate the corresponding initial text based on the target multimodal data.

6. The method according to claim 1, characterized in that, The method further includes: when the concurrency of the task that generates the initial text reaches a preset concurrency threshold and the utilization rate of the computing resources used to execute the task that generates the initial text exceeds a preset resource utilization threshold, adding a computing node used to execute the task that generates the initial text; and releasing the computing node when the task that generates the initial text is completed or the utilization rate of the computing resources is lower than a preset release threshold.

7. The method according to claim 1, characterized in that, The method further includes: performing frequency statistics on the collaborative annotation data to filter out high-frequency annotation data and low-frequency annotation data; after generating the initial text, using a rule engine to immediately correct the erroneous text corresponding to the high-frequency annotation data; constructing a training sample set from the erroneous text, corrected content, and contextual text corresponding to the low-frequency annotation data; incrementally training the target model based on the training sample set; and using the target model to correct the initial text.

8. A multimodal data correction device, characterized in that, include: An acquisition module is used to acquire target multimodal data and generate corresponding initial text based on the target multimodal data, wherein the target multimodal data includes at least speech data; a sending module is used to send the initial text to multiple terminals so that the multiple terminals can perform real-time annotation operations based on the initial text to obtain collaborative annotation data; a receiving module is used to receive the collaborative annotation data sent by the multiple terminals and generate collaborative annotation results based on the collaborative annotation data. The correction module is used to correct the initial text based on the collaborative annotation results and generate corrected text.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a multimodal data correction program stored in the memory to implement the multimodal data correction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the multimodal data correction method according to any one of claims 1 to 7.

Images