Conference real-time translation display method, system, device, storage medium and product

Abstract

The application relates to the technical field of real-time speech recognition, and provides a conference real-time translation display method, system, device, storage medium and product, the method comprising the following steps: receiving a structured subtitle data packet from a cloud server; the structured subtitle data packet is issued by the cloud server after recognizing conference audio and correcting errors by using an artificial intelligence model, and at least comprises an incremental version number and an operation type indication; based on a locally maintained subtitle state table, the incremental version number is compared with a locally corresponding highest applied version number, and subtitle version filtering is performed; when it is determined that the update packet is valid, local anti-jitter refresh rendering is performed in a near-eye display area of a head-mounted display device according to the operation type indication. The application effectively solves the problem that an old version of a subtitle covers a new version caused by network disorder, and avoids visual jitter caused by frequent updating of intermediate results in translation in a limited near-eye display area, thereby improving the reading continuity and comfort of a user.

Description

Technical Field

[0001] This application relates to the field of real-time speech recognition technology, and in particular to methods, systems, devices, storage media, and products for real-time conference translation display. Background Technology

[0002] With the increasing international exchange, head-mounted display devices are being used more and more widely in real-time conference translation scenarios due to their ability to provide a first-person, hands-free experience. However, existing conference captioning solutions based on head-mounted display devices generally suffer from the following shortcomings in practical applications: 1. Near-eye displays exhibit severe text jumps, easily causing visual fatigue. Streaming speech recognition and machine translation continuously output intermediate revision results during processing. Traditional display terminals typically update by redrawing entire sentences or the entire screen after receiving new text. Due to the limited field of view and display area of ​​head-mounted displays, this frequent global redrawing causes the text to jump drastically in front of the user's eyes, severely disrupting reading continuity.

[0003] 2. Out-of-order network transmission can easily lead to older versions overwriting newer versions, causing display errors. In the complex network environment of a conference, data packet transmission often experiences jitter, out-of-order delivery, and retransmissions. Especially when the cloud uses large-scale artificial intelligence models for asynchronous error correction, the final confirmation text of the higher version may arrive before the intermediate text of the lower version. If the terminal passively renders the subtitles only according to the order of reception, the later-arriving erroneous text from the older version will overwrite the already displayed correct text. Summary of the Invention

[0004] This application provides a method, system, device, storage medium, and product for real-time conference translation display, which can adapt to the characteristics of streaming translation and network out-of-order environments, and can achieve stable and smooth rendering and presentation of subtitles on head-mounted devices.

[0005] This application provides a real-time conference translation display method applied to a head-mounted display device. The method includes: receiving a structured subtitle data packet from a cloud server; the structured subtitle data packet is sent by the cloud server after recognizing the conference audio and correcting errors using an artificial intelligence model; the structured subtitle data packet includes at least an incremental version number and an operation type indicator; based on a locally maintained subtitle status table, comparing the incremental version number with the corresponding highest applied version number locally, and performing subtitle version filtering; when the received structured subtitle data packet is determined to be a valid update packet, performing local anti-jump refresh rendering in the near-eye display area of ​​the head-mounted display device according to the operation type indicator.

[0006] According to the real-time conference translation display method provided in this application, the structured subtitle data packet also includes a subtitle segment identifier; performing subtitle version filtering includes: querying the highest applied version number of the corresponding segment recorded in the subtitle status table by parsing the subtitle segment identifier; if the received incremental version number is less than or equal to the highest applied version number, the current data packet is determined to be an old version or a duplicate packet; if the received incremental version number is greater than the highest applied version number, the current data packet is determined to be a valid update packet.

[0007] According to the real-time conference translation display method provided in this application, before performing local anti-jump refresh rendering, anti-out-of-order processing is performed using a local cache, including: putting valid update packets into the local reorder cache; detecting whether the global increment sequence number of the data packets in the reorder cache is consecutive; if there is a sequence number gap, waiting for the missing data packet within a preset out-of-order waiting window; if the missing packet is not received after the preset out-of-order waiting window has expired, retrieving the highest version packet in the current cache and performing rendering, and sending a missing sequence retransmission request to the cloud server.

[0008] According to the real-time conference translation display method provided in this application, local anti-jump refresh rendering is performed, including: when the operation type indicates content update, performing difference calculation on the newly received text and the old text currently displayed on the screen locally to extract the changed text intervals; mapping the text intervals to dirty regions according to the interface layout parameters of the current near-eye display area; and using a double buffering mechanism to redraw the dirty regions and the smallest adjacent regions affected by the dirty regions, while keeping the rendering state of other unchanged regions unchanged.

[0009] According to the real-time translation display method for meetings provided in this application, the method further includes: when a continuous loss of heartbeats or interruption of the subtitle stream is detected, maintaining the display of the most recently confirmed subtitle content and alerting the network status; after the network is restored, re-establishing a connection with the cloud server using the latest parameters in the local subtitle status table and aligning the subtitle status.

[0010] This application provides a real-time conference translation display method applied to a cloud server. The real-time conference translation display method includes: acquiring the conference audio and determining candidate error correction segments based on the conference audio; retrieving the conference knowledge base based on the conference context identifier bound to the current audio and obtaining professional knowledge items matching the candidate error correction segments; assembling the candidate error correction segments and professional knowledge items into structured prompt words and inputting them into an artificial intelligence model to obtain the structured error correction result output by the artificial intelligence model; generating a structured subtitle data package based on the structured error correction result and sending it to a head-mounted display device.

[0011] According to the real-time translation display method for meetings provided in this application, candidate error correction segments are determined based on the audio of the meeting, including: converting the audio of the meeting into a preliminary translation text with a timestamp and intermediate recognition results; extracting low-confidence features from the intermediate recognition results, performing candidate error detection, and determining candidate error correction segments in the preliminary translation text.

[0012] According to the real-time conference translation display method provided in this application, the method for obtaining professional knowledge items that match the candidate error correction segments includes: parsing the conference information bound to the currently processed conference audio stream; the conference information includes at least one of conference number, session number, agenda identifier, and speaker identifier; using the conference information as a filtering condition, and retrieving the corresponding professional knowledge items from the conference knowledge base through text similarity calculation within the corresponding range.

[0013] According to the real-time conference translation display method provided in this application, a structured subtitle data package is generated based on the structured error correction result, including: verifying the structured error correction result, and generating a structured subtitle data package with an incremental version number update after the verification passes; wherein, the verification includes at least one of evidence verification and semantic offset prevention verification.

[0014] According to the real-time conference translation display method provided in this application, when distributing structured subtitle data packets, multi-terminal differentiated distribution is performed, specifically including: obtaining the language subscription preferences and information display level preferences reported by each currently connected head-mounted display device; and within the same time window, pushing structured subtitle data packets of different target languages ​​and / or different information densities to the corresponding head-mounted display devices.

[0015] This application also provides a head-mounted display device using the aforementioned real-time conference translation display method. The head-mounted display device includes: a communication module for establishing a connection with a cloud server, receiving structured subtitle data packets, and reporting local configuration instructions; a near-eye display module for providing the wearer with a visual information display area within the field of view; and a local processing module, including a processor and a memory, for maintaining a subtitle status table, performing subtitle version comparison filtering, calculating dirty areas, and driving the near-eye display module to perform local anti-jump refresh rendering.

[0016] This application also provides a real-time conference translation and display system, including a cloud server and at least one head-mounted display device; the cloud server device is used to execute the real-time conference translation and display method described above, and the head-mounted display device is used to execute the real-time conference translation and display method described above.

[0017] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described real-time conference translation display methods.

[0018] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described real-time conference translation display methods.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described real-time conference translation display methods.

[0020] The real-time conference translation display method, system, device, storage medium, and product provided in this application, applied to a head-mounted display device, include: receiving structured subtitle data packets from a cloud server; the structured subtitle data packets are distributed by the cloud server after recognizing the conference audio and correcting errors using an artificial intelligence model; the structured subtitle data packets include at least an incremental version number and an operation type indicator; based on a locally maintained subtitle status table, comparing the incremental version number with the corresponding highest applied version number locally, and performing subtitle version filtering; when the received structured subtitle data packet is determined to be a valid update packet, performing local anti-jump refresh rendering in the near-eye display area of ​​the head-mounted display device according to the operation type indicator. Through the above methods, this application effectively solves the problem of old version subtitles overwriting new versions caused by network out-of-order delivery, and avoids visual jumps caused by frequent updates of intermediate translation results within a limited near-eye display area, improving the user's reading continuity and comfort. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is one of the flowcharts illustrating the real-time conference translation display method provided in this application embodiment.

[0023] Figure 2 This is the second flowchart illustrating the real-time conference translation display method provided in this application embodiment.

[0024] Figure 3 This is a schematic diagram of the local subtitle refresh and disorder processing flow for AR glasses provided in the embodiments of this application.

[0025] Figure 4 This is a schematic diagram illustrating the relationship between the subtitle data package and the AR local status table provided in the embodiments of this application.

[0026] Figure 5This is a schematic diagram of the dynamic error correction process for large model context and terminology provided in the embodiments of this application.

[0027] Figure 6 This is a schematic diagram of the overall process of the real-time conference translation display system provided in this application embodiment.

[0028] Figure 7 This is a schematic diagram of the conference implementation translation process provided in the embodiments of this application.

[0029] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0032] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0033] In related technologies, conference translation solutions mainly include the following categories: human simultaneous interpreters combined with dedicated equipment such as simultaneous interpreter receivers and headsets; subtitle display solutions based on terminals such as mobile phones and tablets; unified subtitle display solutions based on large screens in the conference venue; and traditional machine translation equipment or portable translation terminal solutions. These solutions generally suffer from the following shortcomings in practical applications: 1. High cost: The cost of human simultaneous interpretation services and dedicated simultaneous interpretation equipment is high, and the overall cost of equipment procurement, operation and maintenance, manpower support, and on-site management is also high in large-scale international conferences.

[0034] 2. Limited language coverage and expansion capabilities: Traditional solutions are usually limited by the number of human translation resources, the number of device channels, and system architecture capabilities, making it difficult to flexibly support a large number of concurrent language services, and also difficult to meet the personalized needs of different users for different target languages.

[0035] 3. Insufficient user experience: Some existing solutions require participants to wear headphones or look down at their phones, which affects their focus and immersion in the meeting; uniform subtitles on large screens are difficult to meet individual reading needs and are greatly affected by seating position.

[0036] 4. Poor experience with traditional streaming input, output, and text refresh: Streaming ASR (Automatic Speech Recognition) and translation typically output intermediate results first, then revise the preceding text based on the subsequent audio context. For example, the same sentence may go through multiple versions: "unsegmented text—revised segmentation—terminology correction—final confirmation." If the terminal redraws the entire sentence or screen each time, it will cause the subtitles in the near-eye display area to jump, affecting reading comfort.

[0037] Furthermore, network transmission is subject to jitter, out-of-order delivery, and retransmissions. If smart glasses only display subtitles in the order they are received, older revision packages may arrive later than newer versions, thus overwriting correctly displayed results. Additionally, smart glasses have a limited display area and can only display a few lines; therefore, subtitle line height, line breaks, and scrolling strategies directly impact the wearer's reading experience. Ordinary subtitle display methods are not designed for the near-eye display characteristics of "stable confirmed line positions, editable current lines, and smooth upward movement when overflowing."

[0038] 5. Insufficient accuracy and stability in professional scenarios: In highly specialized conference scenarios such as science and technology, industry, government affairs, and foreign affairs, there are numerous agenda topics, professional terms, proper nouns, guest names, organization names, and industry abbreviations. Traditional translation systems struggle to dynamically correct errors and optimize professional expressions based on context, resulting in insufficient translation accuracy and unstable expression. Feeding entire texts directly to a large model each time not only leads to high latency but also easily introduces illusory rewritings irrelevant to the meeting.

[0039] 6. Uneven information acquisition in large venues: Existing systems are often affected by far-field sound attenuation, environmental noise, display distance and information transmission path in large venues, resulting in inconsistent clarity and timeliness of information obtained by participants in different locations.

[0040] 7. Traditional conference interpreting often requires a lot of preparatory work such as distributing headsets, configuring channels, pairing equipment, and on-site debugging, resulting in low deployment efficiency and making it particularly unsuitable for high-frequency, multi-venue concurrent conference scenarios.

[0041] Based on this, this application provides a real-time conference translation technology solution that combines cloud-based structured subtitle version management, local partial refresh and out-of-order processing on display terminals, conference knowledge base retrieval, and constrained error correction of large models. It is a conference translation enhancement technology solution that can support multilingual, low-latency, scalable management, and convenient deployment for large-scale conference scenarios.

[0042] This application provides a real-time translation display method for meetings, applied to a head-mounted display device. A head-mounted display (HMD) is an electronic device worn on a user's head that provides visual information in front of the wearer. For example, a head-mounted display device can be augmented reality (AR) glasses, mixed reality (MR) glasses, virtual reality (VR) headsets, smart translation glasses, etc. It should be understood that the head-mounted display device is not limited to the above examples and can be any wearable device with near-eye display capabilities.

[0043] In this application, real-time conference translation display refers to the process of converting a speaker's speech into text subtitles in real time and displaying them on a head-mounted display device during a conference. This process may include speech recognition, text translation, text correction, and subtitle rendering. For example, in an international conference, if a speaker delivers a speech in English, Chinese audience members wearing head-mounted displays can see the translated Chinese subtitles in real time in their near-eye display area. As another example, in a medical academic conference, the technical terms used by the speaker can be accurately displayed on the audience's head-mounted displays after being corrected by an artificial intelligence model. It should be understood that the application scenarios of real-time conference translation display are not limited to the above examples; they can also be applied to various scenarios requiring real-time subtitle translation, such as remote video conferencing, corporate internal training, and simultaneous interpretation in court trials.

[0044] Please see Figure 1 , Figure 1 This is one of the flowcharts illustrating the real-time conference translation display method provided in this application embodiment. In this embodiment, the real-time conference translation display method may include steps S110 to S130, each step being as follows: S110: Receives structured caption data packets from the cloud server; the structured caption data packets are sent by the cloud server after recognizing the conference audio and correcting errors using an artificial intelligence model; the structured caption data packets include at least an incremental version number and an operation type indicator.

[0045] Structured caption data packets refer to data units, including caption content and its associated control information, sent from a cloud server to a head-mounted display device. Unlike traditional plain text caption streams, structured caption data packets in this embodiment utilize structured field design to enable precise version management and partial refresh control of the head-mounted display device.

[0046] The structured subtitle data package includes at least an incremental version number and an operation type indicator, and may also include fields such as subtitle segment identifier, global incrementing sequence number, timestamp, target language identifier, and confidence score.

[0047] Optionally, the structured caption data package may also include auxiliary layout information; when the head-mounted display device performs rendering, it parses the suggested line break position, specified line number, or priority parameter in the auxiliary layout information to reduce the system overhead of local terminal re-layout calculation.

[0048] An incremental version number refers to an incrementing number assigned to each content update of the same subtitle segment. Because speech recognition and AI model error correction are incremental processes, the recognition result of the same passage may undergo multiple corrections, each corresponding to a new incremental version number.

[0049] The operation type indicator is a field in the structured caption data packet used to inform the head-mounted display device what rendering operation should be performed. Operation type indicators may include, for example, types such as adding, updating content, deleting, and confirming finalization, but this application is not limited to these.

[0050] Artificial intelligence models refer to machine learning models used to intelligently correct errors in recognized text. An example of an artificial intelligence model is a Large Language Model (LLM).

[0051] The head-mounted display device establishes a communication connection with a cloud server through its communication module. This communication connection can be, for example, a long connection based on the WebSocket protocol, a message push connection based on the MQTT protocol, or a server push connection based on HTTP / 2, but this application is not limited to these.

[0052] After processing the conference audio through speech recognition, text translation, and artificial intelligence model error correction, the cloud server encapsulates the processing results into structured subtitle data packets and sends them to the head-mounted display device via the aforementioned communication connection.

[0053] The incremental version number in the structured caption data packet is used to identify different corrected versions of the same caption segment. For example, for the same sentence, the speech recognition engine may first output a preliminary recognition result (corresponding to version number 1), and then the artificial intelligence model may output a corrected result (corresponding to version number 2) after correcting the technical terms. In this case, the head-mounted display device will receive two structured caption data packets with version numbers 1 and 2, respectively. The operation type indicator in the structured caption data packet is used to tell the head-mounted display device what rendering operation should be performed.

[0054] S120: Based on the locally maintained subtitle status table, compare the incremental version number with the corresponding highest applied version number locally, and perform subtitle version filtering.

[0055] A subtitle status table is a data structure maintained locally by a head-mounted display device to record the current status of each subtitle segment. The subtitle status table can store information such as the segment identifier, the highest version number applied, the currently displayed content, the display position, and the timestamp for each subtitle segment.

[0056] By using a caption status table, head-mounted display devices can determine whether the received structured caption data packets are valid update packets, thereby avoiding duplicate rendering or rollback rendering.

[0057] The head-mounted display device maintains a local caption status table. This caption status table records the status information of each caption segment currently being displayed or processed, including but not limited to the highest version number applied to each segment.

[0058] When a head-mounted display device receives a new structured caption data packet, it compares the incremental version number in the data packet with the highest applied version number recorded in the corresponding segment in the caption status table to determine whether the data packet is a valid update packet.

[0059] By filtering subtitle versions, we can avoid the problem of old version data overwriting new version data due to network latency, retransmission of data packets, or out-of-order arrival, thus ensuring the consistency and accuracy of the subtitle display content.

[0060] S130: When the received structured caption data packet is determined to be a valid update packet, local anti-jump refresh rendering is performed in the near-eye display area of ​​the head-mounted display device according to the operation type instruction.

[0061] The near-eye display area refers to the area of ​​visual information displayed by a head-mounted display device within the wearer's field of view. The size and position of the near-eye display area can vary depending on the device model and user settings.

[0062] Local anti-jump refresh rendering refers to re-rendering only the local area that has changed when updating the content of the subtitle display, instead of redrawing the entire display area. It also uses mechanisms such as double buffering to ensure the smoothness of the visual transition and avoid poor visual experience such as text jumping or flickering.

[0063] When the subtitle version filter in step S120 determines that the currently received structured subtitle data packet is a valid update packet, the head-mounted display device performs the corresponding rendering operation in the near-eye display area according to the operation type indication in the data packet.

[0064] For example, when the operation type indicator is "Add", a new subtitle line is added to the near-eye display area; when the operation type indicator is "Content Update", the content of an existing subtitle line is partially updated; when the operation type indicator is "Delete", the corresponding subtitle line is removed from the near-eye display area; when the operation type indicator is "Confirm Final", the corresponding subtitle line is marked as final and will no longer accept subsequent updates.

[0065] When performing partial anti-jump refresh rendering, the head-mounted display only re-renders the areas that have changed, while keeping the rendering state of other unchanged areas unchanged. Compared with full-screen redraw, this partial refresh strategy can significantly reduce rendering computation and power consumption, and avoid the overall text jumping or flickering that may occur during full-screen refresh, thereby improving the comfort of users reading subtitles while wearing head-mounted displays for extended periods in meeting scenarios.

[0066] In some embodiments, the structured caption data packet may further include a final state flag indicating whether the current caption is a final confirmation result; the head-mounted display device may control the caption layout according to the following rules: For a subtitle line marked as true in the final state, its physical line position in the near-eye display area is locked and will no longer change with intermediate recognition results; for intermediate revision results marked as false in the final state, text replacement is performed only within the current active line and the locked subtitle line is not pushed; only when a new final confirmation line is received, or the current active line reaches the display rhythm threshold causing the display window to overflow, is the old line in the near-eye display area triggered to move smoothly upward as a whole.

[0067] The real-time conference translation display method applied to head-mounted display devices according to the above embodiments can effectively avoid abnormal subtitle display caused by network transmission problems by receiving structured subtitle data packets sent by the cloud server and performing subtitle version filtering based on the locally maintained subtitle status table; and by performing local anti-jump refresh rendering in the near-eye display area, a smooth and stable visual experience can be provided while ensuring real-time subtitle updates.

[0068] This application implements an anti-out-of-order and version control mechanism to maintain the accuracy and sequential presentation of displayed content. The device parses cloud-based subtitle data and compares it with local records, updating only when the version number of a new data packet is higher than the locally applied version. This helps intercept old data packets that are delayed due to network instability, reducing the problem of old subtitles overwriting new subtitles, and also better complements the error correction results asynchronously distributed from the cloud's large model.

[0069] This application implements partial anti-jump rendering, improving the reading experience. Unlike full-screen or full-line redrawing, after receiving a valid update, the terminal device only refreshes the changed areas based on the operation type (add, revise, confirm, etc.). This method fixes the display position of confirmed text, controls changes within a small range, reduces flickering and jitter in the subtitles, and helps maintain continuity during long-term reading.

[0070] This application embodiment reduces the rendering computational overhead of the display device. Before rendering, the system filters subtitle versions, discards duplicate and expired data packets, and reduces unnecessary graphics rendering processes; combined with a partial refresh mechanism, it further saves the device's computing resources and power consumption.

[0071] Based on any of the above embodiments, the structured subtitle data packet further includes a subtitle segment identifier; the step of performing subtitle version filtering may specifically include: By parsing the subtitle segment identifier, the system queries the subtitle status table for the highest version number already applied to the corresponding segment. If the received incremental version number is less than or equal to the highest version number already applied, the current data packet is determined to be an old version or a duplicate packet. If the received incremental version number is greater than the highest version number already applied, the current data packet is determined to be a valid update packet.

[0072] A subtitle segment identifier is a unique identifier assigned by the cloud server to each individual subtitle segment, used to distinguish different subtitle segments.

[0073] Specifically, the head-mounted display device parses the subtitle segment identifier from the received structured subtitle data packet, and then uses the subtitle segment identifier as an index to look up the corresponding record in the locally maintained subtitle status table. If a record for the segment identifier exists in the subtitle status table, the highest version number applied in that record is read; if a record for the segment identifier does not exist in the subtitle status table, it means that this is a brand new subtitle segment, and its highest version number can be regarded as the initial value (e.g., 0).

[0074] If the received incremental version number is less than or equal to the highest applied version number, the current data packet is determined to be an old version or a duplicate packet. This means that the head-mounted display device has previously received and applied the same or a higher version update of the subtitle segment, therefore there is no need to render the current data packet. The head-mounted display device can directly discard the data packet to avoid unnecessary rendering operations and possible subtitle rollback.

[0075] If the received incremental version number is greater than the highest applied version number, the current data packet is determined to be a valid update packet. This means that the current data packet contains the latest version of the subtitle segment and needs to be rendered and updated. At this time, the head-mounted display device updates the highest applied version number corresponding to the segment in the subtitle status table to the currently received incremental version number and continues to execute subsequent rendering steps.

[0076] Based on any of the above embodiments, the step of performing anti-out-of-order processing using a local cache before performing local anti-jump refresh rendering may specifically include: Place the valid update packet into the local reorder cache; check if the global increment sequence number of the data packets in the reorder cache is consecutive; if there is a sequence number gap, wait for the missing data packet within the preset out-of-order waiting window; if the missing packet is not received after the preset out-of-order waiting window, extract the highest version packet in the current cache, perform rendering, and send a missing sequence retransmission request to the cloud server.

[0077] Because data packets may arrive out of order during network transmission—that is, later-sent packets may arrive at the head-mounted display device before earlier-sent packets—if rendering is performed directly without anti-out-of-order processing, it may cause abrupt changes or logical inconsistencies in the subtitle content. Therefore, this embodiment solves the problem of out-of-order data packets by setting up a local reordering cache.

[0078] The reordering cache is a temporary cache area maintained locally by the head-mounted display device to temporarily store valid update packages that have passed version filtering but have not yet been rendered. The reordering cache can be implemented, for example, using a priority queue data structure, with the globally incrementing sequence number of the data packets as the sorting basis, but this application is not limited to this.

[0079] It should be noted that the global incrementing sequence number is a field in the structured caption data packet. Unlike the incremental version number, the global incrementing sequence number is a globally incrementing number across all caption segments, used to identify the sending order of the data packets.

[0080] The head-mounted display examines the sequence of data packets in the reordering cache starting from the current position to be rendered, determining whether the globally increasing sequence number is consecutively increasing. If a sequence number gap is detected (i.e., data packets corresponding to one or more globally increasing sequence numbers have not yet arrived), a pre-defined out-of-order wait window is initiated. Within the out-of-order wait window, the head-mounted display pauses rendering operations, waiting for the missing data packets to arrive.

[0081] The preset out-of-order waiting window can be set to a duration between 50 milliseconds and 200 milliseconds, but this application is not limited to this. The duration can be adjusted according to the actual network environment and application scenario.

[0082] If the missing packet is not received within the preset out-of-order waiting window, the highest version packet in the current cache is retrieved for rendering, and a missing sequence retransmission request is sent to the cloud server. When the wait times out, the head-mounted display device no longer waits for the missing data packet, but instead retrieves the highest available version data packet from the reordering cache for rendering to ensure the real-time display of subtitles.

[0083] Simultaneously, the head-mounted display device sends a missing sequence retransmission request to the cloud server, informing the server which data packets corresponding to globally incrementing sequence numbers were not received and requesting the server to retransmit them. The missing sequence retransmission request can include a list of missing globally incrementing sequence numbers, subtitle segment identifiers, and other information.

[0084] This embodiment uses a "render first, then fill in" strategy to ensure the real-time display of subtitles and to correct and update the subtitle content after receiving retransmitted data packets.

[0085] Based on any of the above embodiments, the step of performing local anti-jump refresh rendering may specifically include: When the operation type is indicated as content update, the newly received text and the old text currently displayed on the screen are calculated locally to extract the text intervals that have changed. Based on the interface layout parameters of the current near-eye display area, the text intervals are mapped to dirty regions. A double buffering mechanism is used to redraw the dirty regions and the smallest adjacent regions affected by the dirty regions, while keeping the rendering state of other unchanged regions unchanged.

[0086] Specifically, the head-mounted display device extracts new text content from the valid update package and retrieves the old text content of the corresponding subtitle segment currently displayed in the near-eye display area from the subtitle status table. Then, it performs a difference calculation on the new and old texts to extract the changed text intervals.

[0087] Based on the interface layout parameters of the current near-eye display area, the text area is mapped to a dirty region. Specifically, interface layout parameters may include, for example, font size, line spacing, character spacing, starting coordinates of the display area, width and height of the display area, etc.

[0088] The dirty region refers to the rectangular area in the near-eye display area that needs to be redrawn. By mapping text intervals to dirty regions, the range of display pixels that need to be redrawn can be precisely located.

[0089] This embodiment employs a double buffering mechanism, which works by maintaining two frame buffers (a front buffer and a back buffer) in memory. After all drawing operations are completed in the back buffer, the contents of the back buffer are swapped to the front buffer for display at once, thereby avoiding screen tearing and flickering during the drawing process.

[0090] In this embodiment, the head-mounted display device redraws only the dirty region and its least adjacent region affected by the dirty region in the back buffer. The least adjacent region affected by the dirty region may include, for example, the region of subsequent text that needs to be redrawn when a character changes and causes subsequent text to shift; or the region of subsequent lines that needs to be redrawn when a line of text is deleted and causes subsequent lines to shift upwards.

[0091] This embodiment minimizes the amount of rendering computation by redrawing only the dirty areas and the smallest adjacent areas, thereby reducing the power consumption of the head-mounted display device. At the same time, it ensures the visual stability of the unchanged areas and avoids subtitle jumps.

[0092] Based on any of the above embodiments, the real-time translation display method for meetings may further include the following steps: When a continuous loss of heartbeats or an interruption in the subtitle stream is detected, the most recently confirmed subtitle content is displayed, and the network status is alerted. After the network is restored, the connection with the cloud server is re-established and the subtitle status is aligned using the latest parameters in the local subtitle status table.

[0093] This embodiment may include network anomaly handling and recovery steps. Specifically, during communication with the cloud server, the head-mounted display device can monitor the network connection status through a heartbeat mechanism. The head-mounted display device periodically sends heartbeat packets to the cloud server and expects to receive a heartbeat response from the cloud server within a preset time. In addition, the head-mounted display device can also monitor the reception of the subtitle data stream to detect whether the subtitle stream is interrupted.

[0094] When consecutive heartbeat loss or subtitle stream interruption is detected, the most recently confirmed subtitle content is displayed, and the network status is alerted. Specifically, when the head-mounted display device detects that several consecutive heartbeat packets have not received a response, or that no new structured subtitle data packets have been received within a preset time, it determines that the current network connection is abnormal.

[0095] At this time, the head-mounted display device does not clear the currently displayed subtitles in the near-eye display area, but instead maintains the display of subtitles with the most recent status of "confirmed" or "finalized" to prevent users from losing existing subtitle information due to network interruption. Simultaneously, the head-mounted display device displays network status reminders in the near-eye display area, such as "Network connection abnormal, subtitles may be delayed," to inform the wearer of the current network status.

[0096] After network recovery, the device reconnects to the cloud server and aligns the subtitle status using the latest parameters in its local subtitle status table. Specifically, when the head-mounted display detects network connectivity restored (e.g., upon receiving a renewed heartbeat response), it reads the latest parameters from its local subtitle status table, including the highest applied version number of each subtitle segment and the last received globally incrementing sequence number, and sends these parameters to the cloud server. Based on these parameters, the cloud server determines the subtitle updates missing by the head-mounted display during the network outage and resends the missing structured subtitle data packets to the head-mounted display.

[0097] After receiving the resent data packet, the head-mounted display device processes it according to steps S120 and S130 above to update the subtitle content to the latest state. This embodiment ensures the continuity of the user's subtitle reading experience under network fluctuations through this disconnection-reconnection alignment mechanism.

[0098] In some embodiments, the head-mounted display device further includes an interactive feedback and configuration synchronization step: Respond to user interactions on the head-mounted display device and obtain local configuration commands for target language switching, font adjustment, or auxiliary information display on / off. If the local configuration command involves display style, the local rendering parameters are updated directly; if it involves language or data source changes, the command is reported to the cloud server to update the multi-terminal differentiated distribution strategy on the cloud side.

[0099] This application provides a method for real-time conference translation display, applied to a cloud server. Please refer to [link / reference]. Figure 2 , Figure 2 This is a second schematic flowchart of the real-time conference translation display method provided in this application embodiment. In this embodiment, the real-time conference translation display method may include steps S210 to S240, each step being as follows: S210: Acquire the audio of the meeting and determine candidate error correction segments based on the audio.

[0100] The cloud server acquires the audio stream from audio acquisition devices at the meeting venue, such as microphone arrays and recording equipment, via the network. The meeting audio can be, for example, a real-time audio stream from the speaker or a mixed audio stream transmitted through the conference system. The cloud server performs speech recognition processing on the acquired meeting audio, converting the audio into text, and identifying candidate error-correction segments that may contain recognition errors during this process.

[0101] S220: Retrieve the conference knowledge base based on the conference context identifier bound to the current audio, and obtain professional knowledge items that match the candidate error correction segment.

[0102] A cloud server maintains a conference knowledge base, which is a pre-built collection of professional knowledge data associated with a specific conference or field. This knowledge base pre-stores professional knowledge items related to each conference. It can store information such as technical terms and their correct spellings, speaker names and titles, product names and technical concepts involved in the conference agenda, abbreviations and their full names, etc.

[0103] For example, in a medical academic conference, the conference knowledge base could include professional knowledge items such as drug names, disease names, and medical terminology. Similarly, in a technology product launch, the conference knowledge base could include professional knowledge items such as product model, technical specifications, and brand name.

[0104] For example, the cloud server can retrieve professional knowledge items that match the candidate error correction segment from the meeting knowledge base based on the meeting context identifier (such as meeting number, session number, etc.) bound to the currently processed meeting audio.

[0105] S230: Assemble the candidate error correction fragments and professional knowledge items into structured prompt words and input them into the artificial intelligence model to obtain the structured error correction results output by the artificial intelligence model.

[0106] The cloud server assembles the identified candidate error correction segments and professional knowledge items into structured prompts according to a predefined template. The structured prompts may include, for example, a task description (e.g., "Please correct the errors in the following identified text based on the provided professional knowledge"), the original text of the candidate error correction segments, a list of matched professional knowledge items, and a description of the expected output format.

[0107] After the assembled structured prompts are input into the AI ​​model, the AI ​​model, based on its language understanding and generation capabilities and the provided professional knowledge items, corrects any errors that may exist in the candidate error correction segments and outputs the structured error correction results.

[0108] Optionally, the structured error correction results may include fields such as the corrected text, the correction location identifier, the original text before correction, and the basis for correction.

[0109] By inputting professional knowledge items as context into the artificial intelligence model, this embodiment can significantly improve the accuracy of the artificial intelligence model in correcting professional terms, personal names, product names, etc., and avoid the "illusion" problem that may occur when the artificial intelligence model relies solely on its pre-trained knowledge for error correction.

[0110] S240: Generate structured subtitle data packets based on the structured error correction results and send them to the head-mounted display device.

[0111] Based on the structured error correction results output by the artificial intelligence model, the cloud server generates a new structured caption data package. This package includes the updated caption content, the corresponding incremental version number (incrementing from the original version number), an operation type indicator (e.g., "content update"), caption segment identifiers, and a globally incrementing sequence number. The generated structured caption data package is then distributed via a communication connection to the head-mounted display device associated with the current meeting.

[0112] According to the real-time conference translation and display method applied to a cloud server in the above embodiments, this embodiment obtains the audio of the conference and determines the candidate error correction segments, obtains matching professional knowledge items based on the conference knowledge base, and uses an artificial intelligence model for structured error correction, which can significantly improve the recognition accuracy of professional terms in conference subtitles. At the same time, the error correction results are efficiently transmitted to the head-mounted display device by generating structured subtitle data packets.

[0113] The above embodiment provides a method for real-time conference translation display. By extracting candidate segments from the cloud, injecting knowledge based on context, constraining the output of large models, and distributing structured data packets, it improves the problems of large models easily generating illusions, unstable professional terminology, and poor near-eye display reading experience in traditional conference translation solutions.

[0114] By identifying candidate error-correction segments based on the audio from the meeting, this application changes the approach of feeding the entire meeting text into a large model. The system only performs local processing on low-confidence segments with doubts, reducing the amount of input and the frequency of calls to the large model, thereby reducing the data processing latency in the translation process and better meeting the real-time requirements of meeting scenarios.

[0115] By retrieving the meeting knowledge base based on the meeting context identifier bound to the current audio, this embodiment obtains professional knowledge items that match the candidate error-correction segment. This allows the system to acquire specialized terminology (such as specific organization names, industry abbreviations, etc.) related to the current meeting, agenda, or speaker. This enables the system to better adapt to meetings in different professional fields, reducing inconsistencies or misidentifications in the translation of professional terms and improving the overall accuracy of the translation.

[0116] By assembling the candidate error-correction segments with the professional knowledge items into structured prompts and inputting them into an artificial intelligence model, this application obtains the structured error-correction results output by the artificial intelligence model, thus limiting the reference scope for error correction for the large model. This approach allows the large model to primarily make corrections based on the provided professional knowledge items, reducing the introduction of irrelevant information or meaningless rewriting of the original text due to the large model's free divergence, and preserving the objective facts of the translated content.

[0117] Furthermore, this embodiment generates structured data packets, which can assist the terminal in improving the subtitle display effect. The data sent from the cloud includes structured information such as version or timing to assist the device in judgment. This provides the basic data for the head-mounted device to handle network out-of-order issues, perform version comparisons, and perform partial refreshes locally, which helps to reduce subtitle jumps and flickering problems caused by continuous revisions during streaming translation, and improves the user's wearing and reading experience.

[0118] Based on any of the above embodiments, the step of determining candidate error-correction segments based on the audio from the meeting may specifically include: The audio from the meeting was converted into a time-stamped initial translation and intermediate recognition results; low-confidence features were extracted from the intermediate recognition results to perform candidate error detection and identify candidate error-correction segments in the initial translation.

[0119] Specifically, the cloud server inputs the received audio from the meeting into the speech recognition engine for processing. The speech recognition engine outputs a preliminary translated text with timestamps, meaning that each recognized character or word is accompanied by its start and end time information in the original audio. Simultaneously, the speech recognition engine also outputs intermediate recognition results, which include auxiliary information such as confidence scores, candidate word lists, and acoustic features obtained during the recognition process.

[0120] Next, low-confidence features are extracted from the intermediate recognition results to perform candidate error detection and identify candidate error-correction segments in the initial translated text. Specifically, the cloud server extracts low-confidence features from the intermediate recognition results. Low-confidence features may include, for example, words or phrases with recognition confidence scores below a preset threshold, multiple highly similar candidate words in the candidate word list, and inconsistencies between acoustic features and language model features.

[0121] By analyzing these low-confidence features, the cloud server performs candidate error detection to identify text segments in the initial translation that may contain recognition errors, i.e., candidate error-correcting segments. A candidate error-correcting segment can be a word, a phrase, or a complete sentence, and its granularity can be adjusted according to actual needs.

[0122] Based on any of the above embodiments, the step of obtaining professional knowledge items that match the candidate error-correcting segment may specifically include: Parse the meeting information bound to the currently processed meeting audio stream; the meeting information includes at least one of the following: meeting number, session number, agenda identifier, and speaker identifier; use the meeting information as a filter condition, and retrieve the corresponding professional knowledge items from the meeting knowledge base by calculating text similarity within the corresponding range.

[0123] Specifically, the cloud server parses meeting information from the metadata of the currently processed meeting audio stream. This meeting information is pre-configured and bound to the audio stream before the meeting starts or during meeting registration. This meeting information is used to accurately locate professional knowledge items relevant to the current meeting from the meeting knowledge base.

[0124] Using conference information as a filter, and within a corresponding range, text similarity calculation is performed to retrieve relevant professional knowledge items from the conference knowledge base. Specifically, the cloud server uses the parsed conference information (such as conference number, session number, etc.) as a filter to first narrow down the search scope, locating a subset of knowledge related to the current conference in the conference knowledge base. Then, within this subset of knowledge, text similarity calculation is used to match candidate error-correction segments with each professional knowledge item in the subset.

[0125] Text similarity can be calculated using methods such as cosine similarity, Jaccard similarity, edit distance, or semantic similarity based on vector embedding, but this application is not limited to these methods.

[0126] For example, if the candidate error correction fragment "transformer architecture" has a high semantic similarity to the professional knowledge item "Transformer architecture: a deep learning architecture based on self-attention mechanism" in the conference knowledge base, then that professional knowledge item will be retrieved as the matching result.

[0127] This embodiment uses meeting information as a filter for hierarchical retrieval, which ensures that the retrieval results are relevant to the current meeting context, improves retrieval efficiency, and avoids a full search of the entire meeting knowledge base.

[0128] Based on any of the above embodiments, the step of generating structured subtitle data packets based on structured error correction results may specifically include: The structured error correction results are verified, and an incremental version number-updated structured caption data package is generated after the verification passes; the verification includes at least one of evidence verification and semantic offset prevention verification.

[0129] Evidence verification refers to verifying whether the error correction results output by the artificial intelligence model are verifiable. Specifically, the cloud server checks whether each corrective suggestion output by the artificial intelligence model can find corresponding supporting evidence in the retrieved professional knowledge items.

[0130] For example, if an AI model corrects "transformer" to "Transformer," it should be verified whether the correction is supported by corresponding professional knowledge items (such as "Transformer architecture"). If a correction suggestion lacks supporting evidence, it may be a "hallucination" output of the AI ​​model and should be rejected or marked as low confidence.

[0131] Semantic offset prevention verification refers to verifying whether the error correction result output by the artificial intelligence model maintains the original semantics. Specifically, the cloud server compares the semantic similarity between the original text before and after error correction to ensure that the error correction operation only modifies the erroneous parts without changing the overall semantics of the original text.

[0132] For example, if the original text is "We adopted a brand new transformer architecture", the corrected result is "We adopted a brand new Transformer architecture". The two are semantically consistent and the verification passes. However, if the corrected result is "We abandoned the traditional Transformer architecture", the semantics have obviously shifted and the verification fails.

[0133] Optionally, semantic similarity comparison can be achieved, for example, by calculating the cosine similarity between the vector embeddings of two text segments, and determining semantic shift when the similarity is lower than a preset threshold.

[0134] After successful verification, the cloud server generates a structured subtitle data package with an updated incremental version number. Specifically, the cloud server increments the incremental version number of the corresponding subtitle segment and encapsulates the corrected text content, the new incremental version number, operation type indication, and other fields into a new structured subtitle data package.

[0135] If the verification fails, the cloud server can keep the original subtitle content unchanged and not generate a new structured subtitle data package, or mark the error correction result as pending manual review.

[0136] Based on any of the above embodiments, when distributing structured subtitle data packets, multi-terminal differentiated distribution is performed, which may specifically include: Obtain the language subscription preferences and information display hierarchy preferences reported by each currently connected head-mounted display device; within the same time window, push structured subtitle data packets of different target languages ​​and / or different information densities to the corresponding head-mounted display devices.

[0137] In a meeting setting, multiple attendees may simultaneously participate in the meeting wearing head-mounted display devices, and different attendees may have different language preferences and information display needs.

[0138] Specifically, multi-terminal differentiated distribution may include the following steps: First, obtain the language subscription preferences and information display hierarchy preferences reported by each currently connected head-mounted display device. Language subscription preferences refer to the target language of subtitle translations that the wearer wishes to receive, such as Chinese, English, Japanese, Korean, etc.

[0139] Information display hierarchy preference refers to the density and level of detail of information that the wearer desires to be displayed in the near-eye display area. Examples include "Concise Mode" (displaying only the core translation content), "Standard Mode" (displaying the translation content and key terminology annotations), and "Detailed Mode" (displaying the translation content, key terminology annotations, and original text comparison). Each head-mounted display device can report the wearer's language subscription preferences and information display hierarchy preferences to the cloud server when establishing a connection or during a meeting.

[0140] Then, within the same time window, structured subtitle data packages in different target languages ​​and / or with different information densities are pushed to the corresponding head-mounted display devices. Specifically, based on the preference information reported by each head-mounted display device, the cloud server performs multilingual translation and multi-level information organization on the corrected subtitle content, generating differentiated structured subtitle data packages for different terminals.

[0141] For example, within the same time window, the cloud server can push a simplified structured subtitle data package in Chinese to head-mounted display device A that has subscribed to the simplified Chinese mode, push a detailed structured subtitle data package in Japanese to head-mounted display device B that has subscribed to the detailed Japanese mode, and push a standard structured subtitle data package in English to head-mounted display device C that has subscribed to the standard English mode.

[0142] By distributing content across multiple terminals, each participant can receive personalized subtitle services tailored to their language and information density needs, thus enhancing the user experience of conference translation subtitles.

[0143] This application also provides a real-time conference translation display system, including a cloud server and at least one head-mounted display device.

[0144] The cloud server device is used to execute the real-time conference translation and display method described above, namely, acquiring the audio of the conference and identifying candidate error correction segments, retrieving professional knowledge items from the conference knowledge base, performing structured error correction using an artificial intelligence model, and generating and distributing structured subtitle data packages.

[0145] The head-mounted display device is used to perform the real-time conference translation display method described above, namely, receiving structured subtitle data packets, performing subtitle version filtering, and performing local anti-jump refresh rendering.

[0146] The cloud server communicates with each head-mounted display device via a network connection. This network connection can be, for example, a communication connection established through a Wi-Fi network, a 5G mobile network, or a wired local area network, but this application is not limited to these. During system operation, the cloud server continuously receives and processes the audio from the conference, and sends the generated structured subtitle data packets to each head-mounted display device in real time; after receiving the structured subtitle data packets, each head-mounted display device renders and displays the subtitles in its near-eye display area.

[0147] The real-time conference translation and display system provided in this embodiment enables the organic collaboration of conference audio acquisition, cloud-based intelligent recognition and translation, large-scale model context enhancement, multilingual concurrent generation, real-time smooth display on smart glasses terminals, personalized user interaction, and unified management of conference equipment. This improves communication efficiency, deployment efficiency, user experience, and technology accessibility in international conferences and similar scenarios.

[0148] This application also provides a head-mounted display device that uses the aforementioned real-time conference translation display method. The head-mounted display device includes a communication module, a near-eye display module, and a local processing module.

[0149] The communication module is used to establish a connection with the cloud server, receive structured text data packets, and report local configuration commands. The communication module may include, for example, a Wi-Fi communication unit, a Bluetooth communication unit, a 5G / 4G cellular communication unit, etc., but this application is not limited to these.

[0150] The communication module is responsible for maintaining the communication link with the cloud server, including sending heartbeat packets, receiving structured caption data packets, reporting local configuration commands such as the wearer's language subscription preferences and information display hierarchy preferences, and initiating reconnection requests after network interruption.

[0151] Near-eye display modules are used to provide wearers with a visual information display area within their field of view. Near-eye display modules may employ technologies such as optical waveguide display technology, Micro-LED display technology, liquid crystal on silicon (LCoS) display technology, and digital light processing (DLP) projection technology, but this application is not limited to these.

[0152] Driven by the local processing module, the near-eye display module presents the subtitle text content visually within the wearer's field of view. Parameters such as the position, size, and transparency of the near-eye display area can be adjusted according to the wearer's settings.

[0153] The local processing module, including a processor and memory, is used to maintain the subtitle status table, perform subtitle version comparison and filtering, calculate dirty regions, and drive the near-eye display module to perform local anti-jump refresh rendering. The local processing module performs each step in the above-mentioned real-time conference translation display method, including version filtering after receiving structured subtitle data packets, anti-out-of-order processing, differential calculation, dirty region mapping, double-buffered rendering, and other operations.

[0154] In addition to the three modules mentioned above, head-mounted display devices may also include other components, such as battery modules, sensor modules (e.g., IMU inertial measurement units, ambient light sensors, etc.), audio modules (e.g., speakers, microphones, etc.).

[0155] The following describes the complete workflow of the real-time conference translation display system according to an embodiment of this application, using AR glasses as an example.

[0156] This embodiment constructs an AR+AI real-time translation technology system for large-scale conference scenarios. It integrates direct audio transmission from the conference microphone / mixer, cloud-based speech recognition and multilingual large-scale model translation, large-scale model context enhancement and dynamic terminology error correction, structured subtitle generation and incremental refresh for AR near-eye display, multi-terminal differentiated language subscription and distribution, and unified access management for large-scale AR glasses. This enables participants to obtain translated subtitles and / or audio in real time through AR glasses without additional headphones or complex pairing. They can also select the language, font, and auxiliary information display content as needed, thus taking into account low latency, high accuracy, professional terminology adaptation, good reading continuity, and large-scale conference deployment capabilities.

[0157] Specifically, the system uses AR glasses as the main information presentation terminal, artificial intelligence big data models as the core engine for translation and semantic enhancement, and networked device management capabilities as support for large-scale applications. It can realize specific functions such as "audio acquisition at the meeting site - audio access and preprocessing - cloud big data model translation processing - multi-terminal differentiated distribution - real-time display on AR glasses - user interaction feedback - unified device management". This allows participants to directly obtain real-time subtitles and / or real-time translated speech through AR glasses without the need for additional headphones or complicated pairing, and select the target language, display style and auxiliary information content according to their personal needs.

[0158] The key improvements in this embodiment are: the cloud organizes the translation results into structured subtitle data packages with version and timing constraints; the AR glasses terminal performs deduplication, out-of-order buffering, version comparison, and partial refresh based on the local state table; and the cloud automatically retrieves the meeting knowledge base based on the meeting session number and assembles structured prompt words to call the large model for constrained error correction.

[0159] In this embodiment, the specific modules of the system are as follows: 1. Conference audio acquisition unit Used to acquire the voice signals of speakers at a conference. The conference audio acquisition unit may include one or more of the following: handheld microphone; lavalier microphone; gooseneck microphone; array microphone; line output interface of the conference sound system; mixing console output interface; other audio devices capable of acquiring valid voice from the conference venue.

[0160] The conference audio acquisition unit is directly connected to the conference sound reinforcement system, or the audio is transmitted directly through a mixing console to reduce far-field reception attenuation and the introduction of environmental noise, thereby improving the quality of the source audio.

[0161] 2. Audio Input and Control Unit It is used to receive audio signals output from the conference audio acquisition unit and to preprocess, standardize, and control the network transmission of the audio signals.

[0162] The audio access and control unit can perform at least one or more of the following processes: analog-to-digital conversion; audio resampling; automatic gain control; noise suppression; echo cancellation; voice activity detection; framing and packetization; timestamp marking; audio encoding; session identifier binding; bitstream buffering; multiple audio input selection; streaming upload.

[0163] The audio access and control unit can be an edge access device, server, mixing console adapter module, or meeting room gateway device. The audio access and control unit is also used to bind the current meeting number, session number, speaking channel number, speaker information, or agenda information to the audio stream so that the cloud processing module can perform associated processing subsequently.

[0164] 3. Cloud-based intelligent translation processing platform It is used to identify, translate, semantically enhance, correct terminology, organize subtitles, synthesize speech, and distribute results of conference audio uploaded by the audio access and control unit.

[0165] A cloud-based intelligent translation processing platform includes at least one or more of the following modules: 1) Speech recognition module: used to convert conference audio streams into source language text in real time; supports continuous recognition, incremental recognition, sentence segmentation recognition, and revision output.

[0166] 2) Translation module: used to translate source language text into at least one target language text; supports multiple target languages ​​and generation.

[0167] 3) Large Model Semantic Enhancement Module: This module combines conference context, terminology knowledge, historical speeches, guest lists, organization names, and professional domain knowledge to perform semantic enhancement on source language text and / or target language text.

[0168] The semantic enhancement processing includes at least one or more of the following: contextual disambiguation; dynamic error correction; proper noun correction; terminology standardization; abbreviation expansion; sentence segmentation optimization; conversion of spoken language to standard expression; translation polishing; and semantic constraints based on the conference theme.

[0169] 4) Terminology Knowledge Management Module: Used to import, maintain, and access meeting-specific knowledge content before or during the meeting. This knowledge content includes, but is not limited to: a professional terminology database; a list of guest names; organization and company names; meeting theme; agenda name; industry abbreviations; standard translations of proper nouns; and a list of hot words and keywords.

[0170] 5) Subtitle generation and arrangement module: Used to organize the recognition results or translation results into structured subtitle units adapted for AR glasses display.

[0171] Each subtitle unit includes one or more of the following fields: subtitle text; subtitle language identifier; generation timestamp; effective timestamp; expiration timestamp; speaker identifier; statement number; incremental version number; style parameters; priority parameters; and auxiliary information identifier.

[0172] 6) Speech synthesis module: used to convert target language text into target language speech for terminal output.

[0173] 7) Terminal Session and Distribution Module: This module is used to send content in different target languages, display modes, and information levels to the corresponding terminals based on the user configuration of different AR glasses terminals.

[0174] 8) Device management and scheduling module or backend: used for unified authentication, unified access, unified grouping, unified monitoring, unified parameter configuration and unified scheduling of large-scale AR glasses terminals.

[0175] 4. AR glasses terminal It is used to connect to a cloud-based intelligent translation processing platform via a wireless network and display translated text, meeting information, and auxiliary information in a near-eye display module, or output translated speech.

[0176] AR glasses terminals include at least: 1) Communication module: Used to establish network communication connection with the cloud-based intelligent translation processing platform.

[0177] 2) Display module: Used to display translated subtitles, original text subtitles, bilingual subtitles, conference topics, speaker information, terminology prompts and other auxiliary information in the near-eye display area.

[0178] 3) Local processing module: used to parse data packets sent from the cloud and perform subtitle sorting, caching, pagination, partial refresh, redraw rendering and interactive response.

[0179] 4) Local caching module: used to buffer the most recent subtitle content and its version information, maintaining display continuity during network jitter, revision updates, or short interruptions.

[0180] 5) Interaction Module: Used to receive user input. Interaction methods may include: touchpad control; button input; voice control; gesture control; and head movement triggering.

[0181] 6) Parameter configuration module: Used to store and apply the current user's personalized configuration, including but not limited to: target language; whether to display the original text; font size; font color; display position; number of subtitle lines; scrolling mode; voice playback switch; auxiliary information switch.

[0182] 5. Meeting Management and Information Service Unit: Used to store and distribute information such as meeting theme, forum name, speaker's name, speaker's position, affiliated organization, current agenda, notification content, and terminology explanation.

[0183] The meeting management and information service unit is connected to a cloud-based intelligent translation processing platform and dynamically updates content according to the meeting's progress. Information output by the meeting management and information service unit can be displayed synchronously with the translated subtitles on the AR glasses terminal.

[0184] For example, the functional modules and their corresponding functions in an AR glasses terminal can be referred to as follows:

[0185] The cloud-based subtitle generation and arrangement module encapsulates the translation results into a structured subtitle data packet, SubtitlePacket. This data packet is not plain text, but contains fields for stable rendering on AR terminals, version detection, and out-of-order processing. The meanings of each field are as follows:

[0186] Specifically, an example of SubtitlePacket is as follows: { "meetingId": "ZGC2026", "sessionId": "MAIN_FORUM_01", "streamId": "AUDIO_CONSOLE_CH1", "targetLang": "en-US", "sentenceId": "S000128", "segmentId": "S000128-02", "seqNo": 35891, "version": 3, "opType": "UPDATE", "startTs": 1715000123.240, "endTs": 1715000125.860, "validFrom": 1715000124.000, "validTo": 1715000132.000, "finalFlag": false, "text": "large model-enabled AR conference translation system", "layoutHint": {"maxLines": 4, "preferLine": "current", "lockCommittedLines": true}, "evidence": {"termIds": ["TERM-AR-018", "TERM-LLM-003"]} }

[0187] AR glasses terminals maintain three core data structures: a reordering cache, a local subtitle status table, and a display window model. SubtitlePackets pushed from the cloud are not directly drawn to the screen; instead, they undergo validation, filtering, sorting, version comparison, and partial refresh decision-making before being displayed on the screen.

[0188] For example, the data stored in the AR glasses terminal can be referenced as follows:

[0189] Please see Figure 3 , Figure 3This is a schematic diagram illustrating the local subtitle refresh and out-of-order processing flow for AR glasses provided in this application embodiment. The local processing flow of the AR glasses after receiving the SubtitlePacket is as follows: 1. After receiving the cloud-based subtitle data packet (SubtitlePacket), first verify the meetingId, sessionId, language / targetLang, checksum, and user subscription configuration, and filter out non-currently subscribed content, that is, data packets that are not from the current meeting or are not in the current language will not enter the rendering process.

[0190] 2. Read sentenceId, segmentId, seqNo, version, opType, startTs / endTs, validFrom / validTo and finalFlag.

[0191] 3. Query the caption status table using (sentenceId, segmentId, targetLang); if the version of the data packet is less than or equal to the applied version (i.e., maxAppliedVersion in the status table), it is determined to be a duplicate or old packet and discarded; if the version is higher, it enters the candidate update process.

[0192] 4. Place candidate data packets into the reordering cache. If the seqNo is consecutive or exceeds the out-of-order waiting window, retrieve the packet to be applied from the reordering cache; if a seqNo gap is found, wait for the missing packet within the preset out-of-order waiting window Δt; if the missing packet is not received after Δt, continue displaying based on the highest available version and request the cloud to resend the missing range.

[0193] 5. Perform operations based on opType: ADD is used to add new subtitle segments; UPDATE is used to revise existing segments; DELETE is used to remove or mark expired segments; COMMIT is used to confirm the final version and lock the display position and version status of the segment.

[0194] 6. For UPDATE operations, character / word-level differencing is performed, marking only the display area corresponding to the changed text as a dirty region. The visible window is maintained based on the effective / expired timestamp, and the line anchor point is stabilized. The position of the displayed lines is not changed unless necessary. That is, for UPDATE operations, the terminal does not replace the entire line or the entire screen, but performs character-level or word-level differencing on the old and new text to obtain the changed range; the dirty region is generated by mapping the font measurement and line break results to screen coordinates.

[0195] 7. The renderer uses a double-buffering mechanism to avoid full-screen redraws. Only the dirty region and the smallest adjacent region affected by it are redrawn; unchanged text, confirmed lines, and auxiliary information areas remain in their original positions.

[0196] 8. Update the highest version number, final state flag, and display position in the local state table.

[0197] 9. If a version gap or long-term missing package is detected, request a reissue or use the latest available version according to the timeout policy.

[0198] In addition, when the display window overflows, the old row is only moved up smoothly as a whole when a new final confirmed row is added or the display rhythm threshold is reached; intermediate revision results are replaced within the current row by default and do not push confirmed rows.

[0199] Furthermore, after successful application, update maxAppliedVersion, finalFlag, renderLineId, renderRect, and expireTs in the caption status table.

[0200] The above process enables AR glasses to follow the principles of "higher version priority, final state priority, current line variable, confirmed line stable, and local area refresh" when facing ASR intermediate results, translation revision results, and out-of-order network packets.

[0201] The rules for handling AR glasses out-of-order transmissions, retransmissions, and version conflicts are as follows:

[0202] It should be noted that the out-of-order waiting window Δt can be dynamically adjusted according to network quality, for example, 80-150ms in a stable Wi-Fi environment and 150-300ms in congested or weak network scenarios. This parameter is not a limiting condition of this application; its purpose is to strike a balance between real-time performance and display stability.

[0203] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the relationship between the subtitle data package and the AR local status table provided in the embodiments of this application.

[0204] The raw data unit received by the AR glasses is a SubtitlePacket, which contains complete metadata: meetingId (meeting ID), sessionId (session ID), sentenceId (sentence ID), segmentId (segment ID), version number, seqNo (sequence number), opType (operation type), startTs / endTs (start and end timestamps), and finalFlag (final flag). These fields collectively support incremental updates and out-of-order reordering.

[0205] The raw data packets first enter the reordering buffer layer. This layer sorts the packets by seqNo and timestamp, and tolerates out-of-order arrivals caused by network latency through a "wait out-of-order window Δt" mechanism; at the same time, it filters duplicate packets and expired versions to ensure that the data processed downstream is complete and up-to-date.

[0206] The cached data is updated in the subtitle status table, which uses (sentenceId, segmentId) as a composite primary key to maintain the current state of each subtitle segment: maxAppliedVersion (the maximum version applied), renderLineId / rect (the rendering line ID and region), finalFlag (whether it has expired), and expireTs (the expiration time). The status table is the hub connecting the data layer and the display layer, and all rendering decisions are based on the current state of this table.

[0207] The status table data drives two downstream modules: 1. Display Window Model: Used to manage the layout of fixed row anchors (displaying up to N rows), distinguishing between "currently variable rows" (still receiving incremental updates) and "confirmed locked rows" (content terminated, position stable). This model updates its own layout through a status table.

[0208] 2. Renderer: Performs actual text drawing, adopts a "word-level / character-level diff" strategy to minimize the redraw range, and combines "dirty region" detection and "double buffering" technology to achieve efficient and flicker-free real-time rendering.

[0209] It's important to note that the AR glasses have two feedback loops: the caption state table simultaneously outputs data to both the "reordering cache" and the "display window model," while the display window model outputs data to the renderer. The renderer may also influence the state table in reverse (e.g., confirming rendering completion). This design allows the system to adaptively handle incremental updates, ensuring display stability while responding promptly to new caption data.

[0210] The large model in this application does not simply receive entire conference texts and freely rewrite them. Instead, it constructs a closed loop in the cloud: "Conference session knowledge retrieval—candidate error detection—structured prompt word assembly—constrained output of the large model—result verification—revision version distribution." This closed loop ensures that the large model corrects errors only at necessary locations, based on relevant conference knowledge, avoiding the introduction of uncertain content.

[0211] Example as follows:

[0212] The meeting knowledge base can be imported by meeting organizers before the meeting begins or dynamically supplemented during the meeting. Each knowledge item can contain the following fields: termId, sourceTerm, targetTerm, aliases, phoneticAliases, speakerId, organization, agendaId, effectiveTimeRange, priority, and confidenceThreshold. The system determines the meeting scope based on the current meetingId and sessionId, and then retrieves a limited number of relevant knowledge items based on agendaId, speakerId, and candidate text similarity.

[0213] An example prompt word template is shown below. This template is automatically filled in by the cloud, rather than being manually entered one by one: System Role: You are the real-time translation and error correction module for conferences. You can only correct errors in terminology, proper nouns, organization names, abbreviations, and punctuation in the ASR or translation based on the given conference knowledge and context. You may not add new facts outside the knowledge base and context.

[0214] Meeting information: meetingId={meetingId}; sessionId={sessionId}; agenda={agendaTitle}; speaker={speakerName}; targetLang={targetLang}.

[0215] Recent context: {lastKSourceSegments}.

[0216] Candidate input: sourceText={sourceText}; mtText={machineTranslation}; asrConfidence={confidence}.

[0217] Related knowledge: {retrievedTermsAndNames}.

[0218] Output requirements: Output only JSON, with fields including noChange, correctedSource, correctedTranslation, and changes[{before, after, type, evidenceTermId, reason, confidence}]. If you cannot confirm this based on relevant knowledge, please output noChange=true.

[0219] After the large model is output, the result validator performs the following validations: 1. JSON Schema Validation: Fields must be complete, types must be correct, and free text interpretation is prohibited.

[0220] 2. Evidence verification: Any modifications involving terms, guests, or organization names must cite the evidenceTermId from retrievedTermsAndNames.

[0221] 3. Confidence verification: If the confidence level is lower than the threshold, the subtitles will not be updated automatically. The original text can be retained or marked for manual review in the background.

[0222] 4. Length and semantic offset verification: If the length of the translation changes beyond the threshold or a new entity not present in the context is introduced, the modification will be rejected.

[0223] 5. Timestamp Preservation: Error correction only changes the text content, without changing the startTs / endTs corresponding to the original subtitle segment, ensuring a stable display rhythm.

[0224] The error correction results verified are used by the subtitle generation module to generate a new SubtitlePacket. This packet retains the same sentenceId and segmentId, but the version is increased, opType is set to UPDATE or COMMIT, and it carries changes and evidenceTermId. After receiving this higher version packet, the AR glasses update the corresponding words or segments according to the aforementioned local refresh algorithm.

[0225] Please see Figure 5 , Figure 5 This is a schematic diagram of the dynamic error correction process for large model context and terminology provided in the embodiments of this application.

[0226] First, the system receives the conference audio stream or incremental text generated by ASR (Automatic Speech Retrieval) and proceeds to candidate error detection. By identifying low-confidence segments, proper nouns, and terminology, it initially locates text areas where identification errors may occur. Simultaneously, it maintains two key knowledge bases: one is the conference knowledge base, which stores structured information such as standard translations of terminology, alternative names / phonetic similarities, guest lists, organization names, agenda topics, and historical context; the other is a context manager, which records the K most recent statements, the current speaker, the current topic, the target language, and historical revisions to maintain contextual coherence.

[0227] After error detection, relevant knowledge is retrieved from the conference knowledge base according to the session number, and assembled into a structured prompt in combination with the information in the context manager. Only knowledge related to the current scenario is injected to avoid interference from irrelevant information. Subsequently, the large model is invoked for inference to output error-corrected text in JSON format (including the translation and confidence). After the large model outputs the result, a result verification step is performed for verification according to the preset Schema, term constraints and confidence threshold. If the verification is passed, a revised version is generated; if not, the original recognition result is retained.

[0228] In the process of result verification, if the term base is precisely hit, rule-based error correction is directly performed to ensure the absolute accuracy of key terms. Finally, the revised text is distributed in the form of a subtitle version package through UPDATE or COMMIT, while the line anchor point is maintained through AR local partial refresh technology to realize real-time and smooth updated display of subtitles.

[0229] The implementation example of large model error correction is as follows: Taking a technology forum as an example, the standard English translation "llvision AI" of the term "亮亮AI", the standard English translation "AR+AI conference translation system" of the term "AR+AI会议翻译系统", as well as the names of guests and institution names are imported in the foreground and background of the conference. During the conference, the ASR intermediate result recognizes "亮亮" as a word with similar pronunciation, and the initial translation module outputs an unstable translation. The candidate error detector identifies this segment as a candidate error correction object according to the low confidence and similar pronunciation word rules, and retrieves relevant terms according to the meetingId and sessionId. The prompt assembler injects the current agenda, current speaker, recent context and relevant terms into the template, and the large model outputs a JSON error correction result containing evidenceTermId. The result verifier confirms that the error correction comes from the conference knowledge base, and generates a higher-version UPDATE package of the same sentenceId to be distributed to the AR glasses. The glasses end only replaces the display area corresponding to the term with the standard translation, without changing the positions of other subtitles.

[0230] Exemplarily, a conference real-time translation display method may include the following steps: Step 1. Collect conference audio: Collect the audio at the conference site, and perform preprocessing, framing, encoding and timestamp marking through the audio access and control unit.

[0231] Step 2. Perform audio preprocessing: The audio access and control unit performs preprocessing on the audio, and the preprocessing includes one or more of the following: denoising; gain adjustment; echo cancellation; framing; encoding; timestamp stamping; buffering; encapsulating and uploading.

[0232] Step 3: Upload to the cloud processing platform: The pre-processed audio stream is uploaded to the cloud-based intelligent translation processing platform and associated with meeting identification information and / or speaking channel information.

[0233] Step 4: Perform translation processing: The cloud-based translation module generates an initial translation in the target language and feeds the source text, initial translation, ASR confidence score, and timestamp into a candidate error detector. The candidate error detector identifies low-confidence words, suspected technical terms, proper names, organization names, abbreviations, and inconsistent translations. The system retrieves relevant terms, speakers, organizations, and historical context from the conference knowledge base based on meetingId, sessionId, agencyId, and speakerId. A cue word assembler generates structured cue words, calls a large model for constrained error correction, and requires the large model to output results in JSON format.

[0234] Step 5: Generate subtitle data units: The result validator performs format, evidence, confidence, and semantic offset checks on the output of the large model; after passing the checks, a revised structured subtitle package is generated. The subtitle generation and version management module sets sentenceId, segmentId, seqNo, version, opType, startTs / endTs, validFrom / validTo, and finalFlag for each subtitle package.

[0235] Step 7: Perform differentiated distribution based on terminal configuration. The terminal conversation and distribution module pushes relevant content to the corresponding AR glasses terminal based on the language preferences, style preferences, and auxiliary information configurations of different terminal users.

[0236] Step 8: The terminal executes display or playback: After receiving the subtitle package, the AR glasses first perform subscription filtering, verification, and cache reordering, and then compare versions according to the local subtitle status table. If it is an old or duplicate package, it is discarded; if it is a higher version package, a partial update is performed by calling ADD, UPDATE, DELETE, or COMMIT. For UPDATE, the AR device performs text differentiation, only redrawing the dirty region, maintaining the stable position of the confirmed subtitle lines. When the user switches languages, font sizes, or display modes, the terminal updates the layout parameters locally; if necessary, it synchronizes the configuration to the cloud so that subtitle packages of the corresponding language and style can be generated for subsequent distribution.

[0237] Step 9: Receive user interaction instructions: The terminal receives user input via the interaction module, including language switching, font adjustment, display mode switching, auxiliary information switching, and playback on / off.

[0238] Step 10: Dynamically update display and service parameters: For configurations that can be applied locally immediately, the AR glasses terminal directly updates the display interface; for configurations that need to affect the cloud distribution strategy, the instructions are uploaded to the cloud platform, and the cloud updates the subsequent distribution content.

[0239] Step 11: Perform terminal management and status maintenance. The device management and scheduling module continuously acquires the online status, power status, network status, and abnormal status of each terminal to achieve unified supervision and scheduling.

[0240] The above-described embodiment can achieve the following technical effects: 1. By using the sentenceId, segmentId, seqNo and version in the SubtitlePacket, this application can distinguish different revisions of the same subtitle, avoiding network out-of-order transmission or retransmission that could cause the old version to overwrite the new version.

[0241] 2. Through a local subtitle status table and reordering cache, AR glasses can maintain consistent subtitle order and stable display status even in weak network and out-of-order conditions.

[0242] 3. By using character / word-level differentiation, dirty regions, and double-buffered rendering, this application only refreshes the changing areas, reducing full-screen refreshes and sudden changes in line position, thus improving near-eye reading comfort.

[0243] 4. By adopting a display window model that features "variable current row, locked confirmation row, and smooth upward movement upon overflow", this application adapts to the display constraints of small windows and a limited number of rows in AR glasses.

[0244] 5. By using a conference knowledge base, candidate error detection, and structured prompts, this application does not simply feed the entire text into a large model without constraints. Instead, it injects relevant terms, guests, institutions, and context of the current session into the error correction link to improve the accuracy of professional translation.

[0245] 6. Through JSON output and result verification mechanism, this application reduces the risk of large model illusion, makes the error correction results verifiable and traceable, and can be transformed into accurate subtitles to drive partial refresh of the display terminal.

[0246] Please see Figure 6 , Figure 6 This is a schematic diagram of the overall process of the real-time conference translation display system provided in this application embodiment.

[0247] 1) Audio Acquisition and Transmission: The system uses the speaker as the core information source. The speaker's audio is first sent to the control console via the audio transmission link. The control console then uploads the audio to the cloud translation system and is also responsible for the overall management of the devices. Simultaneously, the AR glasses worn by the participants, as terminal devices, also send requests to the cloud translation system, establishing a two-way data channel.

[0248] 2) Cloud Processing and Translation: The cloud translation system is the central node of the entire architecture. It receives audio streams from the console and requests from each AR glasses, performs real-time speech recognition and translation, and then pushes the translated text back to each AR glasses terminal. Furthermore, the system interfaces with the equipment operation and maintenance management layer to monitor and manage the status of terminal devices.

[0249] 3) Subtitle Generation and Knowledge Management: The translated text data flows further to the cloud-based real-time translation module. The output of this module is then passed up to the subtitle generation and version management layer, where subtitle formatting, timeline alignment, and multi-version management are completed. Finally, the subtitle content enters the conference knowledge base and large model constraint layer. This layer not only undertakes the functions of accumulating and storing conference knowledge but also constrains and optimizes translation quality through the capabilities of the large model. The processed results are then output back to the AR glasses display layer, completing the information loop.

[0250] 4) Terminal presentation: Each participant's AR glasses serve as the final display terminal, actively initiating translation requests and receiving translated text pushed from the cloud. At the same time, the system can also receive display content optimized by the knowledge base and large model, realizing a dual information presentation of "real-time translation + intelligent enhancement".

[0251] Please see Figure 7 , Figure 7 This is a schematic diagram of the conference implementation translation process provided in the embodiments of this application.

[0252] In this embodiment, the overall method can be divided into three stages: on-site / cloud-based basic processing, dynamic error correction of large model terminology, and anti-jump display for AR glasses.

[0253] In the on-site / cloud-based basic processing stage, the conference audio is first captured via microphone, mixing console, or sound card (S1), followed by audio preprocessing, including framing, encoding, timestamping, and binding to conference / session / channel information (S2). Next, streaming speech recognition outputs incremental text of the source language, sentence ID (sentenceId), and confidence level (S3), and the translation engine generates an initial translation in the target language, retaining the segment number and timestamp (S4). Afterwards, the candidate error detection stage (S5) identifies low-confidence segments, terminology / property name errors, and segments with inconsistent translations, and determines whether to trigger the large-scale model error correction process.

[0254] When error correction is detected, the large model's dynamic terminology correction phase begins. This phase first performs knowledge retrieval, searching the conference knowledge base based on the conference ID (meetingld), session ID (sessionld), agenda ID (agendald), and speaker ID (speakerld) (S6). Then, structured prompts are assembled, and relevant terms, guests, institutions, historical context, and correction constraints are injected (S7). Subsequently, the large model is invoked to perform terminology, proper noun, sentence segmentation, and translation consistency correction only on candidate segments (S8). The results are then validated, including JSON Schema verification, evidence and confidence assessment, semantic offset detection, and timestamp preservation verification (S9). After successful validation, a higher version of the subtitle package is generated, which also affects the generation and distribution of subsequent subtitle packages.

[0255] Once it is detected that no error correction is needed, or after generating a higher version subtitle package, a structured subtitle package can be generated, such as setting seqNo, version, opType, validFrom / validTo, finalFlag (S10), and the subtitle package can be distributed according to terminal subscriptions.

[0256] During the AR glasses' anti-jump display phase, the AR device receives and filters subtitle packets, verifying the meeting, language, subscription configuration, and checksum (S11). Next, it performs reordering and version comparison, entering the reordering cache by seqNo and comparing the version with the status (S12). If an old version or duplicate packet is detected, it is discarded; if it is a new, higher version packet, it performs operations such as adding, revising, deleting, or confirming subtitle segments (S13). Then, a partial refresh is performed, with character / word-level differentiation during the update to generate a dirty region, and anti-jump display is achieved through double-buffered redrawing (S14). Afterwards, the display status is updated, the subtitle status table is updated, and logic such as current line variability, confirmed line locking, and overflow smooth upward movement is handled (S15), finally completing the display and providing configuration feedback.

[0257] In addition, this embodiment also supports user interaction feedback (S16), including local application or cloud upload of configurations such as language switching, font size, and display mode, forming a complete closed-loop experience.

[0258] In this embodiment, a large model is used to achieve dynamic error correction of terms and proper nouns to ensure translation accuracy. At the same time, an anti-jump mechanism is designed for the special display scenario of AR glasses. Version comparison, partial refresh and double buffering technology are used to ensure a smooth and stable visual experience when the subtitles are updated, avoiding flickering or jumping problems caused by full refresh.

[0259] On the other hand, this application also provides an electronic device, please refer to... Figure 8 , Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the electronic device may include a memory 820, a processor 810, and a computer program stored in the memory 820 and executable on the processor 810. When the processor 810 executes the program, it can implement a real-time conference translation display method, which may include: Receive structured caption data packets from the cloud server; the structured caption data packets are sent by the cloud server after recognizing the conference audio and correcting errors using an artificial intelligence model; the structured caption data packets include at least an incremental version number and an operation type indicator; based on the locally maintained caption status table, the incremental version number is compared with the corresponding highest applied version number locally, and caption version filtering is performed; when the received structured caption data packet is determined to be a valid update packet, local anti-jump refresh rendering is performed in the near-eye display area of ​​the head-mounted display device according to the operation type indicator; Alternatively, the system can acquire the audio from the meeting and determine candidate error correction segments based on the audio; retrieve the meeting knowledge base based on the meeting context identifier bound to the current audio to obtain professional knowledge items that match the candidate error correction segments; assemble the candidate error correction segments and professional knowledge items into structured prompt words and input them into an artificial intelligence model to obtain the structured error correction results output by the artificial intelligence model; generate structured subtitle data packages based on the structured error correction results and send them to the head-mounted display device.

[0260] Optionally, the electronic device may further include a communication bus 830 and a communication interface 840, wherein the processor 810, the communication interface 840, and the memory 820 communicate with each other via the communication bus 830. The processor 810 can call the computer program in the memory 820 to execute the real-time conference translation and display methods provided by the above methods.

[0261] Furthermore, the logical instructions in the aforementioned memory 820 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0262] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the real-time conference translation and display method provided by the above methods. The steps and principles of the method have been described in detail in the above methods and will not be repeated here.

[0263] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the real-time conference translation display method provided by the above methods. The steps and principles of this method have been described in detail in the above methods and will not be repeated here.

[0264] Non-transitory computer-readable storage media can be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0265] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0266] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for real-time translation display in a conference, characterized in that, The real-time conference translation display method, applied to head-mounted display devices, includes: Receive structured caption data packets from a cloud server; the structured caption data packets are sent by the cloud server after recognizing the conference audio and correcting errors using an artificial intelligence model; the structured caption data packets include at least an incremental version number and an operation type indicator; Based on the locally maintained subtitle status table, the incremental version number is compared with the corresponding highest applied version number locally, and subtitle version filtering is performed; When the received structured caption data packet is determined to be a valid update packet, local anti-jump refresh rendering is performed in the near-eye display area of ​​the head-mounted display device according to the operation type indication.

2. The real-time conference translation display method according to claim 1, characterized in that, The structured subtitle data package also includes subtitle segment identifiers; the subtitle version filtering includes: By parsing the subtitle segment identifier, the highest version number of the corresponding segment recorded in the subtitle status table is queried; If the received incremental version number is less than or equal to the highest applied version number, then the current data packet is determined to be an old version or a duplicate packet. If the received incremental version number is greater than the highest applied version number, then the current data packet is determined to be a valid update packet.

3. The real-time conference translation display method according to claim 1, characterized in that, Before performing local anti-jump refresh rendering, anti-out-of-order processing is performed using a local cache, including: Place the valid update package into the local reorder cache; Detect whether the globally increasing sequence numbers of data packets in the reordering buffer are consecutive; if there is a sequence number gap, wait for the missing data packets within a preset out-of-order waiting window; If the missing packet is not received within the preset out-of-order waiting window, the highest version packet in the current cache is extracted and rendered, and a missing sequence retransmission request is sent to the cloud server.

4. The real-time conference translation display method according to claim 1, characterized in that, The execution of local anti-jump refresh rendering includes: When the operation type is indicated as content update, the newly received text is calculated locally and the old text currently displayed on the screen is compared with the difference to extract the text range that has changed. Based on the interface layout parameters of the current near-eye display area, the text interval is mapped to a dirty region; A double buffering mechanism is used to redraw the dirty region and the smallest adjacent region affected by the dirty region, while keeping the rendering state of other unchanged regions unchanged.

5. The real-time conference translation display method according to any one of claims 1 to 4, characterized in that, Also includes: When a continuous loss of heartbeats or an interruption in the subtitle stream is detected, the most recently confirmed subtitle content is displayed, and the network status is alerted. After the network is restored, the latest parameters in the local caption status table are used to re-establish a connection with the cloud server and align the caption status.

6. A method for real-time translation display in a conference, characterized in that, The real-time conference translation display method, applied to a cloud server, includes: Acquire the audio of the meeting and determine candidate error correction segments based on the audio. Based on the meeting context identifier bound to the current audio, the meeting knowledge base is searched to obtain professional knowledge items that match the candidate error correction segment; The candidate error correction fragments are assembled with the professional knowledge items into structured prompt words and input into the artificial intelligence model to obtain the structured error correction results output by the artificial intelligence model; Based on the structured error correction results, a structured subtitle data packet is generated and sent to the head-mounted display device.

7. The real-time conference translation display method according to claim 6, characterized in that, The process of determining candidate error-correction segments based on the conference audio includes: The meeting audio was converted into a time-stamped initial translation and intermediate recognition results. Low-confidence features are extracted from the intermediate recognition results to perform candidate error detection and determine candidate error-correction segments in the initial translated text.

8. The real-time conference translation display method according to claim 6, characterized in that, The acquisition of professional knowledge items matching the candidate error-correcting segment includes: Parse the meeting information bound to the currently processed meeting audio stream; the meeting information includes at least one of the following: meeting number, session number, agenda identifier, and speaker identifier; Using the meeting information as a filter, and calculating text similarity within the corresponding range, the corresponding professional knowledge items are retrieved from the meeting knowledge base.

9. The real-time conference translation display method according to claim 6, characterized in that, The generation of structured subtitle data packets based on the structured error correction results includes: The structured error correction results are verified, and an incremental version number-updated structured subtitle data package is generated after the verification is passed. The verification includes at least one of evidence verification and semantic offset prevention verification.

10. The real-time conference translation display method according to any one of claims 6 to 9, characterized in that, When distributing structured subtitle data packets, differentiated distribution across multiple terminals is performed, specifically including: Obtain the language subscription preferences and information display hierarchy preferences reported by each currently connected head-mounted display device; Within the same time window, structured subtitle data packets of different target languages ​​and / or different information densities are pushed to the corresponding head-mounted display devices.

11. A head-mounted display device, characterized in that, The head-mounted display device, using the real-time conference translation display method as described in any one of claims 1 to 5, comprises: The communication module is used to establish a connection with the cloud server, receive structured caption data packets, and report local configuration commands. Near-eye display module, used to provide the wearer with a visual information display area within the field of view; The local processing module, including a processor and a memory, is used to maintain a subtitle status table, perform subtitle version comparison and filtering, calculate dirty areas, and drive the near-eye display module to perform local anti-jump refresh rendering.

12. A real-time conference translation display system, characterized in that, Includes a cloud server and at least one head-mounted display device; The cloud server device is used to execute the real-time conference translation and display method as described in any one of claims 1 to 5, and the head-mounted display device is used to execute the real-time conference translation and display method as described in any one of claims 6 to 10.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the real-time conference translation display method as described in any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the real-time conference translation display method as described in any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the real-time conference translation display method as described in any one of claims 1 to 10.

Images