Intelligent projector real-time translation interaction method and intelligent terminal thereof

Abstract

The application provides a kind of intelligent projector real-time translation interaction method and its intelligent terminal, it is related to intelligent projection technical field.The method includes the following steps: by synchronous acquisition acoustic signal and the display content with interactive behavior containing visual annotation;Analysis of visual information, extract text data and interactive space-time data, correlate and match target text element;Initial identification text is obtained by identifying acoustic signal, calculate the semantic similarity of its and target text, and according to this, semantic calibration is carried out to obtain calibrated text;Finally, the calibrated text is translated into the target language, and the visual annotation is layered and superimposed to project and display.The method of the application aims to solve the technical problems in the prior art, such as low translation accuracy, interactive lag and student cognitive confusion in complex teaching scenarios due to insufficient matching of acoustic recognition and visual information, and can achieve dynamic semantic calibration of acoustic recognition results based on visual interactive information, significantly improving the translation accuracy of professional terms and complex context.

Description

Technical Field

[0001] This invention relates to the field of intelligent projection technology, and more specifically, to a real-time translation interaction method for an intelligent projector and its intelligent terminal. Background Technology

[0002] With the increasing prevalence of international education, highly specialized technical courses offered by leading academic institutions place extremely high demands on teaching aids. As a core device for classroom interaction, the accuracy of translation and the real-time nature of interaction of smart projectors directly impact the quality of cross-cultural knowledge transfer. However, existing technologies still have significant limitations in handling complex teaching scenarios.

[0003] First, specialized fields contain a large number of industry-specific terms, abbreviations, and cutting-edge technology names, which are often not included in general language models. Instructors frequently use non-standardized spoken language or a mix of languages ​​during lectures, making it difficult for speech recognition systems to accurately map semantics and easily leading to translation bias. Second, classroom teaching encourages high levels of interaction, with multiple students potentially asking questions or commenting in different languages ​​simultaneously. This concurrent input from multiple languages ​​and sound sources poses a significant challenge to sound source localization, speech separation, and language recognition. Furthermore, some teaching environments are limited by architectural acoustics, resulting in severe reverberation and interference, which contaminates the original speech signal during the acquisition stage, further reducing the robustness of speech recognition.

[0004] In the aforementioned complex environments, existing smart projectors often fail to effectively combine visual interactive information with acoustic signals for collaborative processing, resulting in delayed translation results and a high error rate. When erroneous translated subtitles are projected simultaneously with visual annotations, it not only fails to aid student comprehension but also causes cognitive confusion due to semantic misinterpretation. The frequent need for error correction and clarification severely disrupts the teaching pace and hinders the improvement of classroom interaction efficiency.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time translation interaction method for a smart projector and its smart terminal, aiming to solve the technical problems in the prior art where insufficient matching of acoustic recognition and visual information leads to low translation accuracy, delayed interaction, and cognitive confusion among students in complex teaching scenarios. This invention enables dynamic semantic calibration of acoustic recognition results based on visual interaction information, significantly improving the translation accuracy of professional terms and complex contexts.

[0007] In a first aspect, the present invention provides a real-time translation interaction method for a smart projector, comprising the following steps: S1. Synchronously acquire acoustic signals within the target area, as well as visual information including display content and interactive behavior; the display content includes visual annotation information generated based on the interactive behavior; S2. By parsing the visual information, extract the text data in the displayed content and the spatiotemporal data of the interactive behavior, and determine the target text element that matches the interactive behavior based on the correlation between the text data and the spatiotemporal data; S3. Generate initial recognition text by recognizing and processing the acoustic signal; S4. Calculate the semantic similarity between the initial identified text and the target text element, and based on the semantic similarity, use the target text element to perform semantic calibration on the initial identified text to obtain the calibrated text; S5. The calibration text is converted into translation information in the target language, and the translation information and the visual annotation information are presented on the projection interface through layered overlay.

[0008] The real-time translation interaction method for smart projectors provided by this invention has the advantages of achieving high-precision real-time semantic calibration and translation through deep fusion of visual interaction information and acoustic signals, effectively reducing translation deviations in complex teaching environments, and improving classroom interaction efficiency and information transmission quality.

[0009] In a second aspect, the present invention provides a smart terminal, comprising: The acquisition unit is used to simultaneously acquire acoustic signals within the target area, as well as visual information including display content and interactive behavior; the display content includes visual annotation information generated based on the interactive behavior; The parsing unit is used to parse the visual information, extract text data from the displayed content and spatiotemporal data of the interactive behavior, and determine the target text element that matches the interactive behavior based on the correlation between the text data and the spatiotemporal data. The recognition unit is used to generate initial recognition text by recognizing and processing the acoustic signal; The calibration unit is used to calculate the semantic similarity between the initial identified text and the target text element, and based on the semantic similarity, use the target text element to perform semantic calibration on the initial identified text to obtain the calibrated text; The translation unit is used to convert the calibration text into translation information in the target language, and to present the translation information and the visual annotation information on the projection interface through layered overlay.

[0010] As can be seen from the above, the real-time translation interaction method for smart projectors provided by this invention effectively solves the problems of difficulty in recognizing professional terms and interference from concurrent recognition of multiple sound sources in complex teaching environments by deeply correlating visual interaction behavior with acoustic signals and using extracted screen text data as a benchmark for semantic calibration. This method not only achieves real-time calibration and accurate presentation of translation results, but also greatly reduces the difficulty of understanding and cognitive load for students through layered and superimposed interactive information display, ensuring high-density cross-cultural knowledge transfer efficiency and significantly enhancing the auxiliary value of smart projectors in professional classroom teaching.

[0011] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0012] Figure 1 This is a flowchart of a real-time translation interaction method for a smart projector provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of a smart terminal provided in an embodiment of the present invention.

[0014] Label Explanation: 100. Acquisition Unit; 200. Analysis Unit; 300. Recognition Unit; 400. Calibration Unit; 500. Translation Unit; 600. Review Unit. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0016] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] In traditional smart projector voice translation interaction, for highly specialized technical courses, the system faces technical bottlenecks such as low speech recognition accuracy and insufficient real-time performance when handling domain-specific terminology, non-standardized spoken expressions, and concurrent multilingual input. Because general-purpose language models lack deep learning of cutting-edge academic vocabulary, and because the audio signals picked up by microphone arrays suffer from severe signal attenuation and aliasing in complex acoustic environments (such as high reverberation and multi-source interference), the speech recognition engine struggles to accurately map the original semantics, leading to semantic bias and information loss in the translation results. This severely restricts the efficiency of information transmission in cross-cultural teaching environments.

[0018] For example, in advanced materials science teaching scenarios, instructors frequently use industry-specific terminology, such as "grain boundary segregation" or "nanotwin strengthening," interspersed with academic expressions in specific languages. When multiple international students simultaneously ask questions in different native languages, the microphone array of the smart projector must simultaneously process overlapping speech signals from multiple directions in a physical space with severe reverberation. In this situation, the system often fails to separate clear human voices from the mixed sound waves, resulting in large sections of garbled text or semantic deviations in the initial text output by the speech recognition engine. This causes a significant disconnect between the translated subtitles projected on the screen and the instructor's explanation, leading to misunderstandings of key experimental procedures or theoretical concepts among students.

[0019] If the above problems are not solved, the voice translation interaction system will be unable to meet the real-time and accuracy requirements of professional teaching, resulting in frequent interruptions in the classroom teaching process, limited students' understanding of complex academic content, and ultimately causing the teaching aid to lose its due knowledge transfer function and hindering the achievement of teaching objectives.

[0020] For reference, see the appendix. Figure 1 This invention provides a real-time translation interaction method for a smart projector, comprising the following steps: S1. Synchronously acquire acoustic signals within the target area, as well as visual information including display content and interactive behavior; the display content includes visual annotation information generated based on interactive behavior; S2. By analyzing visual information, extract text data and spatiotemporal data of interactive behavior from the displayed content, and determine the target text element that matches the interactive behavior based on the correlation between text data and spatiotemporal data; S3. Generate initial recognition text by recognizing and processing the acoustic signals; S4. Calculate the semantic similarity between the initial recognized text and the target text elements, and based on the semantic similarity, use the target text elements to perform semantic calibration on the initial recognized text to obtain the calibrated text; S5. Convert the calibration text into translation information in the target language, and present the translation information and visual annotation information on the projection interface through layered overlay.

[0021] For ease of understanding, the following explains some key terms in this embodiment: Smart projector: A device that integrates projection, voice recognition, image recognition, translation and interaction functions. It can capture and process acoustic signals and visual information in the environment in real time, and present the processing results on the projection interface in a layered overlay manner.

[0022] Acoustic signals: These refer to sound signals collected by a microphone array within a target area, including human voices and ambient noise. After preprocessing, these signals are used for speech recognition and sound source localization.

[0023] Visual information: refers to image and video data captured by a high-resolution RGB camera, including courseware content displayed on the projection interface and user (such as teacher or student) interaction behaviors (such as gestures and body postures).

[0024] Displayed content: refers to teaching or meeting materials such as images, text, charts, and formulas displayed on the screen by the projector.

[0025] Interactive behavior: refers to the interaction between users and the projection interface or system through physical actions (such as gestures, pointing, drawing) or voice commands.

[0026] Visual annotation information: refers to visual marks generated in real time on the projection interface based on user interaction behavior, used to assist in explanation or emphasize content, such as highlight halos, circles, etc.

[0027] Text data: refers to text information extracted from displayed content through optical character recognition (OCR) technology.

[0028] Spatiotemporal data refers to attribute data that describes interactive behavior in time and space, such as the timestamp of the gesture, the coordinates of the gesture on the screen, and the type of gesture.

[0029] Target text element: refers to the screen text content that is highly matched with the user's interaction behavior in terms of semantics and time and space. It is usually the core professional terms or concepts that the user is currently focusing on or explaining.

[0030] Initial recognized text: refers to the preliminary text result generated after the acoustic signal is processed by speech recognition, which may contain recognition errors or inaccurate technical terms.

[0031] Semantic similarity: refers to the measure of how close two text fragments (such as the initial identification text and the target text element) are in meaning. It is usually calculated by methods such as word vector embedding and cosine similarity.

[0032] Calibrated text: refers to the more accurate text that is more in line with the professional context obtained by semantic calibration of the target text elements based on the initial identified text.

[0033] Translation information: refers to the text content after the calibration text has been converted into the target language.

[0034] Projection interface: refers to the physical screen or wall onto which a smart projector projects processed information (including display content, translation information, and visual annotation information).

[0035] This embodiment provides a real-time translation interaction method for smart projectors, aiming to solve the problems of low accuracy and poor real-time performance of speech translation in specialized teaching scenarios due to difficulties in recognizing professional terms, interference from multiple languages, and complex acoustic environments, which in turn affect the quality of classroom teaching.

[0036] In step S1, the smart projector synchronously acquires acoustic signals within the target area, as well as visual information including displayed content and interactive behavior. The displayed content includes visual annotations generated based on interactive behavior. Specifically, the smart projector captures sound in the classroom through a microphone array and captures images of the projection screen area using a built-in high-resolution RGB camera. This visual information includes not only the teaching materials but also hand gestures from teachers or students within the screen area. For example, when a teacher gestures towards a technical term on the screen during a lecture, the camera captures this gesture. Simultaneously, the system generates a temporary, non-permanent highlight or indicator cursor on the projection screen based on this gesture, precisely following the teacher's gesture trajectory to visually emphasize the speaker's current focus. This highlight is typically a semi-transparent colored halo, achieved by adding a separate overlay layer in the projector's graphics rendering pipeline, with adjustable transparency. If the recognized gesture is a circling or marking action, the system automatically generates a separate, layerable annotation layer for such illustrative gestures. This annotation layer is a separate graphics buffer, independent of the core teaching materials, allowing speakers to freely circle, mark, or write without modifying the original materials. The annotation's color, thickness, and other attributes can be adjusted using preset gestures or voice commands.

[0037] In step S2, visual information is analyzed to extract text data and spatiotemporal data of interactive behavior from the displayed content. Based on the correlation between text data and spatiotemporal data, target text elements matching the interactive behavior are determined. Specifically, the image processing unit analyzes the visual information captured by the camera in real time, uses optical character recognition (OCR) technology to extract the text content displayed on the screen, and uses image recognition algorithms to recognize images, charts, and formulas. For gestures, the system uses a skeleton tracking algorithm to identify key points of the hand and then determines the type of gesture, such as pointing, circling, or simulated clicking. When the system detects that a speaker is giving a speech and their gesture is temporally synchronized and spatially correlated with specific content (such as technical terms, formulas, or diagrams) on the projection screen, the system will activate an intelligent context judgment mechanism. This mechanism determines the correlation by calculating the difference between the timestamp of the speech recognition result and the timestamp of the gesture, as well as the distance between the coordinates of the gesture on the screen and the coordinates of screen content elements (such as text boxes or image areas). If the time difference is less than a preset threshold (e.g., 0.5 seconds) and the spatial distance is less than a preset threshold (e.g., 50 pixels), a strong correlation is considered to exist. At this point, the gesture will be interpreted as a “supporting instruction” or “instructional annotation” rather than a regular screen operation command.

[0038] In step S3, initial recognition text is generated by processing the acoustic signal. Specifically, the audio processing unit performs preliminary acoustic processing on the acquired speech signal, including using an adaptive noise cancellation algorithm to remove background noise, employing acoustic echo cancellation technology to purify the speech, and using the characteristics of the microphone array to estimate the approximate location of the current speaker through sound source localization algorithms such as generalized cross-correlation phase transform, thus initially identifying the language used by the speaker. Subsequently, the speech recognition module recognizes the processed acoustic signal to generate initial recognition text.

[0039] In step S4, the semantic similarity between the initial recognized text and the target text elements is calculated. Based on the semantic similarity, the target text elements are used to semantically calibrate the initial recognized text to obtain calibrated text. The system performs deep semantic association between the real-time recognized speech content (including highly specialized terminology and teachers' unique non-standard expressions) and the synchronously generated gesture annotations. This association is achieved through a multimodal fusion module, which receives text output from the speech recognition module, gesture coordinates and type from the visual recognition module, and courseware text and image information provided by the screen content recognition module. When the speech contains specialized terminology or non-standard expressions that are difficult to accurately recognize or translate, the system will prioritize referencing the screen content pointed to or circled by the synchronously generated gesture annotations. For example, if the speech recognition result vaguely identifies "CRISPR," but the teacher's gesture clearly points to the illustration or text of "CRISPR-Cas9 gene editing technology" on the screen, the system will perform contextual inference based on the current teaching theme of bioinformatics engineering. The inference process calculates the semantic similarity between the speech recognition candidate words and the screen text pointed to by the gesture (e.g., using word vector embedding techniques to calculate cosine similarity), and combines the weights of the current course topic vocabulary to calibrate the speech recognition results, correcting them to "CRISPR-Cas9 gene editing".

[0040] In step S5, the calibration text is converted into translation information in the target language, and the translation information and visual annotation information are presented on the projection interface in a layered overlay. The calibrated speech recognition results are then sent to the translation system. The translation system uses this visually enhanced context to select the translation that best fits the professional context. For example, for the materials science term "grain boundary segregation," if the gesture points to a microstructure diagram, the translation system will prioritize the precise translation specific to the materials science field, rather than the more general "boundary separation," thus ensuring the professionalism and accuracy of the translation. The semantically calibrated and translated multilingual subtitles are displayed on the projection screen in a layered manner along with the real-time gesture annotations. The subtitles are usually displayed at the bottom of the screen, while the annotation layer is overlaid on top of the courseware content. The transparency of the annotation layer can be adjusted as needed, for example, by using gestures or voice commands to control its opacity, ensuring that the annotations do not completely obscure the core teaching content while remaining clearly visible. These temporary annotations can be cleared by students or teachers with a single click using preset simple gestures or voice commands to keep the screen clean. When a user explicitly issues the command to "save annotations," the system binds all graphic data of the current annotation layer, the corresponding audio translation, and the timestamp at the time of generation. This information is then associated with the current presentation page as an independent "annotation version" and saved to the projector's internal memory.

[0041] The following example will provide a more detailed explanation of the above technical solution: Imagine an international bioinformatics engineering class where an instructor is explaining the complex mechanisms of CRISPR-Cas9 gene editing technology in English. The highly specialized nature of this technology, coupled with the instructor's potential use of colloquial and non-standardized expressions, or the inclusion of German terminology in the English explanation, presents challenges to traditional speech recognition and translation systems. Furthermore, the classroom may have echoes, and students from different countries may simultaneously ask questions in their native languages, creating a complex acoustic environment.

[0042] In this scenario, the real-time translation interaction method for smart projectors proposed in this application will operate according to the following process: First, in step S1, the microphone array of the smart projector synchronously collects acoustic signals in the classroom, including the lecturer's voice, students' questions, and ambient noise. Simultaneously, the high-resolution RGB camera captures visual information on the projection screen, including displayed bioinformatics courseware (e.g., illustrations and text containing "CRISPR-Cas9 gene editing technology") and the lecturer's gestures on the screen. For example, when explaining "CRISPR-Cas9," the lecturer might point to the corresponding molecular structure diagram on the screen. At this time, the smart projector will generate a bright halo on the projection interface in real time, precisely following the lecturer's gesture trajectory, as visual annotation information.

[0043] Next, in step S2, the image processing unit of the smart projector analyzes the captured visual information. It uses Optical Character Recognition (OCR) technology to extract text data such as "CRISPR-Cas9 gene editing technology," "Cas9 protein," and "sgRNA" from the courseware. Simultaneously, it uses a skeleton tracking algorithm to recognize the lecturer's gestures and records their spatiotemporal data on the screen, such as the timestamp of the gesture and its coordinates on the screen. The system then determines whether there is a strong correlation between the lecturer's gestures and the text data on the screen. For example, if the lecturer's gesture, while explaining "CRISPR-Cas9," precisely points to the text area of ​​"CRISPR-Cas9 gene editing technology" on the screen, and is highly synchronized in time, the system will determine that "CRISPR-Cas9 gene editing technology" is the target text element matching this interaction.

[0044] Subsequently, in step S3, the audio processing unit of the smart projector performs recognition processing on the acquired acoustic signals. It first performs noise cancellation and echo cancellation, and then converts the lecturer's speech into initial recognition text. Due to the complexity of technical terms and the lecturer's colloquial expressions, the initial recognition text may contain errors, such as recognizing "CRISPR-Cas9" as "Crisper Cass 9" or "CRISPR".

[0045] In step S4, the system calculates the semantic similarity between the initial identified text (e.g., "Crisper Cass 9") and the target text element ("CRISPR-Cas9 gene editing technology"). Since "Crisper Cass 9" and "CRISPR-Cas9 gene editing technology" have a semantic connection, the system uses this similarity and the target text element to perform semantic calibration on the initial identified text. In conjunction with the current teaching theme of bioinformatics engineering, the system calibrates the initial identified text to the more accurate "CRISPR-Cas9 gene editing technology," resulting in the calibrated text.

[0046] Finally, in step S5, the smart projector converts the calibration text "CRISPR-Cas9 gene editing technology" into translation information in the target language (e.g., translated into Chinese as "CRISPR-Cas9 gene editing technology"). Simultaneously, this translation information, along with the visual annotation information generated in step S1 (i.e., the highlighted halo following the lecturer's gestures), is presented on the projection interface in a layered overlay manner. When students watch the projection, they not only see the lecturer's presentation slides but also real-time accurate Chinese translation subtitles and visual annotations indicated by the lecturer's gestures, thus clearly understanding the professional concepts explained by the lecturer.

[0047] As can be seen from the above examples, the real-time translation interaction method for smart projectors in this application effectively solves many challenges faced by traditional translation systems in specialized teaching scenarios by deeply integrating visual and acoustic information. Traditional smart projectors often rely solely on speech recognition results when processing technical terms. When speech recognition is flawed, the translation will also be incorrect, leading to difficulties for students in understanding. For example, if "CRISPR-Cas9" is incorrectly identified as "Crisper Cass 9" based solely on speech recognition and directly translated, students will not be able to accurately understand its meaning.

[0048] However, this application, by introducing the analysis of visual information, particularly the spatiotemporal data analysis of user interaction behaviors (such as gestures), can accurately pinpoint the screen content currently being focused on by the user and use it as an anchor point for semantic calibration. In the example above, even if the initial speech recognition result is biased, because the lecturer's gesture clearly points to the target text element "CRISPR-Cas9 gene editing technology" on the screen, the system can utilize this visual context information to perform precise semantic calibration on the speech recognition result. This multimodal information fusion calibration mechanism significantly improves the accuracy of professional terminology recognition and the professionalism of translation.

[0049] Furthermore, this application presents the calibrated translation information and visual annotations layered and overlaid on the projection interface, making the translation results intuitively aligned with the teaching content. This not only reduces students' cognitive burden but also enhances the clarity of cross-language communication through visual assistance. Compared to the traditional method of simply displaying translated subtitles, this application, by displaying visual annotations synchronized with the explanation in real time, can more effectively guide students' attention and help them associate the translated content with specific illustrations or text on the screen, thereby gaining a deeper understanding of complex professional concepts. This innovative interactive method effectively improves the quality of classroom teaching and students' learning outcomes, providing strong technical support for international professional education.

[0050] In some embodiments, step S2, which involves determining the target text element matching the interactive behavior based on the correlation between text data and spatiotemporal data, includes the following specific steps: S21. Based on the spatiotemporal data, determine the local focus area of ​​the interactive behavior on the displayed content, and extract all text elements and their respective screen coordinates within the local focus area; S22. Evaluate the geometric distance between each text element within the local focal region and the key point coordinates of the interaction behavior in the spatiotemporal data to obtain the first correlation degree; specifically including step S221: S221. Calculate the Euclidean distance between the keypoint coordinates and the screen coordinates, and obtain the first correlation degree by normalizing the Euclidean distance; S23. Obtain the preset topic vocabulary corresponding to the currently displayed content, and calculate the semantic relevance between each text element in the local focus area and the preset topic vocabulary to obtain the second relevance; S24. Calculate the association score based on the first association degree and the second association degree, and determine the text element with the highest association score as the target text element that matches the interaction behavior.

[0051] Spatiotemporal data refers to specific information obtained through the analysis of visual information, describing the temporal and spatial aspects of interactive behavior. This can include the timestamp of the interactive behavior, its two-dimensional or three-dimensional coordinates on the projected interface, the type of the interactive behavior (e.g., pointing, drawing, handwriting), and the trajectory information. For example, a skeleton tracking algorithm can be used to identify key hand points, thereby determining the type of gesture and its real-time coordinates on the screen. A local focus area refers to a defined region in the displayed content that is spatially closely related to the interactive behavior. Determining this area aims to narrow down the range of subsequent text element matching, improving processing efficiency and accuracy. It can be dynamically generated based on the type and scope of the interactive behavior. For example, if the interactive behavior is pointing, the local focus area could be a circular area with a preset radius around the fingertip; if the interactive behavior is drawing, the local focus area could be the smallest bounding rectangle formed by the drawing trajectory. Text elements and their respective screen coordinates refer to the text information extracted from the displayed content using Optical Character Recognition (OCR) technology, such as technical terms, formulas, and titles in courseware. Screen coordinates refer to the positional information of these text elements on the projected interface, typically represented by the coordinates of the top-left and bottom-right corners of the text box or the center point coordinates. Keypoint coordinates refer to specific points in an interactive action that represent the core location of the interactive intent. For example, for a pointing gesture, the keypoint coordinates could be the coordinates of the fingertip; for a drawing gesture, the keypoint coordinates could be the geometric center point or starting point of the drawing trajectory. These coordinates are obtained by analyzing the gestures using image recognition algorithms. Geometric distance is an indicator used to quantify the spatial proximity of text elements and interactive actions. It is obtained by calculating the spatial distance between the screen coordinates of the text element and the keypoint coordinates of the interactive action. For example, Euclidean distance can be used to measure the straight-line distance between two points, or Manhattan distance can be used to measure the distance along the coordinate axes. First affinity is a quantitative indicator reflecting the spatial closeness of text elements and interactive actions, based on geometric distance assessment. It is obtained by normalizing the geometric distance, allowing for a fair comparison of the spatial affinity between text elements of different sizes or positions. For example, the reciprocal of the Euclidean distance or its value after exponential decay can be used as the first degree of relevance; the closer the distance, the higher the relevance. Preset topic vocabulary refers to a set of professional terms highly relevant to the currently displayed content or course chapter. These terms can be pre-defined by the teacher, automatically extracted from the course syllabus, or obtained through text mining of historical teaching content. For example, in a materials science course, preset topic vocabulary might include "grain boundary segregation," "stacking fault energy," and "high-entropy alloy." Semantic relevance is an indicator used to quantify the degree of semantic association between text elements and preset topic vocabulary.The first correlation score can be calculated using natural language processing techniques, such as using word embeddings (e.g., Word2Vec, GloVe, or BERT embeddings) to calculate the cosine similarity between text elements and topic vocabulary, or performing semantic matching based on ontology or knowledge graphs. The second correlation score is a quantitative indicator reflecting the degree of semantic matching between text elements and preset topic vocabulary, obtained based on semantic relevance assessment. The higher the semantic relevance, the greater the second correlation score, indicating that the text element is more consistent with the current teaching topic context. The correlation score is a comprehensive indicator calculated by combining the first and second correlation scores, used to comprehensively evaluate the degree of matching between text elements and interactive behaviors. It can be obtained by weighted summation or product of the first and second correlation scores, for example, correlation score = w1 * first correlation score + w2 * second correlation score, where w1 and w2 are preset weights. The target text element refers to the text that, after multi-dimensional evaluation, is determined to be the most matching text for the interactive behavior among all candidate text elements. It is a key basis for subsequent semantic calibration, ensuring that the system can accurately understand the speaker's intent.

[0052] This application's solution achieves precise positioning of target text elements by introducing a multi-dimensional correlation evaluation mechanism. In the real-time translation interaction method for smart projectors, the system simultaneously collects acoustic signals and visual information containing displayed content and interactive behavior within the target area, and analyzes the visual information to extract text data from the displayed content and spatiotemporal data of the interactive behavior. To address the problem that in highly specialized and visually information-intensive teaching scenarios, simple correlation matching alone is insufficient to accurately pinpoint the core terms being explained by the teacher, and that the target text element remains unclear when the spatiotemporal data of the interactive behavior is correlated with multiple text data in the displayed content, this application further proposes the following working principle. Based on the analyzed spatiotemporal data, the system first determines the local focal area of ​​the interactive behavior on the displayed content. This step effectively filters out irrelevant text on the screen that is far from the interactive behavior, thereby significantly narrowing the scope of subsequent matching and improving processing efficiency. Within this local focal area, the system further extracts all text elements and their respective screen coordinates. Subsequently, to quantify the spatial closeness between interactive behavior and text, the system evaluates the geometric distance between each text element within the local focal area and the key point coordinates of the interactive behavior in the spatiotemporal data. By normalizing the Euclidean distance, a first degree of relevance is obtained. This ensures the relevance of the matching results in physical space, creating a strong spatial association between the interactive action and the content on the screen. Simultaneously, to further filter out professional terms that fit the current teaching context from a semantic perspective, the system obtains preset topic vocabulary corresponding to the currently displayed content and calculates the semantic relevance between each text element within the local focal area and the preset topic vocabulary, obtaining a second degree of relevance. The introduction of this semantic dimension allows the system to consider not only spatial proximity but also content relevance, effectively avoiding misjudgments caused by spatial proximity but semantic discrepancies. Finally, the system calculates a relevance score based on the first and second degrees of relevance and identifies the text element with the highest relevance score as the target text element matching the interactive behavior. This dual-dimensional weighted evaluation method balances spatial location and semantic logic, deeply binding spatial interactive actions with the core terminology in the teaching content. Through this mechanism, the proposed solution effectively addresses the positioning bias problem that easily arises in a single matching mode, ensuring that the subsequent semantic calibration process is based on the most accurate terminology. This precise determination of target text elements, closely integrated with the semantic calibration step (S4) in the basic method, enables the initial identified text to be calibrated more accurately, thereby significantly improving the professionalism and accuracy of the translation, and ultimately enhancing the overall performance and user experience of the entire real-time translation interaction method for smart projectors.

[0053] The following is a concrete example to illustrate this. Suppose that in a materials science course, the lecturer is explaining the "mechanism of plastic deformation" and displays a slide on the projection screen containing technical terms such as "dislocation slip" and "defect structure." When the lecturer mentions a concept in a vague voice (e.g., "DS") while gesturing towards a dense area on the screen containing these terms, the smart projector performs the following steps to determine the target text element. First, the system determines the local focal region of the lecturer's gesture on the projection screen based on the spatiotemporal data of the gesture, such as the real-time coordinates and trajectory of the gesture. For example, if the lecturer points to a point on the screen, the system will define a circular area with a radius of 5 centimeters centered on that point as the local focal region. Next, within this local focal region, the system uses optical character recognition (OCR) technology to extract all text elements and their respective screen coordinates. For example, it identifies the text elements "dislocation slip" and "defect structure" and records their precise positions on the screen. Second, the system evaluates the geometric distance between each text element within the local focal region and the key coordinates of the lecturer's gesture. For example, the system calculates the Euclidean distance between the coordinates of the instructor's fingertip and the center point of the "Dislocation Slip" text box, as well as the Euclidean distance between the fingertip and the center point of the "Defect Structure" text box. Assuming "Dislocation Slip" is closer to the fingertip, its Euclidean distance is smaller. The system then normalizes these Euclidean distances to obtain the first degree of relevance. The closer the distance, the higher the first degree of relevance. Simultaneously, the system retrieves the preset topic vocabulary corresponding to the currently displayed content. Since the current course chapter is "Plastic Deformation Mechanism," the system retrieves relevant professional terms from the preset topic vocabulary and assigns them corresponding weights. For example, "Dislocation Slip" has a high semantic relevance to "Plastic Deformation Mechanism," while "Defect Structure" has a relatively low semantic relevance. The system calculates the semantic relevance of each text element ("Dislocation Slip" and "Defect Structure") within the local focus area to the preset topic vocabulary "Plastic Deformation Mechanism," obtaining the second degree of relevance. Finally, the system calculates a relevance score for "Dislocation Slip" and "Defect Structure" based on the calculated first and second degrees of relevance. For example, the association score can be derived from the weighted sum of the first and second association scores. Since "dislocation slip" is spatially closer to the instructor's gesture and semantically more relevant to the topic of "plastic deformation mechanism," its association score will be higher than "defect structure." The system identifies "dislocation slip," with the highest association score, as the target text element matching the instructor's interaction. Thus, even if the instructor's speech recognition results vaguely identify "DS" as candidate words like "dislocation slip" and "defect structure," the system can accurately calibrate it as "dislocation slip" by combining the spatial information of the gesture and the semantic context of the course.

[0054] Through the aforementioned technical solution, this application effectively addresses the problem that traditional methods struggle to accurately pinpoint the core terminology being explained by the teacher in highly specialized and visually information-intensive teaching scenarios. By introducing the delineation of local focal areas, the matching range of text elements is effectively narrowed, improving processing efficiency. The first degree of relevance, calculated based on geometric distance, ensures a close spatial correspondence between interactive behavior and screen content. Simultaneously, the second degree of relevance, obtained by introducing preset topic vocabulary and calculating semantic relevance, further enhances the accuracy of matching from a semantic perspective, avoiding misjudgments based solely on spatial proximity. This multi-dimensional evaluation mechanism combining spatial and semantic approaches enables the system to more accurately associate the speaker's interactive actions with specific professional terms in the displayed content, thereby precisely identifying the target text element. This accurate identification of the target text element provides a solid foundation for subsequent semantic calibration, significantly improving the calibration accuracy of the initial identified text, thus ensuring the professionalism and accuracy of the translated information. Ultimately, this effectively enhances the overall effect of real-time translation interaction on smart projectors, optimizing students' learning experience and the quality of classroom interaction.

[0055] In some embodiments, the specific steps in step S3 include: S31. Collect the speaker's biometric information and the location information of the sound source; S32. Match biometric information with a preset user list and combine it with sound source location information to determine the speaker's identity; the speaker's identity includes key users and / or non-key users; S33. Based on the speaker's identity and the type of interaction, prioritize the acoustic signals and assign corresponding audio processing channels; wherein, acoustic signals marked as key objects are given the first processing priority and routed to the low-latency audio processing channel, and acoustic signals marked as non-key objects are given the second processing priority and routed to the parsing channel; the first processing priority is higher than the second processing priority.

[0056] This solution effectively addresses the chaotic speech processing issues in complex acoustic environments by introducing speaker identification and priority scheduling mechanisms, achieving refined management of acoustic signals. Biometric information refers to data that uniquely identifies an individual's physiological or behavioral characteristics, such as facial features, voiceprints, iris features, or fingerprints. This information can be acquired through sensor devices, such as capturing facial images with a camera to extract facial features, or collecting speech signals through a microphone array to extract voiceprint features. Sound source location information refers to the speaker's spatial location data relative to the smart projector, such as the speaker's azimuth, distance, or three-dimensional coordinates. This information can be obtained through acoustic localization techniques, such as using the phase difference or time difference of arrival of the microphone array for sound source localization, or combining visual information for multimodal localization. The preset user list is a database storing known user identity information (such as name, role, and biometric templates). The matching process involves comparing the collected biometric information with the templates in this list to identify the specific speaker. Determining a speaker's identity by combining sound source location information refers to further confirming their identity based on biometric matching and the speaker's spatial location information. This is especially important when multiple people are present simultaneously or when biometric information is not entirely clear; sound source location can serve as an auxiliary basis for judgment. The classification of speakers (key speakers and / or non-key speakers) is based on preset rules or scenario requirements. For example, instructors are typically defined as key speakers, while students may be defined as non-key speakers. This classification aims to differentiate the processing of voice information from different roles. The type of interactive behavior refers to the speaker's intentions or actions in a specific context, such as lecturing, asking questions, discussing, or annotating. These behaviors can be identified by analyzing the speaker's head posture, gestures, or voice content. Prioritization refers to assigning different processing orders or resources to different acoustic signals based on the importance of the speaker's identity and the type of interactive behavior. For example, the content of a key speaker's lecture can be given a higher processing priority. Allocating corresponding audio processing channels refers to assigning dedicated hardware or software processing paths to acoustic signals of different priorities. These channels can have different processing capabilities, latency characteristics, or algorithm configurations to meet the processing needs of different signals. The primary processing priority and low-latency audio processing channel are designed to ensure that critical information (such as explanations of key subjects) is processed and transmitted quickly and accurately, minimizing processing latency to guarantee the smoothness of real-time translation. The secondary processing priority and parsing channel is used to process secondary information (such as questions or discussions from non-key subjects). These signals may allow for some processing latency, but still need to be identified and parsed to ensure information integrity. The primary processing priority is higher than the secondary processing priority, clarifying the system's strategy for resource allocation and processing order, ensuring the priority processing of core teaching content.

[0057] This application's solution, by collecting the speaker's biometric information and sound source location information, enables the system to establish a mapping relationship between the speaker's identity and spatial location. This multi-dimensional information collection method provides a reliable data foundation for accurately distinguishing different speakers. Matching biometric information with a preset user list and combining it with sound source location can accurately pinpoint the current speaker's identity attributes, classifying them as key or non-key individuals. This classification mechanism allows the system to treat the voice information of different roles differently according to the actual needs of the teaching scenario. Prioritizing acoustic signals based on the speaker's identity and interaction type, and allocating corresponding audio processing channels, is the core of achieving efficient processing. Assigning signals from key individuals the first processing priority and routing them to low-latency channels ensures the real-time nature and accuracy of core teaching content; assigning signals from non-key individuals the second processing priority and routing them to parsing channels optimizes the allocation of system resources while ensuring information integrity, avoiding interference from non-critical information on the core translation task. Through the aforementioned mechanism, the proposed solution effectively filters and distinguishes acoustic signals from different sources, ensuring that in complex environments with multiple languages ​​and multiple sound sources, the voice signals of key speakers can be prioritized and clearly identified. This provides high-quality input for subsequent initial recognition text generation, semantic calibration, and translation information generation. This not only improves the accuracy of speech recognition but also significantly enhances the response efficiency and user experience of the entire real-time translation interaction system, enabling smart projectors to more effectively assist in cross-cultural communication and knowledge transfer.

[0058] In some embodiments, the specific steps in step S4 include: S41. Obtain the preset topic vocabulary corresponding to the currently displayed content; the preset topic vocabulary includes multiple standard professional terms and their corresponding first preset weights; S42. Calculate the semantic similarity between multiple candidate words corresponding to the initial recognition text and each standard technical term in the target text element; S43. Calculate the comprehensive score of each candidate word based on semantic similarity and the first preset weight, and select the candidate word with the highest comprehensive score as the content in the calibration text.

[0059] The first step, acquiring preset topic vocabulary corresponding to the currently displayed content, aims to provide a domain-specific professional vocabulary library for subsequent semantic calibration. Preset topic vocabulary can be pre-constructed through text mining and keyword extraction of corpora from textbooks, papers, and industry standards in specific disciplines, and stored in the projector's local database or cloud knowledge base; alternatively, it can be dynamically generated based on the content of the current teaching materials using natural language processing technologies (e.g., topic models, keyword extraction algorithms). The standard professional terms included in the preset topic vocabulary refer to standardized terms widely accepted and used within a specific discipline, such as "grain boundary segregation" and "CRISPR-Cas9 gene editing." These terms can be obtained from authoritative academic dictionaries, professional textbooks, or expert-annotated data. The first preset weight is used to quantify the importance or relevance of each standard professional term in the current topic or context. This weight can be manually set by domain experts based on experience, or it can be automatically calculated by an algorithm based on factors such as the frequency of the term's appearance in the teaching materials, its position in titles or key paragraphs, or its semantic distance from the current chapter's topic words.

[0060] Calculating the semantic similarity between multiple candidate words corresponding to the initial recognized text and standard technical terms in the target text elements is to evaluate the semantic matching degree between the possible results of speech recognition and the technical terms in the visual focus content on the screen. The multiple candidate words corresponding to the initial recognized text are typically an N-best list generated by the speech recognition system after processing the acoustic signal, based on the scores of the acoustic and language models—representing multiple highly probable recognition results. The standard technical terms in the target text elements refer to technical vocabulary extracted from the local focus area on the projected interface that matches the user's interaction behavior, obtained through optical character recognition (OCR) or directly from the courseware metadata. Semantic similarity can be calculated using various methods. For example, cosine similarity can be calculated between two words or phrases based on word vector embeddings (such as pre-trained models like Word2Vec, GloVe, and BERT) to quantify their proximity in the semantic space; alternatively, knowledge graphs or ontology can be used to evaluate semantic distance by calculating the path length or the number of shared parent nodes of words in the semantic network.

[0061] The key step in determining the final calibration result is to calculate the comprehensive score of each candidate word based on semantic similarity and a first preset weight, and then select the candidate word with the highest comprehensive score as the content in the calibration text. The comprehensive score is typically calculated by weighting and fusing semantic similarity and the first preset weight. For example, a simple linear weighted summation can be used (Comprehensive Score = α * Semantic Similarity + β * First Preset Weight), where α and β are adjustable coefficients used to balance the influence of semantic matching and the importance of technical terms in the decision. More complex fusion models, such as machine learning-based methods, can also be used to optimize the calculation of the comprehensive score through training. Finally, the system selects the word with the highest comprehensive score from all candidate words as the calibrated text content, ensuring that the calibration result not only conforms to the initial judgment of speech recognition but also highly matches the technical terms in the current teaching context.

[0062] This application's solution refines the calibration process of the initial recognition text by introducing preset topic vocabulary and their corresponding first preset weights. Based on the simultaneous acquisition of acoustic signals and visual information containing displayed content and interactive behavior within the target area, and the parsing of this information to extract text and spatiotemporal data, thereby determining the target text elements matching the interactive behavior, this solution further utilizes subject-specific expertise. First, by acquiring the preset topic vocabulary corresponding to the currently displayed content, the calibration scope is limited to the subject-specific terminology database, effectively filtering out irrelevant words that may appear in the general language model and avoiding misidentification caused by interference from general vocabulary in a professional context. Second, by calculating the semantic similarity between multiple candidate words corresponding to the initial recognition text and each standard professional term in the target text element, the matching degree between the candidate words and the current teaching context can be quantified. Finally, a comprehensive score is calculated by combining the first preset weights. The unique aspect of this step is that it not only considers semantic proximity but also assigns different importance to different professional terms within a specific subject through a weighting mechanism, enabling the system to prioritize terms that are both semantically logical and possess subject-specific authority. This multi-dimensional calibration mechanism, which combines visual interactive information, domain-specific vocabulary, and their importance weights, enables smart projectors to significantly improve the accuracy and professionalism of the calibration text when faced with highly specialized terminology and non-standardized expressions used by teachers, ensuring that the translation results accurately reflect the teaching content.

[0063] In some embodiments, step S5 is followed by: S6A1. Combine calibration text, translation information, and visual annotation information with timestamps to generate structured data associated with the currently displayed content; S6A2. In review mode, in response to the user's backtracking trigger operation on the timeline, the acoustic signals, translation information and visual annotation information in the structured data of the corresponding time node are replayed on the projection interface.

[0064] The calibration text, translation information, and visual annotation information are the results obtained by the smart projector through in-depth processing of acoustic signals and visual information during real-time translation interaction. The calibration text is the speaker's speech recognition result after semantic calibration; the translation information is the target language translation of the calibration text; and the visual annotation information is visual markers generated in real-time on the projection interface that are related to the interactive behavior. These pieces of information together constitute the core content of classroom teaching and are crucial for student understanding and review. A timestamp is precise time information used to mark the moment data is generated or an event occurs. It can be an absolute time value, such as a Unix timestamp or an ISO 8601 format time string; or a relative time value, such as milliseconds relative to the start time of the class. The introduction of timestamps allows all relevant data to be indexed and retrieved in chronological order, providing a time reference for subsequent backtracking and replay. Structured data refers to data organized according to a predefined data model, which has a clear format and structure, facilitating computer storage, management, and retrieval. For example, this data can be stored in JSON (JavaScript Object Notation) format, XML (Extensible Markup Language) format, or relational databases (such as SQLite). This structured storage method ensures data integrity, consistency, and accessibility. Association with the currently displayed content means that the generated structured data does not exist in isolation, but is closely bound to the specific courseware page or content displayed by the projector during the teaching process. This association can be achieved by adding a courseware page identifier or content hash value to each piece of structured data, ensuring accurate location of the corresponding teaching context during review. Review mode is a dedicated operating state provided by smart projectors, designed to help users review and consolidate learning content. In this mode, the projector's display and audio output functions are adjusted to support the retrieval and reproduction of historical teaching content. User-triggered timeline rewind is a user-initiated interactive behavior used to navigate recorded teaching content over time. For example, users can trigger this action via the fast forward / rewind button on the remote control, a swipe on the touchscreen, a voice command (such as "go back five minutes ago"), or by clicking a specific point on the timeline in the graphical user interface (GUI). This operation allows users to flexibly choose any point in time for review. Replaying the acoustic signals, translation information, and visual annotations from the structured data at the corresponding time point means that when the user triggers the backtracking operation, the system retrieves the acoustic signals, translation information, and visual annotations within the stored structured data at that selected time point and its vicinity. Subsequently, the projector synchronously replays this multimodal information on its interface.The acoustic signal playback restores the speaker's original voice, the translated information is presented again as subtitles, and the visual annotations are overlaid on the courseware content in their original graphic form. This synchronous playback mechanism aims to reproduce the classroom scene as closely as possible, providing an immersive review experience.

[0065] This application's solution achieves digital retention and scenario-based reproduction of the classroom teaching process by introducing a structured data storage and backtracking mechanism. The method first combines real-time generated calibration text, translation information, and visual annotation information with precise timestamps to generate structured data associated with the currently displayed content. This structured data includes not only semantically calibrated speaker speech recognition results but also their target language translation, as well as visual markers generated in real-time on the projection interface related to the speaker's interactive behavior. By binding this multimodal information with timestamps and storing it in a structured manner, the originally discrete teaching information can be systematically organized to form a complete temporal data index. In review mode, when a user initiates a backtracking trigger operation on the timeline, the system can accurately retrieve the acoustic signals, translation information, and visual annotation information for the corresponding time period from the stored structured data according to the user-specified time node. Subsequently, the smart projector synchronously replays this multimodal information on the projection interface. This synchronous playback mechanism not only restores the original context of the teacher's explanation but also helps students intuitively map auditory, textual, and visual information to specific teaching content during review by overlaying visual annotations and translation information. Combined with the aforementioned real-time translation interaction method using a smart projector, this solution significantly improves teaching effectiveness. During real-time interaction, the smart projector can simultaneously acquire acoustic signals and visual information, and after parsing, recognizing, semantically calibrating, and translating, generate high-quality calibration text, translation information, and visual annotation information. These high-quality real-time processing results provide reliable input for generating structured data in this solution. For example, by parsing visual information, extracting spatiotemporal data of textual data and interactive behavior, and identifying target text elements, the accuracy and relevance of visual annotation information are ensured. By recognizing and processing acoustic signals, initial recognized text is generated, and further semantic calibration is performed to obtain highly accurate calibration text, which lays the foundation for subsequent translation information and acoustic signal playback. Ultimately, the translated information and visual annotations are layered and overlaid on the projection interface. This real-time presented information is the core content that this solution requires for structured storage and playback. Therefore, based on the real-time translation interaction method, this solution further addresses the problem of difficulty in reviewing classroom content due to its immediacy. By digitally preserving and recreating the multimodal interactive content in the classroom, it greatly improves the efficiency and depth of students' after-class review, enabling them to target specific gaps in their knowledge, effectively consolidate their understanding, and enhance learning outcomes.

[0066] In some embodiments, step S5 is followed by: S6B1. Monitor the explicit interactive actions of users in review mode, calculate the frequency of user interaction, and evaluate the information density score and information update frequency in the corresponding projected interface during the user's explicit interactive actions. S6B2. Calculate the cognitive load coefficient based on user interaction frequency, information density score, and information update frequency; the specific calculation formula is: cognitive load coefficient = (a * information density score) + (b * information update frequency) + (c * user interaction frequency), where a, b, and c are all second preset weights; S6B3. Compare the cognitive load coefficient with a preset dynamic threshold, and when the cognitive load coefficient is greater than the preset dynamic threshold, execute a preset information flow simplification step; the information flow simplification step includes: C1. Increase the acoustic signal volume of the key object and display the visual annotation information of the key object in a preset highlight color and a completely opaque state; C2. Reduce the acoustic signal volume of non-critical objects and increase the transparency of visual annotation information for non-critical objects; C3. Based on the area pointed to by the interaction behavior of the key object, generate a visual focus area on the projection interface, increase the display brightness of the visual focus area, and reduce the display brightness of other areas outside the visual focus area.

[0067] Specifically, in review mode, the system continuously monitors the user's explicit interactions with the projector. These interactions can include pausing, playing, rewinding, fast-forwarding, and selecting specific points on the timeline. These interactions are typically captured via input devices such as remote controls, touchscreens, or gesture recognition modules. Simultaneously, the system assesses the complexity of the information presented on the current projection interface. This includes the number of simultaneously displayed subtitle lines, the spatial density of active visual annotations on the screen, and the frequency of new information (such as subtitles or annotations). The information density score can be obtained by statistically analyzing the proportion of non-transparent pixels on the screen or calculating the overlap of active annotation areas. The information update frequency is measured by recording the number of new or updated subtitles and annotations per second.

[0068] Based on the above monitoring and evaluation results, the system calculates a cognitive load coefficient. This coefficient is a weighted sum of user interaction frequency, information density score, and information update frequency. Here, a, b, and c are second preset weights, which can be adjusted according to the actual application scenario to reflect the degree of influence of different factors on cognitive load. For example, frequent pause and replay operations, or a high information density score and update frequency on the screen, will increase the coefficient value, indicating that the user may be facing a high cognitive load.

[0069] The system then compares the calculated cognitive load coefficient with a preset dynamic threshold. This threshold is not fixed but dynamically adjusted based on the user's past review habits and mastery of different subject matter. For example, if a user frequently experiences difficulty understanding a specific topic (as determined by frequent pausing or replaying), the system automatically lowers the cognitive load threshold for that topic, making it easier to trigger information simplification strategies. When the cognitive load coefficient exceeds this dynamic threshold, the system executes preset information flow simplification steps to reduce the user's cognitive burden.

[0070] In the information flow simplification process, the system employs a multi-layered strategy. First, the volume of acoustic signals for key objects is increased to make them more audibly prominent. Simultaneously, visual annotations for key objects are displayed in a preset highlight color and at full opacity to ensure clear visibility and effectively guide the user's attention. Second, the volume of acoustic signals for non-key objects is decreased to reduce auditory interference. The transparency of visual annotations for non-key objects is increased, making them semi-transparent or temporarily hidden in certain situations, further reducing their distraction to the user. Finally, based on the area indicated by the interaction with key objects, the system generates a visual focus area on the projection interface. The brightness of this area is increased, while the brightness of other areas outside the visual focus area is decreased, thus proactively guiding the user's attention to the core content.

[0071] This application's solution introduces a cognitive load assessment mechanism to achieve adaptive adjustment of teaching information presentation in review mode. By monitoring users' explicit interactive actions and combining interface information density and update frequency, the system can quantify users' cognitive load status, thus proactively intervening and optimizing the information flow when users' information processing capacity reaches its limit. The solution calculates a cognitive load coefficient, weighting and fusing user interaction behavior with interface information characteristics to objectively reflect the user's psychological load level during review. When this coefficient exceeds a dynamic threshold, the system triggers an information flow simplification step, achieving focus on core teaching content through differentiated processing of acoustic signals and visual annotations. Specifically, by increasing the volume of key objects and enhancing the contrast of their visual annotations, while reducing interference from non-key objects, the system effectively guides users' attention resources. Furthermore, by generating a visual focus area on the projection interface and adjusting brightness contrast, the visual guidance of key knowledge points is further strengthened, thereby significantly reducing users' cognitive burden and improving review efficiency while ensuring the integrity of teaching information.

[0072] In one specific implementation, a dedicated processing unit within the smart projector, such as an embedded processor based on the ARM Cortex-A series architecture (e.g., NXP i.MX 8M Plus), tracks user input events from the remote control, touchscreen, or gesture recognition module in real time to monitor explicit user interactions. The projector's graphics rendering engine provides information on the number and location of all active overlays (including subtitle and annotation layers) on the current screen. An information density calculation module calculates an information density score based on this data, for example, by statistically analyzing the proportion of non-transparent pixels on the screen or calculating the overlap of active annotation areas. Simultaneously, it records the number of new or updated subtitles and annotations per second as an indicator of information update frequency. The processing unit maintains a cognitive load indicator, which is a weighted sum of the information density score, information update frequency, and user interaction frequency (e.g., the number of pauses or rewinds per minute), for example, indicator = (0.4 × information density score) + (0.3 × information update frequency) + (0.3 × user interaction frequency). A dynamic threshold management module stores a base threshold (e.g., 0.7) and adjusts it based on the user's comprehension efficiency for different course chapters in historical review data (e.g., time required to complete review, number of replays). For example, if the user's comprehension efficiency is low in the "Grain Boundary Segregation" chapter, the threshold for that chapter will automatically be lowered by 0.1. When the calculated cognitive load indicator exceeds this dynamic threshold, the system activates an information flow simplification strategy. At this point, the projector's audio processing unit (e.g., a Cirrus Logic CS47L90 audio DSP) dynamically adjusts the gain of the lecturer's voice channel, for example, by 3 dB. The graphics rendering pipeline sets the lecturer's annotation layer to 100% opacity and assigns a preset highlight color (e.g., bright yellow). The subtitle rendering module selects a larger font (e.g., 36pt) and a high-contrast color combination (e.g., white text with black outline) for the lecturer's translated subtitles and fixes them at the bottom center of the screen. Simultaneously, the audio processing unit reduces the gain of the assistant's voice channel (e.g., by 6 dB) or temporarily mutes the channel using digital muting. The graphics rendering pipeline sets the opacity of the assistant's annotation layer to 20% and assigns a soft color (e.g., light gray). The assisted captioning rendering module selects a smaller font (e.g., 24pt) and a low-contrast color (e.g., gray text, no stroke) for the assisted translation and renders it as a draggable, closable floating window that defaults to displaying in a non-core area of ​​the screen. Furthermore, the graphics rendering pipeline uses a dynamic mask layer to guide visual focus. This mask layer calculates a rectangular or irregular focus area based on the screen coordinates of the instructor's annotations and the semantically relevant area of ​​the current content being presented (coordinates of text boxes recognized via OCR).A GLSL (OpenGL Shading Language) shader is applied to the screen rendering, increasing the brightness of pixels within the focal area (e.g., by 10%), while decreasing the brightness of pixels outside the focal area (e.g., by 15%), thus creating a visually distinct primary and secondary effect.

[0073] Through the aforementioned technical solution, this application effectively addresses the cognitive overload problem caused by massive amounts of information during review mode. By assessing the user's cognitive load in real time and dynamically adjusting the information presentation, the system intelligently streamlines the information flow, highlighting core content, thereby significantly reducing the user's cognitive burden and helping them focus more efficiently on key knowledge points, improving learning efficiency and review effectiveness. Especially when combined with the aforementioned structured data generation and timeline backtracking functions, this solution provides users with a more personalized and adaptive learning experience when reviewing historical teaching content, avoiding confusion and fatigue caused by excessive information, thus better consolidating learned knowledge.

[0074] Reference Appendix Figure 2 This invention provides a smart terminal (the smart terminal adopts the real-time translation interaction method of the smart projector in the above embodiment, and the specific process is described in the corresponding steps above), including: Acquisition unit 100 is used to synchronously acquire acoustic signals within the target area, as well as visual information including display content and interactive behavior; the display content includes visual annotation information generated based on interactive behavior; The parsing unit 200 is used to analyze visual information, extract text data and spatiotemporal data of interactive behavior from the displayed content, and determine the target text element that matches the interactive behavior based on the correlation between the text data and the spatiotemporal data. The recognition unit 300 is used to generate initial recognition text by recognizing and processing acoustic signals; The calibration unit 400 is used to calculate the semantic similarity between the initial recognition text and the target text elements, and based on the semantic similarity, to obtain the calibration text by semantically calibrating the initial recognition text using the target text elements. The translation unit 500 is used to convert the calibration text into translation information in the target language, and to present the translation information and visual annotation information on the projection interface through layered overlay.

[0075] In some embodiments, a review unit 600 is also included for performing the following steps after the translation unit 500 has run: S6A1. Combine calibration text, translation information, and visual annotation information with timestamps to generate structured data associated with the currently displayed content; S6A2. In review mode, in response to the user's backtracking trigger operation on the timeline, the acoustic signals, translation information and visual annotation information in the structured data of the corresponding time node are replayed on the projection interface.

[0076] In some embodiments, the review unit 600 is further configured to perform the following steps after the translation unit 500 has run: S6B1. Monitor the explicit interactive actions of users in review mode, calculate the frequency of user interaction, and evaluate the information density score and information update frequency in the corresponding projected interface during the user's explicit interactive actions. S6B2. Calculate the cognitive load coefficient based on user interaction frequency, information density score, and information update frequency; S6B3. Compare the cognitive load coefficient with the preset dynamic threshold, and when the cognitive load coefficient is greater than the preset dynamic threshold, execute the preset information flow simplification steps.

[0077] In this context, the units described as separate components may or may not be physically separate. Similarly, the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0078] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0079] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0080] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time translation interaction method for a smart projector, characterized in that, Includes the following steps: S1. Synchronously acquire acoustic signals within the target area, as well as visual information including display content and interactive behavior; the display content includes visual annotation information generated based on the interactive behavior; S2. By parsing the visual information, extract the text data in the displayed content and the spatiotemporal data of the interactive behavior, and determine the target text element that matches the interactive behavior based on the correlation between the text data and the spatiotemporal data; S3. Generate initial recognition text by recognizing and processing the acoustic signal; S4. Calculate the semantic similarity between the initial identified text and the target text element, and based on the semantic similarity, use the target text element to perform semantic calibration on the initial identified text to obtain the calibrated text; S5. The calibration text is converted into translation information in the target language, and the translation information and the visual annotation information are presented on the projection interface through layered overlay.

2. The real-time translation interaction method for an intelligent projector according to claim 1, characterized in that, In step S2, the specific steps for determining the target text element matching the interaction behavior based on the correlation between the text data and the spatiotemporal data include: S21. Based on the spatiotemporal data, determine the local focus area of ​​the interactive behavior on the displayed content, and extract all text elements and their respective screen coordinates within the local focus area; S22. Evaluate the geometric distance between each text element within the local focus area and the key point coordinates of the interaction behavior in the spatiotemporal data to obtain the first correlation degree; S23. Obtain the preset topic vocabulary corresponding to the currently displayed content, and calculate the semantic relevance between each text element in the local focus area and the preset topic vocabulary to obtain the second relevance; S24. Calculate the association score based on the first association degree and the second association degree, and determine the text element with the highest association score as the target text element that matches the interaction behavior.

3. The real-time translation interaction method for an intelligent projector according to claim 2, characterized in that, The specific steps in step S22 include: S221. Calculate the Euclidean distance between the key point coordinates and the screen coordinates, and obtain the first correlation degree by normalizing the Euclidean distance.

4. The real-time translation interaction method for an intelligent projector according to claim 1, characterized in that, The specific steps in step S3 include: S31. Collect the speaker's biometric information and the location information of the sound source; S32. Match the biometric information with a preset user list, and combine the sound source location information to determine the speaker's identity; S33. Prioritize the acoustic signals according to the speaker's identity and the type of interaction behavior, and allocate corresponding audio processing channels.

5. The real-time translation interaction method for an intelligent projector according to claim 4, characterized in that, The speaker's identity includes key personnel and / or non-key personnel; The specific steps in step S33 include: Acoustic signals marked as the priority object are assigned a first processing priority and routed to a low-latency audio processing channel, while acoustic signals marked as the non-priority object are assigned a second processing priority and routed to a resolution channel; the first processing priority is higher than the second processing priority.

6. The real-time translation interaction method for an intelligent projector according to claim 1, characterized in that, The specific steps in step S4 include: S41. Obtain the preset topic vocabulary corresponding to the currently displayed content; the preset topic vocabulary includes multiple standard professional terms and their corresponding first preset weights; S42. Calculate the semantic similarity between multiple candidate words corresponding to the initial identified text and each standard technical term in the target text element; S43. Calculate the comprehensive score of each candidate word based on the semantic similarity and the first preset weight, and select the candidate word with the highest comprehensive score as the content in the calibration text.

7. The real-time translation interaction method for an intelligent projector according to claim 5, characterized in that, Step S5 is followed by: S6A1. Combine the calibration text, the translation information, and the visual annotation information with the timestamp to generate structured data associated with the currently displayed content; S6A2. In review mode, in response to the user's backtracking trigger operation on the timeline, the acoustic signals, translation information and visual annotation information in the structured data of the corresponding time node are replayed on the projection interface.

8. The real-time translation interaction method for an intelligent projector according to claim 7, characterized in that, Step S5 is followed by: S6B1. Monitor the user's explicit interactive actions in the review mode, calculate the user's interaction frequency, and evaluate the information density score and information update frequency in the corresponding projected interface during the user's execution of the explicit interactive actions. S6B2. Calculate the cognitive load coefficient based on the user interaction frequency, the information density score, and the information update frequency; S6B3. Compare the cognitive load coefficient with a preset dynamic threshold, and when the cognitive load coefficient is greater than the preset dynamic threshold, execute a preset information flow simplification step.

9. The real-time translation interaction method for an intelligent projector according to claim 8, characterized in that, The information flow simplification steps include: C1. Increase the acoustic signal volume of the key object and display the visual annotation information of the key object in a preset highlight color and a completely opaque state; C2. Reduce the acoustic signal volume of the non-key object and increase the transparency of the visual annotation information of the non-key object; C3. Based on the area pointed to by the interaction behavior of the key object, generate a visual focus area on the projection interface, increase the display brightness of the visual focus area, and decrease the display brightness of other areas outside the visual focus area.

10. A smart terminal, characterized in that, include: The acquisition unit is used to simultaneously acquire acoustic signals within the target area, as well as visual information including displayed content and interactive behavior. The displayed content includes visual annotation information generated based on the interactive behavior; The parsing unit is used to parse the visual information, extract text data from the displayed content and spatiotemporal data of the interactive behavior, and determine the target text element that matches the interactive behavior based on the correlation between the text data and the spatiotemporal data. The recognition unit is used to generate initial recognition text by recognizing and processing the acoustic signal; The calibration unit is used to calculate the semantic similarity between the initial identified text and the target text element, and based on the semantic similarity, use the target text element to perform semantic calibration on the initial identified text to obtain the calibrated text; The translation unit is used to convert the calibration text into translation information in the target language, and to present the translation information and the visual annotation information on the projection interface through layered overlay.

Images