Multi-screen recording and broadcasting central control fusion device and method based on semantic association and visual attention

Abstract

This invention discloses a multi-screen recording and broadcasting central control fusion device and method based on semantic association and visual attention, belonging to the field of teaching recording and broadcasting equipment technology. The device includes a multi-screen acquisition unit, a visual attention detection unit, a sound pickup and analysis unit, a core control module, a teaching semantic association library, a fusion decision engine, a screen fusion processing module, and a storage and output module. The core control module performs content feature extraction and semantic encoding on the video signals of each screen, generates semantic relevance scores through matching with the teaching semantic association library, and determines the effective gaze state based on dual thresholds of gaze duration and gaze point stability. The fusion decision engine performs weighted fusion decisions, simultaneously outputting central control instructions to the external teaching recording and broadcasting control system and fusion mode instructions to the screen fusion processing module, and achieving adaptive iterative optimization of the fusion mode through a feedback loop of image quality evaluation parameters. This invention enables intelligent acquisition and adaptive fusion of multi-screen teaching content driven by semantics, which is beneficial for improving the teaching relevance and review value of recorded broadcasts.

Description

[0001] summary This invention discloses a multi-screen recording and broadcasting central control fusion device and method based on semantic association and visual attention, belonging to the field of teaching recording and broadcasting equipment technology. The device includes a multi-screen acquisition unit, a visual attention detection unit, a sound pickup and analysis unit, a core control module, a teaching semantic association library, a fusion decision engine, a screen fusion processing module, and a storage and output module. The core control module performs content feature extraction and semantic encoding on the video signals of each screen, generates semantic relevance scores through matching with the teaching semantic association library, and determines the effective gaze state based on dual thresholds of gaze duration and gaze point stability. The fusion decision engine performs weighted fusion decisions, simultaneously outputting central control instructions to the external teaching recording and broadcasting control system and fusion mode instructions to the screen fusion processing module, and achieving adaptive iterative optimization of the fusion mode through a feedback loop of image quality evaluation parameters. This invention enables intelligent acquisition and adaptive fusion of multi-screen teaching content driven by semantics, which is beneficial for improving the teaching relevance and review value of recorded broadcasts. Technical Field

[0002] This invention relates to the field of teaching recording and broadcasting equipment technology, specifically to a multi-screen recording and broadcasting central control fusion device and method based on semantic association and visual attention. Background Technology

[0003] With the deepening development of smart education, multi-screen teaching scenarios are becoming increasingly common. Teachers often use multiple displays to show courseware, blackboard writing, experimental demonstrations, and other content. However, most existing multi-screen recording and broadcasting technologies use simple multi-channel signal synchronous recording or fixed-mode screen splicing, which has the following technical shortcomings: First, there is a lack of intelligent decision-making mechanisms. Existing equipment cannot intelligently switch based on the importance of the teaching content or the user's focus, resulting in the recorded content not highlighting key points and low efficiency in later review.

[0004] Secondly, the fusion mode is singular and fixed. Existing multi-screen fusion solutions usually adopt fixed layouts such as picture-in-picture or rotation, which cannot dynamically adjust the fusion layout according to the logical relationship between teaching content, resulting in poor logic of the synthesized picture.

[0005] Third, there is a lack of quantitative evaluation and feedback on image quality. Existing fusion outputs typically lack objective evaluation of the quality of the composite result. Once generated, the output is fixed and cannot be self-corrected based on image quality feedback, making it difficult to adapt to the dynamic changes in the teaching environment.

[0006] Fourth, the application of visual attention detection technology is limited. Although some recording and broadcasting equipment has introduced visual attention detection technology, it is not stable enough in scenarios such as occlusion, low light, or micro-eye movements (such as blinking or rapid scanning), and it is not deeply integrated with teaching semantic information, making it difficult to accurately reflect the real user attention state.

[0007] The fundamental flaw of the aforementioned existing technologies lies in the fact that semantic content analysis, visual attention tracking, and audio scene recognition work independently, failing to form a collaborative decision-making loop driven by the teaching scenario, and also lacking a feedback correction mechanism for the quality of the integrated output, resulting in the recorded video failing to adaptively match the dynamic changes in the teaching process. Summary of the Invention Technical issues

[0008] The main technical problem that this invention aims to solve is that existing multi-screen recording and broadcasting devices lack an intelligent fusion decision-making mechanism based on the semantics of teaching content and the user's visual attention, resulting in the recording and broadcasting screen not highlighting the key points, having a fixed fusion mode, and lacking quality feedback.

[0009] Secondary technical issues include: insufficient stability of visual attention detection in scenarios with eye occlusion; and the inability of the fusion layout to dynamically adjust based on the logical connections between knowledge points. Technical solution

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A multi-screen recording and broadcasting central control fusion device based on semantic association and visual attention includes a multi-screen acquisition unit, a visual attention detection unit, a sound pickup and analysis unit, a core control module, a teaching semantic association library, a fusion decision engine, a screen fusion processing module, and a storage and output module.

[0011] The multi-screen acquisition unit synchronously acquires video signals from multiple teaching displays and inputs them into the core control module. The visual attention detection unit acquires user eye images and outputs the gaze point coordinate sequence and eye state signals to the core control module. The sound pickup and analysis unit acquires sound from the teaching environment and outputs audio feature analysis results to the core control module.

[0012] The core control module includes a content feature extraction submodule, a semantic encoding submodule, a gaze state determination submodule, and a teaching scenario determination submodule. The content feature extraction submodule extracts text region, graphic region, and screen layout features from each video signal. The semantic encoding submodule converts the extracted features into semantic feature vectors and performs similarity matching with knowledge point vectors in a teaching semantic association library to generate a semantic relevance score for each screen. The gaze state determination submodule calculates the cumulative gaze duration and gaze stability for each screen based on the gaze point coordinate sequence. A screen is considered to have a valid gaze state when the cumulative gaze duration is greater than or equal to a duration threshold and the dispersion of the gaze point coordinates is less than a preset dispersion threshold. The teaching scenario determination submodule determines the current teaching scenario based on the audio feature analysis results.

[0013] The fusion decision engine performs weighted fusion decisions based on semantic relevance scores and effective gaze states, generating screen switching or screen hold commands as central control commands output to the external teaching recording and broadcasting control system. Simultaneously, it generates fusion mode commands output to the screen fusion processing module. The teaching semantic association library stores keywords, knowledge points, and logical relationships between knowledge points in the teaching domain, and interacts bidirectionally with the fusion decision engine.

[0014] The screen fusion processing module fuses multiple video signals according to the fusion mode instructions output by the fusion decision engine, generating a single-screen integrated image. It then calculates image quality evaluation parameters for the generated single-screen integrated image and sends these parameters back to the fusion decision engine for iterative optimization of the fusion mode. The fusion mode includes at least one of picture-in-picture mode, split-screen mode, and single-screen full-screen mode. The storage and output module stores and outputs the single-screen integrated image and the synchronously generated semantic annotation file.

[0015] Furthermore, the weighted fusion decision of the fusion decision engine includes: assigning a first weight to the semantic relevance score and assigning a second weight to the effective gaze state, with the sum of the first weight and the second weight being 1; when the weighted comprehensive score of a screen exceeds the switching threshold and continues to exceed the preset hysteresis time, generating an instruction to switch to that screen; when the weighted comprehensive score of the current output screen is not lower than the hold threshold, generating a screen hold instruction.

[0016] Furthermore, the first and second weights are dynamically adjusted based on the teaching scenario output by the teaching scenario determination submodule: the weight of semantic relevance score is increased in the lecture scenario, and the weight of effective gaze state is increased in the interactive scenario; when the proportion of speech in the audio is higher than the preset value, the teaching scenario determination submodule determines it as a lecture scenario, and when the frequency of interactive responses in the audio is higher than the preset value, it is determined as an interactive scenario.

[0017] Furthermore, the processing flow of the screen fusion module includes: multi-related screen marking, establishing association marks for multiple related screens based on the logical relationships between knowledge points in the teaching semantic association library; core content extraction, extracting the core content area containing knowledge points from each related screen; knowledge graph logical matching, determining the fusion layout of multi-screen content based on the logical relationships between knowledge points; fusion mode selection, selecting picture-in-picture mode, active screen zoom mode, or multi-screen rotation mode based on the layout determination result; and single-screen integrated screen generation, merging the multi-screen content into a single screen according to the selected fusion mode.

[0018] Furthermore, the image quality evaluation parameters include at least one of semantic overlap, layout rationality, and visual balance. When the image quality evaluation parameters are below a preset threshold, the fusion decision engine switches the fusion mode to regenerate the single-screen integrated image. When the image quality evaluation parameters meet the preset threshold or the number of iterations reaches the upper limit, the current image is output.

[0019] Furthermore, the visual attention detection unit employs a dual-view imaging structure, including a direct-view camera and a reflecting lens. The direct-view camera captures a direct image of the user's eye along the optical axis. The reflecting surface of the reflecting lens is positioned at an angle of 30° to 60° to the optical axis of the direct-view camera, forming a reflected image of the eyeball and guiding it into the imaging optical path. The visual attention detection unit also includes an eye feature analysis submodule, which uses multi-frame continuous image acquisition and a 3D shape recovery algorithm to construct 3D models of the iris and cornea. In scenarios with eye occlusion, the 3D model is combined with visible eye features to perform pose estimation to maintain tracking stability.

[0020] Furthermore, the semantic encoding submodule uses a deep learning text encoding model to semantically encode the extracted text region content, resulting in a semantic feature vector with fixed dimensions; the similarity matching uses cosine similarity to calculate the similarity value between the semantic feature vector and the knowledge point vector, and normalizes the similarity value to the interval between 0 and 1 as the semantic relevance score.

[0021] This invention also provides a multi-screen recording and playback fusion method based on semantic association and visual attention, comprising the following steps: S1: Multi-screen signal synchronous acquisition, synchronously acquiring video signals from multiple teaching display screens; S2: Screen content feature extraction, extracting text area, graphic area and screen layout features from each video signal; S3: Semantic encoding and association matching converts the extracted features into semantic feature vectors, performs similarity matching with the knowledge point vectors in the pre-set teaching semantic association library, and generates semantic relevance scores for each screen. S4: Visual attention data acquisition and effective gaze determination. Acquire user eye images to obtain gaze point coordinate sequence and eye state signal, calculate the cumulative gaze duration and gaze point stability of each screen, and determine that the screen is in an effective gaze state when the cumulative gaze duration is greater than or equal to the duration threshold and the dispersion of the gaze point coordinates is less than the preset dispersion threshold. S5: Teaching scenario determination, collect audio features from the teaching scene, and determine whether the current teaching scenario is a lecture scenario or an interactive scenario based on the audio features; S6: Weighted fusion decision, assigning a first weight to the semantic relevance score and a second weight to the effective gaze state. The first weight and the second weight are dynamically adjusted according to the teaching scenario. The weighted comprehensive score of each screen is calculated. When the weighted comprehensive score of a screen exceeds the switching threshold and continues to exceed the preset hysteresis time, a screen switching instruction is generated. When the weighted comprehensive score of the current output screen is not lower than the holding threshold, a screen holding instruction is generated. S7: The central control command output is integrated with the screen. The switching command or hold command is output as a central control command to the external teaching recording and broadcasting control system. At the same time, multiple video signals are integrated according to the determined integration mode to generate a single-screen integrated picture. S8: Quality feedback iteration. Calculates the image quality evaluation parameters for the generated single-screen integrated image. When the image quality evaluation parameters are lower than the preset threshold, switches the fusion mode and regenerates until the image quality evaluation parameters meet the threshold or the number of iterations reaches the upper limit. Beneficial effects

[0022] First, this invention achieves intelligent switching of recording perspectives through weighted fusion decision-making based on semantic association and visual attention, which helps to filter out non-teaching content and reduce recording redundancy.

[0023] Secondly, the present invention introduces a picture quality evaluation feedback mechanism, which can automatically iterate and optimize the fusion layout and mode based on the fusion results, which helps to solve the problem of fixed mode failing when content changes.

[0024] Third, the present invention adopts a dual-view structure combined with three-dimensional shape recovery visual attention detection technology, which helps to overcome the tracking instability caused by eye occlusion and micro-movement, and adapts to the complex lighting environment of the teaching site.

[0025] Fourth, the visual data of this invention is only used for real-time gaze point calculation, does not store original eye images, and does not perform identity recognition, which helps to meet the data compliance requirements of educational scenarios.

[0026] Fifth, this invention constructs a collaborative technology system driven by audio teaching scenario determination, dynamic weighted decision-making based on semantic relevance score and visual effective gaze state, and closed-loop correction based on integrated quality evaluation feedback. The three work together rather than independently, which is conducive to making the recorded broadcast screen adaptively match the dynamic changes in the teaching process. Attached Figure Description

[0027] Figure 1 This is a block diagram of the overall structure of the device of the present invention;

[0028] Figure 2 This is a flowchart of the screen blending process;

[0029] Figure 3 Workflow diagram for the core control module;

[0030] Figure 4 This is a schematic diagram of a weighted fusion decision-making mechanism.

[0032] Figure labeling: 101-Multi-screen acquisition unit; 102-Visual attention detection unit; 103-Sound pickup and analysis unit; 104-Core control module; 105-Teaching semantic association library; 106-Fusion decision engine; 107-Screen fusion processing module; 108-Storage output module; 109-Central control command output; 1041-Content feature extraction submodule; 1042-Semantic encoding submodule; 1043-Gaze ​​state determination submodule; 1044-Teaching scene determination submodule. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Example 1: Overall Equipment Architecture Reference Figure 1 The recording and broadcasting equipment in this embodiment includes a multi-screen acquisition unit 101, a visual attention detection unit 102, a sound pickup and analysis unit 103, a core control module 104, a teaching semantic association library 105, a fusion decision engine 106, a screen fusion processing module 107, a storage output module 108, and a central control command output 109.

[0035] The multi-screen acquisition unit 101 simultaneously acquires video signals from 2 to 4 teaching displays, with each video signal supporting a resolution of 1080P or higher and a frame rate of no less than 25 frames per second. The visual attention detection unit 102 acquires user eye images and outputs a gaze point coordinate sequence and eye state signals. The sound pickup and analysis unit 103 acquires sound from the teaching environment, extracts audio features, and outputs the audio feature analysis results.

[0036] The core control module 104 integrates a content feature extraction submodule 1041, a semantic encoding submodule 1042, a gaze state determination submodule 1043, and a teaching scenario determination submodule 1044, and works collaboratively with the teaching semantic association library 105 and the fusion decision engine 106. Specifically, the content feature extraction submodule 1041 performs text region detection (using text detection methods based on connected component analysis or deep learning), graphic region segmentation, and screen layout feature extraction on each video signal; the semantic encoding submodule 1042 inputs the extracted text region content into a deep learning text encoding model and outputs a semantic feature vector with fixed dimensions; the gaze state determination submodule 1043 receives the gaze point coordinate sequence, calculates the cumulative gaze duration by screen region, and calculates the dispersion of the gaze point coordinates within the sliding time window (using coordinate standard deviation or root mean square deviation as a measure of dispersion); the teaching scenario determination submodule 1044 receives the audio feature analysis results and determines the current teaching scenario based on the proportion of speech and the frequency of interactive responses.

[0037] The fusion decision engine 106 receives semantic relevance scores and effective gaze status, performs a weighted fusion decision, and generates two parallel outputs: one is a central control command, which is sent to the external teaching recording and broadcasting control system via the central control command output 109 to control the recording channel switching of the external recording and broadcasting host; the other is a fusion mode command, which is sent to the screen fusion processing module 107 to generate a local single-screen integrated image. The two outputs work in parallel, serving the external recording and broadcasting control and the local preview / backup output, respectively.

[0038] The screen fusion processing module 107 is responsible for the final image composition and quality feedback. The storage output module 108 generates semantic annotation files simultaneously while storing the integrated single-screen image.

[0039] Example 2: Weighted Fusion Decision-Making Method Reference Figure 4 This embodiment details the working principle of the fusion decision engine 106.

[0040] Suppose the teacher is explaining a complex physics formula (screen 1), and at the same time there is a corresponding diagram on the blackboard (screen 2).

[0041] In the S3 stage, the semantic coding submodule 1042 calculated the highest semantic relevance score for screen 1, followed by screen 2.

[0042] In stage S4, the teacher gazes at screen 1 for an extended period of time. The gaze state determination submodule 1043 calculates that the cumulative gaze duration of screen 1 is greater than or equal to the duration threshold, and the dispersion of the gaze point coordinates is less than the preset dispersion threshold. Screen 1 is then determined to be in a valid gaze state.

[0043] In the S5 stage, the teaching scenario determination submodule 1044 analyzes the audio features output by the sound pickup analysis unit 103, detects that the proportion of speech in the audio is higher than the preset value, and determines that the current scenario is a lecture.

[0044] In stage S6, during the teaching scenario, the fusion decision engine 106 assigns a first weight (e.g., α=0.7) to the semantic relevance score and a second weight (β=0.3, α+β=1) to the effective gaze state. Screen 1 has the highest and most stable weighted composite score, and the system determines that it should switch to a close-up of Screen 1. If the teacher suddenly looks at Screen 2 and begins interacting with the students, the teaching scenario determination submodule 1044 detects that the frequency of interactive responses in the audio is higher than a preset value, and determines that the scenario has switched to an interactive scenario. The fusion decision engine 106 quickly adjusts the weights (e.g., α=0.4, β=0.6). Although the semantic relevance score of Screen 2 is lower than that of Screen 1, its composite score may rise rapidly because it is an interactive hotspot and is effectively gazed upon by the teacher. If the weighted composite score of Screen 2 continues to exceed the switching threshold and exceeds a preset hysteresis time, the system will automatically switch to Screen 2 to capture the moment of interaction.

[0045] The specific values ​​of the aforementioned duration threshold, dispersion threshold, switching threshold, hold threshold, and hysteresis time need to be determined based on the test results in the actual teaching scenario; this embodiment does not impose specific limitations.

[0046] Example 3: Quality Feedback and Adaptive Adjustment Reference Figure 2 After generating the picture-in-picture composite image, the screen blending processing module 107 calculates the image quality evaluation parameters. This embodiment provides the specific calculation method for three parameters: Semantic overlap: Semantic vectors are obtained by semantically encoding the core content regions extracted from each screen. The cosine similarity between the semantic vectors of any two screens is calculated, and the arithmetic mean of all pairwise similarities is taken and normalized to the range of 0 to 1. The higher the semantic overlap, the higher the degree of semantic repetition of the content across multiple screens, and the more suitable it is to use a single-screen magnification mode rather than a multi-screen side-by-side mode.

[0047] Layout rationality: The degree to which the effective display area ratio of the core content area in the single-screen integrated image conforms to the preset optimal ratio. The calculation formula is: Layout rationality = 1 − |S_core / S_total − R_opt| / R_opt, where S_core is the pixel area of ​​the core content area, S_total is the total pixel area of ​​the single-screen integrated image, and R_opt is the preset optimal ratio (which can be taken as 0.6 to 0.8). The closer the layout rationality is to 1, the more reasonable the display ratio of the core content.

[0048] Visual balance: The integrated single-screen image is divided into four quadrants, and the information density of each quadrant is calculated (using the proportion of edge pixels extracted by the edge detection operator to the total number of pixels in the quadrant as the information density). The ratio of the variance to the mean of the information density in each of the four quadrants is calculated as follows: Visual balance = 1 − σ² / μ, where σ² is the variance of the information density in the four quadrants and μ is the mean of the information density in the four quadrants. The closer the visual balance is to 1, the more balanced the information distribution in each area of ​​the image.

[0049] If the system detects that the illustrations on screen 2 are severely obscured, causing the layout to fall below a preset threshold, it will automatically abandon the current picture-in-picture layout and attempt to switch to a split-column layout to simultaneously display formulas and illustrations, thereby optimizing the recording effect. When the image quality evaluation parameters meet the preset threshold or the number of iterations reaches the upper limit (e.g., 3 times), the current image is output.

[0050] Example 4: Preferred Structure for Dual-View Visual Attention Detection This embodiment provides a preferred implementation of the visual attention detection unit 102. The visual attention detection unit 102 employs a dual-view imaging structure, including a direct-view camera and a reflective lens. The direct-view camera faces the user's eyeball along the principal optical axis, capturing a direct image of the eyeball from the front. The reflective surface of the reflective lens is positioned at an angle of 30° to 60° (preferably 45°) to the optical axis of the direct-view camera, located to the side of the camera's field of view. It reflects the reflected image from the side of the eyeball and guides it into the camera's imaging optical path, forming a dual-view image. A single-camera time-division acquisition method (one frame captures the direct-view image, the next frame captures the reflected image) or a dual-camera synchronous acquisition method can be used.

[0051] The eye feature analysis submodule extracts and matches feature points from continuously acquired multi-frame eye images, and reconstructs the 3D geometric model of the iris and cornea using the Structure for Motion Restoration (SfM) algorithm. When eye occlusion occurs (such as eyelashes or eyelids partially obscuring the iris), the pre-constructed 3D model is used as a priori, and combined with eye feature points in the unoccluded area to perform 3D pose estimation. The pose parameters are then extrapolated to the coordinates of the current gaze point, thereby maintaining tracking continuity in partially occluded scenarios.

[0052] Example 5: Semantic annotation file output The storage output module 108 simultaneously generates semantic annotation files while storing the integrated single-screen view. The semantic annotation files are in JSON or XML format, and each record contains the following fields: timestamp (milliseconds), screen_id (screen number), semantic_score (semantic relevance score, 0 to 1), knowledge_point_id (associated knowledge point ID), knowledge_point_name (knowledge point name), gaze_status (effective gaze status, boolean), gaze_duration (cumulative gaze duration, in seconds), teaching_scene (current teaching scene, value is "lecture" or "interaction"), and fusion_mode (fusion mode identifier, value is "pip", "active_zoom", or "rotation"). These semantic annotation files facilitate intelligent retrieval and segment localization based on knowledge points or gaze status in later stages.

[0053] Example 6: Knowledge Graph Logical Matching and Fusion Layout Mapping Rules This embodiment details the specific mapping rules for the knowledge graph logical matching step in the screen fusion processing module 107. The logical relationships between knowledge points stored in the teaching semantic association library include four types: predecessor-successor relationships, inclusion relationships, parallel relationships, and causal relationships. Different relationship types correspond to different fusion layouts. Precursor-successor relationships (such as "Newton's Second Law" and "acceleration calculation"): corresponding to the left and right column layout, the preceding knowledge points are on the left side of the screen, and the succeeding knowledge points are on the right side of the screen, which conforms to the reading order from left to right.

[0054] Containment relationships (e.g., "function" contains "linear function"): corresponding to a picture-in-picture layout, the parent knowledge point is displayed in full screen, and the child knowledge point is displayed as a small window overlaid in the corner of the screen, reflecting the subordinate hierarchy.

[0055] Parallel relationships (such as "addition" and "multiplication" being parallel operations): This corresponds to an equally divided multi-screen layout, where each associated screen equally divides the screen space, reflecting an equal relationship.

[0056] Cause and effect relationship (e.g., "force analysis" leads to "change in motion state"): corresponding to the top and bottom column layout, the cause knowledge points are located in the upper half of the screen, and the result knowledge points are located in the lower half of the screen, which conforms to the cause-effect reading order from top to bottom.

[0057] The blending mode selection step determines the result based on the above layout. Among the picture-in-picture mode, active screen zoom mode, and multi-screen rotation mode, the most matching blending mode is selected to generate a single-screen integrated screen.

Claims

1. A multi-screen recording and broadcasting central control fusion device based on semantic association and visual attention, characterized in that, include: Multi-screen acquisition unit, visual attention detection unit, sound pickup and analysis unit, core control module, teaching semantic association library, fusion decision engine, screen fusion processing module and storage output module; The multi-screen acquisition unit is used to simultaneously acquire video signals from multiple teaching displays and input them into the core control module; The visual attention detection unit is used to acquire user eye images and output the gaze point coordinate sequence and eye state signal to the core control module; The sound pickup and analysis unit is used to collect the sound of the teaching site and output the audio feature analysis results to the core control module; The core control module includes a content feature extraction submodule, a semantic encoding submodule, a gaze state determination submodule, and a teaching scenario determination submodule; the content feature extraction submodule is used to extract text region, graphic region, and screen layout features from each video signal; The semantic encoding submodule is used to convert the extracted features into semantic feature vectors and perform similarity matching with the knowledge point vectors in the teaching semantic association library to generate semantic relevance scores for each screen. The gaze state determination submodule is used to calculate the cumulative gaze duration and gaze stability of each screen based on the gaze point coordinate sequence. When the cumulative gaze duration is greater than or equal to the duration threshold and the dispersion of the gaze point coordinates is less than the preset dispersion threshold, the screen is determined to be in a valid gaze state. The teaching scenario determination submodule is used to determine the current teaching scenario based on the audio feature analysis results; The fusion decision engine is used to perform weighted fusion decision based on the semantic relevance score and the effective gaze state, generate screen switching instructions or screen hold instructions as central control instructions and output them to the external teaching recording and broadcasting control system, and generate fusion mode instructions and output them to the screen fusion processing module. The teaching semantic association library is used to store keywords, knowledge points and logical relationships between knowledge points in the teaching field, and interacts bidirectionally with the fusion decision engine. The screen fusion processing module is used to fuse multiple video signals according to the fusion mode instructions output by the fusion decision engine, generate a single-screen integrated picture, calculate picture quality evaluation parameters for the generated single-screen integrated picture, and send the evaluation parameters back to the fusion decision engine to iteratively optimize the fusion mode. The fusion mode includes at least one of picture-in-picture mode, split-screen mode, and single-screen full-screen mode; The storage output module is used to store and output the single-screen integrated image and the synchronously generated semantic annotation file.

2. The device according to claim 1, characterized in that, The weighted fusion decision of the fusion decision engine includes: assigning a first weight to the semantic relevance score and assigning a second weight to the effective gaze state, wherein the sum of the first weight and the second weight is 1; when the weighted comprehensive score of a screen exceeds the switching threshold and continues to exceed the preset hysteresis time, a switching instruction to that screen is generated; when the weighted comprehensive score of the current output screen is not lower than the hold threshold, a screen hold instruction is generated.

3. The device according to claim 2, characterized in that, The first weight and the second weight are dynamically adjusted according to the teaching scenario output by the teaching scenario determination submodule: the weight of semantic relevance score is increased in the teaching scenario, and the weight of effective gaze state is increased in the interactive scenario; when the proportion of speech in the audio is higher than the preset value, the teaching scenario determination submodule determines it as a teaching scenario, and when the frequency of interactive response in the audio is higher than the preset value, it is determined as an interactive scenario.

4. The device according to claim 1, characterized in that, The processing flow of the screen fusion module includes: multi-associated screen marking, establishing association marks for multiple screens with associations based on the logical association relationships between knowledge points in the teaching semantic association library; core content extraction, extracting core content areas containing knowledge points from each associated screen; knowledge graph logical matching, determining the fusion layout of multi-screen content based on the logical association relationships between knowledge points; fusion mode selection, selecting picture-in-picture mode, active screen zoom mode, or multi-screen rotation mode based on the layout determination result; and single-screen integrated screen generation, merging the multi-screen content into a single screen according to the selected fusion mode.

5. The device according to claim 4, characterized in that, The image quality evaluation parameters include at least one of semantic overlap, layout rationality, and visual balance. When the image quality evaluation parameters are lower than the preset threshold, the fusion decision engine switches the fusion mode to regenerate the single-screen integrated image. When the image quality evaluation parameters meet the preset threshold or the number of iterations reaches the upper limit, the current image is output.

6. The device according to claim 1, characterized in that, The visual attention detection unit adopts a dual-view imaging structure, including a direct-view camera and a reflective lens. The direct-view camera captures a direct image of the user's eye along the optical axis. The reflective surface of the reflective lens is set at an angle of 30° to 60° with the optical axis of the direct-view camera to form a reflected image of the eyeball and guide it into the imaging optical path. The visual attention detection unit also includes an eye feature analysis submodule, which uses multi-frame continuous image acquisition and a three-dimensional shape recovery algorithm to construct a three-dimensional model of the iris and cornea. In scenarios where the eye is occluded, the three-dimensional model is combined with visible eye features to perform pose estimation to maintain tracking stability.

7. The device according to claim 1, characterized in that, The semantic encoding submodule uses a deep learning text encoding model to semantically encode the extracted text region content, resulting in a semantic feature vector with fixed dimensions. The similarity matching uses cosine similarity to calculate the similarity value between the semantic feature vector and the knowledge point vector, and normalizes the similarity value to the interval between 0 and 1 as the semantic relevance score.

8. A multi-screen recording and playback fusion method based on semantic association and visual attention, characterized in that, Includes the following steps: S1: Multi-screen signal synchronous acquisition, synchronously acquiring video signals from multiple teaching display screens; S2: Screen content feature extraction, extracting text area, graphic area and screen layout features from each video signal; S3: Semantic encoding and association matching converts the extracted features into semantic feature vectors, performs similarity matching with the knowledge point vectors in the pre-set teaching semantic association library, and generates semantic relevance scores for each screen. S4: Visual attention data acquisition and effective gaze determination. Acquire user eye images to obtain gaze point coordinate sequence and eye state signal, calculate the cumulative gaze duration and gaze point stability of each screen, and determine that the screen is in an effective gaze state when the cumulative gaze duration is greater than or equal to the duration threshold and the dispersion of the gaze point coordinates is less than the preset dispersion threshold. S5: Teaching scenario determination, collect audio features from the teaching scene, and determine whether the current teaching scenario is a lecture scenario or an interactive scenario based on the audio features; S6: Weighted fusion decision, assigning a first weight to the semantic relevance score and a second weight to the effective gaze state. The first weight and the second weight are dynamically adjusted according to the teaching scenario. The weighted comprehensive score of each screen is calculated. When the weighted comprehensive score of a screen exceeds the switching threshold and continues to exceed the preset hysteresis time, a screen switching instruction is generated. When the weighted comprehensive score of the current output screen is not lower than the holding threshold, a screen holding instruction is generated. S7: The central control command output is integrated with the screen. The switching command or hold command is output as a central control command to the external teaching recording and broadcasting control system. At the same time, multiple video signals are integrated according to the determined integration mode to generate a single-screen integrated picture. S8: Quality feedback iteration. Calculates the image quality evaluation parameters for the generated single-screen integrated image. When the image quality evaluation parameters are lower than the preset threshold, switches the fusion mode and regenerates until the image quality evaluation parameters meet the threshold or the number of iterations reaches the upper limit.

Images