A multimodal visual analysis system for teaching

Abstract

The present invention discloses a multimodal visual analysis system for teaching, which belongs to the field of computer vision and artificial intelligence technology. The system comprises a video acquisition and processing module, which acquires video frames in real time and determines whether there are dynamic changes in the scene; a multi-threaded parallel analysis module, which realizes parallel processing of video acquisition and analysis; a user interface module, which displays the captured real-time video frames by designing the interface layout; an analysis result output and display module, which pushes and displays intermediate results in real time, and sends the final analysis conclusion after the analysis is completed to the user interface module; a system stability and resource management module, which includes multi-threaded management and synchronization and resource overhead monitoring and optimization; a log and error handling module, which records program operation information, records captured exceptions in a log and takes corresponding measures; the multimodal visual analysis system for teaching provided by the present invention improves practical operation efficiency and quality by analyzing real-time video frames.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a multimodal visual analysis system for teaching. Background Art

[0002] Traditional instructional techniques for practical electronic circuit teaching face numerous bottlenecks. From a real-time perspective, manual observation-based instruction is not only inefficient but also prone to errors and missed judgments due to subjective factors such as observer fatigue and distraction, and feedback is significantly delayed. Offline video analysis, on the other hand, requires post-processing, making it impossible to provide timely guidance during the actual operation and difficult to meet the real-time error correction requirements of practical electronic circuit training.

[0003] In terms of intelligence, existing systems often rely on pattern-matching algorithms based on preset rules. For example, they set a fixed threshold for solder joint shape to determine soldering quality. This makes it difficult for the system to accurately identify defects in challenging lighting conditions or when components are slightly offset. This lack of adaptive capabilities often renders it helpless in the face of diverse operating scenarios and personalized operating habits.

[0004] In terms of functional completeness, most tools are relatively limited in functionality. For example, error detection tools can identify operational errors but fail to provide specific correction steps or relevant knowledge points. Operation classification tools can only categorize operational behaviors but fail to provide learners with matching exercises to consolidate their knowledge, making it difficult to provide effective guidance throughout the entire process of practical electronic circuit teaching.

[0005] In terms of domain adaptability, the feature extraction methods used by general-purpose visual models, such as the general-purpose convolution kernels of convolutional neural networks, struggle to accurately capture the subtle features of professional electronic circuit tools and components. For specialized information such as resistor color coding, multimeter pointer scales, solder joint roundness and glossiness, and capacitor polarity markings, general-purpose models have low recognition accuracy, failing to meet the professional requirements of practical electronic circuit teaching.

[0006] Therefore, it is necessary to design a multimodal visual analysis module for teaching, which can intelligently analyze the real-time video stream captured by the camera in the laboratory, help users complete key operations such as circuit soldering, component installation, and line debugging, and provide all-round guidance such as behavior description, error correction, operation classification, knowledge point push and exercise generation to improve the efficiency and quality of practical operation. Summary of the Invention

[0007] The purpose of the present invention is to provide a multimodal visual analysis system for teaching to solve the problems existing in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention provides a multimodal visual analysis system for teaching, comprising:

[0009] The video acquisition and processing module collects video frames in real time and calculates the difference between two adjacent frames through a dynamic frame difference detection algorithm to determine whether there are dynamic changes in the scene;

[0010] Multi-threaded parallel analysis module, including thread design and collaboration and dynamic load balancing mechanism, to achieve parallel processing of video acquisition and analysis;

[0011] The user interface module displays the captured real-time video frames by designing the interface layout;

[0012] The analysis result output and display module pushes and displays the intermediate results of the multi-threaded parallel analysis module in real time, and sends the final analysis conclusion after the analysis is completed to the user interface module;

[0013] System stability and resource management module, including multi-thread management and synchronization and resource overhead monitoring and optimization;

[0014] The log and error handling module records the program's running information, captures exceptions at the main entrance of the program, records them in the log and takes corresponding measures.

[0015] For video capture, use cv2.VideoCapture to open the camera, continuously read video frames in a loop, and use a buffer mechanism to cache the captured video frames. The buffer acts as a temporary storage area. When the camera captures a video frame, it is first stored in the buffer until it is processed later. This can avoid frame drops caused by brief network fluctuations or hardware failures. At the same time, by setting an appropriate frame rate (such as adding time.sleep0.1) to control the frame rate, it can avoid excessive processing and reduce system burden. In actual applications, the frame rate can be dynamically adjusted according to system performance and requirements.

[0016] Preferably, the dynamic frame difference detection algorithm determines whether there is a dynamic change in the scene by analyzing the difference between two adjacent frames. When the difference exceeds a threshold, it is considered that a dynamic change has occurred in the scene, and the current frame is analyzed, specifically:

[0017] Initialization: Initialize a variable to store the previous frame image, with the initial value set to None; this variable will be used in subsequent frame difference calculations;

[0018] Loop reading of frame data: Continuously read frame data in an infinite loop, and obtain the current frame image in each loop;

[0019] Calculate the frame difference:

[0020] ;

[0021] in, Represents the difference image at position The pixel value at ; and Respectively represent the current frame and the previous frame image at position The pixel value at ;

[0022] Grayscale processing: Convert the obtained difference image into a grayscale image; the purpose of grayscale processing is to simplify the subsequent processing steps. Grayscale images have only one channel and are more convenient to process.

[0023] Binarization: Binarize the grayscale image. Binarization is to divide the pixel values ​​in the image into two categories according to a preset threshold. Pixels greater than the threshold are set to 255 (white), and pixels less than or equal to the threshold are set to 0 (black). It is expressed as:

[0024] ;

[0025] in, Represents the difference image after binarization at position The pixel value at ; Represents the grayscale difference image at position The pixel value at is the preset threshold, which is set to 10% here;

[0026] Calculate the difference percentage:

[0027] ;

[0028] in, is the number of non-zero pixels; is the total number of pixels;

[0029] Determine whether to analyze: The calculated difference percentage is compared with a preset threshold. If the difference percentage exceeds the threshold, the change between the two frames is considered large enough to require further analysis of the current frame. In the code, the current frame is encoded into JPEG format byte data and sent via a signal;

[0030] Update the previous frame: assign the current frame to the variable storing the previous frame image, which will be used as the previous frame in the next loop;

[0031] First frame processing: When the variable storing the previous frame image is None, it indicates that the current frame is the first frame. For the first frame, it is encoded into byte data in JPEG format and sent out through a signal for analysis.

[0032] Preferably, thread design and collaboration are achieved by designing two thread classes, CameraThread and AnalysisWorker. CameraThread is used to collect video frames, continuously read video frames from the camera, and pass them to the AnalysisWorker thread for analysis, specifically:

[0033] Video frame decoding: Convert the video frame received by AnalysisWorker in the form of byte data (frame_data) into a numpy array; use OpenCV's imdecode function to decode the numpy array into an image frame;

[0034] Feature extraction: Use OpenCV's resize function to reduce the resolution of the image frame to the specified size; convert the processed image frame to a PIL image object; save the PIL image object as a JPEG format byte stream and perform base64 encoding;

[0035] Model inference: Call the ollama.generate function, passing in the model name, prompt information, and base64 encoding parameters of the image to start streaming inference. Process each output block in the inference process and check whether it has timed out. If so, throw a timeout exception. Splice each output block into a complete analysis result.

[0036] Through multithreading, we can achieve parallel processing of video acquisition and analysis, improving the overall performance of the system. We use the pyqtSignal signal mechanism to achieve communication between threads and return the analysis results to the main thread for display in a timely manner.

[0037] Preferably, the dynamic load balancing mechanism adopts a dual mechanism of timed forced analysis and frame difference triggering to ensure real-time performance; the timed forced analysis analyzes the current video frame at a fixed time interval (10 seconds), and the analysis operation will be triggered even when there is no obvious change in the video picture, so as to ensure that the system can continuously monitor the video content and avoid missing important information due to the limitations of frame difference detection; the dual mechanism of frame difference triggering is consistent with the calculation method of the dynamic frame difference detection algorithm. In actual applications, the time interval and frame difference threshold of the timed forced analysis can be adjusted according to different scenarios and needs.

[0038] Preferably, the interface layout design uses PyQt5 to build the user interface, and the interface layout is performed through layout managers such as QVBoxLayout and QHBoxLayout to ensure that the interface elements are neatly arranged and beautiful. The interface areas are reasonably divided, such as the video display area, the control button area, and the analysis result display area, to facilitate user operation and information viewing. When designing the interface layout, it is necessary to consider the user's usage habits and operational convenience to avoid an overly complex or cluttered interface; through video display and interaction optimization, the captured video frame is converted into QPixmap format and displayed on a QLabel to achieve real-time video display. To improve the display effect, some preprocessing can be performed on the video frame, such as adjusting brightness and contrast. Corresponding slot functions are added to the buttons. When the user clicks the button, the corresponding operation can be triggered in time, such as starting / stopping the camera, forcing analysis, etc. At the same time, the rendering performance of the interface is optimized to reduce interface freezes. Double buffering technology can be used to prepare the image of the next frame in advance to reduce rendering time.

[0039] Preferably, the analysis result output and display module uses a dual-signal output mechanism, using two different signals to push intermediate results and the final analysis conclusion in real time. The intermediate result signal is generated by the AnalysisWorker thread during model inference. Each intermediate result block is sent via a specific pyqtSignal. This signal carries two key pieces of information: the text box index corresponding to the analysis result and the specific intermediate result content. This allows the user interface to receive these intermediate results in real time and update the display, allowing users to track the progress of the analysis. The final analysis conclusion is the deterministic result obtained by the AnalysisWorker thread after a comprehensive and systematic analysis of the video frames. It is obtained after completing a series of complex operations (such as decoding the video frames, extracting key features, and applying the model for inference). Throughout the analysis process, the model comprehensively considers various information in the video frames and reaches the final conclusion through calculation and judgment. This conclusion is presented as a string, containing a detailed interpretation, analysis, and judgment of the video frame content, such as the operation steps, operation type, potential errors or risks, and questions and answers related to relevant knowledge points. Finally, this conclusion is sent to the user interface via a specific signal and displayed in full, allowing users to clearly understand the key information and analysis results contained in the video frame. The final analysis conclusion signal is sent out through another pyqtSignal, which also carries the text box index and the complete analysis result.

[0040] Optimize result presentation: Format analysis results to present them to users in a clear and understandable manner. This can be presented in a variety of formats, such as text and charts, to enhance readability. Detailed explanations and captions are also provided to help users better understand the results.

[0041] Preferably, locking mechanisms and semaphores are used in multi-thread management and synchronization to control thread access to shared resources, avoiding resource competition and deadlock problems; for example, when operating on shared video frame data, a mutex lock is used to ensure that only one thread can access the data at the same time. At the same time, the life cycle of the thread is reasonably designed, and the thread is destroyed in time when it is not needed to release system resources. Thread pool technology can be used to uniformly manage and schedule threads to improve thread reusability and efficiency. The locking mechanism is used to ensure that there is a thread accessing the shared video frame data at the same time to ensure data accuracy and consistency; the semaphore is used to limit the number of AnalysisWorker threads running at the same time, and at the same time, the use of the buffer is controlled by controlling access to the queue.

[0042] Resource Overhead Monitoring and Optimization: Regularly monitor system resource usage, such as CPU utilization and memory usage. When resource usage exceeds a certain threshold, take appropriate optimization measures, such as reducing unnecessary computing tasks and releasing cached data. At the same time, optimize algorithms and data structures to improve code execution efficiency and reduce resource overhead. For example, use more efficient algorithms for image feature extraction to reduce computational effort and memory usage.

[0043] Preferably, the log and error handling module uses the logging module to configure the log, and records the program's running information in the app.log file for logging. Through logging, problems that occur during the program's operation can be easily checked; by setting different log levels, information of different levels of detail can be recorded according to needs. For example, during the development and debugging phase, the log level can be set to DEBUG to record more detailed information; after the official launch, the log level can be set to INFO to only record key information; the try-except statement is used at the main entrance of the program to capture exceptions. When a serious error occurs in the program, the error information is recorded in the log, and a message box pops up to prompt the user. Different types of exceptions are classified and processed, and corresponding recovery measures are taken, such as restarting threads, releasing resources, etc., to improve the system's fault tolerance. For example, when the camera fails to open, the user can be prompted to check the camera device and try to reopen it.

[0044] Therefore, the present invention adopts the above-mentioned multimodal visual analysis system for teaching, which can perform intelligent analysis on the real-time video stream captured by the camera in the laboratory, help users complete key operations such as circuit welding, component installation, and line debugging, and provide all-round guidance such as behavior description, error correction, operation classification, knowledge point push and exercise generation, thereby improving practical operation efficiency and quality.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic structural diagram of a multimodal visual analysis system for teaching according to the present invention. DETAILED DESCRIPTION

[0047] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0048] See also Figure 1 , a multimodal visual analysis system for teaching, comprising:

[0049] 1. Initialization

[0050] Core functions: Complete environment configuration, resource loading and global parameter initialization before program startup.

[0051] Key components:

[0052] 1. Log Configuration

[0053] Use the logging module to write logs to app.log to record system operation status and error information (such as camera connection failure and model analysis timeout).

[0054] Log level: INFO and above. The format includes timestamp, module name, log level and message.

[0055] 2.Qt environment configuration

[0056] Enable high DPI adaptation , to ensure that the interface displays normally on high-definition screen devices.

[0057] Set Fusion styles and custom color palettes to define the visual style (color, border, font) of interface elements (buttons, text boxes, progress bars).

[0058] 3. Memory optimization (Windows platform)

[0059] Call the system function SetProcessWorkingSetSize through ctypes to limit the program's memory usage and avoid performance issues caused by memory leaks.

[0060] 4. Interaction logic: The program entry (if__name__=='__main__') prioritizes executing the initialization logic to ensure that subsequent modules run in a unified configuration environment.

[0061] 2. Interface Interaction

[0062] Core functions: Build user operation interfaces, process user input (such as starting / stopping cameras), and display real-time video and analysis results.

[0063] Key components:

[0064] 1. Main Window Layout (RealTimeAnalysisTool)

[0065] Left video panel:

[0066] The QFrame area displays the real-time camera image (rendered by a QLabel) and supports adaptive scaling. It also includes control buttons (start / stop the camera) and a status indicator. Click events trigger the start and stop of the camera thread.

[0067] Right analysis panel:

[0068] The grid layout (QGridLayout) generates multiple "analysis cards", each card contains:

[0069] Analysis title (such as "Real-time Operation Analysis"), description (analysis prompt word), progress bar (QProgressBar), result text box (QTextEdit, read-only).

[0070] The scroll area (QScrollArea) ensures that the analysis results can be scrolled when they exceed the height of the interface.

[0071] 2. Status bar: Displays system status (such as "Camera running") and current time (updated every second).

[0072] User interaction logic

[0073] 1. Camera control: Click the button to switch the camera status, and use the toggle_camera method to switch the button text, style, and status indicator color.

[0074] Analysis result display: Through the signal-slot mechanism, the analysis thread's streaming output (analysis_stream) and final result (analysis_complete) are received, and the text box content and progress bar status are updated.

[0075] 2. Data Flow

[0076] The camera thread's new_display_frame signal → _update_video_frame slot function → decodes the frame data and renders it to QLabel.

[0077] Analysis thread's analysis_stream signal → _update_analysis_stream slot function → append analysis results to the text box line by line.

[0078] Analysis thread's analysis_complete signal → _finalize_analysis_output slot function → mark analysis completed and update the progress bar to 100%.

[0079] 3. Processing

[0080] Core functions: Real-time acquisition of industrial camera video streams, pre-processing of frame data (frame difference detection), and sending of valid frames (display frames / analysis frames) through signals.

[0081] Key components:

[0082] 1. Camera Thread Class (CameraThread)

[0083] initialization:

[0084] Supports multiple backends (DSHOW / MSMF / ANY), with DSHOW being preferred to ensure Windows compatibility.

[0085] 2. A frame buffer queue (deque, maximum length 5) is used to avoid memory overflow, and a retry mechanism (up to 3 times) is used to handle camera read failures.

[0086] 3. Core logic (run method):

[0087] Frame acquisition: Loops through camera frames, with a fixed resolution of 640x480 and encoded in JPEG format (quality 70% to reduce network transmission pressure).

[0088] Frame difference detection: Calculates the pixel difference between the current frame and the previous frame. If the difference exceeds 10%, it is marked as a "frame that needs analysis" (to avoid meaningless repeated analysis).

[0089] 4. Signal sending:

[0090] new_display_frame: Send all valid frames for real-time display.

[0091] new_analysis_frame: Send only frames with significant changes or the first frame (triggering analysis logic).

[0092] 5. Exception handling: Capture errors such as camera initialization failure and read timeout, and send an empty frame signal to notify the interface to display the error status.

[0093] 6. Interaction Logic

[0094] The main thread controls the life cycle of the camera thread through the start() / stop() method.

[0095] Frame data is transmitted via binary byte streams (bytes) to avoid memory conflicts caused by direct cross-thread operations on OpenCV arrays.

[0096] 4. Intelligent Analysis

[0097] Core functions: Preprocess the analysis frames sent by the camera, call the Ollama model for multimodal reasoning, and return streaming analysis results, including:

[0098] 1. Use multithreading to achieve parallel analysis;

[0099] 2. Image preprocessing: scaling (160×120) and Base64 encoding;

[0100] 3. Streaming API calls and result processing;

[0101] 4. Timeout control (20 seconds) and automatic retry (up to 3 times).

[0102] The implementation code is as follows:

[0103] class AnalysisWorker(QThread):

[0104] def attempt_analysis(self):

[0105] """Attempt to analyze"""

[0106] Image preprocessing

[0107] buffer = np.frombuffer(self.frame_data, dtype=np.uint8)

[0108] frame = cv2.imdecode(buffer, cv2.IMREAD_COLOR)

[0109] frame = cv2.resize(frame, self.frame_resize)

[0110] base64_img = self._frame_to_base64(frame)

[0111] # Construct the request

[0112] request_data = {

[0113] "model": self.ollama_model,

[0114] "prompt": self.prompt,

[0115] "stream": True,

[0116] "images": [base64_img],

[0117] "options": {"temperature": 0.7, "num_predict": 1024}

[0118] }

[0119] # Send the request and handle the streaming response

[0120] full_response = ""

[0121] response = requests.post(

[0122] f"{self.ollama_server} / api / generate",

[0123] json=request_data,

[0124] stream=True,

[0125] timeout=30 )

[0127] for line in response.iter_lines():

[0128] if self.should_stop:

[0129] break

[0130] chunk = json.loads(line.decode('utf-8'))

[0131] if not chunk.get('done', False):

[0132] response_chunk = chunk.get('response', '')

[0133] full_response += response_chunk

[0134] self.analysis_stream.emit(self.box_index, response_chunk)

[0135] if not self.should_stop:

[0136] self.analysis_complete.emit(self.box_index, full_response)

[0137] Key components:

[0138] 1. Analysis Worker

[0139] Input parameters:

[0140] Frame data (binary byte stream), analysis prompt words (such as "Detect possible errors in the current operation"), and target text box index (corresponding to the analysis card on the right).

[0141] Preprocessing logic:

[0142] The decoded frame data is an OpenCV array, scaled to 160x120 (to reduce the model input resolution and improve inference speed).

[0143] Convert to RGB format and encode to a base64 string using PIL.Image (the image input format that conforms to the Ollama model).

[0144] 2. Model call:

[0145] Use the ollama.generate API, specify the model modelscope.cn / lmstudio-community / MiniCPM-o-2_6-gguf:latest, and enable streaming output (returning results word by word).

[0146] 3. Timeout protection: A single analysis is limited to 30 seconds. If exceeded, the system will retry (up to 3 times).

[0147] 4. Signaling mechanism:

[0148] analysis_start: Sent when analysis starts, triggering the interface to display the timestamp and separator line.

[0149] analysis_stream: Returns the model output chunk by chunk, updating the text box content in real time.

[0150] analysis_complete: Sends the final results when the analysis is complete (success / failure).

[0151] 5. Performance Optimization

[0152] Image compression: The analysis frame quality is set to 70% to reduce the amount of data transmitted over the network and processed by the model.

[0153] Parallel processing: Four analysis tasks (corresponding to four text boxes) run in parallel through independent threads, improving throughput.

[0154] 5. Thread Management

[0155] Core functions: Coordinate the lifecycles of camera threads and analysis threads to ensure safe multi-threaded interaction; trigger forced analysis at regular intervals to avoid missed detections.

[0156] Key components:

[0157] Thread safety mechanism:

[0158] Signal-slot communication: All cross-thread data interactions (such as frame data transmission and analysis result updates) are implemented through PyQt's signals and slots, avoiding thread safety issues caused by directly operating UI components.

[0159] Worker thread list (analysis_workers): records all running analysis threads, terminates and cleans them up in batches when the camera stops to avoid memory leaks.

[0160] Timed forced analysis

[0161] The timer (10-second interval) triggers the _force_analyze_frame method, forcing the use of the latest frame for analysis regardless of whether the frame difference meets the standard.

[0162] Application scenario: Prevent analysis stagnation when there is no significant image change for a long time (such as when a worker continues to perform the same operation).

[0163] Exception handling

[0164] When the camera thread fails to read, it will automatically retry to open the device (within 3 times). If the number of times exceeds, the interface will prompt an error and stop the thread.

[0165] When the analysis thread times out or the model call fails, a retry mechanism (3 times) and error message feedback (such as "Analysis timed out, retry failed") are implemented.

[0166] 6. Auxiliary tools

[0167] Core functions: Provide general tool functions, exception capture and resource release logic to enhance system stability.

[0168] Key components:

[0169] 1. Image Tools

[0170] _frame_to_base64 (AnalysisWorker): Converts an OpenCV frame to a base64 string for recognition by the Ollama model.

[0171] cv2.imencode / cv2.imdecode: Converts frame data between binary byte stream and OpenCV array.

[0172] 2. Resource release

[0173] In the window close event (closeEvent), stop the camera thread, release the camera handle (cap.release()), and destroy the OpenCV window (cv2.destroyAllWindows()).

[0174] After the analysis thread is completed, the finished signal triggers _remove_worker to remove the finished thread from the list.

[0175] 3. Logs and error messages

[0176] Each module records key events (such as "camera thread start" and "analysis timeout") through an independent logger instance to facilitate debugging.

[0177] QMessageBox displays user-level errors (such as "Unable to start camera"), combined with technical-level error logging in the log (exc_info=True).

[0178] The workflow of a multimodal visual analysis system for teaching in this embodiment is as follows:

[0179] 1. System initialization

[0180] The main window loads:

[0181] Create a PyQt5 main window and set the resolution (1360×860).

[0182] Initialize the left video panel (live camera image) and the right analysis panel (4 analysis channels).

[0183] Configure the logging system (log to app.log).

[0184] Set Windows memory optimization (SetProcessWorkingSetSize).

[0185] Camera thread initialization:

[0186] Try multiple backends (CAP_DSHOW > CAP_MSMF > CAP_ANY).

[0187] Set the resolution to 640×480@30fps.

[0188] Initialize the frame difference detection mechanism (cv2.absdiff).

[0189] Initialize the analysis thread pool: Four independent threads handle the following tasks: operation step analysis, operation type classification, potential error detection, and knowledge point question generation. Each thread is bound to a separate signal slot for streaming analysis results.

[0190] 2. Video acquisition process

[0191] Camera startup:

[0192] The user clicks "Start Camera", triggering CameraThread.start().

[0193] The thread enters the loop frame reading mode:

[0194] while self.running:

[0195] ret,frame=self.cap.read()

[0196] If not ret:

[0197] self.retry_count += 1 #Failed retry 3 times

[0198] Frame difference detection (dynamic analysis trigger):

[0199] Calculate the difference between the current frame and the previous frame:

[0200] Python

[0201] diff=cv2.absdiff(prev_frame, current_frame)

[0202] diff_gray = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY)

[0203] _, diff_threshold = cv2.threshold(diff_gray, 30, 255, cv2.THRESH_BINARY)

[0204] diff_percentage = (np.count_nonzero(diff_threshold) / diff_threshold.size) 100

[0205] If diff_percentage > 10%, the analysis is triggered.

[0206] Dual-channel output:

[0207] Display frame (JPEG 70% quality) → sent to UI thread for rendering.

[0208] Analyze frame (downsampled to 160×120 + JPEG compression) → send to AI analysis thread.

[0209] 3. AI Analysis Process

[0210] Analysis task distribution:

[0211] Each frame triggers 4 independent analysis tasks (AnalysisWorker).

[0212] Each task is bound to a different prompt:

[0213] Python

[0214] prompts = [

[0215] "Analyze the current electrical operation steps",

[0216] "Identify the type of operation (wiring / measurement / debugging)",

[0217] "Detect potential errors or risks",

[0218] "Generate test questions on relevant knowledge points" ]

[0220] Model call (Ollama API):

[0221] The MiniCPM-o-2_6 model (lightweight multimodal model) is used.

[0222] Image encoded as Base64 for transmission:

[0223] Python

[0224] img = Image.fromarray(frame)

[0225] buffer = io.BytesIO()

[0226] img.save(buffer, format="JPEG", quality=70)

[0227] base64_img = base64.b64encode(buffer.getvalue()).decode("utf-8")

[0228] Streaming call (stream=True):

[0229] Python

[0230] response = ollama.generate(

[0231] model="MiniCPM-o-2_6",

[0232] prompt=prompt,

[0233] images=[base64_img],

[0234] stream=True,

[0235] options={"timeout": 30} )

[0237] Streaming result update:

[0238] Send analysis snippets to the UI thread in real time:

[0239] Python

[0240] for chunk in response:

[0241] if not chunk['done']:

[0242] self.analysis_stream.emit(box_index, chunk['response'])

[0243] The final result is marked as " Analysis Completed".

[0244] 4. UI rendering and interaction

[0245] Video shows:

[0246] Receive the new_display_frame signal and update the QPixmap:

[0247] Python

[0248] q_img=QImage(rgb_data, w, h, bytes_per_line, QImage.Format_RGB888)

[0249] pixmap=QPixmap.fromImage(q_img).scaled(video_label.size(),Qt.KeepAspectRatio)

[0250] video_label.setPixmap(pixmap)

[0251] The analysis results show:

[0252] Progress bar animation (10%→100%).

[0253] Streaming text rendering (with timestamps):

[0254] Error handling:

[0255] Camera Error → "Camera Error" is displayed (red status bar).

[0256] Analysis timed out → "Analysis timed out" (orange warning) is displayed.

[0257] 5. System Shutdown

[0258] Resource release:

[0259] Stop the camera thread (CameraThread.stop()).

[0260] Terminate all analysis threads (worker.stop()).

[0261] Release OpenCV resources (cv2.destroyAllWindows()).

[0262] Logging:

[0263] Log exit status (app.log).

[0264] Therefore, the present invention adopts the above-mentioned multimodal visual analysis system for teaching to improve the efficiency and quality of practical operation by analyzing real-time video frames.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal visual analysis system for teaching, characterized in that: include: The video acquisition and processing module collects video frames in real time and calculates the difference between two adjacent frames through a dynamic frame difference detection algorithm to determine whether there are dynamic changes in the scene; Multi-threaded parallel analysis module, including thread design and collaboration and dynamic load balancing mechanism, to achieve parallel processing of video acquisition and analysis; The user interface module displays the captured real-time video frames by designing the interface layout; The analysis result output and display module pushes and displays the intermediate results of the multi-threaded parallel analysis module in real time, and sends the final analysis conclusion after the analysis is completed to the user interface module; System stability and resource management module, including multi-thread management and synchronization and resource overhead monitoring and optimization; The logging and error handling module records program operation information, captures exceptions at the main entry point of the program, records them in the log, and takes corresponding measures; The dynamic frame difference detection algorithm determines whether there is a dynamic change in the scene by analyzing the difference between two adjacent frames. When the difference exceeds a threshold, it is considered that a dynamic change has occurred in the scene, and the current frame is analyzed as follows: Initialization: Initialize a variable to store the previous frame image, and set the initial value to None; Loop reading of frame data: Continuously read frame data in an infinite loop, and obtain the current frame image in each loop; Calculate the frame difference: ； in, Represents the difference image at position The pixel value at ; and Respectively represent the current frame and the previous frame image at position The pixel value at ; Grayscale processing: convert the obtained difference image into a grayscale image; Binarization: Binarize the grayscale image and express it as: ； in, Represents the difference image after binarization at position The pixel value at ; Represents the grayscale difference image at position The pixel value at is the preset threshold; Calculate the difference percentage: ； in, is the number of non-zero pixels; is the total number of pixels; Determine whether to analyze: Compare the calculated difference percentage with the preset threshold. If it exceeds the threshold, perform analysis. Update the previous frame: assign the current frame to the variable storing the previous frame image, which will be used as the previous frame in the next loop; First frame processing: When the variable storing the previous frame is None, it indicates that the current frame is the first frame. For the first frame, it is encoded into byte data in JPEG format and sent out through a signal for analysis; Thread design and collaboration By designing two thread classes, CameraThread and AnalysisWorker, CameraThread is used to collect video frames, continuously read video frames from the camera, and pass them to the AnalysisWorker thread for analysis. Specifically: Video frame decoding: Convert the video frames received by AnalysisWorker in the form of byte data into numpy arrays; use OpenCV's imdecode function to decode the numpy arrays into image frames; Feature extraction: Use OpenCV's resize function to reduce the resolution of the image frame to the specified size; convert the processed image frame to a PIL image object; save the PIL image object as a JPEG format byte stream and perform base64 encoding; Model inference: Call the ollama.generate function, pass in the model name, prompt information, and base64 encoding parameters of the image, and start streaming inference; process each output block in the inference process and check whether it has timed out. If so, throw a timeout exception; splice each output block into a complete analysis result; The analysis result output and display module adopts a dual-signal output mechanism, using two different signals to push intermediate results and final analysis conclusions in real time; the intermediate result signal is that the AnalysisWorker thread will continuously generate stage-by-stage intermediate results during the model inference process. Every time an intermediate result block is obtained, it will be sent out through pyqtSignal. This signal carries two key information: one is the text box index corresponding to the analysis result, and the other is the specific intermediate result content; the final analysis conclusion is the result obtained by the AnalysisWorker thread analyzing the video frame. The final analysis conclusion signal sends this final analysis conclusion through another pyqtSignal. Similarly, this signal also carries the text box index and the complete analysis result.

2. The multimodal visual analysis system for teaching according to claim 1, characterized in that: For video capture, use cv2.VideoCapture to open the camera, read video frames in a loop, and use a buffer mechanism to cache the captured video frames. The buffer serves as a temporary storage area waiting for subsequent processing. At the same time, add time.sleep to control the frame rate.

3. The multimodal visual analysis system for teaching according to claim 1, characterized in that: The dynamic load balancing mechanism adopts a dual mechanism of timed forced analysis and frame difference triggering; the timed forced analysis analyzes the current video frame at a fixed time interval; the dual mechanism of frame difference triggering is consistent with the calculation method of the dynamic frame difference detection algorithm.

4. The multimodal visual analysis system for teaching according to claim 1, characterized in that: The interface layout design uses PyQt5 to build the user interface, uses the layout manager to layout the interface, and reasonably divides the interface area; through video display and interaction optimization, the captured video frames are converted into QPixmap format and displayed on QLabel to achieve real-time display of the video.

5. The multimodal visual analysis system for teaching according to claim 1, characterized in that: In multi-thread management and synchronization, lock mechanisms and semaphores are used to control thread access to shared resources. The lock mechanism is used to ensure that at least one thread can access shared video frame data at a time. Semaphores are used to limit the number of AnalysisWorker threads running simultaneously. At the same time, access to the queue is controlled to control the use of the buffer.

6. The multimodal visual analysis system for teaching according to claim 1, characterized in that: The logging and error handling module uses the logging module to configure logging, recording the program's running information in the app.log file. By setting different log levels, different levels of detail can be recorded according to needs. The try-except statement is used at the main entrance of the program to catch exceptions. When a serious error occurs in the program, the error information is recorded in the log and a message box pops up to alert the user.

Images