College student classroom automatic sign-in method and system based on multi-frame focusing enhancement and adaptive panoramic stitching

Abstract

The invention provides a college student classroom automatic sign-in method and system based on multi-frame focusing enhancement and adaptive panoramic stitching, and relates to the field of intelligent classroom management. According to the method, a multi-frame focusing enhancement technology is utilized, multi-frame images with different focal lengths are fused, a clear high-quality image is generated, and the problem of focusing difficulty possibly occurring in a large classroom environment is solved. No matter which area the student sits in, it can be ensured that the system captures high-quality image data, and the recognition accuracy is ensured. Besides, the method introduces an adaptive panoramic stitching technology to cover the whole classroom, and adaptive panoramic stitching is carried out on the images of different blocks, so that large-range image coverage can be realized, even a camera with a small visual angle can obtain a complete classroom panorama through the stitching technology, all students can be ensured to be monitored, and the method is suitable for popularization and application. The problem that the visual angle of a single camera is limited is solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent classroom management, and in particular to an automatic classroom sign-in method and system for college students based on multi-frame focus enhancement and adaptive panoramic stitching. Background Art

[0002] In order to improve classroom management efficiency and promote the construction of learning style, some universities and educational institutions have begun to use intelligent means to conduct student attendance and classroom management. In recent years, sign-in methods based on smart devices have been widely used, including mobile phone sign-in and GPS positioning sign-in.

[0003] Mobile sign-in enables students to sign in independently through mobile phone applications or QR code scanning. It is convenient and efficient, but there are also obvious loopholes. For example, students can easily sign in on behalf of others by sharing QR codes or application login information, thus affecting the authenticity of attendance data. In addition, this method is highly dependent on the normal working state of the smartphone. Once the phone battery is low or the network is not smooth, the sign-in process will be disrupted.

[0004] GPS sign-in obtains the geographic location information of students' mobile phones to confirm whether they have signed in in the classroom. However, in indoor environments, GPS signals often have large positioning errors, making it difficult to accurately determine the students' actual locations, thus affecting the accuracy and reliability of sign-in.

[0005] In order to solve these problems, the face recognition sign-in system based on computer vision technology came into being and became an important exploration direction of current intelligent classroom management.

[0006] Traditional camera surveillance lacks recognition accuracy. Although camera-based face recognition systems have been introduced, most existing systems are limited to fixed viewing angles and areas, and the coverage of cameras is limited, which makes it easy for some students to fail to be recognized. In addition, in large classrooms or when students are widely distributed, recognition accuracy and coverage are limited, making it impossible to achieve comprehensive and effective management. Summary of the invention

[0007] Purpose of the invention: To propose an automatic sign-in method for college students' classroom based on multi-frame focus enhancement and adaptive panoramic stitching, and further propose a college students' classroom behavior recognition and monitoring system to solve the above-mentioned problems existing in the prior art.

[0008] The first aspect of the present invention provides a method for automatic classroom attendance of college students based on multi-frame focus enhancement and adaptive panoramic stitching, and the method comprises the following steps:

[0009] S1. The teacher starts the class and starts the camera. After receiving the start command, the camera enters initialization and turns off the default autofocus function of the camera.

[0010] S2, continuously shooting a number of frames of images in a predetermined block at a preset time interval, each frame of image capturing different details at different focal lengths; performing local analysis on each frame of image to identify the clearest area on each focal plane; and synthesizing the clear areas in the multiple images in the block into a composite image;

[0011] S3, if the field of view of the composite image includes the entire classroom, the current composite image is directly used as the image to be analyzed and the process proceeds to step S5;

[0012] If the composite image cannot cover the entire classroom, re-execute step S1 to step S2, collect composite images of several different blocks at multiple viewing angles, and stitch the composite images of several different blocks into a panoramic image, and proceed to step S4;

[0013] S4, cutting out the edge distortion area or blank area of ​​the stitched panoramic image, and performing noise reduction processing on the image using a Gaussian filter, and outputting the image to be analyzed;

[0014] S5. Based on the image to be analyzed, perform face recognition and automatic sign-in management process, match the identity information of each student, and update the attendance record;

[0015] S6. Visual reminders are given for faces that fail to match and students who fail to sign in, and teachers complete the final sign-in and error correction manually;

[0016] S7, behavior monitoring and analysis, access to various behavior monitoring modules through plug-in methods.

[0017] S8. Combine the behavior recognition results with the attendance data to generate a comprehensive behavior analysis report. The report includes students' behavior performance in class, participation scores, concentration, etc., to help teachers manage the classroom and build a good learning style.

[0018] In a further embodiment of the first aspect, since there is more than one focus point in each block captured by the camera, for example, there are three rows of seats in the block, if the camera autofocus is used directly, it will focus directly on one point, and the sharpness of the image in the non-focus area will be reduced. For this reason, the present application turns off the default autofocus function of the camera and uses manual focusing for multiple times block by block. The focal length of the lens is adjusted using contrast detection focusing technology to ensure that the image captured by the camera has sufficient clarity. During the focusing process, the change in image contrast at different focal lengths is calculated by continuously adjusting the focal length of the lens, and the contrast measurement method uses the Laplace operator to calculate the edge information and high-frequency details of the image.

[0019] The contrast of the image position I(x,y) is transformed by Laplace transform to obtain the contrast response value:

[0020]

[0021] Sum or average C(x,y) over the entire image to get the overall contrast C total :

[0022]

[0023] In the formula, W represents the width of the classroom; H ​​represents the height of the classroom;

[0024] If the contrast change no longer increases significantly after adjusting the focus several times in a row, the system is considered to have reached the best focus state and the search is terminated. That is, when the image contrast corresponding to a certain focal length reaches the maximum, the system determines that the focal length is appropriate and the focus is completed.

[0025] In a further embodiment of the first aspect, the local analysis of the focus area is performed by using the Laplace operator to identify the area, specifically, the convolution kernel corresponding to the Laplace operator is convolved with the input image to obtain the Laplace response image function of each pixel, and the clarity of the image is measured by calculating the sum of the absolute values ​​of the Laplace responses in the image. In order to identify the clear area in the image, the image is divided into an N×M grid. I*J represents the camera resolution; d represents the typical size of the target face; each target object should occupy p% of the pixel area in the grid area. Calculate the clarity scores of multiple frames in each grid, and select the grid image with the highest clarity score as the final image for subsequent image fusion.

[0026] In a further embodiment of the first aspect, the process of calculating the clarity scores of multiple frames of images in each grid specifically includes:

[0027] Convolve the convolution kernel corresponding to the Laplace operator with the input image to obtain the Laplace response image function of each pixel:

[0028] L(x,y)=I(x,y)*K

[0029] Where K is the Laplacian convolution kernel, I(x,y) is the input image;

[0030] The clarity of the image is measured by calculating the sum of the absolute values ​​of the Laplace responses in the image. Let S represent the overall clarity score of the image, that is, S = ∑ x,y |L(x,y)|, the higher the value, the higher the clarity;

[0031] Each grid area is counted as R ij , and the clarity score of each grid area is calculated as The clarity score of each sub-region is used to determine whether the region is on the focal plane, and the grid image with the highest clarity score is selected as the final image for subsequent image fusion.

[0032] In a further embodiment of the first aspect, the process of synthesizing clear areas in multiple images within a block into a composite image includes: calculating pixel clarity values ​​through a Laplace operator, and performing the above-mentioned maximum value selection operation on all pixels. The final generated fused image contains the clearest areas from different frames, ensuring that the optimal focal plane information of each pixel position is retained, thereby improving the clarity and detail performance of the final image.

[0033] Multi-frame images I1(x,y),I2(x,y),…,I n (x,y), corresponding to each region R ij , each region of the final composite image has a selected sharpness score of Right now Thus all sharp areas are merged into one high-quality image.

[0034] In a further embodiment of the first aspect, the process of stitching multiple block images into a panoramic image includes: first, calculating the number of blocks Index based on an algorithm of image geometric characteristics and classroom space layout:

[0035] Install the camera in the center of the classroom at a height of h. c , and the horizontal coverage is calculated And vertical coverage Where H represents the height of the classroom; θ h Represents the horizontal viewing angle of the camera; θ v Indicates the vertical viewing angle of the camera;

[0036] Determine the physical size of each block, which is width high where n h and n v are the number of blocks in the horizontal and vertical directions, respectively;

[0037] According to the pixel requirements and physical size, calculate the number of blocks Index = n h ×n v ,in and and So that eventually Where R x Represents the horizontal pixel of the image; R yrepresents the vertical pixel of the image; L represents the length of the classroom; W represents the width of the classroom;

[0038] After the initial calculation of the number of blocks, adjustments are made based on the actual viewing angle of the camera and the size of the classroom to ensure that the number of divided blocks can fully cover every part of the classroom.

[0039] The system registers continuous frame images through the feature point matching algorithm SIFT algorithm. The feature point matching algorithm identifies the key feature points in the image and calculates their corresponding relationship in different images, thereby achieving geometric alignment of the image; secondly, the image is transformed, and the image is perspective transformed according to the image registration result. The perspective transformation adjusts the perspective of the image so that the straight lines in the image remain straight lines, thereby maintaining geometric consistency during the stitching process; then, through pre-processing, panoramic stitching is performed, and the multi-band stitching algorithm is used for stitching to synthesize the transformed images into a seamless panoramic image. During the stitching process, it is necessary to process the overlapping areas between the images, and use image fusion technology to reduce the visual differences at the seams to generate a continuous and natural panoramic effect.

[0040] In a further embodiment of the first aspect, due to different shooting conditions, there may be differences in color, brightness and exposure between images. The color histogram of the image is standardized through histogram matching to make the color distribution of all images more consistent. Secondly, in the overlapping area of ​​the image, bilateral filtering is used to perform brightness gradient adjustment to gradually adjust the brightness so that it has a smooth transition with the adjacent images.

[0041] Since there may be edge distortion or blank areas in the image during stitching, the final panoramic image needs to be properly cropped. At the same time, a Gaussian filter is used to reduce the noise of the image.

[0042] In a further embodiment of the first aspect, the process of executing the face recognition and automatic sign-in management process is as follows: All face areas are detected from the panoramic image using a face detection algorithm. The face detection algorithm generates a candidate area containing the facial information of each student by identifying the facial features in the image. The second step is to extract features from the detected face area. A deep learning model is used to encode each face and extract its unique facial feature vector. The feature extraction model is based on convolutional neural network training, and by learning a large amount of face data, it can extract high-dimensional facial feature information. The third step is feature comparison, which compares the extracted facial feature vector with the facial feature library in the class database to determine the identity of the student. The system matches the student's identity information by calculating the cosine similarity between the feature vectors.

[0043] The second aspect of the present invention discloses a system for identifying and monitoring classroom behavior of college students, which includes a camera component, an image processing module, an attendance management module, a visual reminder module, and a behavior monitoring and analysis module.

[0044] The camera assembly is pre-installed in the classroom and can form an image within a predetermined field of view.

[0045] The image processing module is used to read the image taken by the camera component, and perform local analysis, image synthesis, and composite image stitching on the image, and finally generate the image to be analyzed.

[0046] The attendance management module performs face recognition and automatic sign-in management processes based on the image to be analyzed, matches the identity information of each student, and updates the attendance record.

[0047] The visual reminder module is used to provide visual reminders for faces that have not been matched and the identities of students who have not signed in. The teacher completes the final sign-in and error correction manually.

[0048] The behavior monitoring and analysis module accesses various behavior monitoring functions in a plug-in manner.

[0049] The image processing module performs the following local analysis process: continuously capture a number of frames of images within a predetermined block at a preset time interval, and each frame of image captures different details at different focal lengths; perform local analysis on each frame of image to identify the clearest area on each focal plane.

[0050] The image processing module performs the image synthesis process as follows: the clear areas in the multiple images in the block are synthesized into a composite image; if the field of view of the composite image includes the entire classroom, the current composite image is directly used as the image to be analyzed.

[0051] The image processing module performs the following process of composite image stitching: the camera collects composite images of several different blocks at multiple viewing angles, and stitches the composite images of several different blocks into a panoramic image. The edge distortion area or blank area of ​​the stitched panoramic image is cropped, and the image is subjected to noise reduction processing using a Gaussian filter, and the image to be analyzed is output.

[0052] In a further embodiment of the second aspect, various behavior monitoring modules are accessed through plug-in means, with standardized interfaces and independent module designs. The modules include sub-processes such as preprocessing, feature extraction, behavior recognition, and result output. The recognition results are passed to the system's main control module in a standardized data format for further processing or recording. Each module is designed as an independent functional unit with no direct dependency on each other.

[0053] The standardized interface defines the standardized protocols [D, C, R] between each module, where D is the data format protocol, including video stream format [compressed video stream encoding, frame rate, resolution], image format [number of bits, number of channels, resolution], data container type, feature data [feature vector format, feature matrix format]; C is the communication protocol, which stipulates the communication model and request response specifications; R is the module hot-swap interface protocol, including module registration and deregistration and heartbeat detection standardized interface.

[0054] Compared with the prior art, the present invention has at least the following beneficial effects:

[0055] (1) The present invention adopts a multi-frame focus enhancement algorithm. In order to ensure that the facial information of all students can be clearly captured in a large classroom environment, the present invention uses multi-frame image acquisition technology combined with an autofocus mechanism to ensure that high-quality facial images can be captured even at long distances or different angles. By acquiring and processing multiple consecutive frames of images, the system can automatically select the clearest part of each frame and generate high-quality images for face recognition.

[0056] (2) The present invention introduces adaptive panoramic stitching technology. In a large classroom, the field of view of a single camera is limited and may not cover all the students' seats. Using adaptive panoramic stitching technology, the system continuously takes multiple pictures and uses an image stitching algorithm to merge these images into a complete panoramic image. The panoramic image can include all the students in the classroom and ensure the clarity of each face area. An algorithm based on the geometric characteristics of the image and the spatial layout of the classroom is designed to calculate the number of stitching blocks Index. Under the premise of ensuring the image quality and clear focus of each block, the camera's viewing angle is reasonably allocated to achieve the best coverage of the entire classroom. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of smart sign-in and behavior monitoring in an embodiment of the present invention.

[0058] Figure 2 This is a behavior monitoring plug-in framework diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0060] To facilitate understanding of the embodiments of the present invention, further explanation will be given below in conjunction with the accompanying drawings, and each of the accompanying drawings does not constitute a limitation on the embodiments of the present invention.

[0061] In order to solve the problem of quick and seamless classroom sign-in and intelligent classroom discipline management for college students and promote the construction of college students' study style, this embodiment discloses a method for classroom face and behavior recognition and monitoring for college students based on multi-frame focus enhancement and adaptive panoramic stitching.

[0062] Figure 1 For smart sign-in and behavior monitoring flow chart, such as Figure 1 As shown, the order of operations is:

[0063] Step S101, start the class and start the camera.

[0064] Specifically, after receiving the startup command, the camera enters the initialization state, turns off the default autofocus function of the camera, and manually adjusts the focal length through an algorithm to achieve the autofocus effect.

[0065] Furthermore, the initialization includes hardware initialization, image acquisition parameter initialization, and module initialization, wherein the hardware initialization includes the hardware detection and firmware loading of the camera, after which the camera will enter a standby state, ready to receive further instructions; image acquisition parameter initialization, during this stage the core acquisition parameters of the camera such as resolution, frame rate, exposure control, white balance adjustment, and focus control are configured; module initialization includes multi-frame focus enhancement module initialization, adaptive panoramic stitching module initialization, and behavior monitoring module initialization.

[0066] Furthermore, the multi-frame focus enhancement module is initialized and a frame buffer is set to store continuously captured image frames.

[0067] Furthermore, the adaptive panoramic stitching module is initialized, and the entire field of view is divided into Index areas according to the viewing angle of the camera and the size of the classroom, that is, the number of blocks Index is calculated based on the algorithm of the geometric characteristics of the image and the spatial layout of the classroom.

[0068] Furthermore, the number of blocks Index, the specific field of view coverage calculation, first preset the relevant parameters, classroom length L, width W and height H, camera parameters including focal length f, horizontal viewing angle θ h , vertical viewing angle θ v , image horizontal pixel R x , image vertical pixel R y , install the camera in the center of the classroom, the camera installation height is h c , from which the horizontal coverage is calculated:

[0069]

[0070] Vertical Coverage:

[0071]

[0072] Determine the physical size of each block, which is width high where n h and n v are the number of blocks in the horizontal and vertical directions respectively.

[0073] Calculate the number of blocks based on pixel requirements and physical size:

[0074] Index = n h ×n v

[0075] in and and

[0076] So that eventually

[0077] After the initial calculation of the number of blocks, adjustments are made based on the actual viewing angle of the camera and the size of the classroom to ensure that the number of divided blocks can fully cover every part of the classroom.

[0078] Furthermore, the autofocus effect, the autofocus module uses contrast detection focus to adjust the focal length of the lens to ensure that the image captured by the camera has sufficient clarity. During the focusing process, the camera captures continuous image frames at the current viewing angle, and the contrast of each image frame is calculated by measuring the edge. The contrast is fed back to the lens drive module according to the size of the contrast, and a small focal length adjustment is made. Once the contrast reaches the highest, the lens stops adjusting, indicating that the focus is locked.

[0079] Furthermore, the edge measurement method uses the Laplace operator to calculate the edge information and high-frequency details of the image. Assuming the image is I(x, y), the contrast calculation can obtain the contrast response value through Laplace transform:

[0080]

[0081] The overall contrast can be obtained by summing or averaging C(x,y) over the entire image:

[0082]

[0083] Step S102: collecting multiple frames of focused images within the block.

[0084] Specifically, in order to identify the clear area in the image, the block image is divided into an N×M grid, and the system continuously shoots several frames of images in the block according to the preset time interval T. In a multi-person scene, since the characters may be in different focal planes, a single focal length may not be able to make all people in the grid clear. The depth of field extension technology is used to increase the focus depth range by adjusting the aperture and reduce the aperture of the camera (increase the f-stop value). This will increase the focal plane depth in the grid captured by the camera, so that people at different distances in the grid are within the clear range. By adjusting the f-stop value, each frame of the image can capture different details at different focal lengths in the grid, thereby completing the acquisition of multiple frames of input images for the grid; further grid focus area identification is performed, and each input image is locally analyzed to identify the clearest area on each grid focal plane.

[0085] Furthermore, the local analysis of the grid focus area is performed by using the Laplace operator to perform area recognition, specifically, the convolution kernel corresponding to the Laplace operator is convolved with the input image to obtain the Laplace response image function of each pixel, that is, L(x, y) = I(x, y) * K, where K is the Laplace convolution kernel, I(x, y) is the input image, and the clarity of the image is measured by calculating the sum of the absolute values ​​of the Laplace responses in the image, and let S represent the overall clarity score of the image, that is, S = ∑ x,y |L(x,y)|, the higher the value, the higher the clarity. Each grid area is counted as R ij , and calculate the clarity score of each grid area as S ij =∑ (x,y)∈Rij |L(x,y)|, the clarity score of each sub-region is used to determine whether the region is on the focal plane, and the grid image with the highest clarity score is selected as the final image for subsequent image fusion.

[0086] Furthermore, the grid N*M is determined based on the camera resolution I*J and the typical size d of the target face. Each target object should occupy p% of the pixel area in the grid area, and the result is:

[0087]

[0088] Step S103: synthesize a composite image within the block.

[0089] Specifically, the regions with the highest clarity scores in the multiple images within the block are synthesized into a high-resolution composite image.

[0090] Furthermore, the synthesis is a high-resolution composite image, and the image reconstruction adopts the maximum fusion method. The pixel sharpness value is calculated by the Laplace operator in S102, and the maximum value selection operation is performed on all pixels. The fused image finally generated contains the clearest areas from different frames, ensuring that the best focal plane information of each pixel position is retained. Multi-frame images I1(x,y),I2(x,y),…,I n (x,y), corresponding to each region R ij , each region of the final composite image has a selected sharpness score of Right now Thus all sharp areas are merged into one high-quality image.

[0091] Step S104: Adaptively stitch multiple blocks into a panoramic image.

[0092] Specifically, in the initialization of step S101, the algorithm based on the geometric characteristics of the image and the layout of the classroom space calculates the number of blocks Index, and the system registers the continuous frame images through the feature point matching algorithm SIFT algorithm. The feature point matching algorithm identifies the key feature points in the image and calculates their correspondence in different images, thereby achieving the geometric alignment of the image; secondly, the image is transformed, and the image is perspective transformed according to the image registration result. The perspective transformation adjusts the viewing angle of the image so that the straight lines in the image remain straight lines, thereby maintaining geometric consistency during the stitching process; then, through pre-processing, panoramic stitching is performed, and the multi-band stitching algorithm is used for stitching to synthesize the transformed images into a seamless panoramic image. During the stitching process, it is necessary to process the overlapping areas between the images, and use image fusion technology to reduce the visual differences at the seams to generate a continuous and natural panoramic effect. The code is as follows:

[0093]

[0094]

[0095] Furthermore, in the pre-processing, due to different shooting conditions, there may be differences in color, brightness and exposure between images. The color histogram of the image is standardized through histogram matching to make the color distribution of all images more consistent. Secondly, in the overlapping area of ​​the image, bilateral filtering is used to perform brightness gradient adjustment to gradually adjust the brightness so that it has a smooth transition with the adjacent images.

[0096] Step S105: splicing optimization.

[0097] Specifically, after the panoramic image stitching is completed, some post-processing is performed to improve the quality of the panoramic image and enhance the viewing effect.

[0098] Furthermore, in the post-processing, since there may be edge distortion or blank areas in the image during stitching, it is necessary to appropriately crop the final panoramic image, and at the same time, use a Gaussian filter to perform noise reduction on the image.

[0099] Step S106: face recognition and automatic sign-in management.

[0100] Specifically, firstly, the panoramic image is spliced ​​by S105 to complete face monitoring, and all face areas are detected from the panoramic image using the face detection algorithm. The face detection algorithm generates a candidate area containing the facial information of each student by identifying the facial features in the image. Secondly, feature extraction is performed on the detected face area. A deep learning model is used to encode each face and extract its unique facial feature vector. The feature extraction model is based on convolutional neural network training. By learning a large amount of face data, it can extract high-dimensional facial feature information. Finally, feature comparison is performed to compare the extracted facial feature vector with the facial feature library in the class database to determine the identity of the student. The system matches the student's identity information by calculating the cosine similarity between the feature vectors.

[0101] Calculate the dot product dot_p of vectors:

[0102]

[0103] Among them, B i is the key feature vector of the students’ faces in the classroom, A i is the i-th key feature vector of the feature library in the comparison library. The norm of the key feature vector is:

[0104]

[0105] The cosine similarity cos_similarity between the face feature vector of the students in the classroom and the key feature vector of the feature library in the library is calculated as:

[0106] Step S107: Check if there is any omission.

[0107] Specifically, the list of missing students is verified by comparing it with the student information list in the database.

[0108] Step S108: Manual intervention by teachers.

[0109] Specifically, if there is any omission in step S107, the teacher will manually intervene to complete the student sign-in. The system will visually remind the unmatched faces and the identities of the students who have not signed in, and the teacher will complete the final sign-in and error correction manually.

[0110] Step S109: behavior monitoring and analysis.

[0111] Specifically, if step S107 is not omitted, or the student sign-in is completed after step S108, further, during the teacher's teaching stage, the student's behavior is monitored through the real-time frame stream, and the monitoring module is connected to various behavior monitoring modules in a plug-in manner.

[0112] Furthermore, various behavior monitoring modules are connected in a plug-in manner, with standardized interfaces and independent module designs. The modules include sub-processes such as preprocessing, feature extraction, behavior recognition, and result output. The recognition results are transmitted to the system's main control module in a standardized data format for further processing or recording. Each module is designed as an independent functional unit with no direct dependency on each other.

[0113] Furthermore, the standardized interface defines standardized protocols [D, C, R] between various modules, where D is the data format protocol, including video stream format [compressed video stream encoding, frame rate, resolution], image format [number of bits, number of channels, resolution], data container type, feature data [feature vector format, feature matrix format]; C is the communication protocol, which stipulates the communication model and request response specifications; R is the module hot-swap interface protocol, including module registration and deregistration and heartbeat detection standardized interface.

[0114] Step S109: Generate a classroom analysis report.

[0115] Specifically, the behavior recognition results are combined with attendance data to generate a comprehensive behavior analysis report, which includes students' behavioral performance in class, participation scores, concentration, etc., to help teachers with classroom management and academic style construction.

[0116] Furthermore, a college student classroom face and behavior recognition monitoring system based on multi-frame focus enhancement and adaptive panoramic stitching, in the embodiment of the present invention, various behavior monitoring algorithms are accessed in a plug-in manner to support the incremental development and integration of the system. This embodiment designs a highly scalable behavior monitoring framework. The framework allows various behavior monitoring algorithms to be accessed in a plug-in manner, thereby supporting the incremental development and integration of the system. In practical applications, different behavior monitoring algorithm models can be gradually embedded in the system according to teaching needs and technological progress. After each algorithm model is embedded, it identifies and classifies specific behavioral features through the combination of deep learning and computer vision technology, and generates corresponding behavior analysis reports. This architectural design not only improves the flexibility and scalability of the system, but also ensures that in future technical iterations, new behavior recognition algorithms can be continuously integrated and optimized to adapt to the ever-changing classroom management needs and achieve accurate monitoring of students' non-learning behaviors, which directly affect classroom participation and teaching effectiveness.

[0117] like Figure 2 As shown, the plug-in core framework includes:

[0118] Step S201: Plug-in loading.

[0119] Specifically, the specified plug-in directory is scanned, the plug-in configuration file is read and the plug-in is dynamically loaded.

[0120] Furthermore, the configuration file describes metadata information of the plug-in, including the name, version, supported behavior types, dependent external libraries, and plug-in entry point of the plug-in, and is persistently stored in the JSON structured data format.

[0121] Furthermore, for the dynamic loading of plugins, the plugin manager first scans the plugin directory and reads the configuration file of each plugin. By parsing the configuration file, the plugin manager obtains the metadata and dependency information of the plugin. Before loading a plugin, the plugin manager needs to check whether the external library that the plugin depends on has been installed. Once the dependency check and installation of the plugin are completed, the plugin manager will dynamically load and execute the plugin code according to the specified fields in the configuration file. The plugin must implement a standardized execute() function interface for the plugin manager to call.

[0122] Step S202: Install / uninstall.

[0123] Specifically, in order to ensure the system's dynamic and scalability, the plugin manager supports dynamic uninstallation, updating, and installation of plugins. When uninstalling a plugin, the system cleans up the resources occupied by the plugin and reloads the new version of the plugin when updating. The plugin update process includes uninstalling the old plugin and loading the new version of the plugin configuration and code.

[0124] Step S203: Execute scheduling.

[0125] Specifically, the scheduling management of plug-ins includes the control of the execution order of plug-ins, as well as concurrency processing, exception handling and resource management. The plug-in execution unit determines the execution order by setting the priority; data transmission between plug-ins is through message queues and shared memory;

[0126] Step S204: interactive communication.

[0127] Specifically, the interactive communication between plug-ins and data flow management transfer data through message queues, which is suitable for asynchronous operations and ensures the reliability and sequentiality of data transfer.

[0128] Step S205: plug-in module.

[0129] Specifically, the plug-in module standardizes the plug-in interface API, data flow management, data preprocessing, and plug-in life cycle management.

[0130] Furthermore, the plug-in standardized interface API defines an init function for initializing the plug-in, an execute function for processing behavior monitoring data, a shutdown function for releasing plug-in resources, and a get_metadatah function for obtaining plug-in metadata.

[0131] Furthermore, the data flow management is to store the image and video data in a unified format according to timestamps and location tags and pass them to the monitoring algorithm.

[0132] Furthermore, the data preprocessing includes image sharpening, denoising, color adjustment, frame processing, etc. Before all data are passed to the plug-in, ensure that their format is consistent with the input format required by the plug-in.

[0133] Furthermore, the plug-in lifecycle management includes the management of various stages of plug-in loading, execution, unloading, updating and resource cleanup during operation, initialization of required resources, and communication interface with the main system.

[0134] In summary, the present invention proposes a method and system for automatic classroom attendance of college students based on multi-frame focus enhancement and adaptive panoramic stitching:

[0135] The system has a high degree of automation. Through technologies such as autofocus, multi-frame focus enhancement, and image stitching, it realizes fully automatic classroom roll call and behavior monitoring, reducing the management burden on teachers. Without excessive human intervention, the system can automatically complete the entire process from image acquisition to analysis after the camera is started, greatly improving management efficiency.

[0136] Multi-frame focus enhancement ensures image clarity. Multi-frame focus enhancement technology is used to fuse multiple frames of images with different focal lengths to generate clear, high-quality images, solving the problem of focusing difficulties that may occur in large classroom environments. No matter where the students sit, the system can capture high-quality image data and ensure the accuracy of recognition.

[0137] Adaptive panoramic stitching technology is introduced to cover the entire classroom. By adaptively stitching images from different blocks, a large range of image coverage can be achieved. Even a camera with a smaller viewing angle can obtain a complete classroom panorama through stitching technology, ensuring that all students can be monitored, overcoming the problem of limited viewing angle of a single camera.

[0138] Accurate face recognition sign-in, combined with multi-frame focus enhancement and panoramic stitching technology, the system can capture clear facial images of each student in real time, and authenticate identity through face recognition technology to achieve accurate and efficient smart sign-in, avoiding human errors and dishonest behavior in traditional sign-in methods and mobile phone sign-in.

[0139] Behavior monitoring has strong scalability. The present invention supports modular access to various behavior monitoring algorithms. Various behavior algorithms are accessed through standardized interfaces. The system can be incrementally developed and expanded as needed to adapt to different scenarios and requirements and meet diverse classroom management needs.

[0140] The hardware requirements are reduced and the cost is low. By optimizing the algorithm and using multi-frame focusing and stitching technology, the present invention can achieve high-quality image acquisition and recognition on relatively low-cost hardware. It does not require high-end camera equipment to achieve intelligent management functions in large-scale classrooms, and has a high cost-effectiveness.

[0141] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for automatic college student classroom attendance based on multi-frame focus enhancement and adaptive panoramic stitching, characterized in that: The steps include: S1. The teacher starts the class and starts the camera. After receiving the start command, the camera enters initialization and turns off the default autofocus function of the camera. S2, continuously shooting a number of frames of images in a predetermined block at a preset time interval, each frame of image capturing different details at a different focal length; Perform local analysis on each frame to identify the clearest area on each focal plane; combine the clear areas in multiple images within the block into a composite image; S3, if the field of view of the composite image includes the entire classroom, the current composite image is directly used as the image to be analyzed and the process proceeds to step S5; If the composite image cannot cover the entire classroom, re-execute step S1 to step S2, collect composite images of several different blocks at multiple viewing angles, and stitch the composite images of several different blocks into a panoramic image, and proceed to step S4; S4, cutting out the edge distortion area or blank area of ​​the stitched panoramic image, and performing noise reduction processing on the image using a Gaussian filter, and outputting the image to be analyzed; S5. Based on the image to be analyzed, perform face recognition and automatic sign-in management process, match the identity information of each student, and update the attendance record; S6. Visual reminders are given for faces that fail to match and students who fail to sign in, and the teacher completes the final sign-in and error correction manually.

2. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 1 is characterized in that: Each frame of the image captures different details at different focal lengths, including: During the focusing process, the focal length of the lens is continuously adjusted to calculate the change in image contrast at different focal lengths; the contrast measurement method uses the Laplace operator to calculate the edge information and high-frequency details of the image; The contrast of the image position I(x,y) is transformed by Laplace transform to obtain the contrast response value: Sum or average C(x,y) over the entire image to get the overall contrast C total : In the formula, W represents the width of the classroom; H ​​represents the height of the classroom; If the contrast change value is within the predetermined range after adjusting the focus several times in succession, the system is considered to have reached the optimal focus state, the search is terminated, and the current focus is locked.

3. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 1 is characterized in that: The local analysis of each frame of the image is performed to identify the clearest area on each focal plane, specifically including: Divide the tile image into an N×M grid: Where I*J represents the camera resolution; d represents the typical size of the target face; each target object should occupy p% of the pixel area in the grid area; The clarity scores of multiple frames in each grid are calculated separately, and the grid image with the highest clarity score is selected as the final image for subsequent image fusion.

4. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 3 is characterized in that: The calculation of the clarity scores of the multiple frames of images in each grid specifically includes: Convolve the convolution kernel corresponding to the Laplace operator with the input image to obtain the Laplace response image function of each pixel: L(x,y)=I(x,y)*K Where K is the Laplacian convolution kernel, I(x,y) is the input image; The clarity of the image is measured by calculating the sum of the absolute values ​​of the Laplace responses in the image. Let S represent the overall clarity score of the image, that is, S = ∑ x,y |L(x,y)|, the higher the value, the higher the clarity; Each grid area is counted as R ij , and the clarity score of each grid area is calculated as The clarity score of each sub-region is used to determine whether the region is on the focal plane, and the grid image with the highest clarity score is selected as the final image for subsequent image fusion.

5. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 4 is characterized in that: The step of synthesizing the clear areas in the multiple images in the block into a composite image specifically includes: Multi-frame images I1(x,y),I2(x,y),…,I n (x,y), corresponding to each region R ij , each region of the final composite image has a selected sharpness score of Right now Thus all sharp areas are merged into one high-quality image.

6. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 1 is characterized in that: The step of stitching the composite images in different blocks into a panoramic image specifically includes: Calculate the number of blocks Index, register consecutive frame images, identify key feature points in the image, and calculate their correspondence in different images to achieve geometric alignment of the image; Perform perspective transformation on the image based on the image registration result; perspective transformation adjusts the viewing angle of the image so that straight lines in the image remain straight lines, thereby maintaining geometric consistency during the stitching process; After pre-processing, panoramic stitching is performed, and a multi-band stitching algorithm is used to stitch the transformed images together to form a seamless panoramic image; The pre-treatment comprises: Normalize the color histogram of the image by histogram matching; In the overlapping area of ​​the image, bilateral filtering is used to adjust the brightness gradient so that it transitions smoothly with the adjacent images.

7. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 6 is characterized in that: The calculation of the block number Index specifically includes: Install the camera in the center of the classroom at a height of h. c , and the horizontal coverage is calculated And vertical coverage Where H represents the height of the classroom; θ h Represents the horizontal viewing angle of the camera; θ v Indicates the vertical viewing angle of the camera; Determine the physical size of each block, which is width high where n h and n v are the number of blocks in the horizontal and vertical directions, respectively; According to the pixel requirements and physical size, calculate the number of blocks Index = n h ×n v ,in and and So that eventually Where R x Represents the horizontal pixel of the image; R y represents the vertical pixel of the image; L represents the length of the classroom; W represents the width of the classroom; After the initial calculation of the number of blocks, adjustments are made based on the actual viewing angle of the camera and the size of the classroom to ensure that the number of divided blocks can fully cover every part of the classroom.

8. The automatic sign-in method for college students in a classroom based on multi-frame focus enhancement and adaptive panoramic stitching according to claim 1 is characterized in that: The facial recognition and automatic sign-in management process is executed to match the identity information of each student, specifically including: By identifying the facial features in the image, a candidate region containing the facial information of each student is generated; Extract features from the detected face area, use a deep learning model to encode each face, and extract its unique facial feature vector; the feature extraction model is based on convolutional neural network training; The extracted facial feature vectors are compared with the facial feature library in the class database to determine the identity of the students, and the identity information of the students is matched by calculating the cosine similarity between the feature vectors.

9. A college student classroom behavior recognition and monitoring system, characterized in that: include: Camera assembly; The camera assembly is pre-installed in the classroom and can form an image within a predetermined field of view; Image processing module; The image processing module is used to read the image captured by the camera assembly, and perform local analysis, image synthesis, and composite image stitching on the image, and finally generate an image to be analyzed; Attendance management module; The attendance management module performs face recognition and automatic sign-in management processes based on the image to be analyzed, matches the identity information of each student, and updates the attendance record; Visual reminder module: The visual reminder module is used to visually remind the faces that have not been matched and the identities of students who have not signed in, and the teacher completes the final sign-in and error correction manually; Behavior monitoring and analysis module; the behavior monitoring and analysis module is connected to various behavior monitoring functions in a plug-in manner; The image processing module can automatically execute the process of steps S2, S3, and S4 as claimed in claim 1.

Images