Intelligent invigilation method based on multi-modal perception

Abstract

This application provides a smart proctoring method based on multimodal perception, relating to the field of image processing technology. The method includes: acquiring continuous image frames of the examination room using a rolling shutter; determining the global periodic brightness component formed by public lighting using spatially separated reference regions; correcting brightness variations in each examination desk area; extracting local periodic bright and dark fringes from local brightness residuals; determining the leakage location and light propagation area based on the peak intensity of the fringes and the outward attenuation range; and then combining the inter-frame phase continuity state, fringe intensity, and the examination desk attribution to the light propagation area to generate a prompt result indicating the presence of a light-emitting display device in the corresponding examination desk area. This application can reduce the interference of artificial lighting fringes on examination desk positioning when the device outline is obstructed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a smart proctoring method based on multimodal perception. Background Technology

[0002] Smart proctoring typically uses fixed camera devices to collect video of the examination room and uses image processing to identify the shape or screen area of ​​electronic devices such as mobile phones and smartwatches. The identified location is then matched with the examination desk area so that proctors can verify the information.

[0003] During the exam, electronic devices may be partially or completely obstructed by objects such as exam papers or hands. The outline of the device and the screen image that the camera can capture decrease as the degree of obstruction increases. Overhead lighting, sunlight, personnel movement, and reflections from the desktop and exam paper can also cause changes in the brightness of the exam area, making it difficult for image features near the obstructed area to stably correspond to the electronic device itself.

[0004] Existing intelligent proctoring methods mainly rely on visible shapes, screen images, or behavioral characteristics. When these characteristics are incomplete due to obstruction, the local brightness changes of the screen light near the obstructed area will be superimposed on the periodic brightness changes of the public lighting. It is difficult to determine the individual examination desk corresponding to the light-emitting display device based on the superimposed image brightness, which easily leads to missed detections or examination desk positioning errors. Summary of the Invention

[0005] This application provides a smart proctoring method based on multimodal perception to solve the problem that when a light-emitting display device is blocked, the local brightness change of the screen light is superimposed with the periodic brightness change of the public lighting, making it difficult to determine the desk where the light-emitting display device is located.

[0006] To address the aforementioned technical problems, this application provides a smart proctoring method based on multimodal perception, comprising: The examination room is captured by a rolling shutter progressive exposure method, and each examination desk area and at least two spatially separated reference areas are determined in the continuous image frames. The global periodic brightness component is determined based on the periodic brightness changes that occur together in each of the reference areas along the line-by-line exposure direction. The brightness change of any of the examination table areas along the line-by-line exposure direction is corrected according to the global periodic brightness component to obtain the local brightness residual of the corresponding examination table area, and local periodic bright and dark stripes are extracted from the local brightness residual. The leakage location is determined based on the peak position of the stripe intensity in the continuous image frames according to the local periodic bright and dark stripes, and the spatial range in which the stripe intensity attenuates outward from the leakage location is determined as the light propagation area. In response to the phase of the local periodic bright and dark stripes remaining continuous in the consecutive image frames, the stripe intensity of the local periodic bright and dark stripes exceeding the non-periodic brightness fluctuation in the local brightness residual, and the light propagation area being located within a single examination table area, a prompt result is generated to indicate the presence of a light-emitting display device within the examination table area where the light propagation area is located.

[0007] Optionally, the step of acquiring continuous image frames of the examination room using a rolling shutter progressive exposure method, and determining each examination desk area and at least two spatially separated reference areas within the continuous image frames, includes: The examination room continuous image frames are acquired with an exposure duration less than the period corresponding to the highest frequency in the preset detection frequency band; Extract the edges of the exam desks in the continuous image frames, and connect the edges of the exam desks belonging to the same exam desk to obtain each exam desk region; select at least two image regions outside each exam desk region that have consistent brightness changes along the line-by-line exposure direction and do not overlap in spatial position to obtain at least two reference regions.

[0008] Optionally, determining the global periodic brightness component based on the periodic brightness changes that occur together in each of the reference regions along the line-by-line exposure direction includes: Based on the pixel row exposure order under the rolling shutter line exposure mode, the brightness change of each reference area along the line exposure direction is converted into a reference row brightness sequence; Based on the pixel row position of each reference region, the brightness sequence of each reference row is phase aligned to obtain the aligned reference row brightness sequence; Extract periodic components with consistent periodicity and phase change relationship from the brightness sequences of each of the alignment reference rows to obtain common periodic components; The common periodic components are fused according to their brightness amplitudes to obtain the global periodic brightness component, and the global periodic brightness component is set to zero when no common periodic component is extracted.

[0009] Optionally, the step of correcting the brightness change of any of the examination table areas along the progressive exposure direction based on the global periodic brightness component to obtain the local brightness residual of the corresponding examination table area, and extracting local periodic bright and dark stripes from the local brightness residual, includes: Select any of the aforementioned exam desk areas as the target exam desk area; Based on the pixel row position of the target exam table area, the global periodic brightness component is phase-aligned and amplitude-matched with the brightness change of the target exam table area along the line-by-line exposure direction to obtain the aligned periodic brightness component; The brightness variation of the target examination table area along the line-by-line exposure direction is corrected based on the alignment period brightness component to obtain the initial local brightness residual. Initial local periodic bright and dark fringes are extracted from the initial local brightness residual, and the initial leakage area is determined based on the position of the peak intensity of the initial local periodic bright and dark fringes. The brightness data covered by the initial leak area is excluded from the brightness change along the line-by-line exposure direction of the target examination table area, and the amplitude of the alignment period brightness component is corrected using the remaining brightness data to obtain the corrected period brightness component. The brightness variation of the target examination table area along the progressive exposure direction is corrected according to the corrected periodic brightness component to obtain the local brightness residual of the target examination table area, and local periodic bright and dark stripes are extracted from the local brightness residual.

[0010] Optionally, determining the location of the leak includes: The continuous image frames are divided into multiple decision time windows consisting of consecutively arranged image frames; Spatially smooth the stripe intensity distribution of the local periodic bright and dark stripes in each image frame within any of the determination time windows to obtain multiple smooth stripe intensity distributions; Based on the spatial distance between the peak positions of each stripe intensity in the multiple smooth stripe intensity distributions, the peak positions of stripe intensity located within the same smooth window coverage area are grouped into at least one stable peak position group, and the center position of each stripe intensity peak position in the stable peak position group with the largest sum of stripe intensity is determined as the leakage position according to the weighted average of the corresponding stripe intensity.

[0011] Optionally, determining the light propagation area includes: Centered on the leak location, the propagation boundaries in each direction are determined based on the range in which the intensity of the local periodic bright and dark stripes continuously decreases along multiple directions as the distance from the leak location increases, and the light propagation area is formed by the enclosing of each of the propagation boundaries.

[0012] Optionally, determining the propagation boundaries in each direction based on the range in which the intensity of the local periodic bright and dark stripes continuously decreases along multiple directions with increasing distance from the leak location includes: The fringe intensity of the local periodic bright and dark fringes is extracted and smoothed along each direction to obtain a smoothed directional intensity sequence. The end of the continuous attenuation sequence segment in the smoothed directional intensity sequence where the fringe intensity decreases continuously and the cumulative decrease reaches a preset attenuation amount is determined as the propagation boundary in the corresponding direction.

[0013] Optionally, determining the inter-frame phase continuity state of the local periodic bright and dark fringes includes: The phase of the local periodic bright and dark stripes is extracted according to the arrangement order of the continuous image frames, and the theoretical phase increment is determined according to the detection frequency of the local periodic bright and dark stripes and the acquisition time interval of adjacent image frames. The minimum difference between the measured phase difference of adjacent image frames and the theoretical phase increment within one phase period is determined to obtain the phase ring residual. When all the phase ring residuals are within the preset phase residual range, it is determined that the phase of the local periodic bright and dark stripes remains continuous in the continuous image frames.

[0014] Optionally, the generation of the prompt result includes: The continuous image frames are divided into multiple decision time windows that are sequentially adjacent and each contains a preset number of image frames. Within each decision time window, the phase continuity state between frames, the stripe intensity of the local periodic bright and dark stripes, and the light propagation area are jointly determined. When the joint determination results of adjacent decision time windows all correspond to the same examination table area, a prompt result is generated to indicate that there is a light-emitting display device in the same examination table area.

[0015] Optionally, the determination that the intensity of the local periodic bright and dark stripes exceeds the non-periodic brightness fluctuation includes: The peak-to-valley brightness difference of the local periodic bright and dark stripe within one stripe period is compared with the peak-to-valley brightness difference of the non-periodic brightness fluctuation within the pixel row range corresponding to the same stripe period of the local brightness residual. When the peak-to-valley brightness difference of the local periodic bright and dark stripe is greater than the peak-to-valley brightness difference of the non-periodic brightness fluctuation, it is determined that the stripe intensity of the local periodic bright and dark stripe exceeds the non-periodic brightness fluctuation.

[0016] The technical solution provided in this application has the following beneficial effects: This application utilizes rolling shutter line-by-line exposure to convert the periodic brightness changes of the light-emitting display device into spatial periodic bright and dark fringes. It then corrects the examination table area using the global periodic brightness component obtained from the reference area, suppressing the common fringes formed by public lighting while preserving the local periodic bright and dark fringes within each individual examination table. By combining the peak fringe intensity, attenuation range, and inter-frame phase continuity, it can reduce mispositioning caused by public lighting and transient reflections when the device outline is obstructed, and generate corresponding prompts for the examination table area. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a smart proctoring method based on multimodal perception, provided for an embodiment of this application; Figure 2 A logic diagram for generating the prompt result of the light-emitting display device provided in the embodiments of this application is presented. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To illustrate the image acquisition and region processing process described below, we will use a fixed examination room used in standardized tests as an example. Multiple rows of desks are set up in the examination room. A rolling shutter camera is installed above or to the side of each desk, capturing images covering the surface of each desk, as well as the area containing the exam paper and hands. After installation, the camera maintains its position, focal length, frame rate, line readout time, and exposure duration, ensuring that the same pixel position in consecutive image frames corresponds to the same examination room location. The image sensor then exposes images sequentially according to a fixed pixel row direction.

[0021] To clarify the image features used in subsequent joint judgment, the multimodal perception in this embodiment refers to the joint perception of temporal, intensity, and spatial features of images from the same source. All three types of features come from the same set of continuous image frames of the examination room acquired by the same rolling shutter camera device. Among them, the inter-frame phase of local periodic bright and dark stripes represents temporal features, stripe intensity represents intensity features, and leakage location and light propagation area represent spatial features. The three types of features correspond to the same examination table area within the same judgment time window and jointly participate in the generation of the prompt result.

[0022] Regarding the above image acquisition conditions and feature relationships, such as Figure 1 As shown, this embodiment provides a smart proctoring method based on multimodal perception, including the following steps: S110 acquires continuous image frames of the examination room using a rolling shutter progressive exposure method, and determines each examination desk area and at least two spatially separated reference areas in the continuous image frames.

[0023] S120 determines the global periodic brightness component based on the periodic brightness changes that occur together in each reference area along the line-by-line exposure direction.

[0024] S130, corrects the brightness change of any examination table area along the line-by-line exposure direction according to the global periodic brightness component, obtains the local brightness residual of the corresponding examination table area, and extracts the local periodic bright and dark stripes from the local brightness residual.

[0025] S140, the leakage location is determined based on the peak position of the local periodic bright and dark stripes in the continuous image frame, and the spatial range of the stripe intensity decaying outward from the leakage location is determined as the light propagation area.

[0026] S150, in response to the phase of the local periodic bright and dark stripes remaining continuous in consecutive image frames, the stripe intensity of the local periodic bright and dark stripes exceeding the non-periodic brightness fluctuation in the local brightness residual, and the light propagation area being located within a single examination table area, a prompt result is generated to indicate the presence of a light-emitting display device within the examination table area where the light propagation area is located.

[0027] To enable comparison of subsequent fringe positions and phases across different image frames, S110 first establishes fixed image acquisition conditions. Rolling shutter line-by-line exposure is an imaging method where the image sensor begins exposure sequentially according to pixel rows, with a fixed exposure time difference between adjacent pixel rows. Continuous examination room image frames are multiple frames of examination room images output by the camera device in chronological order; during a single device judgment period, each image frame uses the same frame rate, line readout time, and exposure duration.

[0028] The progressive exposure method is used because the screen brightness of the light-emitting display device can vary with the refresh cycle or pulse width modulation cycle. When screen light leaks from the edge of the exam paper or between the hands and illuminates the exam table surface, the intensity of the reflected light decreases, but its periodic variation over time can still enter the imaging device. In progressive exposure, different pixel rows begin exposure at staggered times, thus each pixel row records the brightness of the same period of light at different phases. By arranging the brightness values ​​according to the pixel row positions, periodic bright and dark stripes can be formed along the progressive exposure direction in a single frame image, thereby converting the temporal periodic brightness variation into a spatial periodic brightness variation.

[0029] Compared to imaging methods that expose all pixels approximately simultaneously, line-by-line exposure is more suitable for capturing periodic leakage light in this embodiment. When all pixels are exposed approximately simultaneously, each pixel integrates the periodic light within the same exposure time period. Periodic brightness changes easily manifest as approximately synchronous brightness changes within the same illuminated area, and when the exposure time covers multiple brightness change stages, brightness averaging also occurs. If the above periodic changes are analyzed only based on adjacent image frames, the available time sampling positions are limited by the video output frame rate. Line-by-line exposure provides multiple staggered line exposure times within a single image frame. The interval between line exposure times is smaller than the interval between adjacent image frames. Therefore, while maintaining the output frame rate of the examination room monitoring video, the sampling positions for periodic light can be increased, reducing the dependence of periodic leakage light detection on increasing the video output frame rate.

[0030] The spatial fringes formed by line-by-line exposure can also preserve the position of the leaked light on the exam desks. After local reflection, the leaked light from the screen typically only affects a portion of the image area of ​​a single exam desk, while the periodic light from the public lighting can simultaneously affect multiple exam desks and the exam room background. Since each exam desk area and reference area are exposed by the same image sensor in the same pixel row sequence, phase alignment of the periodic brightness changes in different areas can be performed based on the pixel row position. First, the global periodic brightness component commonly contained in each reference area is extracted, and then the local periodic bright and dark fringes retained after correction in the exam desk area are observed. When all pixels are exposed approximately simultaneously, a single frame image lacks the spatial phase sequence formed by the line exposure time difference, making it difficult to directly obtain the fringe positions and phase relationships used for the aforementioned line brightness sequence processing.

[0031] In this embodiment, when the line-by-line exposure method is used, the line readout time determines the exposure time difference between adjacent pixel lines and the number of pixel lines spanned by the same periodic brightness change in the image, while the exposure duration determines the integral degree of a single pixel line with respect to the periodic brightness change. Keeping the line readout time fixed allows periodic light of the same frequency to correspond to stable stripe intervals in consecutive image frames; an exposure duration shorter than the period corresponding to the highest frequency in the preset detection frequency band reduces stripe attenuation caused by the averaging of brightness changes within the same exposure time. With the position of the imaging device, the line-by-line exposure direction, and the acquisition sequence remaining unchanged, the same pixel line corresponds to the same examination room position and the same exposure order in different image frames, thereby providing a unified spatial and temporal reference for subsequent inter-frame phase comparison and light propagation area positioning.

[0032] The line-by-line exposure direction can extend from the top to the bottom of the image. For example, the line readout time can be set from 8μs to 20μs, selected based on the line timing capability of the image sensor. Increasing the line readout time reduces the number of pixel lines corresponding to one screen brightness cycle, thus decreasing the stripe interval; conversely, decreasing the readout time spans more pixel lines. The frame rate can be set from 25 frames / s to 30 frames / s, a range corresponding to the commonly used fixed video output specifications of surveillance cameras, and remains unchanged during continuous judgment.

[0033] The preset detection frequency band limits the periodic brightness changes on the screen that the camera device needs to retrieve, and is the total frequency band set for continuous image frames of the examination room. For example, the preset detection frequency band can be from 100Hz to 1000Hz, and is set according to the refresh rate or pulse width modulation frequency of the display screen to be tested. The highest frequency of 1000Hz corresponds to a 1ms period, and the exposure time can be selected within 0.1ms to 0.8ms and kept less than 1ms; when the examination table image is dark, an exposure time close to the upper limit can be selected to increase the amount of light entering the screen; when the periodic change is significantly averaged during exposure, the exposure time can be adjusted downwards. The effective detection frequency band actually participating in the extraction of periodic components at a certain image location is also limited by the effective pixel rows of the local image block.

[0034] After fixing the acquisition parameters, the exam desk region and reference region are divided from consecutive image frames. The exam desk region corresponds to a single exam seat and includes the exam desk surface, exam paper, and hand images. The exam desk edges can be obtained using existing Canny edge detection; edges belonging to the same exam desk are closed and connected to form each exam desk region.

[0035] Reference regions are used to obtain the periodic brightness variations formed by public lighting in different spatial locations. These can be selected from walls, floors, or fixed surfaces of the podium outside the examination desk areas, and at least two reference regions must not overlap in the image and must be spatially separated. For each candidate region, the periodic component with the largest amplitude within a preset detection frequency band is taken as the principal period. The same principal period means that the frequency difference of corresponding periodic components in the row brightness sequence formed by the candidate regions is no greater than the frequency resolution interval of the row brightness sequence. Candidate regions that satisfy this frequency relationship and exhibit a consistent order of brightness peaks and valleys after alignment are determined as reference regions.

[0036] After the examination table area and the reference area are determined, S120 extracts the global periodic brightness component from the reference area. The global periodic brightness component is the periodic brightness variation commonly contained in the spatially separated reference areas. During processing, the brightness of the same pixel row in each reference area is collected along a direction perpendicular to the line-by-line exposure direction to form a reference row brightness sequence arranged according to the pixel row position; the same pixel row can use average brightness to reduce local differences caused by wall or floor texture.

[0037] Each reference region is located in a different pixel row, and the same common illumination change will be recorded at different exposure times. Based on the starting pixel row position and row readout time of the reference region, the brightness sequences of each reference row are transformed to the same exposure time reference, resulting in aligned reference row brightness sequences. The frequency resolution interval of the reference row brightness sequences is determined by the number of pixel rows contained in the sequence and the row readout time. For example, existing fast Fourier transforms can be used to obtain the periodic component with the largest amplitude within a preset detection frequency band from each aligned reference row brightness sequence; when the frequency difference of the periodic components of multiple reference regions is not greater than the corresponding frequency resolution interval, and the peak-valley order is the same after alignment, a common periodic component is formed.

[0038] After obtaining the common periodic components, they are fused according to their brightness amplitudes to form a global periodic brightness component. During fusion, the ratio of the amplitude of each common periodic component to the sum of the amplitudes of all common periodic components can be used as the fusion ratio, so that reference areas with more sufficient brightness response can have a corresponding impact on the fusion result. When no common periodic components are formed in any reference area, the global periodic brightness component is set to zero, and the global periodic brightness component is not deducted from the subsequent examination table area; instead, local components are directly extracted from the brightness changes of the examination table.

[0039] In S130, the target exam table area is processed in the order of initial calibration, elimination of the influence of the initial leakage area, and recalibration. The target exam table area is any exam table area currently undergoing calibration and determined by the light-emitting display device. To preserve the spatial location of brightness changes within the exam table, the target exam table area is not merged into a single line brightness sequence, but rather local image blocks are set around each image position. The center of the local image block moves gradually within the target exam table area by one pixel position; the local image block extends to both sides along the line-by-line exposure direction to the boundary of the target exam table area or the image boundary, based on the center position, thereby obtaining a continuous line of pixels available at the current position. The brightness of the same pixel line in the local image block is collected along a direction perpendicular to the line-by-line exposure direction to form a local line brightness sequence corresponding to the center position of the local image block.

[0040] The spatial observation duration is the product of the effective pixel rows of a local image patch and the row reading time. The frequency resolution interval of the local row brightness sequence is the reciprocal of this spatial observation duration. To confirm that brightness changes repeat within the local image patch, the local row brightness sequence must contain at least two complete fringe periods. The first fringe period is used to obtain a set of periodic brightness changes, and the second fringe period is used to confirm that the corresponding periodic brightness changes reappear. The lowest frequency that can form two complete fringe periods within the spatial observation duration is determined as the local frequency lower limit. The higher of the local frequency lower limit and the preset detection frequency band lower limit is taken as the effective detection frequency band lower limit, and the effective detection frequency band upper limit is the preset detection frequency band upper limit. When approaching the boundary of the target examination table area, the effective detection frequency band is re-determined based on the cropped effective pixel rows. When the effective detection frequency band lower limit is higher than the effective detection frequency band upper limit, no periodic component detection result is output at the current position.

[0041] For each local image patch, based on the pixel row corresponding to the center position, the global periodic brightness component is shifted to the exposure time corresponding to the local row brightness sequence, and amplitude matching is performed to obtain the aligned periodic brightness component. Amplitude matching can use existing least squares matching to minimize the sum of squares of the brightness differences between the local row brightness sequence and the aligned periodic brightness component in the corresponding pixel row. Subsequently, the aligned periodic brightness component is subtracted from the local row brightness sequence to obtain the initial local brightness residual sequence associated with the center position of the local image patch; all initial local brightness residual sequences together constitute the initial local brightness residual of the target examination table area.

[0042] Periodic components and phases are extracted from the effective detection frequency band of each initial local brightness residual sequence. When the periodic component is obtained through frequency domain amplitude, an inverse transformation is performed on the selected periodic component according to the amplitude normalization method used in spectral analysis to obtain a brightness sequence corresponding to a fringe period. The difference between the maximum and minimum brightness values ​​in this brightness sequence is determined as the fringe intensity. Therefore, subsequent peak localization, attenuation comparison, and intensity determination all use the peak-valley brightness difference to represent the fringe intensity. The phase is used as the fringe phase at the corresponding center position. The fringe intensity and fringe phase are arranged according to the two-dimensional coordinates of each center position in the target examination table area to form the initial fringe intensity distribution and the initial fringe phase distribution. The positions where the fringe intensity exceeds the non-periodic brightness fluctuation within the same local image patch constitute the initial local periodic bright and dark fringes.

[0043] Starting from the initial peak position of the fringe intensity, the region expands to adjacent positions according to the eight-neighbor connectivity relationship. The eight-neighbor region consists of adjacent image positions that share edges or vertices with the current image position. When the frequency difference between the periodic component of the adjacent position and the periodic component of the peak position is not greater than the smaller value of the frequency resolution interval between the two local row brightness sequences, and the fringe intensity of the adjacent position exceeds the non-periodic brightness fluctuation within the same local image block, the adjacent position is incorporated into the initial leakage region until the adjacent position no longer meets the above conditions.

[0044] After the initial leakage area is formed, local image patches whose center positions fall into or intersect with the initial leakage area are excluded, and the local row brightness sequences of the remaining local image patches are used as amplitude correction data. The amplitude scaling factor represents the brightness response ratio of the global periodic brightness component formed by public lighting in the target examination table area. Maintaining the phase correspondence between the global periodic brightness component and the target examination table area, least squares fitting is performed using all amplitude correction data to minimize the sum of squares of the brightness differences between the amplitude correction data and the global periodic brightness component adjusted according to the amplitude scaling factor. This yields an amplitude scaling factor corresponding to the target examination table area. For example, if the peak-to-valley brightness difference of the global periodic brightness component within one cycle is 10 gray levels, and the peak-to-valley brightness difference of the same-frequency periodic component in the amplitude correction data is 6 gray levels, the amplitude scaling factor can be set to 0.6, adjusting the peak-to-valley brightness difference of the corrected periodic brightness component within the corresponding cycle to 6 gray levels. After phase alignment of the corrected periodic brightness component according to the pixel row position of each local image block, it is uniformly applied to all local image blocks within the target examination table area, including the local image blocks within the initial leakage area. Each local image block is then corrected again and the periodic component is extracted. The obtained fringe intensity and fringe phase are mapped to the corresponding center position to form the local brightness residual, two-dimensional fringe intensity distribution, and fringe phase distribution of the target examination table area. The positions where continuous periodic brightness changes constitute local periodic brightness and darkness fringes.

[0045] After recalibration, the local periodic bright and dark fringes have been mapped to the two-dimensional coordinates of the target examination table area. S140 determines the leakage location and light propagation area based on the spatial variation of the fringe intensity. The fringe intensity uses the aforementioned peak-valley brightness difference, and the fringe intensity at the center of each local image block is arranged according to the two-dimensional coordinates to form a fringe intensity distribution, which is used to characterize the change of leaked light from the strong response location to the surrounding area.

[0046] To reduce the impact of individual noise locations on peak locations, spatial smoothing is performed on the stripe intensity distribution in consecutive image frames. Spatial smoothing can employ a moving average window, with the window side length selected based on the image width of a single local periodic bright and dark stripe, and using an odd number of pixels with a side length no greater than the distance between adjacent stripe intensity peaks. This allows the window to weaken isolated noise without merging adjacent stripe intensity peaks.

[0047] After spatial smoothing, the peak positions of fringe intensity are extracted from each image frame. Peak positions falling within the same smoothing window coverage area within the same decision time window form a stable peak position group, which is the set of peak positions that repeatedly appear in the same local area of ​​consecutive image frames. When multiple stable peak position groups exist, the fringe intensities corresponding to the peak positions within each group are accumulated, and the stable peak position group with the largest sum of fringe intensities is selected. The two-dimensional coordinates are then weighted using the fringe intensities of each peak position within the group, and the resulting center position is taken as the leakage position. The leakage position corresponds to the location where periodic light forms the strongest continuous response on the visible reflective surface. It can be located at the edge of the exam paper, on the surface of a hand, or on a desktop, and can be in a different image position from the geometric center of the light-emitting display device.

[0048] Once the leak location is determined, stripe intensities are extracted along multiple directions covering the area surrounding the leak location. The directional intervals are determined based on the boundary spatial resolution to be preserved within the target area. Each directional intensity sequence is smoothed using the aforementioned moving average window to obtain a smoothed directional intensity sequence, ensuring that the leak location and the propagation boundary use the same spatial scale.

[0049] Along each smooth direction, the intensity sequence is compared outward from the leak location with the intensity of adjacent fringes. A continuous attenuation sequence segment is formed when the fringe intensity continuously decreases and the cumulative decrease reaches a preset attenuation amount. The end of this continuous attenuation sequence segment is the propagation boundary for the corresponding direction. The preset attenuation amount is determined by the fringe intensity data of the current decision time window before determining the propagation boundary. Specifically, along each direction, the first position outside the initial leak area where the fringe intensity does not exceed the aperiodic brightness fluctuation within the same local image patch is found and designated as the background reference position. The positive difference between the fringe intensity at the leak location and the fringe intensity at each background reference position is calculated, and the minimum of these positive differences is determined as the preset attenuation amount. Using the minimum positive difference ensures that all effective directions use the same attenuation scale and avoids incorporating brightness changes outside the background reference positions into the light propagation area. Directions where no background reference position is found are not included in the propagation boundary determination.

[0050] Directions that can form a continuous attenuation sequence are considered valid directions, and the propagation boundaries of each valid direction are connected in order. If reflection is interrupted in some directions due to the edges of the exam paper, hands, or the surface material, the corresponding direction is not used to determine the propagation boundary; if the remaining propagation boundaries can enclose a closed area, that closed area is determined as the light propagation region. If all propagation boundaries fall within the same exam table area and do not cross other exam table areas, the light propagation region is determined to be within a single exam table area; if the remaining propagation boundaries cannot form a closed area, the current image frame does not participate in the generation of the prompt result.

[0051] Reference Figure 2The determination logic shown involves establishing the spatial assignment of the light propagation region, followed by S150 checking the temporal continuity and intensity distinguishability of local periodic bright and dark fringes. Within a determination time window, the first image frame forming a closed light propagation region is used as the reference image frame, and the two-dimensional coordinates of the local image block where the leakage location is located and its center pixel row are determined as the fixed phase reference position. Subsequent image frames extract the detection frequency and phase from the local image block corresponding to the fixed phase reference position; if a valid local row brightness sequence cannot be formed at this position, the current determination time window does not output the inter-frame phase continuity state. The detection frequency is the periodic component frequency extracted from the local brightness residual corresponding to the fixed phase reference position, and the acquisition time interval between adjacent image frames is determined by the image frame timestamp. Based on the proportion of the period that the detection frequency traverses within the acquisition time interval, the theoretical phase increment between adjacent image frames is determined.

[0052] The measured phase difference is determined based on the phase extracted from adjacent image frames at a fixed phase reference position, and the measured phase difference and theoretical phase increment are converted to the same phase period. When the two are on opposite sides of the period start point, the smaller phase difference value after bypassing the period start point is taken as the phase ring residual. The phase ring residual represents the deviation of the actual phase advance of local periodic bright and dark fringes from the theoretical phase advance. The phase ring residual corresponding to stable periodic light is close to 0°, so the preset phase residual range is centered at 0° to accommodate image noise and phase extraction errors. For example, when the line reading time, exposure time, and frame rate are consistent with the examination room acquisition conditions, continuous image frames of stable periodic light are acquired, and image frames with image saturation or no periodic components are extracted are excluded. The phase ring residual of the remaining image frames is calculated. When the phase ring residual is distributed between -12° and 10°, the maximum value of the absolute value of the phase ring residual, 12°, can be used as the range boundary, and the preset phase residual range is set to -12° to 12°. If the range is too narrow, normal phase changes caused by imaging noise will be judged as discontinuous; if the range is too wide, the ability to distinguish instantaneous reflections will be reduced. Since each image frame uses the same two-dimensional coordinates and the same center pixel row, the spatial phase difference caused by line-by-line exposure with rolling shutter is not included in the inter-frame phase change. When the phase annular residual of each adjacent image frame is within the preset phase residual range, the inter-frame phase continuity is continuous.

[0053] After confirming the temporal relationship, a fringe period is selected from the local periodic bright and dark fringes. The corresponding pixel row range is determined, and the peak-valley brightness difference of the periodic bright and dark fringes within this range is obtained. The extracted periodic component is subtracted from the same local brightness residual, and then the peak-valley brightness difference of the non-periodic brightness fluctuation within the same pixel row range is obtained. When the former is greater than the latter, the fringe intensity of the local periodic bright and dark fringes exceeds the non-periodic brightness fluctuation. The two peak-valley brightness differences come from the same image frame, the same examination table area, and the same pixel row range, and can be compared under the same brightness unit.

[0054] Since single-frame reflections are accidental, consecutive image frames are divided into multiple sequentially adjacent decision time windows, each containing a preset number of image frames. The preset number is determined based on the fixed video output frame rate of the camera device and the continuous confirmation duration, and is defined as the number of complete image frames that can be captured within the continuous confirmation duration. For example, when the video output frame rate is 25 frames / s and the continuous confirmation duration is 0.2s, the preset number is set to 5 frames. Increasing the preset number extends the continuous confirmation time and reduces the impact of momentary reflections, while decreasing the preset number shortens the time required to generate the prompt result.

[0055] Within each decision window, the inter-frame phase continuity state, the fringe intensity of local periodic bright and dark fringes, and the light propagation area are jointly determined. When the inter-frame phase continuity state is continuous, the fringe intensity exceeds the non-periodic brightness fluctuation, and the light propagation area is located within a single examination table area, the current decision window outputs the joint decision result for the corresponding examination table area. When the joint decision results of adjacent decision windows all correspond to the same examination table area, a prompt result is generated to indicate the presence of a light-emitting display device within that examination table area.

[0056] The following parameter settings are provided to illustrate the relationship between the processed data. A rolling shutter camera is installed above and in front of the exam desk, with the line-by-line exposure direction extending from the top to the bottom of the image. The line readout time is set to 10μs, taken from the fixed line timing of the camera; the frame rate is set to 25 frames / s, and the exposure duration is set to 0.5ms. The preset detection frequency band is set to 100Hz to 1000Hz, with 0.5ms being less than the 1ms period corresponding to the highest frequency. Taking a location within the target exam desk area where 600 consecutive effective pixel lines can be obtained as an example, the spatial observation duration is 6ms; the lower limit of the local frequency, determined by including at least two complete fringe periods, is approximately 333Hz. The effective detection frequency band at this location is 333Hz to 1000Hz, and no detection results are output for periodic components within the 100Hz to 333Hz range. The mobile phone is located below the exam paper, and the leaked light from the edge of the phone can illuminate the desktop of the same exam desk. The wall and the aisle floor serve as two reference areas. The parameters mentioned above are only used to illustrate the connection relationship between the processed data and do not constitute a limitation on the scope of protection of this application.

[0057] After data acquisition begins, reference row brightness sequences are generated from the wall and aisle floor, respectively. The two sequences are then phase-aligned according to pixel row positions. When the frequency difference of corresponding periodic components is no greater than the frequency resolution interval of the reference row brightness sequences, and the peak-to-valley order is consistent, the corresponding periodic components are fused into a global periodic brightness component. Subsequently, the center of the local image patch is gradually moved one pixel at a time within the target exam table area. The corresponding effective detection frequency band is determined based on the number of consecutive pixel rows available at each position. The global periodic brightness component is then moved to the exposure time corresponding to each local row brightness sequence and amplitude matching is performed. Initial correction yields an initial local brightness residual sequence associated with each center position.

[0058] Periodic components and phases are extracted from the effective detection frequency bands of each initial local brightness residual sequence. The peak-valley brightness difference within a fringe period obtained by inverse transformation of the selected periodic components is determined as the fringe intensity. The fringe intensity and phase are then mapped to the center position of the local image patch, thereby forming a two-dimensional initial fringe intensity distribution and an initial fringe phase distribution. When a fringe intensity peak appears near the edge of the exam paper, the position satisfying the frequency resolution and intensity comparison conditions is expanded from the peak position according to the eight-neighbor connectivity relationship, forming the initial leakage region. After excluding the local image patches that intersect with the initial leakage region, an amplitude scaling factor corresponding to the target exam table region is refitted using the remaining local row brightness sequences. The obtained corrected periodic brightness component is applied to all local image patches within the target exam table region, and the two-dimensional fringe intensity distribution is generated again after correction.

[0059] After completing the two-dimensional fringe intensity distribution, if the peak values ​​of the fringe intensity fall within the same smooth window coverage area in consecutive image frames within the same decision time window, a stable peak position group is formed at the corresponding peak positions. The stable peak position group with the largest sum of fringe intensities is selected, and the two-dimensional coordinates are weighted according to the fringe intensity of the peak positions within the group to obtain the leakage position. Subsequently, the fringe intensity decrease process is tracked along multiple directions, and the propagation boundary of each effective direction is determined according to the aforementioned preset attenuation amount; when each propagation boundary can be enclosed and located within the same examination table area, a light propagation region is formed.

[0060] For adjacent image frames, the local image block where the leakage location of the reference image frame within the judgment time window is located is used as a fixed phase reference position. The phase is extracted from the same two-dimensional coordinates and the same center pixel row of subsequent image frames. The theoretical phase increment is determined based on the detection frequency of the local periodic bright and dark stripes at the fixed phase reference position and the image frame timestamp. The phase ring residual between the measured phase difference and the theoretical phase increment is then used to determine the phase continuity state between frames. When the phase ring residual is within a preset phase residual range, the peak-valley brightness difference of the local periodic bright and dark stripes is greater than the peak-valley brightness difference of the non-periodic brightness fluctuation within the same pixel row, and the light propagation area is located within a single examination table area, the current judgment time window outputs the joint judgment result corresponding to the examination table area. When the joint judgment results of adjacent judgment time windows all correspond to the same examination table area, a prompt result is generated to indicate the presence of a light-emitting display device within that examination table area.

[0061] When sunlight intensifies or the gaps in the shading decrease, the peak-to-valley brightness difference of local periodic bright and dark stripes may no longer be greater than the peak-to-valley brightness difference of non-periodic brightness fluctuations, and the current judgment time window will not generate a prompt result. When artificial lighting alone produces stripes, the common periodic component of the wall surface and the aisle floor enters the global periodic brightness component; after the target examination table is corrected, no local light propagation area is formed, and the corresponding judgment time window will not generate a prompt result.

[0062] The above detection is based on the formation of local periodic bright and dark stripes on the visible reflective surface by periodically modulated light entering the camera device. Effective detection requires that the light-emitting display device is in an emitting state, the screen light has periodic brightness changes within a preset detection frequency band, some screen light can enter the camera device through gaps or surface reflection, and the corresponding image area is not continuously saturated.

[0063] When the light-emitting display device has its screen off, the screen light does not have a recognizable periodic change, or the device is completely enclosed by an opaque structure, continuous image frames may not form local periodic bright and dark stripes. Strong sunlight causing image saturation, camera distance resulting in local stripes being below the image resolution, or image defocusing or motion blur may also prevent the generation of prompts. The absence of a prompt cannot be used alone as confirmation that no electronic device exists in the examination area.

[0064] The prompts generated in this embodiment can be reviewed by the invigilator in conjunction with the on-site information.

[0065] The above provides a detailed description of a smart proctoring method based on multimodal perception provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A smart proctoring method based on multimodal perception, characterized in that, include: The examination room is captured by a rolling shutter progressive exposure method, and each examination desk area and at least two spatially separated reference areas are determined in the continuous image frames. The global periodic brightness component is determined based on the periodic brightness changes that occur together in each of the reference areas along the line-by-line exposure direction. The brightness change of any of the examination table areas along the line-by-line exposure direction is corrected according to the global periodic brightness component to obtain the local brightness residual of the corresponding examination table area, and local periodic bright and dark stripes are extracted from the local brightness residual. The leakage location is determined based on the peak position of the stripe intensity in the continuous image frames according to the local periodic bright and dark stripes, and the spatial range in which the stripe intensity attenuates outward from the leakage location is determined as the light propagation area. In response to the phase of the local periodic bright and dark stripes remaining continuous in the consecutive image frames, the stripe intensity of the local periodic bright and dark stripes exceeding the non-periodic brightness fluctuation in the local brightness residual, and the light propagation area being located within a single examination table area, a prompt result is generated to indicate the presence of a light-emitting display device within the examination table area where the light propagation area is located.

2. The method according to claim 1, characterized in that, The process of acquiring continuous image frames of the examination room using a rolling shutter progressive exposure method, and determining each examination desk area and at least two spatially separated reference areas within the continuous image frames, includes: The examination room continuous image frames are acquired with an exposure duration less than the period corresponding to the highest frequency in the preset detection frequency band; Extract the edges of the exam desks in the continuous image frames, and connect the edges of the exam desks belonging to the same exam desk to obtain each exam desk region; select at least two image regions outside each exam desk region that have consistent brightness changes along the line-by-line exposure direction and do not overlap in spatial position to obtain at least two reference regions.

3. The method according to claim 2, characterized in that, The step of determining the global periodic brightness component based on the periodic brightness changes that occur together in each of the reference regions along the line-by-line exposure direction includes: Based on the pixel row exposure order under the rolling shutter line exposure mode, the brightness change of each reference area along the line exposure direction is converted into a reference row brightness sequence; Based on the pixel row position of each reference region, the brightness sequence of each reference row is phase aligned to obtain the aligned reference row brightness sequence; Extract periodic components with consistent periodicity and phase change relationship from the brightness sequences of each of the alignment reference rows to obtain common periodic components; The common periodic components are fused according to their brightness amplitudes to obtain the global periodic brightness component, and the global periodic brightness component is set to zero when no common periodic component is extracted.

4. The method according to claim 3, characterized in that, The step of correcting the brightness change of any of the examination table areas along the progressive exposure direction based on the global periodic brightness component to obtain the local brightness residual of the corresponding examination table area, and extracting local periodic bright and dark stripes from the local brightness residual, includes: Select any of the aforementioned exam desk areas as the target exam desk area; Based on the pixel row position of the target exam table area, the global periodic brightness component is phase-aligned and amplitude-matched with the brightness change of the target exam table area along the line-by-line exposure direction to obtain the aligned periodic brightness component; The brightness variation of the target examination table area along the line-by-line exposure direction is corrected based on the alignment period brightness component to obtain the initial local brightness residual. Initial local periodic bright and dark fringes are extracted from the initial local brightness residual, and the initial leakage area is determined based on the position of the peak intensity of the initial local periodic bright and dark fringes. The brightness data covered by the initial leak area is excluded from the brightness change along the line-by-line exposure direction of the target examination table area, and the amplitude of the alignment period brightness component is corrected using the remaining brightness data to obtain the corrected period brightness component. The brightness variation of the target examination table area along the progressive exposure direction is corrected according to the corrected periodic brightness component to obtain the local brightness residual of the target examination table area, and local periodic bright and dark stripes are extracted from the local brightness residual.

5. The method according to claim 4, characterized in that, The determination of the leak location includes: The continuous image frames are divided into multiple decision time windows consisting of consecutively arranged image frames; Spatially smooth the stripe intensity distribution of the local periodic bright and dark stripes in each image frame within any of the determination time windows to obtain multiple smooth stripe intensity distributions; Based on the spatial distance between the peak positions of each stripe intensity in the multiple smooth stripe intensity distributions, the peak positions of stripe intensity located within the same smooth window coverage area are grouped into at least one stable peak position group, and the center position of each stripe intensity peak position in the stable peak position group with the largest sum of stripe intensity is determined as the leakage position according to the weighted average of the corresponding stripe intensity.

6. The method according to claim 5, characterized in that, The determination of the light propagation region includes: Centered on the leak location, the propagation boundaries in each direction are determined based on the range in which the intensity of the local periodic bright and dark stripes continuously decreases along multiple directions as the distance from the leak location increases, and the light propagation area is formed by the enclosing of each of the propagation boundaries.

7. The method according to claim 6, characterized in that, The method of determining the propagation boundaries in each direction based on the range in which the intensity of the local periodic bright and dark stripes continuously decreases along multiple directions with increasing distance from the leak location includes: The fringe intensity of the local periodic bright and dark fringes is extracted and smoothed along each direction to obtain a smoothed directional intensity sequence. The end of the continuous attenuation sequence segment in the smoothed directional intensity sequence where the fringe intensity decreases continuously and the cumulative decrease reaches a preset attenuation amount is determined as the propagation boundary in the corresponding direction.

8. The method according to claim 4, characterized in that, The determination of the inter-frame phase continuity state of the local periodic bright and dark fringes includes: The phase of the local periodic bright and dark stripes is extracted according to the arrangement order of the continuous image frames, and the theoretical phase increment is determined according to the detection frequency of the local periodic bright and dark stripes and the acquisition time interval of adjacent image frames. The minimum difference between the measured phase difference of adjacent image frames and the theoretical phase increment within one phase period is determined to obtain the phase ring residual. When all the phase ring residuals are within the preset phase residual range, it is determined that the phase of the local periodic bright and dark stripes remains continuous in the continuous image frames.

9. The method according to claim 8, characterized in that, The generation of the prompt result includes: The continuous image frames are divided into multiple decision time windows that are sequentially adjacent and each contains a preset number of image frames. Within each decision time window, the phase continuity state between frames, the stripe intensity of the local periodic bright and dark stripes, and the light propagation area are jointly determined. When the joint determination results of adjacent decision time windows all correspond to the same examination table area, a prompt result is generated to indicate that there is a light-emitting display device in the same examination table area.

10. The method according to claim 4, characterized in that, The determination that the intensity of the local periodic bright and dark fringes exceeds the non-periodic brightness fluctuation includes: The peak-to-valley brightness difference of the local periodic bright and dark stripe within one stripe period is compared with the peak-to-valley brightness difference of the non-periodic brightness fluctuation within the pixel row range corresponding to the same stripe period of the local brightness residual. When the peak-to-valley brightness difference of the local periodic bright and dark stripe is greater than the peak-to-valley brightness difference of the non-periodic brightness fluctuation, it is determined that the stripe intensity of the local periodic bright and dark stripe exceeds the non-periodic brightness fluctuation.

Images