Multi-view paperless online examination cheating behavior identification method

Abstract

The invention discloses a multi-view paperless online examination cheating behavior identification method, which specifically comprises the following steps of: 1, obtaining a facial image of an examinee through a PC screen camera, preprocessing the facial image, and constructing an examinee sight direction identification module by utilizing transfer learning in combination with a SwinTransform network; 2, obtaining an examination desktop environment image through a side overlook angle camera, and constructing a desktop environment article recognition module by utilizing transfer learning and combining with a YOLOX network; 3, obtaining a first visual angle image of the examinee through a head-mounted camera, and constructing an abnormal target recognition module of the examinee under the first visual angle by utilizing transfer learning and combining with YOLOX; and 4, according to the detection result, constructing an online paperless standard examination suspected cheating behavior classification module, judging the suspected cheating type of the examinee in the online examination, and capturing cheating evidence images in three perspectives. According to the invention, the system can automatically detect common examination abnormal behaviors during paperless online examination, provides an objective invigilation basis for invigilators, and reduces the investment of manpower cost.

Description

Technical Field

[0001] The invention relates to a multi-view paperless online examination cheating behavior recognition method, belonging to the technical field of multi-view visual sensor data fusion. Background Art

[0002] Since the end of the 20th century, online education has emerged with the development of technology. In order to maintain the normal operation of education and teaching, schools at different stages of education have adjusted their teaching work to online, which has forced the rapid development of online education and the demand for efficient online examination invigilation has also emerged. However, the development of online examinations has also led to the issue of examination integrity. Therefore, timely, efficient, and automated online examinations as behavior recognition are of great significance to the development of online education.

[0003] At present, the research perspectives of common online paperless examination cheating behavior recognition are relatively single, and are mainly divided into the following two categories. "Single-view cheating behavior detection": in the main view image, after calibration, the line of sight focus is detected, and the facial information is used for identity authentication to realize online examination cheating behavior detection; "Dual-view cheating behavior detection": in the main view image, the line of sight direction is estimated, and in the side view image, the environmental object detection and the examinee's posture estimation are performed, and the examination behavior is judged by combining multi-view information.

[0004] In summary, the existing online examination cheating behavior recognition has the following problems: ① The monitoring range of online examination behavior recognition is imperfect, there are monitoring blind spots, and the visual field image of the examinee's line of sight is missing; ② The online examination cheating behavior classification is not detailed and comprehensive enough, and the behavior judgment is inaccurate. In view of the above problems, the present invention involves proposing an online paperless examination cheating behavior recognition and evidence collection method based on a multi-view visual sensor. Summary of the invention

[0005] In view of the problems existing in the above-mentioned prior art, the present invention provides a method for identifying and collecting evidence of cheating behavior in online paperless examinations based on a multi-view visual sensor, thereby solving the above-mentioned technical problems.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a multi-view paperless online examination cheating behavior identification method, comprising the following steps:

[0007] Step S1: Obtain the examinee's facial image through the PC screen camera (main perspective) and preprocess it. Use transfer learning combined with the SwinTransformer network to build an examinee's line of sight direction recognition module to detect the examinee's abnormal line of sight direction (sneak glance) in the main perspective;

[0008] Step S2: Obtain the test desktop environment image through the side-view camera (side view), use transfer learning combined with the YOLOX network to build a desktop environment object recognition module, and detect common cheating-related items on the desktop;

[0009] Step S3: Obtain the first-person perspective image of the examinee through the head-mounted camera (first-person perspective), use transfer learning combined with the YOLOX network to build an abnormal target recognition module under the examinee's first-person perspective, and detect common cheating-aiding targets;

[0010] Step S4: Based on the detection results of steps S1, S2, and S3, a classification module for suspected cheating behavior in online paperless standard examinations is constructed to identify the types of suspected cheating in online examinations by candidates and capture cheating evidence images from three perspectives.

[0011] Furthermore, the step S1 specifically includes:

[0012] S11: First, collect the examinee's main perspective image based on the PC screen camera, and use the feature points numbered 1, 8, 15, and 20 in the Dlib function HOG algorithm to preprocess the image to obtain the facial image;

[0013] S12: Secondly, a two-dimensional coordinate axis is established with the center of the screen as the origin, and each sight line direction is divided with 45° as the threshold, and the sight line directions include nine directions: upper left, left, lower left, upper, middle, lower, upper right, right, and lower right;

[0014] S13: Finally, transfer learning is used to establish the optimal model for gaze direction recognition based on the SWIN-T pre-trained weights of the Swin Transformer network. The input is the pre-processed facial image, and the output is the confidence of each gaze direction of the image. A gaze direction recognition module for the examinee is constructed to detect the abnormal gaze direction (sneak glance) of the examinee in the main perspective.

[0015] Furthermore, the step S2 specifically includes:

[0016] S21: First, we collect a dataset of abnormal items in the desktop environment based on a side-view camera, which includes three common abnormal items in online paperless exams: books, papers, and electronic devices.

[0017] S22: Secondly, transfer learning is used to establish the optimal model for detecting abnormal items on the desktop based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of each abnormal item, and a desktop environment item recognition module is built to detect common exam cheating-related items on the desktop.

[0018] Furthermore, the step S3 specifically includes:

[0019] S31: First, the candidate's first-person perspective data collection hardware is based on a self-assembled head-mounted bracket and a USB camera. The first-person perspective abnormal target dataset is obtained based on the head-mounted camera, which includes four types of online paperless examination targets: people, books, paper, and electronic devices;

[0020] S32: Secondly, transfer learning is used to establish the optimal model for first-person target detection based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of the target, and an abnormal target recognition module is constructed under the first-person perspective of the candidate to detect common targets that assist cheating.

[0021] Furthermore, the step S4 specifically includes:

[0022] S41: First, perform MAX-MIN normalization based on the obtained line of sight direction detection confidence, the specific method is as follows:

[0023]

[0024] Among them, p θi It is the confidence (probability) of the nine directions of the examinee's sight under the main viewing angle. θmax is the maximum confidence among the 9 sight directions, p θmin is the minimum confidence (probability) among the 9 sight directions. After normalizing the image, the confidence value x of the i-th sight direction i Between (0, 1);

[0025] S42: Secondly, using the output of the examinee's line of sight direction recognition module described in step S1 and the output of the desktop environment object recognition module described in step S2 as input, a "main perspective-side perspective" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed.

[0026] S43: Again, using the output of the examinee's line of sight direction recognition module described in step S1 and the output of the examinee's first-person perspective abnormal target recognition module described in step S3 as input, a examinee's "main perspective-first perspective" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed.

[0027] S44: Finally, based on the output results of the above two multi-layer perceptrons, a classification module for suspected cheating behaviors in online paperless standard examinations was constructed to identify 13 common suspected cheating behaviors of candidates in online examinations and capture cheating evidence images from three perspectives.

[0028] Furthermore, the step S42 specifically includes:

[0029] S42-1 is a "main perspective-side perspective" joint cheating behavior discrimination network based on a multi-layer perceptron. The network includes an input layer, a hidden layer, and an output layer.

[0030] S42-2 has 12 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41. They include: the probability of the upper left field of view x1, the probability of the left field of view x2, the probability of the lower left field of view x3, the probability of the upper field of view x4, the probability of the middle field of view x5, the probability of the lower field of view x6, the probability of the upper right field of view x7, the probability of the right field of view x8, and the probability of the lower right field of view x9. The last 3 neurons are the three confidences of abnormal items output by step S2, including: the confidence of the book x1, the probability of the book x2, the probability of the book x3, the probability of the book x4, the probability of the book x5, the probability of the book x6, the probability of the book x7, the probability of the book x8, and the probability of the book x9. 10 , Paper confidence x 11 , electronic equipment confidence x 12 ;

[0031] The number of neurons in the single hidden layer of S42-3 is the same as that in the output layer, that is, it includes 12 neurons.

[0032] S42-4 has 9 output layer neurons, representing 9 cheating behaviors near the desktop, including looking at the paper on the desktop and below y1, looking at the books on the desktop and below y2, looking at the electronic devices on the desktop and below y3, looking at the paper behind the computer y4, looking at the books behind the computer y5, looking at the electronic devices behind the computer y6, looking at the desktop and below for multiple illegal items y7, looking at the back of the computer for multiple illegal items y8, and normal side view behavior y9;

[0033] S42-5 uses the error back propagation method to train and obtain the optimal weight value of the three-layer perceptron;

[0034] Furthermore, the step S43 specifically includes:

[0035] S43-1 is a "main perspective-first perspective" joint cheating behavior discrimination network based on a multi-layer perceptron, which consists of three layers: input layer, hidden layer and output layer;

[0036] S43-2 has 13 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41. They include: the probability of the upper left field of view x1, the probability of the left field of view x2, the probability of the lower left field of view x3, the probability of the upper field of view x4, the probability of the middle field of view x5, the probability of the lower field of view x6, the probability of the upper right field of view x7, the probability of the right field of view x8, and the probability of the lower right field of view x9. The last 4 neurons are the four abnormal target confidences output by step S3, including: book confidence x1, book confidence x2, book confidence x3, book confidence x4, book confidence x5, book confidence x6, book confidence x7, book confidence x8, book confidence x9. 10 , Paper confidence x 11 , electronic equipment confidence x 12 , person confidence x 13 ;

[0037] The number of neurons in the single hidden layer of S43-3 is the same as that in the output layer, i.e., it includes 13 neurons;

[0038] S43-4 has 4 output layer neurons, representing 4 types of cheating behaviors near the table, including helping cheaters to lift books y1, helping cheaters to lift papers y2, helping cheaters to lift electronic devices y3, and the first-person perspective of the examinee is normal y4;

[0039] S43-5 uses the error back propagation method to train and obtain the optimal weight values ​​of the three-layer perceptron.

[0040] The beneficial effects of the present invention are as follows: by using the method of the present invention, (1) the areas where cheating may occur in the exam are covered by using a PC screen camera, a desktop overhead camera, and a wearable camera on the examinee's head; (2) an online cheating determination classification network is constructed to perform detailed classification and identification of different types of cheating behaviors; and (3) real-time online capture and evidence collection is performed to provide effective intelligent proctoring technical support for online paperless exams. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0042] Figure 2 (a) is a schematic diagram of the main viewing angle image before Dlib processing of the present invention, Figure 2 (b) is a schematic diagram of the main viewing angle image after Dlib processing of the present invention;

[0043] Figure 3 The SwinTransformer network diagram of the present invention;

[0044] Figure 4 A schematic diagram of the YOLOX network of the present invention;

[0045] Figure 5 It is a schematic diagram of a module for classifying suspected cheating behaviors in a paperless standard examination of the present invention;

[0046] Figure 6 A schematic diagram of placing the three-viewing angle vision sensor of the present invention;

[0047] Figure 7 (a) is a schematic diagram of the structure of the "side-view cheating behavior identification network" of the present invention, Figure 7 (b) is a schematic diagram of the structure of the “candidate first-person perspective cheating behavior identification network” of the present invention;

[0048] Figure 8The present invention is a schematic diagram of an example of capturing and collecting evidence of abnormal behavior from three perspectives, wherein (a) is the appearance of paper and the person is looking at the desktop and below the desktop; (b) is the appearance of paper and the person is looking at the desktop and below the desktop; (c) is the appearance of a book and the person is looking at the desktop and below the desktop; (d) is the appearance of an electronic device and the person is looking at the desktop and below the desktop; (e) is the appearance of a book and a collection and the person is looking behind the computer; (f) is the appearance of paper and the person is looking behind the computer; (g) is the appearance of a book and the person is looking behind the computer; (h) is the appearance of a book and paper and the person is looking behind the computer; (i) is the appearance of an electronic device with the assistance of a third party; (j) is the appearance of a book with the assistance of a third party; and (k) is the appearance of a paper with the assistance of a third party. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0051] like Figure 1 As shown, the present invention is a multi-view paperless online examination cheating behavior identification method, comprising the following steps:

[0052] Step S1: Obtain the examinee's facial image through the PC screen camera (main perspective) and preprocess it. Use transfer learning combined with the SwinTransformer network to build an examinee's line of sight direction recognition module to detect the examinee's abnormal line of sight direction (sneak glance) in the main perspective;

[0053] Specifically, S11: first, collect the examinee's main perspective image based on the PC screen camera, and use the feature points numbered 1, 8, 15, and 20 in the Dlib function HOG algorithm to preprocess the image to obtain a facial image;

[0054] S12: Secondly, a two-dimensional coordinate axis is established with the center of the screen as the origin, and each sight line direction is divided with 45° as the threshold, and the sight line directions include nine directions: upper left, left, lower left, upper, middle, lower, upper right, right, and lower right;

[0055] S13: Finally, transfer learning is used to establish the optimal model for gaze direction recognition based on the SWIN-T pre-trained weights of the Swin Transformer network. The input is the pre-processed facial image, and the output is the confidence of each gaze direction of the image. The candidate's gaze direction recognition module is constructed to detect the abnormal gaze direction (sneak glance) of the candidate in the main perspective.

[0056] Step S2: Obtain the test desktop environment image through the side-view camera (side view), use transfer learning combined with the YOLOX network to build a desktop environment object recognition module, and detect common cheating-related items on the desktop;

[0057] Specifically, S21: First, a dataset of abnormal items in the desktop environment is collected based on a side-view camera, which includes three common abnormal items for online paperless examinations: books, papers, and electronic devices;

[0058] S22: Secondly, transfer learning is used to establish the optimal model for detecting abnormal items on the desktop based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of each abnormal item, and a desktop environment item recognition module is built to detect common exam cheating-related items on the desktop.

[0059] Step S3: Obtain the first-person perspective image of the examinee through the head-mounted camera (first-person perspective), use transfer learning combined with the YOLOX network to build an abnormal target recognition module under the examinee's first-person perspective, and detect common cheating-aiding targets;

[0060] Specifically, S31: First, the candidate's first-person perspective data collection hardware is based on a self-assembled head-mounted bracket and a USB camera, and a first-person perspective abnormal target data set is obtained based on the head-mounted camera, which includes four types of online paperless examination targets: people, books, paper, and electronic devices;

[0061] S32: Secondly, transfer learning is used to establish the optimal model for first-person target detection based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of the target, and an abnormal target recognition module is constructed under the first-person perspective of the candidate to detect common targets that assist cheating.

[0062] Step S4: Based on the detection results of steps S1, S2, and S3, a classification module for suspected cheating behavior in online paperless standard examinations is constructed to identify the types of suspected cheating in online examinations by candidates and capture cheating evidence images from three perspectives.

[0063] Specifically, the step S4 includes:

[0064] S41: First, perform MAX-MIN normalization based on the obtained line of sight direction detection confidence, the specific method is as follows:

[0065]

[0066] Among them, p θi It is the confidence (probability) of the nine directions of the examinee's sight under the main viewing angle. θmax is the maximum confidence among the 9 sight directions, p θmin is the minimum confidence (probability) among the 9 sight directions. After normalizing the image, the confidence value x of the i-th sight direction i Between (0, 1);

[0067] S42: Secondly, using the output of the examinee's line of sight direction recognition module described in step S1 and the output of the desktop environment object recognition module described in step S2 as input, a "main perspective-side perspective" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed.

[0068] S43: Again, using the output of the examinee's line of sight direction recognition module described in step S1 and the output of the examinee's first-person perspective abnormal target recognition module described in step S3 as input, a examinee's "main perspective-first perspective" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed.

[0069] S44: Finally, based on the output results of the above two multi-layer perceptrons, a classification module for suspected cheating behaviors in online paperless standard examinations was constructed to identify 13 common suspected cheating behaviors of candidates in online examinations and capture cheating evidence images from three perspectives.

[0070] In this embodiment, preferably, step S42 is as follows:

[0071] S42-1 is a "main perspective-side perspective" joint cheating behavior discrimination network based on a multi-layer perceptron. The network includes an input layer, a hidden layer, and an output layer.

[0072] S42-2 has 12 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41. They include: the probability of the upper left field of view x1, the probability of the left field of view x2, the probability of the lower left field of view x3, the probability of the upper field of view x4, the probability of the middle field of view x5, the probability of the lower field of view x6, the probability of the upper right field of view x7, the probability of the right field of view x8, and the probability of the lower right field of view x9. The last 3 neurons are the three confidences of abnormal items output by step S2, including: the confidence of the book x1, the probability of the book x2, the probability of the book x3, the probability of the book x4, the probability of the book x5, the probability of the book x6, the probability of the book x7, the probability of the book x8, and the probability of the book x9. 10 , Paper confidence x 11 , electronic equipment confidence x 12 ;

[0073] The number of neurons in the single hidden layer of S42-3 is the same as that in the output layer, that is, it includes 12 neurons.

[0074] S42-4 has 9 output layer neurons, representing 9 cheating behaviors near the desktop, including looking at the paper on the desktop and below y1, looking at the books on the desktop and below y2, looking at the electronic devices on the desktop and below y3, looking at the paper behind the computer y4, looking at the books behind the computer y5, looking at the electronic devices behind the computer y6, looking at the desktop and below for multiple illegal items y7, looking at the back of the computer for multiple illegal items y8, and normal side view behavior y9;

[0075] S42-5 uses the error back propagation method to train and obtain the optimal weight value of the three-layer perceptron;

[0076] In this embodiment, preferably, step S43 is as follows:

[0077] S43-1 is a "main perspective-first perspective" joint cheating behavior discrimination network based on a multi-layer perceptron, which consists of three layers: input layer, hidden layer and output layer;

[0078] S43-2 has 13 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41. They include: the probability of the upper left field of view x1, the probability of the left field of view x2, the probability of the lower left field of view x3, the probability of the upper field of view x4, the probability of the middle field of view x5, the probability of the lower field of view x6, the probability of the upper right field of view x7, the probability of the right field of view x8, and the probability of the lower right field of view x9. The last 4 neurons are the four abnormal target confidences output by step S3, including: book confidence x1, book confidence x2, book confidence x3, book confidence x4, book confidence x5, book confidence x6, book confidence x7, book confidence x8, book confidence x9. 10 , Paper confidence x 11 , electronic equipment confidence x 12 , person confidence x 13 ;

[0079] The number of neurons in the single hidden layer of S43-3 is the same as that in the output layer, that is, it includes 13 neurons.

[0080] S43-4 has 4 output layer neurons, representing 4 types of cheating behaviors near the table, including helping cheaters to lift books y1, helping cheaters to lift papers y2, helping cheaters to lift electronic devices y3, and the first-person perspective of the examinee is normal y4;

[0081] S43-5 uses the error back propagation method to train and obtain the optimal weight value of the three-layer perceptron;

[0082] Example:

[0083] Step 1) Obtain the candidate's facial image through the PC screen camera (main perspective) and pre-process it. Use transfer learning combined with the SwinTransformer network to build a candidate's line of sight direction recognition module to detect the candidate's abnormal line of sight (sneak glance) in the main perspective; the specific content is:

[0084] First, collect the examinee's main perspective image based on the PC screen camera, such as Figure 2 As shown in (a), the feature points numbered 1, 8, 15, and 20 in the Dlib function HOG algorithm are used to preprocess the image to obtain a facial image, as shown in Figure 2 (b)

[0085] Secondly, a two-dimensional coordinate axis is established with the center of the screen as the origin, and each sight line direction is divided with 45° as the threshold. The sight lines include nine directions: upper left, left, lower left, upper, middle, lower, upper right, right, and lower right.

[0086] Finally, transfer learning is used to establish the optimal model for gaze direction recognition based on the SWIN-T pre-trained weights of the Swin Transformer network. Figure 3 As shown, the input is the preprocessed facial image, the output is the confidence of each gaze direction of the image, and a candidate's gaze direction recognition module is constructed to detect the abnormal gaze direction (sneak glance) of the candidate in the main perspective.

[0087] Step 2) Obtain the image of the test desktop environment through the side-view camera (side view), use transfer learning combined with the YOLOX network to build a desktop environment object recognition module, and detect common cheating-related items on the desktop; the specific content is:

[0088] Firstly, we collect a dataset of abnormal objects in the desktop environment based on a side-view camera, which includes three common abnormal objects in online paperless examinations: books, papers, and electronic devices.

[0089] Secondly, transfer learning is used to establish the optimal model for detecting abnormal items on the desktop based on the YOLOX-s pre-trained weights of YOLOX. Figure 4 As shown, the output is the bounding box and confidence score of each abnormal object, and a desktop environment object recognition module is built to detect common exam cheating-related items on the desktop.

[0090] Step 3) Obtain the first-person perspective image of the examinee through the head-mounted camera (first-person perspective), use transfer learning to build an abnormal target recognition module under the examinee's first-person perspective, and detect common cheating-aiding targets; the specific content is:

[0091] First, the candidate's first-person perspective data collection hardware is based on a self-assembled head-mounted bracket and a USB camera. The head-mounted camera is used to obtain a first-person perspective abnormal target dataset, which includes four types of online paperless test targets: people, books, paper, and electronic devices.

[0092] Secondly, transfer learning is used to establish the optimal model for first-person target detection based on the YOLOX-s pre-trained weights of YOLOX. The output is the bounding box and confidence of the target. An abnormal target recognition module from the first-person perspective of the candidate is constructed to detect common targets that assist cheating.

[0093] Step 4) Based on the detection results of steps 1, 2, and 3, a classification module for suspected cheating behavior in online paperless standard examinations is constructed, such as Figure 5 As shown in the figure, the type of suspected cheating in the online exam is determined, and the cheating evidence images under three perspectives are captured, such as Figure 6 The specific contents are as follows:

[0094] First, perform MAX-MIN normalization based on the obtained line of sight direction confidence.

[0095] Secondly, a side-view cheating behavior discrimination network based on a multi-layer perceptron is constructed, such as Figure 7 As shown in (a), it includes input layer, hidden layer and output layer. There are 12 neurons in the input layer, including: probability of upper left visual field x1, probability of left visual field x2, probability of lower left visual field x3, probability of upper visual field x4, probability of middle visual field x5, probability of lower visual field x6, probability of upper right visual field x7, probability of right visual field x8, probability of lower right visual field x9, book confidence x1, probability of upper left visual field x2, probability of lower left visual field x3, probability of upper right visual field x4, probability of middle visual field x5, probability of lower visual field x6, probability of upper right visual field x7, probability of right visual field x8, probability of lower right visual field x9, book confidence x1, probability of upper left visual field x1, probability of upper right visual field x2, probability of lower left visual field x3, probability of upper left visual field x4, probability of middle visual field x5, probability of lower visual field x6, probability of upper right visual field x7, probability of right visual field x8, probability 10 , Paper confidence x 11 , electronic equipment confidence x 12 ; There are 9 neurons in the output layer, including, looking at the paper on the table and below y1, looking at the book on the table and below y2, looking at the electronic device on the table and below y3, looking at the paper behind the computer y4, looking at the book behind the computer y5, looking at the electronic device behind the computer y6, looking at the table and below for multiple illegal items y7, looking at the back of the computer for multiple illegal items y8, and normal behavior from the side perspective y9; training to obtain the optimal weight value;

[0096] Secondly, a first-person perspective cheating behavior discrimination network based on a multi-layer perceptron is constructed, such as Figure 7As shown in (b), it consists of three layers: input layer, hidden layer and output layer. There are 13 neurons in the input layer, including: probability of upper left visual field x1, probability of left visual field x2, probability of lower left visual field x3, probability of upper visual field x4, probability of middle visual field x5, probability of lower visual field x6, probability of upper right visual field x7, probability of right visual field x8, probability of lower right visual field x9, book confidence x1, probability of upper left visual field x2, probability of lower left visual field x3, probability of upper right visual field x4, probability of middle visual field x5, probability of lower visual field x6, probability of upper right visual field x7, probability of right visual field x8, probability of lower right visual field x9, book confidence x1, probability of upper left visual field x1, probability of lower left visual field x1, probability of upper right visual field x2, probability of lower right visual field x3, probability of upper left visual field x4, probability of upper right visual field x5, probability of lower right visual field x6, probability of upper 10 , Paper confidence x 11 , electronic equipment confidence x 12 , person confidence x 13 ; There are 4 neurons in the output layer, including, assisting the cheater to lift the book y1, assisting the cheater to lift the paper y2, assisting the cheater to lift the electronic device y3, and the examinee's first-person perspective behavior is normal y4; training to obtain the optimal weight value;

[0097] Finally, based on the output results of the two multi-layer perceptrons, a classification module for suspected cheating behavior in online paperless standard examinations was constructed to identify the types of suspected cheating in online examinations by candidates, capture cheating evidence from three perspectives, and capture forensic images of common abnormal behavior types. Figure 8 shown.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-view paperless online examination cheating behavior recognition method, characterized in that: The steps include: Step S1: Obtain the examinee's facial image through the PC screen camera (main perspective) and pre-process it. Use transfer learning combined with the SwinTransformer network to build an examinee's line of sight direction recognition module to detect the examinee's abnormal line of sight direction (sneak glance) under the main perspective; Step S2: Obtain the test desktop environment image through the side-view camera (side view), use transfer learning combined with the YOLOX network to build a desktop environment object recognition module, and detect common cheating-related items on the desktop; Step S3: Obtain the first-person perspective image of the examinee through the head-mounted camera (first-person perspective), use transfer learning combined with the YOLOX network to build an abnormal target recognition module under the examinee's first-person perspective, and detect common cheating-aiding targets; Step S4: Based on the detection results of steps S1, S2, and S3, a classification module for suspected cheating behavior in online paperless standard examinations is constructed to identify the types of suspected cheating in online examinations by candidates and capture cheating evidence images from three perspectives.

2. According to claim 1, a multi-view paperless online examination cheating behavior identification method is characterized in that: The step S1 specifically includes: S11: First, collect the examinee's main perspective image based on the PC screen camera, and use the feature points numbered 1, 8, 15, and 20 in the Dlib function HOG algorithm to preprocess the image to obtain the facial image; S12: Secondly, a two-dimensional coordinate axis is established with the center of the screen as the origin, and each sight line direction is divided with 45° as the threshold, and the sight line directions include nine directions: upper left, left, lower left, upper, middle, lower, upper right, right, and lower right; S13: Finally, transfer learning is used to establish the optimal model for gaze direction recognition based on the SWIN-T pre-trained weights of the Swin Transformer network. The input is the pre-processed facial image, and the output is the confidence of each gaze direction of the image. A gaze direction recognition module for the examinee is constructed to detect the abnormal gaze direction (sneak glance) of the examinee in the main perspective.

3. According to claim 1, a multi-view paperless online examination cheating behavior identification method is characterized in that: The step S2 specifically includes: S21: First, we collect a dataset of abnormal items in the desktop environment based on a side-view camera, which includes three common abnormal items in online paperless exams: books, papers, and electronic devices. S22: Secondly, transfer learning is used to establish the optimal model for detecting abnormal items on the desktop based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of each abnormal item, and a desktop environment item recognition module is built to detect common exam cheating-related items on the desktop.

4. According to claim 1, a multi-view paperless online examination cheating behavior identification method is characterized in that: The step S3 specifically includes: S31: First, the candidate's first-person perspective data collection hardware is based on a self-assembled head-mounted bracket and a USB camera. The first-person perspective abnormal target dataset is obtained based on the head-mounted camera, which includes four types of online paperless examination targets: people, books, paper, and electronic devices; S32: Secondly, transfer learning is used to establish the optimal model for first-person target detection based on the YOLOX-s pre-trained weights of YOLOX, where the output is the bounding box and confidence of the target, and an abnormal target recognition module is constructed under the first-person perspective of the candidate to detect common targets that assist cheating.

5. According to claim 1, a multi-view paperless online examination cheating behavior identification method is characterized in that: The step S4 specifically includes: S41: First, perform MAX-MIN normalization based on the obtained line of sight direction detection confidence, the specific method is as follows: Among them, p θi is the confidence (probability) of the examinee's line of sight in the nine directions under the main viewing angle; p θmax is the maximum confidence among the 9 sight directions, p θmin is the minimum confidence (probability) among the 9 sight directions; after normalizing the image, the confidence value x of the ith sight direction i Between (0, 1); S42: Secondly, using the output of the examinee's sight direction recognition module described in step S1 and the output of the desktop environment object recognition module described in step S2 as input, a "main view-side view" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed; S43: Again, using the output of the examinee's sight direction recognition module described in step S1 and the output of the examinee's first-person perspective abnormal target recognition module described in step S3 as input, a examinee's "main perspective-first-person perspective" joint cheating behavior discrimination network based on a multi-layer perceptron is constructed; S44: Finally, based on the output results of the above two multi-layer perceptrons, a classification module for suspected cheating behaviors in online paperless standard examinations was constructed to identify 13 common suspected cheating behaviors of candidates in online examinations and capture cheating evidence images from three perspectives.

6. The multi-view paperless online examination cheating behavior identification method according to claim 1 is characterized in that: The step S42 specifically includes: S42-1 A "main view-side view" joint cheating behavior discrimination network based on a multi-layer perceptron. The network includes an input layer, a hidden layer, and an output layer. S42-2 has 12 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41; they include: the probability of the upper left visual field x1, the probability of the left visual field x2, the probability of the lower left visual field x3, the probability of the upper visual field x4, the probability of the middle visual field x5, the probability of the lower visual field x6, the probability of the upper right visual field x7, the probability of the right visual field x8, and the probability of the lower right visual field x9; the last 3 neurons are the three confidences of abnormal items output by step S2, including: the book confidence x1, the probability of the book confidence x2, the probability of the book confidence x3, the probability of the book confidence x4, the probability of the book confidence x5, the probability of the book confidence x6, the probability of the book confidence x7, the probability of the book confidence x8, and the probability of the book confidence x9. 10 , Paper confidence x 11 , electronic equipment confidence x 12 ; The number of neurons in the single hidden layer of S42-3 is the same as that in the output layer, i.e., it includes 12 neurons; There are 9 neurons in the output layer of S42-4, representing 9 cheating behaviors near the desktop, including looking at the paper on the desktop and below y1, looking at the books on the desktop and below y2, looking at the electronic devices on the desktop and below y3, looking at the paper behind the computer y4, looking at the books behind the computer y5, looking at the electronic devices behind the computer y6, looking at the desktop and below for multiple illegal items y7, looking at the back of the computer for multiple illegal items y8, and normal side view behavior y9; S42-5 uses the error back propagation method to train and obtain the optimal weight values ​​of the three-layer perceptron.

7. The multi-view paperless online examination cheating behavior identification method according to claim 1 is characterized in that: The step S43 specifically includes: S43-1 is a "main perspective-first perspective" joint cheating behavior discrimination network based on a multi-layer perceptron, which consists of three layers: input layer, hidden layer and output layer; S43-2 has 13 input layer neurons. The first 9 neurons are the normalized values ​​of the 9 sight direction recognition results of step S1, as described in step S41, including: the probability of the upper left visual field x1, the probability of the left visual field x2, the probability of the lower left visual field x3, the probability of the upper visual field x4, the probability of the middle visual field x5, the probability of the lower visual field x6, the probability of the upper right visual field x7, the probability of the right visual field x8, and the probability of the lower right visual field x9; the last 4 neurons are the four abnormal target confidences output by step S3, including: book confidence x1, 10 , Paper confidence x 11 , electronic equipment confidence x 12 , person confidence x 13 ; The number of neurons in the single hidden layer of S43-3 is the same as that in the output layer, i.e., it includes 13 neurons; There are 4 neurons in the output layer of S43-4, representing 4 types of cheating behaviors near the desk, including helping cheaters to lift books y1, helping cheaters to lift papers y2, helping cheaters to lift electronic devices y3, and the first-person perspective of the examinee is normal y4; S43-5 uses the error back propagation method to train and obtain the optimal weight values ​​of the three-layer perceptron.

Images