Conference discipline real-time monitoring method based on computer vision and deep learning

Abstract

The invention relates to the field of computer processing, in particular to a conference discipline real-time monitoring method based on computer vision and deep learning, which comprises the following steps: firstly, establishing a conference discipline real-time monitoring model by using a neural network, acquiring a video stream of conference live broadcast in a conference process, and acquiring a picture sample corresponding to a current frame in real time according to the video stream; and inputting the obtained picture sample into a conference discipline real-time monitoring model, and judging whether the behavior of the participants in the picture sample is illegal or not until the conference is ended. Through computer vision and deep learning technologies, a conference discipline real-time monitoring model is established, and behaviors of conference participants are intelligently identified, so that a conference host and an organizer can more accurately know discipline conditions of conference participants in a conference place in real time, the conference discipline is reorganized in time, and the efficiency and the quality of the conference are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer processing, and in particular to a real-time monitoring method for conference discipline based on computer vision and deep learning. Background Art

[0002] With the continuous development of modern society, Internet video conferencing has been widely used as an important communication and decision-making method. This remote technology-based conference method plays an important role in promoting information exchange, decision-making, teamwork, problem solving, resource optimization, relationship building and innovative development. However, because the meeting is conducted remotely, it is often impossible to supervise the various inappropriate behaviors of the participants. Participants often sleep, make phone calls, lie on the table, and play with their mobile phones, which seriously affects the efficiency and quality of the meeting. Although it is possible to manually watch the conference video to check the meeting discipline of the participants, the workload is huge and time-consuming and laborious.

[0003] There is an urgent need for a method that saves time and effort and can ensure meeting discipline without manual supervision, so as to improve the quality and efficiency of remote meetings. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time monitoring method for conference discipline based on computer vision and deep learning in response to the corresponding deficiencies in the prior art. Through computer vision and deep learning technology, a real-time monitoring model for conference discipline is established, and the behavior of conference participants is intelligently identified, so that conference hosts and organizers can more accurately understand the discipline of conference participants in real time, rectify conference discipline in a timely manner, and improve the efficiency and quality of the meeting.

[0005] The purpose of the present invention is to adopt the following scheme to achieve: a real-time monitoring method of conference discipline based on computer vision and deep learning, comprising the following steps:

[0006] 1) Use neural networks to establish a real-time monitoring model for meeting discipline;

[0007] 2) Obtain the video stream of the conference live broadcast, and obtain the image sample corresponding to the current frame in real time based on the video stream;

[0008] 3) Input the image sample obtained in step 2) into the real-time monitoring model of conference discipline to determine whether the behavior of the participants in the image sample violates the rules:

[0009] 3-1) If there is no violation, repeat steps 2) to 3) until the meeting ends;

[0010] 3-2) If there is a violation, the monitoring results will be sent to the meeting manager, and steps 2) to 3) will be repeated until the meeting ends.

[0011] Preferably, in step 1), the method of establishing the meeting discipline real-time monitoring model is as follows:

[0012] 1-1) Collect pictures of several participants behaving in violation of regulations to form a violation sample set;

[0013] 1-2) Extract features from all illegal samples in the illegal sample set to obtain an illegal feature set;

[0014] 1-3) Divide the violation feature set into a training set and a test set;

[0015] 1-4) Iteratively train a neural network model using the training set until the performance of the neural network model is verified to be the highest using the test set.

[0016] Preferably, in step 1-2), before extracting features from all illegal samples in the illegal sample set, all image samples in the illegal sample set are preprocessed to improve the computer's recognition effect on illegal samples.

[0017] Preferably, in step 3), before the image samples obtained in step 2) are input into the real-time monitoring model for conference discipline, the image samples are preprocessed to improve the computer's judgment effect on the illegal behavior of participants.

[0018] Preferably, the preprocessing method for the image samples includes clearing text information, unifying image size, randomly flipping images, image segmentation processing, and randomly adjusting image brightness, contrast, saturation, and hue.

[0019] Preferably, in step 1-2), the algorithm for extracting features from all illegal samples in the illegal sample set is a SURF (Speeded Up Robust Features) algorithm.

[0020] Preferably, in steps 1-4), the neural network model is a deep residual network or a deep residual convolutional network.

[0021] Preferably, the neural network model is ResNet18, or ResNet34, or ResNet50.

[0022] Preferably, a violation database is also provided for storing picture samples of behavior violations.

[0023] The advantages of the present invention are as follows:

[0024] ① The present invention establishes a real-time monitoring model for conference discipline through data collection, preprocessing, feature extraction, model training, model testing and optimization, and post-processing, which is used for the identification and processing of illegal image samples, and can timely obtain the illegal information of participants during the meeting and monitor the illegal situation in the venue in real time;

[0025] ② The present invention can more accurately extract violation information by classifying and identifying each pixel, reduce misjudgment and missed judgment, and achieve high-precision identification of venue violations;

[0026] ③ The present invention uses machine learning and deep learning methods. As the data set continues to expand, the recognition performance and accuracy of image samples will continue to improve. It can completely break away from manual work and automatically classify and identify violations at the venue, greatly reducing manual intervention and operation, and improving efficiency and accuracy;

[0027] ④ The present invention can not only extract violation information more comprehensively by fusing and analyzing various information in the image samples, but also more accurately monitor the violation of remote conference participants by combining target detection, target positioning and other technologies;

[0028] ⑤ Various data collected by the present invention during the meeting (such as violation information, etc.) can be used to analyze the meeting effect, such as audience attention curve, hot spots, etc., to provide data support for subsequent activity planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the present invention;

[0030] Figure 2 Schematic diagram of the neural network structure used in the embodiment of the present invention;

[0031] Figure 3 A sample schematic diagram of a violation feature set in an embodiment of the present invention;

[0032] Figure 4 The accuracy and loss changes of the training set and the test set in the embodiment of the present invention;

[0033] Figure 5 It is a schematic diagram of identifying mobile phone playing through real-time video in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] like Figures 1 to 5 As shown, a real-time monitoring method for conference discipline based on computer vision and deep learning includes the following steps:

[0035] 1) A real-time monitoring model for conference discipline is established using a neural network. The establishment method of the real-time monitoring model for conference discipline is as follows:

[0036] 1-1) Collect pictures of several participants' illegal behaviors to form a violation sample set; for example, participants sleeping, making phone calls, lying on the table, and playing with mobile phones are all considered illegal behaviors; in this embodiment, the number of samples in the violation sample set is 128. In order to improve the computer's recognition effect on violation samples, all violation samples in the violation sample set need to be preprocessed;

[0037] 1-2) Using the SURF algorithm to extract features from all illegal samples in the illegal sample set, and obtaining the illegal feature set;

[0038] In this embodiment, the purpose of feature extraction is to extract information useful for image recognition, classification, retrieval and other tasks from all illegal samples. These features are usually invariant to factors such as transformation, illumination, and scale, making subsequent tasks more stable and reliable. By extracting features, complex image data can be converted into a representation that can be understood and processed by computers, thereby realizing automatic image recognition.

[0039] 1-3) Divide the violation feature set into a training set and a test set;

[0040] 1-4) Iteratively train a neural network model using the training set until the performance of the neural network model is verified to be the highest using the test set. The specific method is as follows:

[0041] 1-4-1) Define the model training function;

[0042] 1-4-2) Modify the transfer learning strategy to unfreeze all layers (i.e. the parameters of all layers are automatically trained);

[0043] 1-4-3) Repeat step 1-4-2) to output the training results until the training results are stable, and use the test set to verify the performance of the neural network model;

[0044] 1-4-4) Obtain the optimal illegal image recognition model according to step 1-4-3);

[0045] The neural network model is a deep residual network or a deep residual convolutional network. This embodiment uses ResNet18, or ResNet34, or ResNet50.

[0046] 2) Obtain the video stream of the conference live broadcast, obtain the image sample corresponding to the current frame in real time according to the video stream, and pre-process the image sample to improve the computer's judgment effect on the illegal behavior of participants;

[0047] In this embodiment, the VideoCapture class of OpenCV is used to capture the video stream. Generally, the real-time video stream is captured by the stream address (i.e., the URL of the video stream) synchronously transmitted by the device (camera, etc.). Moreover, after obtaining the video stream, it is necessary to determine whether the video stream can be opened. The specific implementation code is as follows:

[0048] import cv2;cap=cv2.VideoCapture(URL);if not cap.isOpened():print("Error opening video stream");exit()

[0049] 3-2) Set the frame extraction interval, that is, determine the monitoring interval. This embodiment uses a method of extracting one frame every 5 frames for monitoring. The specific implementation code is as follows:

[0050] frame_interval=5; current_frame=0;

[0051] 3-3) Read and process frames: During the meeting, read the video frame and save the current frame image according to the monitoring interval. The specific implementation code is as follows:

[0052] Loop to read video frames while True:ret,frame=cap.read()

[0053] Determine whether the video frame is read correctly if ret:

[0054] Check if the frame interval has been reached if current_frame%frame_interval==0:

[0055] Save the obtained video slice cv2.imwrite('Frame',frame)

[0056] #Increase the current frame count current_frame += 1; else:

[0057] #If there is no frame to read, exit the loop break;

[0058] 3-4) Release resources: After completing all operations, make sure to release the resources occupied by the VideoCapture object. The specific implementation code is as follows:

[0059] #Release resources cap.release(); cv2.destroyAllWindows();

[0060] 3) Input the image sample obtained in step 2) into the real-time monitoring model of conference discipline to determine whether the behavior of the participants in the image sample violates the rules:

[0061] 3-1) If there is no violation, repeat steps 2) to 3) until the meeting ends;

[0062] 3-2) If there is a violation, the monitoring results will be sent to the meeting manager, and steps 2) to 3) will be repeated until the meeting ends.

[0063] This embodiment also provides a violation database for storing sample images of behaviors that violate the rules. That is, in this embodiment, if a participant is found to have violated the rules, the corresponding sample images are saved in the violation database, otherwise the sample images obtained this time (i.e., the image corresponding to the current frame) are deleted. Then, during the entire meeting, the violation database is dynamically scanned to see if there are new violation images. If there are new violation images, the violation images are sent to the meeting supervisor, or directly displayed on the large screen of the remote meeting to remind the participants to abide by the meeting disciplines.

[0064] It is worth noting that, whether in step 1-1) or step 2), the image sample preprocessing method is preferably consistent, so as to maximize the accuracy of the real-time monitoring model of conference discipline in judging the violation of participants. These image sample preprocessing methods include:

[0065] ① Clear text information:

[0066] 1.1 Since KerasOCR provides a ready-made OCR text recognition model and an end-to-end training pipeline, this embodiment directly uses the API provided by KerasOCR to obtain the text information in the sample image and the coordinate information of the four vertices of the text in the sample image;

[0067] 1.2 Get the area where the text needs to be cleared from the four vertex coordinate information, and directly use the repair function provided in OpenCV to process the area into rgb (255, 255, 255) (white);

[0068] ② Unify image size:

[0069] The size of all illegal samples in the illegal sample set is uniformly set to 96×96 pixels, so that the sample images can adapt to the input nodes of the neural network;

[0070] ③ Randomly flip the image:

[0071] In order to better identify entities at different angles, this embodiment uses the cv2.flip function () in the OpenCV library to randomly flip the sample image;

[0072] ④Image segmentation processing:

[0073] This embodiment uses the Conny operator method to perform image segmentation, and the specific method is as follows:

[0074] 4.1 Image grayscale

[0075] 4.2 Perform Gaussian smoothing filter on the image:

[0076] 4.2.1 First, generate a two-dimensional Gaussian distribution matrix:

[0077] p(x,y) = p(x)p(y)

[0078] In the formula, p(x,y) represents the joint probability density function of two random variables x and y, p(x) represents the marginal probability density function of random variable x, and p(y) represents the marginal probability density function of random variable y. Since x and y are independent, their joint probability density function can be decomposed into the product of their respective marginal probability density functions. In this formula, since p(x,y)=p(x)p(y), this implies that x and y are independent random variables. The specific code is as follows:

[0079] frac{1}{2pi}exp(-frac{x^2+y^2}{2})tag{5};

[0080] Here, 2pi is a normalizing constant for the probability density function, ensuring that the integral of the entire probability density function over the domain is 1. In this two-dimensional Gaussian distribution, since the variances of x and y are both 1 (and independent of each other), this constant, together with the part inside the exp function, ensures that the total integral of the probability density function is 1. exp represents the natural exponential function, which is the power function of e. For example, exp(-2x 2 +y 2 ),

[0081] -2x 2 +y 2 is the parameter of the exponential function, which determines the shape of the probability density function. In this example, since x 2 +y 2 is the square of the distance from x and y to the origin. This expression describes the probability density of a two-dimensional Gaussian distribution (or normal distribution) centered at the origin and with a variance of 1.

[0082] In order to vectorize the formula, this embodiment uses the vector textbf{v}=[xy]^T;

[0083] 4.2.2 Then, convolve it with the grayscale image according to the following formula to achieve image filtering:

[0084] fs(x,y)=f(x,y)*p(x,y)

[0085] Where fs(x,y) is the filtered image, f(x,y) is the original image, and p(x,y) is the filter (also called kernel or mask).

[0086] 4.3 Calculate the gradient magnitude and direction:

[0087] This embodiment uses a first-order finite difference approximation method to calculate the gradient amplitude (rate of change) of the grayscale value. When calculating the rate of change, for a one-variable function, the derivative is calculated, and for a two-variable function, the partial derivative is calculated.

[0088] 4.4 Perform non-maximum suppression (NMS) on the gradient amplitude:

[0089] Find the local maximum of the sample image pixel point, and compare the gradient amplitudes in front and behind it along the gradient direction. The area with the largest gradient amplitude along its gradient direction is retained, otherwise it is suppressed. This step mainly excludes non-edge pixels and only retains some thin lines (i.e. candidate edges).

[0090] 4.5 Detecting and connecting edges using dual threshold method:

[0091] The upper limit value of the gradient amplitude threshold is set as the high threshold value TH, the lower limit value is set as the low threshold value TL, and TH:TL is 2:1 or 3:1;

[0092] If the gradient magnitude at a certain pixel position exceeds TH, the pixel is retained as an edge pixel.

[0093] If the gradient magnitude at a certain pixel position is less than TL, the pixel is excluded.

[0094] If the gradient magnitude at a pixel position is between TH and TL, the pixel is retained only if it is connected to a higher pixel than the original one.

[0095] ⑤ Randomly adjust the brightness, contrast, saturation, and hue of the image:

[0096] In this embodiment, the cv2.convertScaleAbs() function in the OpenCV library is used to randomly adjust the brightness and contrast of the image, and the cv2.cvtColor() function is used to randomly adjust the hue and saturation of the image.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification made to the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A real-time monitoring method for conference discipline based on computer vision and deep learning, characterized in that: The following steps are involved: 1) Use neural networks to establish a real-time monitoring model for meeting discipline; 2) Obtain the video stream of the conference live broadcast, and obtain the image sample corresponding to the current frame in real time based on the video stream; 3) Input the image sample obtained in step 2) into the real-time monitoring model of conference discipline to determine whether the behavior of the participants in the image sample violates the rules: 3-1) If there is no violation, repeat steps 2) to 3) until the meeting ends; 3-2) If there is a violation, the monitoring results will be sent to the meeting manager, and steps 2) to 3) will be repeated until the meeting ends.

2. The method for real-time monitoring of conference discipline according to claim 1 is characterized in that: In step 1), the method of establishing the meeting discipline real-time monitoring model is as follows: 1-1) Collect pictures of several participants behaving in violation of regulations to form a violation sample set; 1-2) Extract features from all illegal samples in the illegal sample set to obtain an illegal feature set; 1-3) Divide the violation feature set into a training set and a test set; 1-4) Iteratively train a neural network model using the training set until the performance of the neural network model is verified to be the highest using the test set.

3. The method for real-time monitoring of conference discipline according to claim 2 is characterized in that: In step 1-2), before extracting features from all illegal samples in the illegal sample set, all image samples in the illegal sample set are preprocessed to improve the recognition effect of illegal samples.

4. The method for real-time monitoring of conference discipline according to claim 1 is characterized in that: In step 3), before the image samples obtained in step 2) are input into the real-time monitoring model of conference discipline, the image samples are preprocessed to improve the judgment effect of the violation behavior of the participants.

5. The method for real-time monitoring of conference discipline according to claim 3 or 4, characterized in that: The preprocessing methods for image samples include clearing text information, unifying image size, randomly flipping images, image segmentation, and randomly adjusting image brightness, contrast, saturation, and hue.

6. The method for real-time monitoring of conference discipline according to claim 2 is characterized in that: In step 1-2), the algorithm used to extract features from all illegal samples in the illegal sample set is the SURF algorithm.

7. The method for real-time monitoring of conference discipline according to claim 2 is characterized in that: In step 1-4), the neural network model is a deep residual network or a deep residual convolutional network.

8. The method for real-time monitoring of conference discipline according to claim 2 or 7, characterized in that: The neural network model is ResNet18, or ResNet34, or ResNet50.

9. The method for real-time monitoring of conference discipline according to claim 1, characterized in that: There is also a violation database that stores sample images of behavior violations.

Images