Online class inspection method and class inspection device based on ROI coding
Through the ROI encoding method, LBP value feature extraction and denoising processing are used to distinguish and code areas of interest and non-interested areas, and the problem of unclear video images when the existing online course inspection system is limited in network bandwidth, it solves the problem that the video image is unclear when the network bandwidth is limited. , improves the experience and efficiency of class tours.
Patent Information
- Application Number
- CN202210441491.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-25
AI Technical Summary
The existing online course inspection system has a limited network bandwidth, and the video picture is not clear and the area of interest cannot be effectively tracked, which affects the course inspection experience.
Using ROI encoding-based method, LBP value feature extraction and denoising processing is used to distinguish between the regions of interest and non-interested areas, and high-code and low-code encoding and decode are performed respectively to ensure the video image quality of the regions of interest.
It improves the clarity of video images in the area of interest, highlights the focus of class inspections, and improves the efficiency and experience of online class inspections.
Smart Images

Figure CN114745550B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of education management, and in particular to an online class inspection method and a class inspection device based on ROI coding. Background Art
[0002] The rapid development of network and information technology is driving the reform of teaching methods in colleges and universities. More and more colleges and universities use online class inspection systems to remotely supervise and manage the teaching process of teachers, so that supervisors are no longer restricted by geographical location. The existing online class inspection system can obtain new class inspection group requests and class inspection group information, complete the formation of class inspection groups, and inspect the classroom list in the class inspection group based on the class inspection group information, display the information such as the teaching building to which the classroom belongs, support supervisors to inspect specific classrooms in the classroom list, and display the teaching details of specific classrooms.
[0003] However, the prior art has the following deficiencies:
[0004] First, the video images collected in the classroom need to be encoded with the same compression ratio. When the network bandwidth is limited, there are problems such as unclear classroom inspection interface and unclear key points, which seriously affects the observation effect.
[0005] Second, the existing class inspection system is unable to track the parts of interest in the screen according to supervision needs, is unable to place the "important departments" in the center of the screen, and even jumps out of the screen, seriously affecting the class inspection experience. Summary of the invention
[0006] The present invention aims to solve at least one of the technical problems existing in the prior art.
[0007] To this end, the present invention proposes an online class inspection method and a class inspection device based on ROI coding, aiming to improve the existing online class inspection system. When the network bandwidth is limited, the clarity of the video image of the region of interest can be effectively improved, and the online class inspection efficiency and class inspection experience can be improved.
[0008] According to a first aspect of the present application, there is provided an online class inspection method based on ROI coding, comprising the following steps:
[0009] Acquire a first video image;
[0010] selecting a region of interest in the first video image;
[0011] Extracting features from the first video image by calculating the LBP value of the first video image; and extracting features from the video image in the region of interest by calculating the LBP value of the video image in the region of interest;
[0012] Tracking the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image according to the extracted features of the first video image and the video image in the region of interest, and the LBP values of the first video image and the video image in the region of interest;
[0013] Distinguishing the region of interest and the non-region of interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size of the video image in the region of interest;
[0014] The video image in the region of interest is encoded and decoded using a first encoding and decoding method, and the video image in the non-region of interest is encoded and decoded using a second encoding and decoding method; and the encoding and decoding image quality level of the first encoding and decoding method is higher than the encoding and decoding image quality of the second encoding and decoding method.
[0015] In the above method, before the step of extracting features from the first video image by calculating the LBP value of the first video image, the method further includes: denoising the first video image.
[0016] In the above method, the step of denoising the first video image includes:
[0017] Select Haar wavelet basis, decompose the first video image into three layers, and obtain the low-frequency part L of each layer n and high frequency part H n ; Where n is the number of decomposition levels, n = 1, 2, 3;
[0018] According to the threshold calculation formula Calculate the threshold T of each layer n ; Wherein, n = 1, 2, 3; σ represents the estimated noise standard deviation; M*N represents the size of the first video image;
[0019] Reconstruct the new coefficients according to the threshold function formula. The threshold function formula is as follows:
[0020]
[0021] In the formula, represents the i-th H in the n-th layer n Coefficient; if illustrate If it is the noise coefficient, it is set to 0; otherwise, it means is the useful signal coefficient;
[0022] The new coefficients to be reconstructed And the third layer low-frequency coefficients are reconstructed into wavelet coefficients and then inverse wavelet transform is performed to obtain the first video image after denoising.
[0023] In the above method, the step of extracting features from the first video image by calculating the LBP value of the first video image includes:
[0024] Segmenting the first video image after denoising into a plurality of circular areas with a radius of 1 pixel;
[0025] Calculate the LBP value of each circular area, the calculation formula is as follows:
[0026]
[0027]
[0028] Where P c Represents the pixel value of the center point of the circular area; P j Indicates the pixel values of other points in the circular area; taking the number of sampling points in the circular area as 8 as an example, the pixel values are P1, P2, ..., P8 in clockwise order;
[0029] All circular areas are traversed and the LBP value of each circular area is calculated to obtain the LBP value of the first video image.
[0030] In the above method, the step of tracking the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image includes:
[0031] Creating a region of interest recognition template according to the LBP value of the video image in the region of interest;
[0032] The sum of the squares of the distances between the video image in the region of interest and the LBP values of the region of interest recognition template is calculated, and the calculation formula is as follows:
[0033]
[0034] Where m is the width of the region of interest, n is the height of the region of interest; lbp(a,b) is the LBP value at the coordinate (a,b) in the entire image, a=i+row, b=j+line; row, line respectively represent the X-axis and Y-axis coordinates of the origin of the current region of interest recognition template in the coordinate system of the entire image; lbp roi (i, j) is the LBP value at the coordinate (i, j) in the region of interest; Dist represents the sum of the squares of the distances between the region of interest recognition template and the region of interest in the entire image;
[0035] Move the region of interest recognition template, repeatedly calculate the sum of squares of the distances between the video image in the region of interest and the LBP value of the region of interest recognition template, until the entire image is traversed, and return the Dist value in real time;
[0036] Determine whether Min_Dist>Dist holds true; if so, set Min_Dist=Dist, where Min_Dist represents the minimum value among all Dist values, and the origin position coordinates corresponding to the Min_Dist value are the position coordinates of the origin of the video image in the region of interest in the entire image;
[0037] The position coordinates of the center point of the video image within the region of interest in the first video image are determined according to the position coordinates of the origin of the video image within the region of interest in the first video image.
[0038] In the above method, the step of encoding and decoding the video image in the region of interest using a first encoding and decoding method, and encoding and decoding the video image in the non-region of interest using a second encoding and decoding method, comprises:
[0039] High bit rate coding is used for the video images in the region of interest; low bit rate coding is used for the video images in the non-region of interest;
[0040] The coded bit stream of the region of interest is decoded at a high bit rate to obtain a corresponding decoded image; the coded bit stream of the region of non-interest is decoded at a low bit rate to obtain a corresponding decoded image;
[0041] The decoded image of the region of interest and the decoded image of the region of no interest are spliced.
[0042] In the above method, the rotation of the camera is adjusted according to the position coordinates of the center point of the video image in the region of interest in the first video image, so that the video image in the region of interest is located at the center of the class inspection interface.
[0043] According to a second aspect of the present application, an online class inspection device based on ROI coding is provided, comprising:
[0044] A first video image acquisition module, used to acquire a first video image;
[0045] An area of interest selection module, used for selecting an area of interest in the first video image;
[0046] a feature extraction module, configured to extract features from the first video image by calculating the LBP value of the first video image; and to extract features from the video image in the region of interest by calculating the LBP value of the video image in the region of interest;
[0047] a position tracking module, configured to track the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image according to the extracted features of the first video image and the video image in the region of interest and the LBP values of the first video image and the video image in the region of interest;
[0048] A region division module, configured to distinguish the region of interest from the region of non-interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size of the video image in the region of interest;
[0049] The image coding module is used to encode and decode the video image in the region of interest using a first coding method, and to encode and decode the video image in the non-region of interest using a second coding method; and the coding and decoding image quality level of the first coding and decoding method is higher than the coding and decoding image quality of the second coding and decoding method.
[0050] According to a third aspect of the present application, a terminal is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the above-mentioned class inspection method when running the computer program.
[0051] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the terminal where the storage medium is located is controlled to execute the above-mentioned class inspection method.
[0052] According to the technical solution provided in the present application, at least the following beneficial effects are achieved: in actual application, the content of the first video image will change. Under the premise that the content of the video image in the area of interest is determined to remain unchanged, the position of the video image in the area of interest in the first video image and the content of the video image in the area of no interest will both change; therefore, it is necessary to track the position of the video image in the area of interest in real time to determine the position coordinates of its center point in the first video image in real time; then, the area of interest and the area of no interest are re-distinguished in the first video image according to the position coordinates and the size of the video image in the area of interest; thereafter, the video image in the area of interest is encoded and decoded using the first encoding and decoding method, and the video image in the area of no interest is encoded and decoded using the second encoding and decoding method; the encoding and decoding image quality level of the first encoding and decoding method is higher than the encoding and decoding image quality of the second encoding and decoding method; adopting different encoding and decoding methods is conducive to improving the clarity of the video image in the area of interest, highlighting the key points of the class inspection, and improving the efficiency and experience of online class inspection when the network bandwidth is limited.
[0053] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0055] Figure 1 This is an application scenario of the embodiment of the present application;
[0056] Figure 2 This is a collection effect diagram of the classroom end provided in the embodiment of the present application;
[0057] Figure 3 A flowchart of an online class inspection method provided in an embodiment of the present application;
[0058] Figure 4 This is a feature extraction effect diagram of the entire image provided in the embodiment of the present application;
[0059] Figure 5 The effect diagram of the region of interest identification template provided in the embodiment of the present application;
[0060] Figure 6 A coding and decoding flow chart provided for an embodiment of the present application;
[0061] Figure 7A diagram showing the position coordinates of the center point of the video image in the region of interest and the position coordinates of the center point of the entire image provided by an embodiment of the present application;
[0062] Figure 8 A schematic diagram of the structure of a class inspection device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The meaning of "several" is one or more, the meaning of "many" is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.
[0065] The application scenarios of the embodiments of the present application are as follows: Figure 1 As shown, it includes a classroom end 10, a class inspection device 20 and a class inspection end 30, wherein the classroom end 10 and the class inspection device 20, and the class inspection device 20 and the class inspection end 30 are interconnected through a communication network, and the connection can be a wired connection or a wireless connection.
[0066] The classroom end 10 can collect video images of teachers and students and video images on the teaching screen through the class patrol device 20, and present them through the class patrol end 30. The collection effect is as follows: Figure 2 shown.
[0067] The classroom end 10 can be a classroom or lecture hall for classes; the class inspection device 20 can be a camera and its background processing center; the class inspection end 30 can be an electronic terminal device such as a desktop computer, a laptop computer, a smart phone, a tablet computer, an e-book reader, etc. The class inspection end 301 includes a display device, which can be an independent display, such as an external display, which can be connected to an electronic terminal device such as a desktop computer, a laptop computer, a smart phone, a tablet computer, etc. to display or expand the display data of the class inspection end 30; the display can also be a display integrated with the class inspection end 30, such as a built-in screen of a laptop computer, a smart phone, or a tablet computer. The class inspection end 30 also includes a pointer device, which can be a built-in pointer device, such as a touchpad on a laptop computer; or it can be a device external to the class inspection end 30 and connected to it for communication, such as a mouse, an external touchpad, or a keyboard with a touchpad. The patrol terminal 30 is installed with an operating system (OS), at least one browsing program (App) and at least one execution program; the operating system may be a Microsoft operating system (Microsoft Windows OS) or an Apple operating system (Mac OS), etc.; the operating system is provided with a display program for displaying a cursor on a display device, and the user controls the position change of the cursor on the display device through a pointer device.
[0068] like Figure 3 As shown, the first aspect of the present application provides an online class inspection method based on ROI coding, and the method includes the following steps:
[0069] Step S110: Acquire a first video image.
[0070] In this application, the video images of teachers and students and the video images on the teaching screen are collected at the classroom end by the class patrol device, and transmitted to the class patrol end for display via the communication network. The content displayed on the screen interface of the class patrol end is the first video image. It should be noted that the content of the first video image may change due to the activities of teachers and students and changes in the teaching content.
[0071] Step S120: Select a region of interest in the first video image.
[0072] In the present application, the class inspector selects the region of interest (ROI) in the first video image by clicking or touching the screen interface of the class inspector end by a mouse or other means; the rest are non-regions of interest. It should be noted that the video image in the region of interest is the content that the class inspector focuses on, such as the activities of the teacher. If in the first video image, the video image of the relevant activities of the teacher is selected as the region of interest, then only the video image of the relevant activities of the teacher is displayed in the region of interest, which can be understood as the content of the video image in the region of interest does not change; that is, when the content of the first video image changes, the content of the video image in the region of interest does not change, but the position of the video image in the region of interest in the first video image will change. It should be noted that after the region of interest is selected, the size information of the region of interest, such as the height and width of the region of interest, can be obtained.
[0073] Step S130, extracting features from the first video image by calculating the LBP value of the first video image; and extracting features from the video image in the region of interest by calculating the LBP value of the video image in the region of interest.
[0074] In the present application, the feature information of the first video image and the video image in the region of interest is extracted by respectively calculating the LBP values of the first video image and the video image in the region of interest. It should be noted that the content of the first video image displayed on the screen interface of the class inspection end changes in real time, and therefore, the LBP value of the first video image changes. After the region of interest is selected, the content of the video image located in the region of interest will not change, and therefore, it can be understood that the LBP value of the video image located in the region of interest is fixed. It can be understood that the LBP value of the first video image is calculated in real time and continuously, while the LBP value of the video image in the region of interest only needs to be calculated once.
[0075] In the present application, before extracting features from the first video image and the video image within the region of interest, it is necessary to perform overall noise reduction on the video image to eliminate irrelevant information in the video image and increase the reliability of feature extraction from the video image.
[0076] In the present application, a method for image denoising based on wavelet transform adaptive threshold is preferred, and the method for image denoising comprises the following steps:
[0077] Step S131: Select Haar wavelet basis and decompose the first video image into three layers to obtain the low-frequency part L of each layer. n and high frequency part H n ; Where n is the number of decomposition levels, n = 1, 2, 3.
[0078] Step S132: Calculate the threshold value according to the formula Calculate the threshold T of each layer n ; Wherein, n=1,2,3; σ represents the estimated noise standard deviation; M*N represents the size of the first video image.
[0079] Step S133, quantizing the wavelet coefficient threshold, and reconstructing new coefficients according to the threshold function formula;
[0080] The threshold function formula is as follows:
[0081]
[0082] In the formula, represents the i-th H in the n-th layer n Coefficient; if illustrate If it is the noise coefficient, it is set to 0; otherwise, it means is the useful signal coefficient.
[0083] Step S134: The new coefficients reconstructed in step S133 The third layer low-frequency coefficients are reconstructed into wavelet coefficients and then inverse wavelet transform is performed to obtain the denoised first video image.
[0084] In the present application, after the first video image is denoised, a method based on a circular LBP algorithm is preferably used to extract features from the denoised overall image. The feature extraction method includes the following steps:
[0085] Step S135: Segment the denoised first video image into a plurality of circular areas with a radius of 1 pixel.
[0086] Step S136: Calculate the LBP value of each circular area. The calculation formula is as follows:
[0087]
[0088]
[0089] Where P c Represents the pixel value of the center point of the circular area; P j Represents the pixel values of other points in the circular area. This paper takes 8 sampling points in the circular area as an example, and its pixel values are P1, P2, ..., P8 in clockwise order.
[0090] Step S137, traverse all circular areas and execute step S136 for each circular area to obtain the LBP feature value of the entire image. The feature extraction effect is as follows: Figure 4 shown.
[0091] In the present application, after the overall video image is denoised, features of the video image in the region of interest are extracted and the corresponding LBP value is obtained according to the region of interest selected by the patrol personnel in the first video image. The extraction steps are consistent with the above-mentioned feature extraction method and will not be repeated here.
[0092] Step S140: Track the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image based on the extracted features of the first video image and the video image in the region of interest, as well as the LBP values of the first video image and the video image in the region of interest.
[0093] Based on the above description, under the premise that the content of the video image in the region of interest remains unchanged, the position of the video image in the region of interest in the first video image will change. Therefore, it is necessary to track the position of the center point of the video image in the region of interest in real time to determine its position coordinates in real time.
[0094] In the present application, a template matching algorithm based on an LBP operator is preferably used to track the video image in the region of interest. The tracking method comprises the following steps:
[0095] Step S141: Create a region of interest recognition template according to the LBP value of the video image in the region of interest.
[0096] like Figure 5 As shown, a region of interest recognition template is made based on a human facial feature region. The initial position of the region of interest recognition template can be set at the upper left corner of the entire video image, or at the lower left corner, upper right corner, or lower right corner, which is not specifically limited in this application. It should be noted that the size of the region of interest recognition template is consistent with the size of the region of interest.
[0097] In the present application, the ROI recognition template moves in the entire video image, and the position coordinates of the center point of the video image in the ROI in the first video image are determined according to the distance between the ROI recognition template and the LBP value in the ROI.
[0098] Step S142, calculating the sum of the squares of the distances between the video image in the region of interest and the LBP values of the region of interest recognition template, the calculation formula is as follows:
[0099]
[0100] Where m is the width of the region of interest, n is the height of the region of interest; lbp(a,b) is the LBP value at the coordinate (a,b) in the entire image, a=i+row, b=j+line, row, line respectively represent the X-axis and Y-axis coordinates of the origin of the current region of interest recognition template in the coordinate system of the entire image; lbp roi (i, j) is the LBP value at the coordinate (i, j) in the region of interest; Dist represents the sum of the squares of the distances between the region of interest recognition template and the region of interest in the entire image.
[0101] For example, the LBP value of the video image of the region of interest, that is, the LBP value of the region of interest recognition template, can be expressed in matrix form as follows:
[0102]
[0103] The LBP value of the entire video image can be expressed in matrix form as
[0104]
[0105] The ROI recognition template starts from the upper left corner of the entire video image and moves to the right. The Dist value of the ROI recognition template at the initial position is calculated. Dist = (3-1) 2 +(5-2) 2 +(2-3) 2 +(1-4) 2 =23.
[0106] Step S143, move the region of interest identification template, repeat step S142 until the entire image is traversed, and return the Dist value in real time.
[0107] After calculating the Dist value of the ROI recognition template at the initial position, the ROI recognition template moves to the right and calculates the Dist value of the position after the first movement, Dist = (5-1) 2 +(1-2) 2 +(1-3) 2 +(3-4) 2 =22; and so on to calculate the Dist value after each move.
[0108] Step S144, determine whether Min_Dist>Dist holds; if so, set Min_Dist=Dist, where Min_Dist represents the minimum value of all Dist values, and the origin position coordinates corresponding to the Min_Dist value at this time are the position coordinates of the origin of the video image in the region of interest in the entire image.
[0109] For example, first, the Dist value of the region of interest identification template at the initial position is used as the Min_Dist value. If the Dist value of the position after the first movement is greater than or equal to the Min_Dist value, it is ignored, and the region of interest identification template continues to move to the right. If the Dist value of the position after the first movement is less than the Min_Dist value, the Dist value of the position after the first movement is used as the Min_Dist value, and the system updates the Min_Dist value. The region of interest identification template continues to move to the right. If the Dist value of the position after the second movement is greater than or equal to the Dist value of the position after the first movement, it is ignored, and the region of interest identification template continues to move to the right. If the Dist value of the position after the second movement is less than the Dist value of the position after the first movement, the Dist value of the position after the second movement is used as the Min_Dist value, and the system updates the Min_Dist value. By analogy, each calculated Dist value is compared with each updated Min_Dist value until the final Min_Dist value is obtained, and the origin position coordinates corresponding to the final Min_Dist value are recorded, that is, the position coordinates (row, line) of the origin of the video image in the region of interest in the entire image.
[0110] In the present application, by real-time calculation and real-time updating of the Min_Dist value, it is beneficial to determine the position coordinates of the origin of the video image in the region of interest in the entire image while determining the Min_Dist value; to avoid the situation where the Min_Dist value is obtained but the position coordinates of the origin of the video image in the region of interest cannot be determined.
[0111] Step S145, determining the position coordinates of the center point of the video image in the region of interest in the first video image according to the position coordinates of the origin of the video image in the region of interest in the first video image;
[0112] The calculation formula is as follows:
[0113] (X CR , Y CR ) = (row+m / 2, line+n / 2)
[0114] In the formula, (X CR , Y CR ) represents the position coordinates of the center point of the video image within the region of interest in the first video image; (row, line) represents the position coordinates of the origin of the video image within the region of interest in the first video image; m represents the width of the video image within the region of interest; n represents the height of the video image within the region of interest.
[0115] Step S150: distinguishing the region of interest from the region of non-interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size of the video image in the region of interest.
[0116] Based on the above description, under the premise that the content of the video image in the region of interest is determined to be unchanged, the position of the video image in the region of interest in the first video image and the content of the video image in the non-interest region will change. Therefore, it is necessary to re-distinguish the region of interest and the non-interest region in the first video image according to the position coordinates of the center point of the video image in the region of interest and the size (width and height) of the video image in the region of interest in real time.
[0117] Step S160: The video image in the region of interest is encoded and decoded using a first encoding and decoding method, and the video image in the non-region of interest is encoded and decoded using a second encoding and decoding method; the encoding and decoding image quality level of the first encoding and decoding method is higher than the encoding and decoding image quality of the second encoding and decoding method.
[0118] In the present application, the quality of the encoded image can be evaluated in a variety of ways, including but not limited to resolution, color depth, image distortion, etc. After distinguishing the region of interest and the region of non-interest in the first video image, the video images in the region of interest and the region of non-interest can be encoded and decoded respectively, and different encoding and decoding parameters and encoding and decoding strategies are given, so that the quality level of the video image in the region of interest is higher than the quality level of the video image in the region of non-interest, so as to ensure the clarity of the video image in the region of interest when the network bandwidth is limited.
[0119] In this application, a coding and decoding algorithm based on an area of interest of an encoder is preferably used to encode and decode a video image. The coding and decoding process is as follows: Figure 6 As shown, the encoding and decoding method includes the following steps:
[0120] Step S161: high bit rate coding is used for the video images in the region of interest; low bit rate coding is used for the video images in the region of no interest.
[0121] In this application, the encoder can be an encoder with adjustable compression ratio such as H264, and the model of the encoder is not specifically limited in this application. Taking the H264 encoder as an example, the first encoding method is H264 high bit rate encoding; the second encoding method is H264 low bit rate encoding.
[0122] Step S162: The encoded bit stream of the region of interest is decoded at a high bit rate to obtain a corresponding decoded image; the encoded bit stream of the region of non-interest is decoded at a low bit rate to obtain a corresponding decoded image.
[0123] In the present application, taking the H264 encoder as an example, the first decoding method is H264 high bit rate decoding; the second decoding method is H264 low bit rate decoding.
[0124] Step S163: splice the decoded image of the region of interest and the decoded image of the non-region of interest to obtain a complete video image.
[0125] In the spliced video images, the clarity of the video images in the region of interest is significantly higher than the clarity of the video images in the non-region of interest.
[0126] It should be noted that, in the actual implementation process, a flag bit can be set for the region of interest. When the flag bit is set to 1, it indicates that the video image belongs to the region of interest and requires high bit rate encoding and decoding; otherwise, low bit rate encoding and decoding are performed; the bit rate size can be manually adjusted as needed. In addition, the course inspector can choose whether to display the video image in the non-interested region as needed. When the video image in the non-interested region does not need to be displayed, step S163 can be skipped, and the final image splicing is not performed, and only the decoded image in the region of interest is displayed. In addition, in some cases, only the video image in the region of interest can be encoded, and the non-interested region can be discarded.
[0127] In some specific embodiments of the present application, the online class inspection method further includes:
[0128] The rotation of the camera is adjusted according to the position coordinates of the center point of the video image in the region of interest in the first video image, so that the video image in the region of interest is always located at the center of the class inspection interface.
[0129] In the present application, after obtaining the position coordinates of the center point of the video image in the region of interest, the deviation and direction between the position coordinates and the preset position of the camera are calculated, and the rotation of the camera is automatically controlled through serial communication.
[0130] It should be noted that, when obtaining the position coordinates (X CR , Y CR ), first check whether the automatic tracking function of the camera is normal, if not, start the manual control function; then check whether the automatic focusing function of the camera is normal, if not, start the manual control focusing function. If the above functions are normal, then according to the position coordinates (X CR , Y CR ) to control the movement of the camera so that the video image in the area of interest is always in the center of the class inspection interface.
[0131] The specific implementation process is as follows Figure 7As shown, for an image with a width of W and a height of H, we regard the upper left corner of the image as the origin, the right is the positive direction of the horizontal coordinate, and the downward is the positive direction of the vertical coordinate. The coordinates of the center point of the image are Finally, according to and (X CR , Y CR )The angle of the camera can be adjusted by the position difference between the two center points so that the center point of the video image in the area of interest coincides with the center point of the entire image.
[0132] like Figure 8 As shown, the second embodiment of the present application provides an online class inspection device based on ROI coding, the class inspection device 20 includes a first video image acquisition module 21, an area of interest selection module 22, a feature extraction module 23, a position tracking module 24, a region division module 25 and an image encoding and decoding module 26;
[0133] in,
[0134] A first video image acquisition module 21, used to acquire a first video image;
[0135] An area of interest selection module 22, used to select an area of interest in the first video image;
[0136] The feature extraction module 23 extracts features from the first video image by calculating the LBP value of the first video image; and extracts features from the video image in the region of interest by calculating the LBP value of the video image in the region of interest;
[0137] A position tracking module 24 is used to track the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image according to the extracted features of the first video image and the video image in the region of interest and the LBP values of the first video image and the video image in the region of interest;
[0138] A region division module 25, configured to distinguish a region of interest and a non-region of interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size (width and height) of the video image in the region of interest;
[0139] The image coding module 26 is used to codec the video image in the region of interest using a first coding method, and codec the video image in the non-region of interest using a second coding method; and the coding image quality level of the first coding method is higher than the coding image quality of the second coding method.
[0140] In some specific embodiments of the present application, the first video image acquisition module 21 includes a video image acquisition unit and a video image presentation unit;
[0141] A video image acquisition unit, used for acquiring a first video image at the classroom end;
[0142] The video image presenting unit is used to display the first video image on the class inspection terminal.
[0143] In some specific embodiments of the present application, the feature extraction module 23 includes a video image noise reduction unit, a first video image feature extraction unit, and an area of interest feature extraction unit;
[0144] A first video image noise reduction unit, used for performing overall noise reduction on the first video image;
[0145] A first video image feature extraction unit, used to extract features from the first video image;
[0146] The region of interest feature extraction unit is used to extract features from the video image in the region of interest.
[0147] In some specific embodiments of the present application, the position tracking module 24 includes a recognition template making unit, a distance calculation unit, a recognition template moving unit, an origin coordinate determination unit, and a center point coordinate determination unit;
[0148] The recognition template making unit is used to make a region of interest recognition template according to the LBP value of the video image in the region of interest;
[0149] A distance calculation unit, used for calculating the sum of squares of distances between the video image in the region of interest and the LBP value of the region of interest recognition template;
[0150] A recognition template moving unit, used for moving the region of interest recognition template in the first video image;
[0151] An origin coordinate determining unit, used for selecting a position coordinate corresponding to a minimum value from a plurality of distance square sum values as the position coordinate of the origin of the video image within the region of interest in the first video image;
[0152] The center point coordinate determining unit is used to determine the position coordinates of the center point of the video image in the region of interest in the first video image according to the position coordinates of the origin of the video image in the region of interest in the first video image.
[0153] In some specific embodiments of the present application, the image encoding and decoding module 26 includes a first encoding unit, a second encoding unit, a first decoding unit, a second decoding unit and an image splicing unit;
[0154] A first encoding unit, used for encoding the video image in the region of interest using H264 high bit rate;
[0155] The second encoding unit is used to encode the video image in the non-interest area using H264 low bit rate;
[0156] A first decoding unit, configured to decode the encoded bit stream of the region of interest using H264 high bit rate to obtain a corresponding decoded image;
[0157] A second decoding unit is used to decode the encoded code stream of the non-interested area using H264 low bit rate to obtain a corresponding decoded image;
[0158] The image stitching unit is used to stitch the decoded image of the area of interest and the decoded image of the area of no interest to obtain a complete video image.
[0159] The above-mentioned online class inspection device based on ROI coding can be implemented through an integrated circuit or a chip, and is not specifically limited in this application.
[0160] The third aspect of the present application also provides a terminal including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the above-mentioned class inspection method when running the computer program.
[0161] Specifically, the processor may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0162] Specifically, the processor is connected to the memory via a bus, and the bus may include a path for transmitting information. The bus may be a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.
[0163] The memory can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.
[0164] Optionally, the memory is used to store the code of the computer program for executing the solution of the present application, and the execution is controlled by the processor. The processor is used to execute the application program code stored in the memory to realize the action of the above-mentioned online class inspection device.
[0165] The fourth aspect of the present application also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the terminal where the storage medium is located is controlled to execute the above-mentioned class inspection method.
[0166] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0167] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present application.
Claims
1. An online class inspection method based on ROI coding, characterized in that: The following steps are involved: Acquire a first video image; selecting a region of interest in the first video image; De-noising the first video image, wherein the de-noising step comprises: Select Haar wavelet basis, decompose the first video image into three layers, and obtain the low-frequency part L of each layer n and the high frequency part H n ; Where n is the number of decomposition levels, n = 1, 2, 3; According to the threshold calculation formula Calculate the threshold T of each layer n ; Wherein, n = 1, 2, 3; σ represents the estimated noise standard deviation; M*N represents the size of the first video image; Reconstruct the new coefficients according to the threshold function formula. The threshold function formula is as follows: In the formula, represents the i-th H in the n-th layer n Coefficient; if illustrate If it is the noise coefficient, it is set to 0; otherwise, it means is the useful signal coefficient; The new coefficients to be reconstructed and reconstructing the wavelet coefficients using the third layer low-frequency coefficients and then performing inverse wavelet transform to obtain the first video image after denoising; Performing feature extraction on the first video image by calculating the LBP value of the first video image, the feature extraction step comprising: Segmenting the first video image after denoising into a plurality of circular areas with a radius of 1 pixel; Calculate the LBP value of each circular area, the calculation formula is as follows: Where P c Represents the pixel value of the center point of the circular area; P j Represents the pixel values of other points in the circular area; the number of sampling points in the circular area is 8, and their pixel values are P1, P2, ..., P8 in clockwise order; Traversing all circular areas and calculating the LBP value of each circular area to obtain the LBP value of the first video image; And, performing feature extraction on the video image in the region of interest by calculating the LBP value of the video image in the region of interest; According to the extracted features of the first video image and the video image in the region of interest, and the LBP values of the first video image and the video image in the region of interest, the video image in the region of interest is tracked to obtain the position coordinates of the center point of the video image in the region of interest in the first video image, and the step of obtaining the position coordinates includes: Creating a region of interest recognition template according to the LBP value of the video image in the region of interest, wherein the size of the region of interest recognition template is consistent with the size of the region of interest; The sum of the squares of the distances between the video image in the region of interest and the LBP values of the region of interest recognition template is calculated, and the calculation formula is as follows: Where m is the width of the region of interest, n is the height of the region of interest; lbp(a,b) is the LBP value at the coordinate (a,b) in the entire image, a=i+row, b=j+line; row, line respectively represent the X-axis and Y-axis coordinates of the origin of the current region of interest recognition template in the coordinate system of the entire image; lbp roi (i, j) is the LBP value at the coordinate (i, j) in the region of interest; Dist represents the sum of the squares of the distances between the region of interest recognition template and the region of interest in the entire image; Move the region of interest recognition template, repeatedly calculate the sum of squares of the distances between the video image in the region of interest and the LBP value of the region of interest recognition template, until the entire image is traversed, and return the Dist value in real time; Determine whether Min_Dist>Dist holds true; if so, set Min_Dist=Dist, where Min_Dist represents the minimum value among all Dist values, and the origin position coordinates corresponding to the Min_Dist value are the position coordinates of the origin of the video image in the region of interest in the entire image; Determine the position coordinates of the center point of the video image in the region of interest in the first video image according to the position coordinates of the origin of the video image in the region of interest in the first video image; distinguish the region of interest from the non-region of interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size of the video image in the region of interest; The video image in the region of interest is encoded and decoded using a first encoding and decoding method, and the video image in the non-region of interest is encoded and decoded using a second encoding and decoding method; and the encoding and decoding image quality level of the first encoding and decoding method is higher than the encoding and decoding image quality of the second encoding and decoding method.
2. The class inspection method according to claim 1, characterized in that: The step of encoding and decoding the video image in the region of interest by using a first encoding and decoding method, and encoding and decoding the video image in the non-region of interest by using a second encoding and decoding method, comprises: High bit rate coding is used for the video images in the region of interest; low bit rate coding is used for the video images in the non-region of interest; The coded bit stream of the region of interest is decoded at a high bit rate to obtain a corresponding decoded image; the coded bit stream of the region of non-interest is decoded at a low bit rate to obtain a corresponding decoded image; The decoded image of the region of interest and the decoded image of the region of no interest are spliced.
3. The class inspection method according to claim 1, characterized in that: Also includes: The rotation of the camera is adjusted according to the position coordinates of the center point of the video image in the region of interest in the first video image, so that the video image in the region of interest is located at the center of the class inspection interface.
4. An online class inspection device based on ROI coding, characterized in that: include: A first video image acquisition module, used to acquire a first video image; An area of interest selection module, used for selecting an area of interest in the first video image; a feature extraction module, configured to denoise the first video image and perform the denoising step as claimed in claim 1; and extract features from the first video image by calculating the LBP value of the first video image and perform the feature extraction step as claimed in claim 1; And, performing feature extraction on the video image in the region of interest by calculating the LBP value of the video image in the region of interest; a position tracking module, configured to track the video image in the region of interest to obtain the position coordinates of the center point of the video image in the region of interest in the first video image according to the extracted features of the first video image and the video image in the region of interest, and the LBP values of the first video image and the video image in the region of interest, and to perform the step of obtaining the position coordinates as claimed in claim 1; A region division module, configured to distinguish the region of interest from the region of non-interest in the first video image according to the position coordinates of the center point of the video image in the region of interest in the first video image and the size of the video image in the region of interest; The image coding module is used to encode and decode the video image in the region of interest using a first coding method, and to encode and decode the video image in the non-region of interest using a second coding method; and the coding and decoding image quality level of the first coding and decoding method is higher than the coding and decoding image quality of the second coding and decoding method.
5. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the class inspection method according to any one of claims 1 to 3 is executed.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the terminal where the storage medium is located is controlled to execute the class inspection method described in any one of claims 1 to 3.
