A teaching system

Abstract

A teaching system is provided. The teaching system comprises: a camera device configured to capture a video; a processing device electrically connected to the camera device, the processing device configured to process the video to obtain a new video; and a display device electrically connected to the processing device, the display device configured to present the new video. The teaching system solves the problem of shaking of handwriting in the picture by associating the board writing images of consecutive frames.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of electronic teaching, in particular to a teaching system. BACKGROUND

[0002] With the popularity and popularity of online classes, teaching videos have become an indispensable educational resource. A teaching video is a video obtained by a camera shooting a teaching scene. In the teaching scene, a teacher teaches by writing content on a writing board, and the board content is crucial for understanding the video. SUMMARY

[0003] Therefore, the present disclosure provides a teaching system for organizing board content in a video.

[0004] According to a first aspect of the present disclosure, a teaching system is provided, comprising:

[0005] a camera device for shooting a video;

[0006] a processing device electrically connected to the camera device, the processing device being configured to process the video to obtain a new video;

[0007] a display device electrically connected to the processing device, configured to present the new video.

[0008] Optionally, the processing device processes the video to obtain a new video by:

[0009] obtaining a continuous frame of board images from the board-unoccluded video;

[0010] for a current frame of board images, calculating a difference area between the current frame of board images and a reference image;

[0011] for each difference area, replacing a corresponding image block in the current frame of board images with a corresponding image block in the reference image to obtain a new board image;

[0012] outputting the new board image to form a new video,

[0013] wherein the current frame of board images is each board image after a first frame of board image in the continuous frame of board images, and the reference image is a board image associated with a previous frame of board image.

[0014] Optionally, for a second frame of board image in the continuous frame of board image as the current frame of board image, the reference image is the first frame of board image, and for the current frame of board image after the second frame of board image, the reference image is a new board image calculated from a previous frame of board image.

[0015] Optionally, for a second frame of the continuous frame of the board-writing image, the current frame of the board-writing image, the reference image is an image obtained by fusing a plurality of images of the board-writing content without the board in the video.

[0016] Optionally, the board-writing content without the board in the video is obtained by the following steps: extracting the board-writing content from an original video; completing the board-writing content occluded by the teacher; and synthesizing the completed board-writing content with the original video.

[0017] Optionally, the difference area between the current frame of the board-writing image and the reference image comprises:

[0018] calculating the pixel-level difference between the current frame of the board-writing image and the reference image, and obtaining the difference area based on the pixel points with the difference greater than the difference threshold, or

[0019] dividing the current frame of the board-writing image and the reference image into a plurality of image blocks respectively, and taking the image blocks with the difference greater than the difference threshold as the difference area.

[0020] Optionally, it further comprises: expanding the irregularly shaped difference area into a regular shape.

[0021] Optionally, it further comprises: adjusting the color of the writing board in the new board-writing image.

[0022] Optionally, the adjustment of the color of the writing board in the new board-writing image comprises:

[0023] obtaining a corrected writing board image, which only contains the effective area that can be written, does not contain the frame of the writing board and has no board-writing content;

[0024] calculating the average value of each color channel of the corrected writing board image or obtaining the fixed value of each color channel preset by the user;

[0025] for a dark board, modifying the pixel value of each color channel in the new board-writing image that is less than the first threshold to the average value of the color channel or the fixed value of each color channel preset by the user;

[0026] for a light board, modifying the pixel value of each color channel in the new board-writing image that is greater than the second threshold to the average value of the color channel or the fixed value of each color channel preset by the user.

[0027] Optionally, it further comprises: determining the color of the writing board in the new board-writing image based on user input.

[0028] The present disclosure provides a teaching system, which solves the problem of shaking of handwriting in a picture by associating the board writing images of continuous frames. Further, a control method of the background color of the writing board is implemented, which makes the board writing image in the transparent display process realize the control of the background color of the writing board, and makes the picture closer to the real scene.

[0029] It should be noted that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure;

[0032] Figure 2 is an exemplary frame image in a video for illustrating unobstructed board writing;

[0033] Figure 3a and Figure 3b is an image block corresponding to a difference region on two images for illustrating;

[0034] Figure 4 is a flowchart of a method for converting a board writing image into a board writing image with a writing board background color provided by an embodiment of the present disclosure;

[0035] Figure 5 is a schematic diagram of a video processing device provided by an embodiment of the present disclosure;

[0036] Figure 6 is a schematic diagram of an electronic device for implementing various embodiments of the present disclosure;

[0037] Figure 7 is a schematic diagram of a teaching system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to facilitate the understanding of the present disclosure, the present disclosure will be described more fully below with reference to the related drawings. The preferred embodiments of the present disclosure are shown in the drawings. However, the present disclosure can be implemented in different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present disclosure more thorough and comprehensive.

[0039] The following concepts are used herein:

[0040] Writing board: a board for writing content in many scenarios such as teaching or meeting, including but not limited to blackboard, greenboard, blueboard and whiteboard. The writing board involved in the present disclosure includes but is not limited to blackboard, greenboard, blueboard and whiteboard, and even electronic board. For the convenience of description in the following, these board writing media are collectively referred to as writing board. The white or light color writing board is collectively referred to as light color board; the dark color writing board such as blackboard and greenboard is collectively referred to as dark color board.

[0041] Board content: content written on the writing board, including but not limited to in the form of text, symbols and charts, etc. The present disclosure collectively refers to the forms of board content as characters.

[0042] Gray image: an image represented by gray scale. Any non-gray image can be converted into a gray image. Specifically, since any color is composed of three primary colors of red, green and blue, if the color of a certain pixel point is represented as (R, G, B) in the RGB space, the gray scale value Gray of the pixel point after conversion to a gray image is the weighted sum of R, G and B.

[0043] However, the teaching video mentioned in the background art may cause the video to fail to completely present the board content due to the teacher blocking the writing board. Therefore, in the scheme of board content extraction and occluded content completion, the complete board content is obtained by extracting the board content from the live video and completing the occluded content, and finally the complete board content and the live video are synthesized to obtain a board-occluded-free video. In this way, students watching the course through the video will not miss important content due to the teacher's body blocking the board, greatly improving the students' classroom experience. Of course, this processing can also be applied to videos in scenarios such as meetings and training.

[0044] However, there will be some subtle differences between the board images of consecutive frames in the live video, and the differences in the handwriting edges of the board content will cause the viewer to see the handwriting of the board content shaking when playing the video. The handwriting shaking refers to the same handwriting at the same position constantly flashing when watching, thereby affecting the visual effect.

[0045] Figure 1 is a flowchart of the video processing method provided by the first embodiment of the present disclosure. Referring to the figure, the following steps are included.

[0046] In step S100, a board image in a continuous frame is taken out from a board-occluded-free video.

[0047] In step S110, the first board image in the continuous frame is taken as a reference image.

[0048] The board content in each frame of the corresponding video should be completely presented. Since the teacher blocks the board content at a certain moment in the teaching process, the original video captured by the camera is not a board-unblocked video. The board-unblocked video can be obtained by the board content extraction and occlusion completion solution mentioned above. An exemplary frame of the board-unblocked video obtained by the solution is shown in FIG. 11. As can be seen from the figure, the board content can be completely presented, and the teacher as a transparent layer does not affect the complete presentation of the board content. Figure 2

[0049] The operations of steps S100 and S110 include: first taking the board-unblocked video as the input of the present step, taking the board images of the continuous frames from the moment when the video has the board content, then taking the first frame of the board images in the continuous frames as the reference image of the initial cycle, and then taking a frame of the board images from the continuous frames as the current frame of the board image. Here, the board image refers to the image with the board content.

[0050] In step S120, the differences between the current frame of the board image and the reference image are calculated, and at least one difference area with a difference greater than the difference threshold is obtained.

[0051] We know that different video streams use different color spaces, such as RGB and YUV. Take RGB as an example to illustrate how to calculate it in this step. RGB color space uses R, G and B values to represent each pixel point in the frame image. Each frame image includes, for example, 640*480 pixel points, and the maximum value of R, G and B is 255 and the minimum value is 0. When RGB is 255, the color presented is white, and when RGB is 0, the color presented is black.

[0052] Based on RGB, as an embodiment of the present step, the RGB average value of each pixel of the extracted board image and the reference image is calculated respectively, and the formula is: RGB average value = (R value + G value + B value) / 3. Then, the RGB average values of the corresponding positions of the extracted board image and the reference image are compared. If the difference value is greater than the difference threshold set in advance, the corresponding pixel position is recorded in the first set. Finally, the difference area is constructed according to the first set. Figure 3a and Figure 3b ​The image blocks X1 and X2 shown in the two images are used to illustrate a difference region. However, it should be noted that while the image blocks X1 and X2 shown in the images are regular in shape, the difference region obtained in this embodiment can be irregular in shape. Optionally, to facilitate subsequent steps, the irregularly shaped difference region can be expanded into a regular shape. In addition, this embodiment can output two or more difference regions.

[0053] Based on RGB, as another implementation of this step, the extracted blackboard image and the reference image are converted into grayscale images respectively. Then, based on the above implementation, the corresponding pixels in the two grayscale images are compared to obtain the difference region.

[0054] Based on RGB, as the third implementation method of this step, since the difference between two adjacent whiteboard images is very small, the extracted whiteboard image and the reference image can be divided into multiple image blocks respectively. Then, the average value of all R, G, and B values ​​of each image block is calculated. The average values ​​of the corresponding image blocks are compared. Then, the image blocks with differences greater than a first threshold are extracted and divided into multiple image blocks. The average value of the sum of all R, G, and B values ​​of each image block is calculated. The average values ​​of the corresponding image blocks are compared to obtain the image blocks with differences greater than a second threshold. This process is repeated until the image blocks with differences greater than the Nth threshold are obtained. The difference region is then obtained based on the image blocks with differences greater than the Nth threshold. It should be understood that the first threshold to the Nth threshold can be different, for example, gradually increasing.

[0055] In step S130, for each difference region in at least one difference region, the corresponding image block in the obtained blackboard image is replaced with the corresponding image block in the reference image to obtain a new blackboard image.

[0056] In step S140, the next frame of the whiteboard image is extracted from the consecutive frames.

[0057] Steps S130 and S140 can be continued for reference. Figure 3b As shown, assuming X2 is the image block obtained by the difference region corresponding to the reference image, and X1 is the image block obtained by the difference region corresponding to the obtained blackboard image, X2 is used to replace X1 to obtain a new blackboard image, and the reference image is replaced by the new blackboard image. Then, the next frame of the blackboard image is extracted from the video with the blackboard unobstructed.

[0058] In step S150, it is determined that there is a next frame of whiteboard image. The final operation of step S140 is to extract the next frame of whiteboard image from the consecutive frames. If this operation is successful, it means that the currently obtained whiteboard image is not the last frame of whiteboard image and there is a next frame of whiteboard image. Then, jump to step S120. If it fails, it means that the currently obtained whiteboard image is the last frame of whiteboard image and there is no next frame of whiteboard image. Exit the loop and execute step S160.

[0059] In step S160, the plurality of new board writing images are outputted to form a new video.

[0060] In the present embodiment, the reference image is the board writing image associated with the board writing image of the current frame, which is changed in the loop body, so that a new reference image can be obtained at the end of each loop. In combination with the drawings, the reference image of the first loop in the loop body composed of steps S120 to S150 is the board writing image of the first frame in the continuous frames, but at the end of each loop, a new reference image can be obtained, which is still the board writing image associated with the board writing image of the current frame, and then the next loop is based on the new reference image. The end condition of the loop is to take out all the board writing images. After the loop is executed, a plurality of board writing images will be obtained, and the new board writing images are arranged in time sequence to form a new video, which is associated, so that there is no phenomenon of handwriting jitter.

[0061] In order to illustrate the above embodiment, a more specific example (Example 1) can be described. For example, the board writing video without occlusion Video_input = {f1, f2, …, fn}, n is a positive integer, f1 to fn represent the frame images in time sequence, then in the first loop, f1 and f2 are compared to obtain the difference area, the image blocks corresponding to the difference area in f1 and f2 are divided into f1_patch and f2_patch, then f2_patch is used to replace f1_patch in f1 to obtain f2'; then in the second loop, f2' and f3 are compared to obtain the difference area, the image blocks corresponding to the difference area in f2' and f3 are divided into f2'_patch and f3_patch, then f3_patch is used to replace f2'_patch in f2' to obtain f3'; in this way, finally f2' to fn' (n-1) frame images are obtained, and then the sequence Video_output = {f1, f2', …, fn'} is outputted as a new video.

[0062] In other examples, the time interval can be lengthened, and the initial reference image is selected every n1 frames. For example, if the board is not blocked in the video Video_input = {f1, f2,..., fn1, fn1+1,..., fn}, n1 is a certain frame in the middle, and the previous fn1 frames are processed according to the first example to obtain {f1, f2',..., fn1'}. When fn1+1 is processed, no difference calculation is performed (no processing), and the output is directly obtained (the role of f1 in the previous processing stage is filled, as the reference image). The sequence is processed again according to the first example to obtain {fn1+1,..., fn'}. The final time sequence is {f1, f2',..., fn1', fn1+1,..., fn'}. fn1+1 is equivalent to the refreshed frame, and the overall de-jittering effect is achieved.

[0063] In some embodiments, the reference image in the first cycle can be other images, for example, the images of several frames of the board-unblocked video without board content are fused to obtain a fused image as the reference image, or an image without board content is captured by a camera device with appropriate light and angle as the reference image.

[0064] In some embodiments, for each frame image in the sequence {f2',..., fn'} described above, edge smoothing and enhancement processing of the characters can also be performed. The edge smoothing and enhancement processing of the characters makes the lines of the characters as uniform as possible and black within the strokes. The edge smoothing and enhancement processing of the characters makes the lines of the characters in the board content more robust and clear, thereby avoiding the phenomenon that the board handwriting is unclear due to the image block replacement step.

[0065] Since the writing board and the board content in the board-unblocked video are often replaced by a white writing board and black board content or a black writing board and white board content, which is deviated from the original color of the writing board, the immersion and interest are slightly poor when transparently displayed. Therefore, as a preferred embodiment, the color of the writing board can also be controlled, and the color of the writing board can be used as the background color of the entire video. The control of the color of the writing board can be set as an optional item and controlled through a corresponding command parameter. For example, if the color control of the writing board is started, the display is performed using the writing board background color or the user preset writing board color, the step of Figure 4 is executed to obtain an image with the writing board background color or the user preset writing board color, otherwise, the default writing board color and board content color are used to output the video. If it is a dark board, the default background color is black and the handwriting is white. If it is a light board, the default background color is white and the handwriting is black.

[0066] In step S410, the corrected writing board image Irgb The writing board image only contains the effective area image of the writing board that can be written on, does not contain the frame of the writing board, and does not have the board content. The above-mentioned corrected writing board image I rgb .

[0067] In step S420, the corrected writing board image I rgb The average value of each color channel, wherein the average value of the R channel is R_mean, the R value of all pixels is accumulated and then divided by the total number of pixels, the average value of the G channel is G_mean, the G value of all pixels is accumulated and then divided by the total number of pixels, and the average value of the B channel is B_mean, the B value of all pixels is accumulated and then divided by the total number of pixels.

[0068] In step S430, according to the average value of each color channel, the board image with the background color being the color of the writing board is obtained, for example, for each pixel point in the dark board, if the R value of the pixel point is less than the first threshold value, the R value of the pixel point is replaced by the above-mentioned R_mean filling, otherwise the R value of the pixel point is retained, and the same scheme is used to execute the judgment and replacement steps of the G value and B value of each pixel point, and finally the writing board in the board image is replaced by the dark board. But for each pixel point in the light board, if the R value of the pixel point is greater than the second threshold value, the R value of the pixel point is replaced by the above-mentioned R_mean filling, otherwise the R value of the pixel point is retained, and the same scheme is used to execute the judgment and replacement steps of the G value and B value of each pixel point, and finally the writing board in the board image is replaced by the light board. The first threshold value and the second threshold value can be set by oneself, for example, the middle value between 0 and 255.

[0069] In addition, if it is desired to set the writing board to the user's preset writing board color, for the dark board, if the R value, G value or B value of the pixel point is less than the corresponding threshold value, the R value, G value or B value of the pixel point is replaced by the fixed value of the corresponding color channel preset by the user, and for the light board, if the R value, G value or B value of the pixel point is greater than the corresponding threshold value, the R value, G value or B value of the pixel point is replaced by the fixed value of the corresponding color channel preset by the user.

[0070] The present disclosure also proposes a processing device of a video, as shown in Figure 5 The device 500 includes a board handwriting de-jittering module 510 and a board background color control module 520. The board handwriting de-jittering module 510 can execute the video processing method mentioned in the above various embodiments (excluding the embodiment of Figure 4 The new video does not have the problem of handwriting jitter in the picture because the association between the consecutive board frame images is made.

[0071] The board background color control module 520 is configured to control the board background color in the output video. Specifically, the color of the board can be controlled by one command parameter, which can be input by a user. For example, the command parameter can determine whether to use a default background color or an original board background color. If the original board background color is used, the steps in the above method can be performed. The board background color control module 520 enables the board image to be controlled in the transparent display process, so that the picture is closer to the real scene. Figure 4

[0072] It should be understood that the board handwriting de-jittering module 510 and the board background color control module 520 can be independent of each other when implemented as computer programs, and both of the two functional modules can be in the form of a plug-in library for calling by a user and a required hardware and software platform.

[0073] In summary, through the above modules or methods, the generated video not only has a stable picture display but also has a presentation effect close to the real scene, thereby ensuring that the understanding and memory of students on the board content are not interrupted, and enhancing the sense of immersion in the classroom.

[0074] The embodiments of the present disclosure further provide an electronic device 1300, as shown in Figure 6 which includes a memory 1310, a processor 1320, and a program stored in the memory 1310 and executable on the processor 1320. The program is executed by the processor 1320 to implement the processes of each embodiment of the above video processing method, and achieve the same technical effects. To avoid repetition, details are not described herein. Of course, the electronic device can further include a power component 1330, a network interface 1340, and an input / output interface 1350, and the like.

[0075] Those skilled in the art can understand that all or part of the steps in the above embodiments can be completed by instructions, or by related hardware controlled by the instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, the embodiments of the present disclosure further provide a computer-readable storage medium having a computer program or instructions stored thereon, which are executed by a processor to implement the processes of each embodiment of the above video processing method. The computer-readable storage medium includes, for example, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0076] ​Since the instructions stored in the readable storage medium can execute the steps in any of the video processing methods provided by the embodiments of the present disclosure, the beneficial effects that can be achieved by any of the video processing methods provided by the embodiments of the present disclosure can be achieved. Details are described in the foregoing embodiments, which will not be repeated here. The specific implementation of each operation can refer to the foregoing embodiments, which will not be repeated here.

[0077] Figure 7 is a schematic diagram of a teaching system provided by the embodiments of the present disclosure. The teaching system includes a camera device 701, a processing device 702 and a display device 703. The camera device 701 is used to shoot a video. The processing device 702 is electrically connected to the camera device, receives the video from the camera device, and can convert the video into a board book unoccluded video according to the method provided in the specification or other methods, then execute the video processing method provided by each of the above embodiments, and output a new video. The display device 703 is used to display the new video output by the processing device 702 on the display screen. It can be understood that, on the one hand, the teaching system can be integrated into an electronic product including a camera, a display and a processor, and the like, and on the other hand, the teaching system can also be realized by a plurality of devices communicating via a network (such as a local area network).

[0078] It should be noted that, in the description of each embodiment in the specification, the difference from other embodiments is mainly explained, and the same or similar parts between each embodiment can be understood by mutual reference. For the system embodiment, since it is basically similar to the method embodiment, the relevant parts can refer to the description of the method embodiment part.

[0079] In addition, it should be pointed out that, in the device and method of the present disclosure, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be regarded as equivalent solutions of the present disclosure. And, the steps of executing the above series of processes can naturally be executed in time sequence according to the order of description, but it is not necessary to be executed in time sequence, and some steps can be executed in parallel or independently of each other. For those skilled in the art, it can be understood that all or any steps or components of the method and device of the present disclosure can be realized in any computing device (including processor, storage medium, etc.) or network of computing devices, in hardware, firmware, software or their combination, which can be realized by those skilled in the art by using their basic programming skills after reading the description of the present disclosure.

[0080] Finally, it should be noted that the above-mentioned embodiments are merely intended for the purpose of illustration, and are not intended to limit the embodiments. Based on the above descriptions, those skilled in the art can further make other variations and changes to the embodiments. Here, it is not necessary or possible to enumerate all the embodiments. The obvious variations and changes derived therefrom are still within the protection scope of the disclosure.

Claims

1. A teaching system comprising: a camera device configured to capture a video; a processing device electrically connected to the camera device, the processing device configured to process the video to obtain a new video, the processing device configured to process the video to obtain a new video by: obtaining n consecutive board image frames from a board-unoccluded video; performing n-1 rounds of replacement steps, each round of replacement step comprising: calculating a difference region between an i-th board image frame and a reference image; replacing image blocks of the difference region of the reference image with image blocks of a corresponding difference region of the i-th board image frame to obtain a new board image, and taking the new board image as the reference image in the next round of replacement step, wherein n is a positive integer greater than or equal to 3, i is an integer from 2 to n in turn, and when i is equal to 2, the 1st board image frame is taken as the reference image, and the difference region has a difference greater than a difference threshold; combining the 1st board image frame and the new board image output by the n-1 rounds of replacement steps to obtain a new board video; adjusting a color of a writing board in the new board image. The board-unoccluded video is obtained by the following steps: extracting board content from an original video; completing the board content occluded by a teacher; and synthesizing the completed complete board content with the original video.

2. The teaching system according to claim 1, wherein, The calculating a difference region between an i-th board image frame and a reference image comprises:

3. The teaching system according to claim 1, wherein, calculating a pixel-level difference between the i-th board image frame and the reference image, and obtaining the difference region based on pixel points having a difference greater than the difference threshold, or dividing the i-th board image frame and the reference image into a plurality of image blocks respectively, and taking image blocks having a difference greater than the difference threshold as the difference region. Extending the irregularly shaped difference region to a regular shape.

4. The teaching system of claim 3, further comprising: The adjusting a color of a writing board in the new board image comprises:

5. The teaching system according to claim 1, wherein, obtaining a corrected writing board image, the corrected writing board image only containing a valid area that can be written, not containing a frame of the writing board and having no board content; calculating an average value of each color channel of the corrected writing board image or obtaining a fixed value of each color channel preset by a user; for a dark board, modifying pixel values less than a first threshold in each color channel of the new board image to the average value of the color channel or the fixed value of each color channel preset by the user; for a light board, modifying pixel values greater than a second threshold in each color channel of the new board image to the average value of the color channel or the fixed value of each color channel preset by the user. Determining a color of a writing board in the new board image based on user input.

6. The teaching system of claim 1, further comprising: ​

Images